Monitoring your infrastructure is essential for performance, reliability, and capacity planning.

In this guide, we’ll set up Prometheus (for metrics collection) and Grafana (for visualization) on Ubuntu 20.04, then connect Rocky Linux 8 and Ubuntu clients using Node Exporter.

🧩 What is Prometheus?

Prometheus is an open-source monitoring tool developed by SoundCloud, now part of the CNCF (Cloud Native Computing Foundation).
It collects metrics from systems and applications and stores them in a time-series database, allowing you to create alerts and analyze performance.

📊 What is Grafana?

Grafana is an open-source data visualization platform.
It connects to Prometheus (and many other sources) to visualize data in interactive dashboards and graphs.

⚙️ Step 1: Update Ubuntu 20.04 Server

# apt update &&  apt upgrade -y

🧠 Step 2: Install Prometheus on Ubuntu 20.04

1️⃣ Create Prometheus user and directories

# useradd --no-create-home --shell /bin/false prometheus
# mkdir /etc/prometheus /var/lib/prometheus

2️⃣ Download Prometheus

# cd /tmp
# wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
# tar xvf prometheus-2.53.0.linux-amd64.tar.gz
# cd prometheus-2.53.0.linux-amd64

3️⃣ Move binaries and set permissions

#  mv prometheus /usr/local/bin/
#  mv promtool /usr/local/bin/
#  mv consoles /etc/prometheus/
# mv console_libraries /etc/prometheus/
# mv prometheus.yml /etc/prometheus/
# chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
# chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

🧾 Step 3: Create Prometheus Systemd Service

# tee /etc/systemd/system/prometheus.service > /dev/null <<EOF
[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF

Then enable and start Prometheus:

# systemctl daemon-reload
# systemctl enable prometheus
# systemctl start prometheus
# systemctl status prometheus

Access it in your browser:
👉 http://gra01.darole.org:9090/

📈 Step 4: Install Grafana on Ubuntu 20.04

1️⃣ Add Grafana APT repository

# apt install -y apt-transport-https software-properties-common curl gpg
# mkdir -p /usr/share/keyrings/
# curl -fsSL https://packages.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg
# echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://packages.grafana.com/oss/deb stable main" | 
sudo tee /etc/apt/sources.list.d/grafana.list

2️⃣ Install Grafana

# apt update
# apt install grafana -y

3️⃣ Enable and Start Grafana

# systemctl enable grafana-server
# systemctl start grafana-server

Open Grafana in a browser:
👉 http://gra01.darole.org:3000/

(Default credentials: admin / admin)

🔗 Step 5: Connect Grafana to Prometheus

Login to Grafana → http://gra01.darole.org:3000
Go to Connections → Data Sources → Add Data Source
Choose Prometheus
In URL → http://gra01.darole.org:9090
Click Save & Test

🖥️ Step 6: Install Node Exporter on Clients

➤ On Rocky Linux 8 & Ubuntu Clients

Run these commands on each client system (both Rocky 8 and Ubuntu):

Download Node Exporter
# cd /tmp
# wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
# tar xvf node_exporter-1.8.2.linux-amd64.tar.gz
# sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/

Create a user
# sudo useradd --no-create-home --shell /bin/false node_exporter

Create systemd service
# sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

# sudo systemctl daemon-reload
# sudo systemctl enable node_exporter
# sudo systemctl start node_exporter

Default metrics endpoint →
👉 http://lamp01.darole.org:9100/metrics

🔧 Step 7: Add Clients to Prometheus Server

Edit Prometheus config on your main Ubuntu 20.04 server:

root@gra01:~# sudo vi /etc/prometheus/prometheus.yml

Add client targets at the bottom:

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets:

          - "kub01.darole.org:9100"
          - "kub02.darole.org:9100"
          - "kub03.darole.org:9100"
          - "dock01.darole.org:9100"
          - "lamp01.darole.org:9100"
          - "zap01.darole.org:9100"
          - "pup01.darole.org:9100"
          - "web01.darole.org:9100"
          - "db01.darole.org:9100"
          - "ans01.darole.org:9100"
          - "tomd01.darole.org:9100"
          - "tomp01.darole.org:9100"
          - "jen01.darole.org:9100"
          - "son01.darole.org:9100"

Replace the IPs with your Rocky 8 and Ubuntu client IPs.

Then restart Prometheus:

# promtool check config /etc/prometheus/prometheus.yml
# sudo systemctl restart prometheus

✅ Step 8: Verify Monitoring Setup

Prometheus Targets:
👉 http://gra01.darole.org:9090/targets

Grafana Dashboards:
👉 http://gra01.darole.org:3000

You should now see metrics from all your Ubuntu and Rocky Linux clients visualized beautifully in Grafana.

🎯 Summary

Component	Description	Default Port
Prometheus	Metrics collection & storage	9090
Grafana	Data visualization	3000
Node Exporter	System metrics on clients	9100

🧠 Use: Prebuilt Dashboards

In Grafana:

Go to Google and search for "grafana dashboard"

Select the dashboard required and default values exported by Prometheus node exporter graphed

Go to Create → Import Dashboard and Paste Copy ID 1860

Select the data source Prometheus

Below is Dash Board for Linux

Monitoring Docker containers with Prometheus + Grafana

Monitoring Docker containers with Prometheus + Grafana is typically done using cAdvisor. cAdvisor collects CPU, memory, disk, filesystem, and network metrics from Docker containers, while Prometheus scrapes those metrics and Grafana visualizes them.

Architecture

Step 1: Run cAdvisor


$ docker run -d \
  --name=cadvisor \
  --restart=unless-stopped \
  -p 8080:8080 \
  -v /:/rootfs:ro \
  -v /var/run:/var/run:ro \
  -v /sys:/sys:ro \
  -v /var/lib/docker/:/var/lib/docker:ro \
  -v /dev/disk/:/dev/disk:ro \
  --privileged \
  gcr.io/cadvisor/cadvisor:v0.49.1

Verify:


$ curl http://localhost:8080/metrics

Step 2: Configure Prometheus

Edit:


sudo vi /etc/prometheus/prometheus.yml

global:

  scrape_interval: 15s

scrape_configs:

  - job_name: 'prometheus'
    static_configs:
      - targets:
          - localhost:9090

  - job_name: 'cadvisor'
    static_configs:
      - targets:
          - dock01:8080
Reload Prometheus
$ sudo systemctl restart prometheus
Step 3: Verify Target
http://gra01.darole.org:9090/targets
You should see
cadvisor State: UP

Step 4: Import Dashboard
Go to google and search for grafana dashboard and then search for docker 

Then import the Dashboard as done for Linux. 
Recommended Dashboard IDs:
Dashboard ID
Docker Monitoring 193
Docker & Host Monitoring 179
cAdvisor Dashboard 14282
Docker Container Metrics 10619

Dashboard	ID
Docker Monitoring	193
Docker & Host Monitoring	179
cAdvisor Dashboard	14282
Docker Container Metrics	10619

Below is Grafana Dashboard for Docker

Kubernetes Monitoring with Prometheus and Grafana

Introduction

As part of my On-Premises DevOps Infrastructure Project, I configured Kubernetes monitoring using Prometheus and Grafana.

The Kubernetes cluster consists of one control-plane node and two worker nodes, while Prometheus and Grafana are hosted on a dedicated monitoring server.

Monitoring Architecture

The monitoring design uses the existing host-level Node Exporter services running on the Kubernetes nodes. Prometheus on gra01 collects these metrics over TCP port 9100.

1. Verify Node Exporter on Kubernetes Nodes

Node Exporter is already installed and running directly on the Kubernetes hosts.

On kub01, I verified the service using:

# ss -lntp | grep :9100

The output confirmed that Node Exporter was listening:

LISTEN 0 4096 *:9100 *:* users:(("node_exporter",pid=935,fd=3))

This confirms that kub01 is already exposing Linux system metrics through Node Exporter.

The same verification can be performed on the worker nodes:

# ssh kub02.darole.org "ss -lntp | grep :9100"

# ssh kub03.darole.org "ss -lntp | grep :9100"

The expected result is a Node Exporter process listening on port 9100.

2. Configure Prometheus

Since Node Exporter is already running on the Kubernetes hosts, Prometheus on gra01 can directly scrape the three nodes.

Edit the Prometheus configuration:

# vi /etc/prometheus/prometheus.yml

Add the Kubernetes node targets:

scrape_configs:

  - job_name: 'kubernetes-nodes'
    static_configs:
      - targets:
          - 'kub01:9100'
          - 'kub02.darole.org:9100'
          - 'kub03.darole.org:9100'

This configuration tells Prometheus to collect metrics from:

kub01:9100
kub02.darole.org:9100
kub03.darole.org:9100

3. Validate Prometheus Configuration

Before restarting Prometheus, validate the configuration:

# promtool check config /etc/prometheus/prometheus.yml

If the configuration is valid, restart Prometheus:

# systemctl restart prometheus

Check the service:

# systemctl status prometheus

4. Verify Prometheus Targets

Open the Prometheus web interface:

http://gra01:9090

Navigate to:

Status → Targets

The Kubernetes nodes should appear as:

The UP state confirms that Prometheus is successfully collecting metrics from the Kubernetes nodes.

5. Metrics Collected from Kubernetes Nodes

Node Exporter provides Linux host-level metrics that can be used to monitor:

CPU

CPU utilization
CPU idle time
CPU system time
CPU user time

Memory

Total memory
Available memory
Used memory
Memory utilization

Filesystem

Filesystem capacity
Available space
Filesystem utilization

Network

Network receive traffic
Network transmit traffic
Network errors
Network packets

These metrics allow the infrastructure team to monitor the health and resource utilization of the Kubernetes nodes.

Kubernetes Monitoring with kube-state-metrics

Node Exporter provides information about the Linux operating system, but it does not provide detailed information about Kubernetes objects.

For example, Node Exporter cannot tell us:

How many Pods are running
How many Deployments exist
Whether desired replicas are available
How many Pods have restarted
Which namespaces exist
Whether a DaemonSet has all required Pods

To obtain this Kubernetes-specific information, I added kube-state-metrics.

1. Install Helm

The first step was to install Helm on kub01.

root@kub01:~# # snap install helm --classic

Installation completed successfully:

helm 4.2.4 from Snapcrafters✪ installed

root@kub01:~# helm version

I then verified the installation:

Output:

version.BuildInfo{
  Version:"v4.2.4",
  GitCommit:"3900f434fd3ef2b84065dc04508df48f288dba00",
  GitTreeState:"clean",
  GoVersion:"go1.26.5",
  KubeClientVersion:"v1.36"
}

2. Add Prometheus Community Helm Repository

I added the Prometheus Community Helm repository:

root@kub01:~# helm repo add prometheus-community \
https://prometheus-community.github.io/helm-charts

The repository was added successfully.

Then I updated the repository information:

root@kub01:~# helm repo update

The update completed successfully:

Successfully got an update from the "prometheus-community" chart repository
Update Complete.

I verified the repository:

root@kub01:~#  helm repo list

Output:

NAME                    URL
prometheus-community    https://prometheus-community.github.io/helm-charts

3. Create Monitoring Namespace

The Kubernetes cluster already had a monitoring namespace.

I verified this while attempting to create it:

root@kub01:~# kubectl create namespace monitoring

The result was:

namespaces "monitoring" created

This confirmed that the namespace was already available.

4. Install kube-state-metrics

I installed kube-state-metrics using the Prometheus Community Helm chart:

root@kub01:~# helm install kube-state-metrics \
  prometheus-community/kube-state-metrics \
  -n monitoring

The installation completed successfully:

NAME: kube-state-metrics
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete

kube-state-metrics listens to the Kubernetes API server and generates metrics representing the current state of Kubernetes objects.

The metrics endpoint is exposed at:

kube-state-metrics.monitoring.svc.cluster.local:8080/metrics

5. Verify kube-state-metrics Pod

I verified the Pod status:

root@kub01:~# kubectl get pods -n monitoring

The result was:

NAME                              READY   STATUS    RESTARTS   AGE
kube-state-metrics-db99d4-5787j   1/1     Running   0          14m

The important values are:

READY    1/1
STATUS   Running

This confirms that kube-state-metrics was successfully deployed and is running inside the Kubernetes cluster.

6. Kubernetes Metrics Provided by kube-state-metrics

kube-state-metrics exposes Kubernetes-specific metrics for resources such as:

Nodes

Node status
Node readiness
Node conditions
Node information

Pods

Pod status
Pod phase
Pod readiness
Pod scheduling information
Container information

Deployments

Desired replicas
Available replicas
Updated replicas
Deployment status

ReplicaSets

Desired replicas
Current replicas
Ready replicas

DaemonSets

Desired Pods
Current Pods
Ready Pods
Available Pods

Namespaces

Namespace information
Namespace status

Container Restarts

Container restart counts
Pod/container state information

This information complements the host-level metrics collected by Node Exporter.

7. Prometheus Configuration Validation

After making the Prometheus configuration changes, I validated the configuration again:

root@kub01:~# kubectl get svc -n monitoring

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kube-state-metrics NodePort 10.98.157.194 <none> 8080:30434/TCP 38h

root@kub01:~#

Update the configuration file.

root@gra01:~#vi /etc/prometheus/prometheus.yml

- job_name: 'kube-state-metrics'

static_configs:

- targets:

- 'kub01:30434'

Check the configuration

root@gra01:~# promtool check config /etc/prometheus/prometheus.yml

The result was:

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

This confirms that the Prometheus configuration has valid syntax.


Building a Kubernetes Pod Monitoring Dashboard in Grafana
Visualizing pod health and lifecycle events is essential for maintaining cluster stability and debugging application issues before they escalate. Follow this step-by-step walkthrough to set up a dedicated Kubernetes pod overview dashboard using Grafana and Prometheus.

Step 1: Create a New Dashboard
Navigate to the Grafana home page, open the top-right dashboard menu, and click New dashboard.
Step 2: Add a Panel
From the empty dashboard view, click the + (Add panel) placeholder to insert your first visualization component.
Step 3: Configure Title and Visualization
In the panel edit view:
Provide a clear Title (e.g., Total Pods or Running Pods) in the right-side options panel.
Click Configure visualization to choose how the metric will be displayed (such as Stat, Gauge, or Time series).
Step 4: Add Data Transformations (Optional)
For tabular displays or aggregated single-value summaries:


Switch to the Transformations tab below the query editor.
Click Add transformation.
Search for Reduce to aggregate multiple time-series data points into single representative values (such as Last, Mean, or Max).

Essential Prometheus (PromQL) Queries for Pod Monitoring
Use the following queries in your panels to track pod health, distribution, and restart churn across your cluster:

Select your Prometheus data source.

I recommend creating the dashboard with the following sections.
PromQL Queries
CPU Utilization
100 - (
  avg by(instance) (
    rate(node_cpu_seconds_total{mode="idle"}[5m])
  ) * 100
)


Memory Utilization
100 * (
  1 -
  node_memory_MemAvailable_bytes /
  node_memory_MemTotal_bytes
)



Filesystem Utilization
100 * (
  1 -
  node_filesystem_avail_bytes /
  node_filesystem_size_bytes
)


Final Dashboard:

Install Prometheus and Grafana on Ubuntu 20.04

🧩 What is Prometheus?

📊 What is Grafana?

⚙️ Step 1: Update Ubuntu 20.04 Server

🧠 Step 2: Install Prometheus on Ubuntu 20.04

1️⃣ Create Prometheus user and directories

2️⃣ Download Prometheus

3️⃣ Move binaries and set permissions

🧾 Step 3: Create Prometheus Systemd Service

📈 Step 4: Install Grafana on Ubuntu 20.04

1️⃣ Add Grafana APT repository

2️⃣ Install Grafana

3️⃣ Enable and Start Grafana

🔗 Step 5: Connect Grafana to Prometheus

🖥️ Step 6: Install Node Exporter on Clients

➤ On Rocky Linux 8 & Ubuntu Clients

🔧 Step 7: Add Clients to Prometheus Server

✅ Step 8: Verify Monitoring Setup

🎯 Summary

🧠 Use: Prebuilt Dashboards

Monitoring Docker containers with Prometheus + Grafana

Architecture

Step 1: Run cAdvisor

Step 2: Configure Prometheus

Step 3: Verify Target

Step 4: Import Dashboard

Kubernetes Monitoring with Prometheus and Grafana

Introduction

Monitoring Architecture

1. Verify Node Exporter on Kubernetes Nodes

2. Configure Prometheus

3. Validate Prometheus Configuration

4. Verify Prometheus Targets

5. Metrics Collected from Kubernetes Nodes

CPU

Memory

Filesystem

Network

Kubernetes Monitoring with kube-state-metrics

1. Install Helm

2. Add Prometheus Community Helm Repository

3. Create Monitoring Namespace

4. Install kube-state-metrics

5. Verify kube-state-metrics Pod

6. Kubernetes Metrics Provided by kube-state-metrics

Nodes

Pods

Deployments

ReplicaSets

DaemonSets

Namespaces

Container Restarts

7. Prometheus Configuration Validation

PromQL Queries

CPU Utilization

Memory Utilization

Filesystem Utilization

Final Dashboard:

No comments:

Post a Comment