📊 Node-exporter + Prometheus + Grafana

In this guide, we’ll set up a monitoring system to collect metrics from all servers and visualize them in Grafana.

We will use a dedicated server to host Prometheus, Grafana, and Node Exporter, while only Node Exporter will be installed on other servers.

• Prometheus is an open-source time-series database written in Go. It pulls metrics from defined services, eliminating data queue bottlenecks and ensuring reliable monitoring.

• Node Exporter exports server metrics in a format compatible with Prometheus. While Prometheus supports many exporters, Node Exporter is ideal for server monitoring.

• Grafana is a web-based tool for visualizing time-series data from Prometheus and other databases. Although Prometheus has its own web interface, Grafana is recommended for advanced visualization.

The setup flow is as follows: Node Exporter collects server metrics → Prometheus stores the data → Grafana visualizes it in dashboards.

Node Exporter

Install and Set Up Node Exporter

# Download and extract Node Exporter
cd $HOME && \
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz && \
tar xvf node_exporter-1.8.2.linux-amd64.tar.gz && \
rm node_exporter-1.8.2.linux-amd64.tar.gz && \
sudo mv node_exporter-1.8.2.linux-amd64 node_exporter && \
chmod +x $HOME/node_exporter/node_exporter && \
sudo mv $HOME/node_exporter/node_exporter /usr/local/bin && \
rm -Rvf $HOME/node_exporter/

# Create a system user with restricted permissions for Node Exporter
sudo useradd --no-create-home --shell /bin/false node_exporter

Create and Enable the exporterd Service

# Create the systemd service file
sudo tee /etc/systemd/system/node_exporterd.service > /dev/null <<EOF
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=3
LimitNOFILE=65535

[Install]
WantedBy=default.target
EOF

# Reload systemd, enable, and start the service
sudo systemctl daemon-reload
sudo systemctl enable node_exporterd
sudo systemctl restart node_exporterd

sudo systemctl status node_exporterd

Verify Logs

sudo journalctl -u node_exporterd -f

Access Node Exporter Metrics

Open a web browser and navigate to: http://<server_IP>:9100/

Replace <server_IP> with the actual IP address of your server running node_exporter.

Prometheus

Update Repositories and Upgrade Packages

sudo apt update && sudo apt upgrade -y

Install Necessary Utilities

sudo apt install -y curl iptables build-essential git wget jq make gcc nano tmux htop nvme-cli pkg-config libssl-dev libleveldb-dev tar clang bsdmainutils ncdu unzip python3-pip
pip install yq

# Create a Prometheus User
sudo useradd -m -s /bin/bash prometheus
sudo groupadd --system prometheus
sudo usermod -aG prometheus prometheus

Download and Install Prometheus

# Create Necessary Directories
sudo mkdir /var/lib/prometheus
for i in rules rules.d files_sd; do sudo mkdir -p /etc/prometheus/${i}; done
mkdir -p /tmp/prometheus && cd /tmp/prometheus

#Download Prometheus
curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest | grep browser_download_url | grep linux-amd64 | cut -d '"' -f 4 | xargs wget

# Extract and Move Prometheus Binaries
tar xvf prometheus*.tar.gz
cd prometheus*/
sudo mv prometheus promtool /usr/local/bin/

# Check Installed Versions
prometheus --version
promtool --version

# Move Configuration Files
sudo mv prometheus.yml /etc/prometheus/prometheus.yml
sudo mv consoles/ console_libraries/ /etc/prometheus/

Create the Prometheus Service File

sudo tee /etc/systemd/system/prometheus.service > /dev/null <<EOF
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP \$MAINPID
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries \
--web.listen-address=0.0.0.0:9090 \
--web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target
EOF

You can replace the port 9090 with a custom port if needed (e.g., 8080).

Set Permissions

for i in rules rules.d files_sd; do sudo chown -R prometheus:prometheus /etc/prometheus/${i}; done
for i in rules rules.d files_sd; do sudo chmod -R 775 /etc/prometheus/${i}; done
sudo chown -R prometheus:prometheus /var/lib/prometheus/

Start and Enable the Prometheus Service

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Access Prometheus

Open a web browser and navigate to: http://<server_IP>:9090/ Replace <server_IP> with the IP address of the server running Prometheus.

Add Additional Servers to Prometheus Configuration

To monitor multiple servers, update the prometheus.yml configuration file with the relevant settings for each server.

Open the Prometheus Configuration File

sudo nano /etc/prometheus/prometheus.yml

Add a New Job for Monitoring

Here’s an example configuration for a job named celestia-consensus that monitors the target at localhost:26660.

- job_name: celestia-consensus
  static_configs:
  - targets: ['localhost:26660']

You can replace localhost:26660 with the IP address and port of the server or service you want to monitor.

For example, if monitoring another server:

- job_name: my-second-server
  static_configs:
    - targets: ['192.168.1.100:9100'] 

Restart Prometheus

Restart the Prometheus service to apply the new configuration:

sudo systemctl restart prometheus

Verify Prometheus is Running Properly

Check the service status to ensure Prometheus is running without issues:

sudo systemctl status prometheus

Access Prometheus

Navigate to http://<server_IP>:9090/targets in your web browser to see the list of active targets and ensure that the new job is being monitored.

Grafana

Install Required Dependencies

sudo apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -

Add the Grafana Repository

echo "deb https://packages.grafana.com/enterprise/deb stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Create a User for Grafana

sudo useradd -m -s /bin/bash grafana
sudo groupadd --system grafana
sudo usermod -aG grafana grafana

Install Grafana Enterprise

#Install Additional Utilities
sudo apt-get install -y adduser libfontconfig1

#Download and Install Grafana Enterprise
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_9.3.2_amd64.deb
sudo dpkg -i grafana-enterprise_9.3.2_amd64.deb

Start and Enable the Grafana Server

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Access Grafana

Open a web browser and navigate to: http://<server_IP>:3000

Replace <server_IP> with your server's IP address.

• Default username: admin

• Default password: admin (You will be prompted to change it after the first login).

After changing the password, click "Configuration" and then "Data Sources".

Now click on Add data source and select the Prometheus data source.

Access Prometheus

To access Prometheus, enter the IP address followed by port 9090 .

• If you are running Prometheus and Grafana on the same server, use: http://localhost:9090

• If Prometheus and Grafana are on separate servers, enter the IP address of the Prometheus server, for example: http://<Prometheus_IP>:9090

Replace <Prometheus_IP> with the actual IP address of your Prometheus server.

Save the settings.

Download Grafana Dashboard JSON Files

To set up dashboards in Grafana, you need the JSON files of the dashboards. These files can either be:

1. Downloaded from a public source like Grafana's Dashboard Library.

2. Shared by someone else.

3. Created by you directly in Grafana.

• If you’re using Grafana's library, search for the dashboard by its ID and download the JSON file.

• Once downloaded, you can import the JSON file into Grafana through the Import Dashboard option in the Grafana interface.

Now click on Import and then on Upload JSON file.

Upload the JSON file you downloaded earlier, then select the Prometheus data source you just configured and click the +Import button.

Now you have a fully functional monitoring system that can scale to include additional servers and services as needed.