Prometheus

Useful Prometheus commands for reloading configuration and enabling management endpoints.

Reload configuration without restart

# Via the /-/reload HTTP endpoint (requires --web.enable-lifecycle flag)
curl -s -XPOST http://localhost:9090/-/reload

Enable the lifecycle API (edit startup flags)

# Run Prometheus with lifecycle endpoint enabled
./prometheus --web.enable-lifecycle --config.file=prometheus.yml

Or in /etc/systemd/system/prometheus.service:

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.enable-lifecycle

sudo systemctl daemon-reload && sudo systemctl restart prometheus

Check Prometheus health

curl -s http://localhost:9090/-/healthy
curl -s http://localhost:9090/-/ready

Query the API

# Instant query
curl 'http://localhost:9090/api/v1/query?query=up'

# Range query
curl 'http://localhost:9090/api/v1/query_range?query=rate(http_requests_total[5m])&start=2024-01-01T00:00:00Z&end=2024-01-01T01:00:00Z&step=60s'

The prometheus.yml file defines global settings, scrape targets, alerting rules, and Alertmanager connections.

Full example configuration

global:
  scrape_interval: 15s       # How often to scrape targets
  evaluation_interval: 15s   # How often to evaluate rules
  scrape_timeout: 10s        # Per-scrape timeout

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Load alerting rules
rule_files:
  - /etc/prometheus/rules/*.yml

# Scrape configurations
scrape_configs:
  # Prometheus monitors itself
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter (system metrics)
  - job_name: 'node'
    static_configs:
      - targets:
          - 'server1.example.com:9100'
          - 'server2.example.com:9100'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

  # Blackbox Exporter (HTTP endpoint monitoring)
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115   # Blackbox exporter address

Reload after changes

curl -s -XPOST http://localhost:9090/-/reload

The php-fpm_exporter exposes PHP-FPM process metrics (active workers, idle workers, requests/sec) to Prometheus for alerting and dashboards.

Step 1 – Enable the PHP-FPM status page

Edit your FPM pool config (e.g., /etc/php/8.1/fpm/pool.d/www.conf):

pm.status_path = /status

sudo systemctl reload php8.1-fpm

Test: curl http://127.0.0.1/status?json (adjust Nginx/Apache to allow the path on localhost).

Step 2 – Install php-fpm_exporter

wget https://github.com/hipages/php-fpm_exporter/releases/latest/download/php-fpm_exporter_linux_amd64
chmod +x php-fpm_exporter_linux_amd64
sudo mv php-fpm_exporter_linux_amd64 /usr/local/bin/php-fpm_exporter

Step 3 – Run the exporter

php-fpm_exporter server --phpfpm.scrape-uri tcp://127.0.0.1:9000/status

Or create a systemd service at /etc/systemd/system/php-fpm-exporter.service:

[Unit]
Description=PHP-FPM Exporter
After=network.target

[Service]
User=nobody
ExecStart=/usr/local/bin/php-fpm_exporter server   --phpfpm.scrape-uri tcp://127.0.0.1:9000/status
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now php-fpm-exporter

Step 4 – Add to Prometheus scrape config

  - job_name: 'php-fpm'
    static_configs:
      - targets: ['localhost:9253']

Key metrics

• phpfpm_active_processes — workers currently handling requests
• phpfpm_idle_processes — workers waiting for requests
• phpfpm_max_active_processes — highest active worker count since start
• phpfpm_slow_requests — requests exceeding request_slowlog_timeout

Install Prometheus from the official GitHub releases and configure it as a managed systemd service.

Step 1 – Create the Prometheus user

sudo useradd -rs /bin/false prometheus

Step 2 – Download and install

wget https://github.com/prometheus/prometheus/releases/download/v3.5.0/prometheus-3.5.0.linux-amd64.tar.gz
tar xvfz prometheus-3.5.0.linux-amd64.tar.gz
cd prometheus-3.5.0.linux-amd64

sudo mv prometheus promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo mv consoles console_libraries /etc/prometheus/
sudo mv prometheus.yml /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Step 3 – Create the systemd service

Create /etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus   --config.file=/etc/prometheus/prometheus.yml   --storage.tsdb.path=/var/lib/prometheus   --web.console.templates=/etc/prometheus/consoles   --web.console.libraries=/etc/prometheus/console_libraries   --web.enable-lifecycle   --storage.tsdb.retention.time=30d
Restart=always

[Install]
WantedBy=multi-user.target

Step 4 – Start and enable

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheus

Step 5 – Verify

curl -s http://localhost:9090/-/healthy
# Open http://YOUR_SERVER_IP:9090 in a browser

The Blackbox Exporter probes external endpoints (HTTP, TCP, ICMP, DNS) and exposes the results as Prometheus metrics — useful for uptime monitoring and SSL certificate expiry alerting.

Step 1 – Create service user and install

sudo useradd -rs /bin/false blackbox_exporter

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.27.0/blackbox_exporter-0.27.0.linux-amd64.tar.gz
tar xvfz blackbox_exporter-0.27.0.linux-amd64.tar.gz
cd blackbox_exporter-0.27.0.linux-amd64

sudo mv blackbox_exporter /usr/local/bin/
sudo mv blackbox.yml /etc/prometheus/
sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
sudo chown blackbox_exporter:blackbox_exporter /etc/prometheus/blackbox.yml

Step 2 – Default blackbox.yml modules

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: []   # 2xx range
      fail_if_ssl: false
      fail_if_not_ssl: false

  http_post_2xx:
    prober: http
    http:
      method: POST

  tcp_connect:
    prober: tcp

  icmp:
    prober: icmp

Step 3 – Create systemd service

Create /etc/systemd/system/blackbox_exporter.service:

[Unit]
Description=Prometheus Blackbox Exporter
After=network.target

[Service]
User=blackbox_exporter
ExecStart=/usr/local/bin/blackbox_exporter   --config.file=/etc/prometheus/blackbox.yml
Restart=always

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now blackbox_exporter

Step 4 – Add to Prometheus scrape config

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://yourdomain.com
          - https://api.yourdomain.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Key metrics

• probe_success — 1 if probe succeeded, 0 if failed
• probe_duration_seconds — total probe time
• probe_ssl_earliest_cert_expiry — SSL cert expiry timestamp (alert when < 30 days)
• probe_http_status_code — HTTP response code

Grafana provides dashboards and alerting for Prometheus (and many other data sources). Install it alongside Prometheus for a complete monitoring stack.

Step 1 – Add the Grafana repository

wget -q -O - https://apt.grafana.com/gpg.key |   gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" |   sudo tee /etc/apt/sources.list.d/grafana.list

Step 2 – Install and start

sudo apt-get update
sudo apt-get install -y grafana
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server
sudo systemctl status grafana-server

Step 3 – Access the web UI

Open http://YOUR_SERVER_IP:3000 in a browser.
Default credentials: admin / admin (you'll be prompted to change the password on first login).

Step 4 – Add Prometheus as a data source

1. Configuration → Data Sources → Add data source.
2. Select Prometheus.
3. URL: http://localhost:9090
4. Click Save & Test.

Step 5 – Import a dashboard

Go to Dashboards → Import and enter a dashboard ID from grafana.com/grafana/dashboards:

• 1860 — Node Exporter Full (system metrics)
• 4271 — Redis Dashboard
• 9628 — PostgreSQL