Blackbox Exporter Probes

Jump to: probe modules, the Prometheus scrape config, alert rules or troubleshooting.

Before you start

Node Exporter tells you what's happening inside a machine: CPU, memory, disk. It can't tell you whether your website actually answers, whether its certificate is about to expire, or whether DNS still resolves. The Blackbox Exporter answers those questions by probing endpoints from the outside, the way a user would, and reporting the results as Prometheus metrics.

This guide adds it to the Prometheus, Grafana and Alertmanager stack from the Prometheus Grafana Monitoring tutorial. You need that stack running, or any Prometheus server you can edit, plus Docker Engine with the Compose plugin.

One idea makes the rest of the setup make sense. Unlike most exporters, the Blackbox Exporter doesn't have a fixed list of things to check. Prometheus tells it what to probe on every scrape, by passing a target and a module as URL parameters:

http://blackbox:9115/probe?target=https://techwilla.com&module=http_2xx

The exporter runs the probe and returns the result. So the list of targets lives in prometheus.yml, and the exporter's own config file only describes how to probe.

Add the exporter to the stack

Add a service to the existing compose.yaml in your monitoring project:

  blackbox:
    image: prom/blackbox-exporter:v0.29.0
    restart: unless-stopped
    command:
      - --config.file=/etc/blackbox/blackbox.yml
    volumes:
      - ./blackbox.yml:/etc/blackbox/blackbox.yml:ro
    ports:
      - "127.0.0.1:9115:9115"

Pin the version you deploy. Check the project's releases page for the current one and read its changelog before upgrading, since module options are occasionally renamed. Prometheus reaches the exporter as blackbox:9115 on the project network. The published port, bound to 127.0.0.1, is only there so you can test probes by hand from the host.

Probe modules

A module is a named recipe for one kind of probe. Create blackbox.yml next to compose.yaml:

modules:
  http_2xx:
    prober: http
    timeout: 10s
    http:
      preferred_ip_protocol: ip4
      valid_status_codes: []          # empty means any 2xx
      follow_redirects: true
      fail_if_not_ssl: true

  http_keyword:
    prober: http
    timeout: 10s
    http:
      preferred_ip_protocol: ip4
      fail_if_body_not_matches_regexp:
        - "TechWilla"

  tcp_connect:
    prober: tcp
    timeout: 5s
    tcp:
      preferred_ip_protocol: ip4

  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: ip4

  dns_a:
    prober: dns
    timeout: 5s
    dns:
      query_name: techwilla.com
      query_type: A
      valid_rcodes: [NOERROR]
      preferred_ip_protocol: ip4

• http_2xx passes when the final response, after redirects, has a 2xx status. fail_if_not_ssl makes it fail if the site ends up on plain HTTP, which catches a broken HTTPS redirect.
• http_keyword also requires a string in the body. Use it for apps that return 200 OK with an error page when they're broken. Replace TechWilla with text that only a working page contains.
• tcp_connect only proves a TCP handshake completed. Use it for databases, SSH or mail ports.
• icmp is a ping. It needs extra privileges; see ICMP in a container.
• dns_a sends the query to the resolver given as the target. The target here is a DNS server, not a website.

preferred_ip_protocol: ip4 matters on hosts without working IPv6. The default prefers IPv6, and a name that resolves to an AAAA record then fails on a machine that can't route it.

Start the exporter and check the config loaded:

docker compose up -d blackbox
docker compose logs blackbox | tail -n 5       # "Loaded config file" and "Listening on"

Test a probe by hand

Before involving Prometheus, run a probe directly. debug=true adds a log of every step to the output:

curl -s "http://127.0.0.1:9115/probe?target=https://techwilla.com&module=http_2xx" | grep -E '^probe_(success|http_status_code|duration_seconds|ssl_earliest_cert_expiry)'

curl -s "http://127.0.0.1:9115/probe?target=https://techwilla.com&module=http_2xx&debug=true" | head -n 40

The metrics you'll use most:

The phase breakdown is the useful part when a site feels slow. A large connect time points at the network, a large tls time at the TLS handshake, and a large processing time at the application itself.

The Prometheus scrape config

Each probe target needs three things from Prometheus: the target passed as a URL parameter, a sensible instance label, and the scrape sent to the exporter rather than to the target itself. Relabelling does all three. Add a job per module to prometheus.yml:

  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://techwilla.com
          - https://rss.example.net
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

  - job_name: blackbox-tcp
    metrics_path: /probe
    params:
      module: [tcp_connect]
    static_configs:
      - targets:
          - 192.168.1.20:22
          - 192.168.1.30:5432
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

  - job_name: blackbox-exporter
    static_configs:
      - targets: [blackbox:9115]

Replace the targets with your own. Reading the relabel rules in order:

1. Copy the address you listed, such as https://techwilla.com, into the target URL parameter.
2. Copy it into the instance label too, so graphs and alerts show the site's address instead of the exporter's.
3. Replace the address Prometheus actually connects to with the exporter.

The last job scrapes the exporter's own metrics, so you notice if the exporter itself stops working. Check the file and reload Prometheus:

docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
docker compose kill -s SIGHUP prometheus

In the Prometheus web interface, Status → Targets should list the new jobs as UP. That only means the scrape of the exporter worked. Whether the site passed is in probe_success:

probe_success{job=~"blackbox-.*"}

Alert rules

Add these to the rules file your Prometheus loads, such as alert_rules.yml from the base tutorial:

groups:
  - name: blackbox
    rules:
      - alert: EndpointDown
        expr: probe_success == 0
        for: 3m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} is failing its {{ $labels.job }} probe"

      - alert: EndpointSlow
        expr: avg_over_time(probe_duration_seconds[10m]) > 2
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} averaged {{ $value | humanizeDuration }} over 10 minutes"

      - alert: CertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate for {{ $labels.instance }} expires in {{ $value | humanize }} days"

      - alert: BlackboxExporterDown
        expr: up{job="blackbox-exporter"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "The Blackbox Exporter isn't responding, so probe alerts are blind"

• for: 3m means the probe must fail on every scrape for three minutes before the alert fires. That filters out a single dropped packet without delaying a real outage much.
• The certificate alert gives two weeks of warning. Automated renewals normally happen 30 days before expiry, so an alert at 14 days means renewal has been failing for over two weeks.
• The last rule matters as much as the others. If the exporter dies, every probe_success series stops, and probe_success == 0 never fires because there's no data to compare.

docker compose exec prometheus promtool check rules /etc/prometheus/alert_rules.yml
docker compose kill -s SIGHUP prometheus

Test the path end to end by adding a target that can't succeed, such as https://does-not-exist.techwilla.com. After three minutes it should appear in Alerts as firing and reach your Alertmanager receiver. Remove it once you've seen the notification.

ICMP in a container

Sending ping packets needs a raw socket, which ordinary processes can't open. Two options, in order of preference:

• Let unprivileged processes use ICMP datagram sockets. On most current distributions, the net.ipv4.ping_group_range sysctl already allows this, and Docker sets it inside containers by default. In that case the icmp module works as is.
• If probes fail with socket: operation not permitted, add the NET_RAW capability to the service with cap_add: [NET_RAW].

Ping only proves a host's network stack answers. Prefer a TCP or HTTP probe on a service the host actually provides.

Where the probes run from

A probe measures the path from the exporter to the target. Running the exporter on your monitoring server tells you whether services are reachable from inside your network. That's different from whether users on the internet can reach them: an expired DNS record or a broken port forward can leave a site working internally and unreachable from outside.

For public services, run a second exporter somewhere outside your network, such as a small cloud VM, and scrape it from your Prometheus over a VPN. Add a probe_location label to each job so alerts say which vantage point failed. A failure from outside only points at DNS, the router or the ISP. A failure from both points at the service.

Scrape intervals and timeouts

Three timeouts interact, and getting them out of order produces probes that fail for no visible reason:

• The module's timeout is how long the exporter gives the probe itself.
• Prometheus's scrape_timeout is how long Prometheus waits for the exporter's answer. It defaults to 10 seconds.
• The scrape_interval is how often the whole thing runs.

The module timeout must be shorter than the scrape timeout, or Prometheus gives up first and records a failed scrape instead of a failed probe. The exporter helps here: it reads the X-Prometheus-Scrape-Timeout-Seconds header Prometheus sends and caps the probe slightly below it. Setting both explicitly still makes the intent clear. A probe every 30 to 60 seconds is plenty for most services. Probing every 5 seconds won't detect an outage meaningfully sooner once the alert's for: duration is applied, and it adds load to every target.

Slow endpoints, such as a login page that queries a database, deserve their own job with a longer interval and timeout, rather than a long timeout on every probe.

A Grafana panel

A table of every probe with its last result covers most needs. Create a panel with these queries, set it to the Table visualisation, and use the Join by field transformation on instance:

probe_success{job=~"blackbox-.*"}
probe_duration_seconds{job=~"blackbox-.*"}
(probe_ssl_earliest_cert_expiry - time()) / 86400

Add value mappings so 1 shows as Up in green and 0 as Down in red. The community dashboard published for the exporter is a quick alternative, but check which metric names it expects against your version before relying on it.

Troubleshooting

The target is UP in Prometheus but probe_success is 0

That's normal when the probe fails: the scrape of the exporter worked, the probe didn't. Run the same probe by hand with debug=true to see which step failed.

probe_success is 0 and the debug log mentions IPv6

The name resolves to an AAAA record and the exporter tried IPv6 first. Set preferred_ip_protocol: ip4 in the module, or fix IPv6 routing on the host.

Every target in a job shows the exporter's address as instance

The relabel rules are in the wrong order, or the second rule is missing. The order must be: copy to __param_target, copy to instance, then replace __address__.

HTTPS probes to internal services fail with certificate errors

The service uses a certificate from your own CA. Mount the CA certificate into the container and point the module at it with tls_config: { ca_file: /etc/blackbox/internal-ca.pem }. Don't set insecure_skip_verify: the probe would then pass for a broken or spoofed certificate, which is exactly what it should catch.

Next steps

• Prometheus Grafana Monitoring: the base stack, including Alertmanager receivers.
• Uptime Kuma Uptime Monitoring: a simpler, web-based alternative for the same kind of checks.
• The exporter's configuration reference lists every module option.