cAdvisor Container Monitoring with Prometheus and Grafana

Jump to: running cAdvisor, the Prometheus scrape job, PromQL for containers or alert rules.

Before you start

docker stats shows what each container is using right now. It can't tell you what a container was doing at 03:00, whether its memory has been creeping up for a week, or how close it came to its limit before it was killed. cAdvisor (Container Advisor) fills that gap. It reads the kernel's control-group statistics for every container on a host and exposes them as Prometheus metrics, so Prometheus can record them and Grafana can graph them.

This guide adds cAdvisor to the stack from Prometheus Grafana Monitoring. You need that stack, or another Prometheus server you can edit, and Docker Engine with the Compose plugin on each host you want to watch.

It helps to know where the numbers come from. Docker places every container in its own cgroup, and the kernel accounts for the CPU time, memory, block I/O and process count of each cgroup. cAdvisor reads those counters from /sys/fs/cgroup and the network counters from each container's network namespace. Node Exporter, by contrast, reports the host as a whole. Run both: one tells you the machine is short of memory, the other tells you which container took it.

stat -fc %T /sys/fs/cgroup     # cgroup2fs on current distributions (cgroup v2)
docker info --format '{{.CgroupVersion}} {{.CgroupDriver}}'

Every distribution released in the last few years uses cgroup v2 with the systemd driver. Recent cAdvisor releases support it fully. Very old releases don't, so use a current one.

Running cAdvisor

cAdvisor needs read access to several host paths to see every container. Add it to the monitoring project's compose.yaml, or to a small project of its own on each additional host:

compose.yaml (add under services:)

  cadvisor:
    image: ghcr.io/google/cadvisor:0.60.6
    restart: unless-stopped
    privileged: true
    devices:
      - /dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    command:
      - --docker_only=true
      - --housekeeping_interval=15s
      - --store_container_labels=false
      - --whitelisted_container_labels=com.docker.compose.project,com.docker.compose.service
      - --disable_metrics=percpu,sched,tcp,udp,advtcp,hugetlb,referenced_memory,cpu_topology,resctrl
    ports:
      - "127.0.0.1:8081:8080"

Check the project's releases page for the current version and pin it. Current images are published on the GitHub Container Registry (ghcr.io), and the tag has no leading v. Older tutorials point at a Docker Hub or gcr.io image that is no longer updated.

What the options do:

The read-only mounts give cAdvisor the cgroup tree, Docker's container metadata and the disk devices it uses to label filesystem metrics. The mounts are the same as in the project's own run instructions.
privileged: true is in cAdvisor's documented setup and is needed on many hosts to read every counter. It also gives the container full access to the host, so treat it accordingly: use the official image, pin the version and never publish the port to the network.
--docker_only=true reports Docker containers and the root cgroup only, instead of every systemd service on the host.
--housekeeping_interval=15s matches a typical Prometheus scrape interval. The default of one second costs CPU and adds nothing if Prometheus only scrapes every 15.
The label options stop cAdvisor copying every container label into every metric. Without them, images with many labels create a large number of metric series, which slows Prometheus down. Only the two Compose labels, project and service, are kept.
--disable_metrics turns off the families most people never query. The per-CPU family on its own multiplies the series count by the number of cores.

docker compose up -d cadvisor
docker compose logs cadvisor | tail -n 5
curl -s http://127.0.0.1:8081/metrics | grep -c '^container_'

cAdvisor also serves a basic web interface on the same port at /containers/. It's useful for a quick look from the host with an SSH tunnel, but Grafana is the better place for day-to-day use.

The Prometheus scrape job

prometheus.yml (add under scrape_configs:)

  - job_name: cadvisor
    scrape_interval: 15s
    static_configs:
      - targets: ['cadvisor:8080']
        labels:
          host: monitor-01
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'container_(tasks_state|memory_failures_total|blkio_device_usage_total)'
        action: drop

Prometheus reaches cAdvisor by service name on the project network, on its internal port 8080. The host label tells hosts apart once several are scraped; set it to each machine's name. The metric_relabel_configs block drops a few more high-volume families at scrape time, as a second line of defence.

For other hosts, publish cAdvisor on the host's private or VPN address only, for example "10.8.0.5:8081:8080", and add each one as a target. cAdvisor has no authentication of its own, and its metrics list every container name and image on the host, so keep it off public interfaces. Docker's port publishing bypasses ufw, as Docker Networking Explained describes.

docker compose exec prometheus promtool check config /etc/prometheus/prometheus.yml
docker compose kill -s SIGHUP prometheus

In Prometheus, Status → Targets should show the cadvisor job as UP.

Which series are containers?

cAdvisor reports the root cgroup and some system cgroups alongside the containers. Container series have a non-empty name label, the container's name, and an image label. Add name!="" to every query, or you'll count the host's total along with each container in it:

count(container_last_seen{name!=""})      # how many containers cAdvisor sees

With the label options above, Compose containers also carry container_label_com_docker_compose_project and container_label_com_docker_compose_service. Those are steadier than container names for dashboards, because Compose can change the numeric suffix when it recreates a container.

PromQL for containers

CPU

# CPU used by each container, in cores (1 = one full core)
sum by (name) (rate(container_cpu_usage_seconds_total{name!=""}[5m]))

# As a share of the container's CPU limit, where one is set
sum by (name) (rate(container_cpu_usage_seconds_total{name!=""}[5m]))
  / on (name) (container_spec_cpu_quota{name!=""} / container_spec_cpu_period{name!=""})

# Time spent throttled because the container hit its CPU limit
sum by (name) (rate(container_cpu_cfs_throttled_seconds_total{name!=""}[5m]))

Throttling is the CPU number people miss. A container with a cpus: 1.0 limit that's throttled for much of each second responds slowly, even though host CPU looks idle.

Memory

# Working set: what the kernel counts against the memory limit
container_memory_working_set_bytes{name!=""}

# Working set as a share of the limit (only for containers with a limit)
container_memory_working_set_bytes{name!=""}
  / (container_spec_memory_limit_bytes{name!=""} > 0)

Use the working set rather than container_memory_usage_bytes. Usage includes page cache that the kernel can reclaim, so it climbs towards the limit on any container that reads files, without meaning anything. The working set excludes the reclaimable part, and it's what the OOM killer acts on.

Network and disk

# Received and transmitted bytes per second
sum by (name) (rate(container_network_receive_bytes_total{name!=""}[5m]))
sum by (name) (rate(container_network_transmit_bytes_total{name!=""}[5m]))

# Block I/O read and written per second
sum by (name) (rate(container_fs_reads_bytes_total{name!=""}[5m]))
sum by (name) (rate(container_fs_writes_bytes_total{name!=""}[5m]))

Containers that share the host's network with network_mode: host have no network series of their own. Their traffic appears in the host's Node Exporter figures instead.

Restarts

cAdvisor doesn't count restarts directly, but a container that restarts gets a new start time:

changes(container_start_time_seconds{name!=""}[1h])

A value above zero means the container started again within the last hour.

Alert rules

alert_rules.yml (add a group)

groups:
  - name: containers
    rules:
      - alert: ContainerNearMemoryLimit
        expr: |
          container_memory_working_set_bytes{name!=""}
            / (container_spec_memory_limit_bytes{name!=""} > 0) > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.name }} on {{ $labels.host }} is using {{ $value | humanizePercentage }} of its memory limit"

      - alert: ContainerCpuThrottled
        expr: |
          sum by (name, host) (rate(container_cpu_cfs_throttled_periods_total{name!=""}[5m]))
            / sum by (name, host) (rate(container_cpu_cfs_periods_total{name!=""}[5m])) > 0.25
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.name }} is CPU-throttled in {{ $value | humanizePercentage }} of periods"

      - alert: ContainerRestarting
        expr: changes(container_start_time_seconds{name!=""}[15m]) > 2
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.name }} on {{ $labels.host }} restarted {{ $value }} times in 15 minutes"

      - alert: CadvisorDown
        expr: up{job="cadvisor"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "cAdvisor on {{ $labels.host }} isn't being scraped, so container alerts are blind"

A container that hits its limit is killed in an instant, so an alert only after the fact isn't much use. Alerting at 90% for ten minutes gives warning of a slow climb, which is how most leaks behave. The last rule is there because every other rule here goes silent if cAdvisor stops.

Containers that should be running but aren't are a different question. cAdvisor stops reporting a container once it's gone, and an absent series can't trip a threshold. Use absent() for the few that matter:

absent(container_last_seen{container_label_com_docker_compose_service="miniflux"})

Check and reload as before:

docker compose exec prometheus promtool check rules /etc/prometheus/alert_rules.yml
docker compose kill -s SIGHUP prometheus

Grafana dashboard

Build a dashboard with a name variable so one dashboard serves every container. Under Dashboard settings → Variables, add a Query variable named container with:

label_values(container_last_seen{name!=""}, name)

Turn on Multi-value and Include All option, then use name=~"$container" in each panel's query. Four time-series panels cover most needs: CPU cores, memory working set, network in and out, and disk read and write. Add the throttling query as a fifth when you set CPU limits.

Community dashboards for cAdvisor exist on Grafana's dashboard site and are a quick start, but many were written for cgroup v1 and query metrics or labels that are now empty. If a panel shows No data, open its query and compare the metric names with what curl -s http://127.0.0.1:8081/metrics returns.

Monitoring several hosts

Each Docker host runs its own cAdvisor, and one Prometheus scrapes them all. Keep the scrape path private. The simplest arrangements are:

A private network or VPN between the hosts, such as Tailscale or WireGuard, with each cAdvisor published only on that interface's address.
The host firewall, if the machines share a trusted LAN. Remember that a port published by Docker bypasses ufw, so bind it to the LAN address and filter in DOCKER-USER if other devices share that network.

Give every target a host label in prometheus.yml, and include that label in every alert summary, as the rules above do. A notification that says "miniflux is near its memory limit" is far less useful at 3 am than one that names the machine. In Grafana, add a second dashboard variable for host and filter the container variable by it, so you can pick a machine and then one of its containers.

Keeping cAdvisor cheap

cAdvisor itself uses some CPU and memory, and every metric series it exposes costs Prometheus memory and disk. On a host with fifty containers, the defaults can produce tens of thousands of series. To check your numbers:

# In Prometheus: series per metric from cAdvisor, largest first
topk(15, count by (__name__) ({job="cadvisor"}))

Keep --docker_only=true and the label whitelist. Container labels copied into metrics are the biggest single source of unnecessary series.
Add metric families you don't use to --disable_metrics, and drop individual metrics at scrape time with metric_relabel_configs.
Don't lower --housekeeping_interval below the scrape interval.
docker stats cadvisor shows its own usage. A few percent of one core is normal on a busy host.

Troubleshooting

Metrics exist but the `name` and `image` labels are empty

cAdvisor can see the cgroups but not Docker's metadata. Check that /var/lib/docker and /var/run are mounted, and that Docker's data directory is the default. If you moved it with data-root in daemon.json, mount that path instead.

`Failed to get system UUID` or `Could not configure a source for OOM detection` in the log

These are warnings about optional features and usually harmless. The OOM warning goes away when /dev/kmsg is passed in with devices:, as in the example.

Memory panels show values close to the limit for every container

The query uses container_memory_usage_bytes, which includes reclaimable cache. Switch to container_memory_working_set_bytes.

cAdvisor's CPU use is high

Raise --housekeeping_interval, disable more metric families, and check whether you have very many short-lived containers, such as CI jobs, each creating new series.

Next steps

Prometheus Grafana Monitoring: the base stack, with Node Exporter for host metrics.
Hardening Docker Containers: set the memory, CPU and PID limits these alerts measure against.
Linux Performance Monitoring: command-line tools for the moment an alert fires.
cAdvisor's Prometheus metrics reference lists every metric it exposes.

Container Metrics with cAdvisor