Linux Performance Monitoring

Jump to: the first minute, CPU, memory, disk I/O, network or history with sar.

Before you start

When a Linux server feels slow, the cause is almost always one of four resources running out: CPU, memory, disk I/O or the network. This guide works through the standard command-line tools for each one, in the order that finds the problem fastest. Everything here reads system state and changes nothing, so it's safe to run on a production machine.

Most of the tools are already installed. The rest come from two packages, sysstat (for iostat, mpstat, pidstat and sar) and htop:

sudo apt update
sudo apt install sysstat htop iotop

sudo dnf install sysstat htop iotop

A method helps more than any single tool. For each resource, check three things, a framework Brendan Gregg calls USE:

• Utilisation: how busy is it?
• Saturation: is work queueing up, waiting for it?
• Errors: is anything failing?

Saturation is the one people skip, and it's usually the answer. A disk that's 60% busy with a long queue is a bigger problem than one at 95% with no queue at all.

The first minute

Versions: the options and columns described here are those documented in the vmstat, iostat and sar manual pages linked under Next steps. Output layouts differ a little between sysstat releases and between distributions, so check iostat -V and the man pages on your own system before relying on a specific column position.

Run these first. Between them they point at the right resource most of the time.

uptime
dmesg -T | tail -n 20
vmstat 1 5
free -h
df -h

uptime prints the load average for the last 1, 5 and 15 minutes. On Linux, load counts processes that are running, waiting for a CPU, or waiting for disk I/O in uninterruptible sleep. Compare it with the number of CPUs from nproc. A load of 4 on a 4-core machine is full; 12 means work is queueing. Because disk waits count too, a high load with idle CPUs points at storage. The three numbers also show direction: 8.1, 4.2, 1.9 is getting worse, 1.9, 4.2, 8.1 is recovering.

dmesg shows kernel messages. Look for Out of memory: Killed process, disk errors such as I/O error or ata1: hard resetting link, and network drivers resetting. A single line here can explain everything else you'll see.

df -h catches the most common cause of a "slow" server, which is actually a full disk making applications fail and retry. df -i checks inodes, which can run out while space is still free.

Reading vmstat

vmstat 1 5 prints a line every second, five times. Ignore the first line: it's an average since boot.

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  0      0 412308  91240 2810344    0    0     4    18  612 1103  7  2 90  1  0
 9  4      0 398112  91240 2810512    0    0  8204   112 2954 5120 22  6 31 41  0

The second line above tells a clear story: four processes blocked, 41% I/O wait and 8 MB per second being read. That's the disk, not the CPU, even though the load average will look like a CPU problem.

CPU

top and htop sort processes by CPU use. In htop, press F6 to choose the sort column, F4 to filter by name and t for a tree view that shows which parent started a swarm of workers. Two numbers are easy to misread:

• A process at 350% isn't a bug in the display. It's using three and a half cores.
• %CPU for a whole machine of 25% can still mean one core is completely busy. A single-threaded program can't use more than one core, so it's stuck at 100% of that core while the rest sit idle.

To see each core separately:

mpstat -P ALL 1 3

One core near 100% while the others are idle points at a single-threaded bottleneck, or at network interrupts all landing on one core. To see which processes use CPU over a few seconds, rather than at one instant:

pidstat 1 5               # per-process CPU, every second
pidstat -t -p 1234 1 5    # per-thread, for one process ID

On a virtual machine, check st in vmstat or %steal in mpstat. Steal time is CPU your VM wanted but the hypervisor gave to someone else. You can't fix it from inside the VM; it needs a larger instance or a less busy host.

Memory

free -h

               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       3.1Gi       312Mi       184Mi       4.2Gi       4.1Gi
Swap:          2.0Gi          0B       2.0Gi

A small free number isn't a problem. Linux uses spare memory as a disk cache (buff/cache) and gives it back the moment programs need it. The number that matters is available: an estimate of how much memory programs can use without swapping. In the example, 4.1 GiB is available, so the machine is fine despite only 312 MiB being "free".

Signs of real memory pressure:

• available is a small fraction of total and keeps shrinking.
• vmstat shows sustained swap-out (so). Swap being used is fine; memory being actively moved to swap is what hurts.
• dmesg shows the OOM killer stopping processes.

Newer kernels report pressure directly. cat /proc/pressure/memory shows the share of time tasks were stalled waiting for memory. A some avg10 value above a few percent means memory is slowing things down right now. The same files exist for cpu and io.

To find what's using the memory, sort by resident size:

ps -eo pid,user,rss,comm --sort=-rss | head -n 10      # RSS in KiB
systemd-cgtop -m                                       # memory per systemd service and container

RSS counts shared libraries in every process that uses them, so the column can add up to more than total memory. For services, systemd-cgtop gives cleaner totals per service, including Docker containers.

Disk I/O

iostat is the main tool. -x adds the extended columns, -z hides idle devices:

iostat -xz 1 5

Latency (await) is what users feel. A hard drive handles about 100 to 200 random requests per second. A workload doing 400 random writes a second will queue on it however the other numbers look.

To find which process is doing the I/O:

sudo iotop -oPa          # only processes doing I/O, accumulated totals
pidstat -d 1 5           # per-process read and write rates

Common culprits are backups, database checkpoints, log files growing quickly, and swap. If iotop shows the kernel thread kswapd busy, the disk problem is really a memory problem.

Network

ip -s link show eth0           # packet counts, errors and drops per interface
sar -n DEV 1 5                 # throughput per interface, per second
ss -s                          # socket summary by state
ss -tnp state established      # established TCP connections and their processes

Replace eth0 with your interface from ip -brief link. Things to check:

• Errors and drops in ip -s link that keep rising point at a bad cable, a duplex mismatch or a driver problem. ethtool eth0 shows the negotiated speed; a gigabit link that came up at 100 Mb/s is a classic.
• Throughput near the link speed in sar -n DEV: the link is full.
• Retransmits. nstat -az TcpRetransSegs shows the counter. A steadily rising rate means packet loss somewhere between you and the clients.
• Thousands of sockets in TIME-WAIT or CLOSE-WAIT in ss -s. Many CLOSE-WAIT sockets usually mean an application that isn't closing its connections.

Slow responses aren't always the server. Name resolution is a frequent hidden delay: if getent hosts example.net takes seconds, every outbound connection does too.

History with sar

Every tool so far shows the present. Problems at 03:00 have usually gone by the time anyone looks. sar from the sysstat package records CPU, memory, disk and network figures every ten minutes and keeps them for about a month.

On Debian and Ubuntu, collection is switched off after install. Turn it on:

sudo sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl enable --now sysstat

On Fedora and RHEL, enabling the service is enough: sudo systemctl enable --now sysstat. Collection is driven by systemd timers, and systemctl list-timers 'sysstat*' shows them.

After a few hours, you can look back:

sar -u                          # CPU for today
sar -r                          # memory
sar -d -p                       # disk, with readable device names
sar -n DEV                      # network
sar -q -s 02:00:00 -e 04:00:00  # load and run queue between 02:00 and 04:00
ls /var/log/sysstat/            # one file per day (/var/log/sa on Fedora and RHEL)
sar -u -f /var/log/sysstat/sa15 # CPU on the 15th

sar is a record of what happened, not an alerting system. For alerts and graphs, the Prometheus Grafana Monitoring stack collects the same kernel figures through Node Exporter.

Containers

Containers are ordinary processes, so every tool above sees them. To attribute load to a container rather than a process ID:

docker stats --no-stream        # CPU, memory, network and block I/O per container
systemd-cgtop                   # every cgroup, including each container

A container hitting its memory limit is killed even when the host has plenty free; docker inspect --format '{{.State.OOMKilled}}' name shows whether that happened. For history per container, see Container Metrics with cAdvisor.

Cheat sheet

Troubleshooting the tools

Cannot open /var/log/sysstat/saNN: No such file or directory

Collection isn't running, or hasn't run yet today. Check ENABLED in /etc/default/sysstat on Debian and Ubuntu, then systemctl status sysstat.

iotop says CONFIG_TASK_DELAY_ACCT not enabled

Some kernels ship with per-task delay accounting off. Add delayacct to the kernel command line, or use pidstat -d, which doesn't need it.

The load average is high but every tool shows idle CPUs

Processes are stuck in uninterruptible sleep, usually waiting on storage or a hung network filesystem. ps -eo state,pid,comm | awk '$1=="D"' lists them. An NFS mount that has lost its server is a frequent cause.

Next steps

• Prometheus Grafana Monitoring: the same figures, recorded continuously and with alerts.
• SMART Disk Health Monitoring: if dmesg showed disk errors, check the drive itself.
• Container Metrics with cAdvisor: per-container history.
• The man pages for vmstat, iostat and sar define every column.