SMART Disk Health Monitoring

Jump to: reading smartctl output, self-tests, automatic monitoring with smartd or what to do when a drive fails a check.

Before you start

Hard drives and SSDs record their own health data: error counters, temperature, hours of use and, for SSDs, how much of their write endurance is left. This is S.M.A.R.T. (Self-Monitoring, Analysis and Reporting Technology). Reading it regularly gives you a chance to replace a drive while it still works, instead of after it doesn't.

Set expectations first. SMART is a warning system, not a prediction. Plenty of drives fail without any warning in their SMART data, and some drives run for years with a few bad sectors. It improves your odds; it doesn't replace backups. If a drive holds anything you can't lose, it needs a copy somewhere else whatever SMART says.

The tool is smartctl, from the smartmontools project. Install it from your distribution's repositories:

sudo apt update
sudo apt install smartmontools

sudo dnf install smartmontools

These commands talk to the hardware directly, so run them on the physical machine or on a VM host. A virtual disk inside a VM has no SMART data, and the host is where you need to look.

smartctl --version | head -n 1     # smartmontools 7.4 or later reads current NVMe drives well

Find your drives

lsblk -d -o NAME,MODEL,SERIAL,SIZE,ROTA,TRAN
sudo smartctl --scan

ROTA is 1 for spinning disks and 0 for SSDs. TRAN shows the connection: sata, nvme or usb. smartctl --scan lists the device paths and the device type smartctl will use for each.

Device names such as /dev/sda can change between reboots. When you set up monitoring, refer to drives by their stable names under /dev/disk/by-id/, which include the model and serial number:

ls -l /dev/disk/by-id/ | grep -v -- -part

Note each drive's serial number now. When one fails, it's how you know which physical drive to pull out of the machine.

Drives behind USB adapters

Many USB enclosures and adapters don't pass SMART commands through by default. If smartctl -a /dev/sdb reports Unknown USB bridge or shows no attributes, tell it to use SCSI-to-ATA translation:

sudo smartctl -d sat -a /dev/sdb

If that fails too, the bridge chip doesn't support SMART at all. The smartmontools project keeps a list of tested USB bridges on its website. For drives you rely on, a SATA connection or an enclosure known to pass SMART through is worth the trouble.

Reading smartctl output

Start with the overall verdict, then read the details:

sudo smartctl -H /dev/sda          # overall health self-assessment
sudo smartctl -a /dev/sda          # everything: identity, attributes, error and self-test logs
sudo smartctl -x /dev/sda          # extended, including the device statistics page

SMART overall-health self-assessment test result: PASSED means only that no attribute has crossed the manufacturer's failure threshold. Those thresholds are set so low that drives often pass right up until they fail. A FAILED result is serious, but PASSED on its own tells you very little. The useful information is in the attribute table.

SATA hard drives and SSDs

SATA drives report numbered attributes. Each has a normalised VALUE, a WORST and a THRESH, all on a vendor scale, plus a RAW_VALUE. For the attributes that matter most, read the raw value: it's the actual count.

Large raw values in attributes 1 (Raw_Read_Error_Rate) and 7 (Seek_Error_Rate) on Seagate drives are normal. Those values pack several counters into one number and look alarming when read as a single count. Compare against the normalised value and the threshold instead.

The trend matters more than any single reading. Five reallocated sectors that have stayed at five for two years is a different situation from five that were zero last month.

NVMe drives

NVMe drives use a standard health log instead of numbered attributes, which makes them easier to read:

sudo smartctl -a /dev/nvme0

Running self-tests

Attributes show what the drive has noticed during normal use. A self-test makes it actively check itself. Both kinds run inside the drive, in the background, while the system keeps working:

sudo smartctl -t short /dev/sda       # a couple of minutes: electrical and mechanical checks, a sample read
sudo smartctl -t long /dev/sda        # reads every sector; hours on a large hard drive

smartctl prints the estimated duration and returns immediately. Check the result once that time has passed:

sudo smartctl -l selftest /dev/sda

Completed without error is what you want. Completed: read failure includes an LBA_of_first_error, a sector the drive couldn't read. A long test that keeps aborting at the same point is a strong sign of a failing drive.

Self-tests compete with normal disk use and run slower on a busy drive. Some USB enclosures put the drive to sleep partway through, which aborts the test. On NVMe drives, self-tests need firmware support; sudo smartctl -c /dev/nvme0 shows whether the drive offers them.

Automatic monitoring with smartd

Reading SMART by hand works once. Monitoring means checking every day without having to remember. The smartmontools package includes smartd, a daemon that polls your drives, runs scheduled self-tests and sends a message when something changes.

Its configuration file is /etc/smartd.conf (/etc/smartmontools/smartd.conf on Fedora and RHEL). By default it contains a single DEVICESCAN line. Replace it with one line per drive, so a drive that disappears is noticed instead of silently skipped:

# One line per drive. Use your own /dev/disk/by-id names.
/dev/disk/by-id/ata-WDC_WD40EFPX-68C6CN0_WD-WX12D3456789 -a -o on -S on -n standby,q -s (S/../.././02|L/../../6/03) -W 4,45,55 -m root
/dev/disk/by-id/nvme-Samsung_SSD_990_PRO_2TB_S7KHNJ0X123456 -a -s (S/../.././02) -W 4,60,70 -m root

• -a turns on the standard set of checks: health, attribute changes, error and self-test logs.
• -o on -S on enable the drive's automatic offline data collection and attribute autosave. They only apply to SATA drives.
• -n standby,q skips the check if the drive is spun down, so smartd doesn't wake a sleeping disk every half hour.
• -s (S/../.././02|L/../../6/03) runs a short test every day at 02:00 and a long test every Saturday at 03:00. The fields are type, month, day of month, day of week and hour.
• -W 4,45,55 logs a temperature change of 4 °C, sends an informational message at 45 °C and a warning at 55 °C. Use higher numbers for NVMe drives, which run hotter.
• -m root sends warnings to root's local mailbox. See below for getting them somewhere you'll see them.

Test the configuration, then enable the service:

sudo smartd -q onecheck            # parses the config, checks every drive once and exits
sudo systemctl enable --now smartd     # the unit is called smartmontools on some Debian releases
journalctl -u smartd -n 30

Getting smartd's warnings to you

-m root only works if the machine can send mail, which most home servers can't. smartd can run a script instead, with -M exec. It runs the script with the warning in environment variables such as SMARTD_DEVICE, SMARTD_FAILTYPE and SMARTD_MESSAGE. This one sends a push notification through ntfy:

#!/bin/sh
curl -fsS -m 10 \
  -H "Title: SMART warning on $(hostname): ${SMARTD_DEVICE}" \
  -H "Priority: high" \
  -d "${SMARTD_FAILTYPE}: ${SMARTD_MESSAGE}" \
  https://ntfy.sh/your-private-topic-name

Replace the topic with a long, unguessable name, since anyone who knows it can read the messages, or point it at your own ntfy server. Make the script executable with sudo chmod 755, then change -m root on each line to:

-m <nomailer> -M exec /usr/local/bin/smartd-notify.sh

Add -M test once, restart smartd, and it sends a test message for every drive at startup. Remove it when you've seen the notification arrive.

Feeding SMART into Prometheus

If you run the stack from Prometheus Grafana Monitoring, you can graph SMART values over time and alert on trends. The smartctl_exporter from the Prometheus community runs smartctl and exposes the results as metrics. It needs raw access to the disks, so it runs privileged or with the specific devices passed in:

  smartctl-exporter:
    image: prometheuscommunity/smartctl-exporter:v0.15.0
    restart: unless-stopped
    privileged: true
    user: root
    ports:
      - "127.0.0.1:9633:9633"

Check the project's releases for the current version before deploying. A privileged container has full access to the host, so only run images you trust in this role, and keep the port off the network. Alert rules then become simple PromQL, for example:

smartctl_device_attribute{attribute_name="Reallocated_Sector_Ct", attribute_value_type="raw"} > 0
smartctl_device_percentage_used > 80

Metric names have changed between exporter versions. Check them against what your version actually exposes, with curl -s http://127.0.0.1:9633/metrics | grep smartctl_device, before writing rules.

What to do when a drive fails a check

1. Make sure your backup is current before doing anything else that stresses the drive, including a long self-test.
2. Check the cable first if the only rising counter is UDMA_CRC_Error_Count. Reseat or replace the SATA cable, then watch whether the count keeps climbing.
3. Run a long self-test if you haven't. A clean result with a stable reallocated count often means the drive found and replaced a few weak sectors. Keep watching it.
4. Plan a replacement if pending or uncorrectable sectors are non-zero, reallocations keep growing, a long test fails, or an NVMe drive reports a critical warning. In a RAID or ZFS pool, replace the drive while the pool is still redundant; the TrueNAS Storage Setup guide covers that.
5. Match serial numbers against what you recorded earlier before pulling a drive, so you remove the right one.

Troubleshooting

SMART support is: Unavailable - device lacks SMART capability

Usually a USB bridge that doesn't pass commands through, or a hardware RAID controller. Try -d sat for USB. For RAID controllers, smartctl has device types such as -d megaraid,N that address each disk behind the controller.

smartd starts but logs Device: ... opened, not capable of SMART self-check

The drive or its bridge doesn't support part of what -a asks for. Run sudo smartctl -a on it to see what is supported, and drop the unsupported options from that line.

Drives never spin down any more

smartd's polling is waking them. Make sure every line for a spinning drive has -n standby,q, and consider a longer interval with -i 3600 on the smartd command line.

A self-test shows Aborted by host

The drive was reset or went to sleep during the test, often because of a USB enclosure's power saving or a system suspend. Run it again with the system idle and awake.

Next steps

• Linux Performance Monitoring: check whether a slow system is waiting on its disks.
• Docker Volumes and Backups: make sure the data on those drives has a second copy.
• TrueNAS Storage Setup: redundant pools and scheduled SMART tests in one place.
• The smartmontools documentation and the smartd.conf man page list every option.