Running Your Scripts as Real Services with Systemd

[Unit]
Description=My App
After=network.target

[Service]
Type=exec
User=appuser
ExecStart=/usr/bin/node /opt/app/server.js
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

[Unit]
Description=Cleanup temp files

[Service]
Type=oneshot
ExecStart=/opt/scripts/cleanup.sh

[Unit]
Description=Run cleanup daily

[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true

[Install]
WantedBy=timers.target

Systemd is PID 1 on most Linux distributions, including Debian, Ubuntu, Fedora, RHEL and Arch. It starts services, tracks their state, restarts them when they crash, and collects their output in the journal. You interact with it through systemctl and journalctl. The reference for everything below is the systemd manual: systemd.service(5) for the [Service] section and systemd.unit(5) for [Unit] and [Install]. systemctl --version shows which systemd release you have; the "latest" pages occasionally describe options newer than your distribution's, and man systemd.service on your machine matches what's installed. Everything on this page works on the systemd versions in Debian 12 and 13, Ubuntu 24.04 and 26.04, and RHEL 9 and 10. Ubuntu 26.04 ships systemd 259, and its release notes say it's the last Ubuntu release that runs old System V init scripts, which is one more reason to write native units.

Custom services work the same way as built-in ones. You write a unit file (a plain text config that tells systemd what binary to run, which user to run it as, and what to do when it dies) and drop it into /etc/systemd/system/.

What you need: a Linux system that runs systemd, sudo access to create unit files in /etc/systemd/system/, and a script or program you want to run as a service.

Your First Service File

Say you have a Python script at /home/alice/mybot/bot.py that you want to run as a service. The service file:

[Unit]
Description=My Discord Bot
Wants=network-online.target
After=network-online.target

[Service]
Type=exec
User=alice
WorkingDirectory=/home/alice/mybot
ExecStart=/usr/bin/python3 /home/alice/mybot/bot.py
Environment=PYTHONUNBUFFERED=1
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Each section does something specific:

The [Unit] Section

• Description — Human-readable name. Shows up in systemctl status.
• Wants=/After=network-online.target — Wait until the network management software reports the network as up before starting. A bot that has to connect out at start-up needs this. Plain After=network.target, which you'll see in many examples, only means the network stack has been started; systemd's network targets page says it doesn't guarantee any interface is configured. Server software that just listens for connections is fine with After=network.target.

The [Service] Section

• Type=exec — The process stays in the foreground, and systemd considers it started once the binary has actually been executed. With Type=simple (the default), systemd considers it started right after forking, before it even tries to run the binary, so systemctl start reports success even if the executable is missing or the User= doesn't exist. The manual recommends exec for long-running services (it was added in systemd 240; if systemctl --version shows something older, check that man systemd.service lists exec, since some distributions backport it, or use simple). Type=notify is better still for programs that support it, because the program itself tells systemd when it's ready.
• Type=forking — For processes that daemonize themselves (fork and exit the parent). systemd waits for the parent to exit, then tracks the child. The manual discourages it for new services.
• User — Run as this user instead of root. Don't run application services as root unless they need privileged ports or resources.
• WorkingDirectory — Sets the working directory before executing. Required when your process reads relative paths.
• ExecStart — The command to execute. Use an absolute path: a bare name is only looked up in a fixed system search path, not your shell's PATH. Shell features like pipes and redirects don't work here (use a wrapper script for those). For a Python virtualenv, point at its interpreter, e.g. /home/alice/mybot/venv/bin/python; there's no need to "activate" it.
• Environment=PYTHONUNBUFFERED=1 — Python buffers stdout when it isn't a terminal, so print() output can show up in the journal late or not at all. This makes it appear immediately.
• Restart=on-failure — Restart on a non-zero exit code, a signal, a timeout or a watchdog failure. Restart=always restarts regardless of exit status.
• RestartSec=10 — Delay between restart attempts. The default is 100 ms, which turns a crash loop into a tight loop.

The [Install] Section

• WantedBy=multi-user.target — Start this service when the system reaches "multi-user mode" (normal operation). systemctl enable creates the symlink that makes this happen, which is what makes it start on boot.

The Commands You'll Use Over and Over

Once you've created the service file, here's how to use it:

# Reload systemd so it sees your new file
sudo systemctl daemon-reload

# Start the service
sudo systemctl start mybot

# Check if it's running
sudo systemctl status mybot

# Stop the service
sudo systemctl stop mybot

# Restart (stop then start)
sudo systemctl restart mybot

# Enable start on boot (add --now to also start it)
sudo systemctl enable mybot

# Disable start on boot
sudo systemctl disable mybot

# View logs for this service
sudo journalctl -u mybot

# Follow logs in real-time
sudo journalctl -u mybot -f

The daemon-reload step trips people up constantly. If you edit a service file and nothing changes, you forgot to reload: systemd keeps using the version it loaded until you tell it to re-read unit files. Recent versions print a warning in systemctl status when the file on disk has changed.

To change a unit that came with a package (say, add an environment variable to nginx.service), don't edit the file under /usr/lib/systemd/system/, which the next package update overwrites. Run sudo systemctl edit nginx instead: it creates a drop-in override in /etc/systemd/system/nginx.service.d/ and reloads for you. systemctl cat nginx shows the unit together with every drop-in that applies.

Reading the Status Output

systemctl status output has the same shape on every unit. Here it is with the variable parts replaced by placeholders in angle brackets (the exact wording differs slightly between systemd versions):

● mybot.service - My Discord Bot
     Loaded: loaded (/etc/systemd/system/mybot.service; enabled; preset: enabled)
     Active: active (running) since <start time>; <how long ago>
   Main PID: <pid> (python3)
      Tasks: <n>
     Memory: <size>
        CPU: <cpu time>
     CGroup: /system.slice/mybot.service
             └─<pid> /usr/bin/python3 /home/alice/mybot/bot.py

<date> <host> systemd[1]: Started mybot.service - My Discord Bot.
<date> <host> python3[<pid>]: <whatever your program printed>

Fields worth reading:

• Active: active (running) — Process is alive. failed means it exited non-zero, was killed, timed out or hit the start limit. inactive (dead) means it's not running and wasn't expected to be. activating (auto-restart) means it's waiting for RestartSec before the next attempt.
• enabled / disabled — Whether it starts on boot. This is the symlink in /etc/systemd/system/multi-user.target.wants/.
• Main PID — The tracked process. If this is wrong, your Type= is wrong.
• The last few journal lines appear at the bottom, usually enough to spot the problem without running journalctl separately. For a failed unit, the Process: or Main PID: line also shows code=exited, status=N/NAME, which the next section decodes.

When Things Go Wrong

Common failure modes and what causes them:

Error: "Failed to start" with status 203/EXEC

systemd couldn't execute the command in ExecStart. Check that:

• The path is correct and absolute (ls -l /path/to/binary)
• The file is executable (chmod +x script.sh) and the service user can reach it: every parent directory needs execute permission for that user, which is a common catch with scripts under someone else's home directory (see Understanding Linux Permissions)
• If it's a script, it has the right shebang line at the top (#!/usr/bin/python3) and Unix line endings (a file saved with Windows CRLF endings fails here too)

Two related codes from the systemd.exec exit code table: 200/CHDIR means changing to the WorkingDirectory failed (it doesn't exist or isn't accessible), and 217/USER means systemd couldn't look up or switch to the User= account, for example because it doesn't exist.

Error: Exit code 1 / Service keeps restarting

Your program is starting and then crashing. Check the logs:

sudo journalctl -u mybot -b -n 50 --no-pager

-b limits output to the current boot and -n 50 shows the last 50 lines. Look for Python tracebacks or error messages. If there's nothing from your program at all, the output may be buffered (see PYTHONUNBUFFERED above).

Script works manually but not as a service

This is usually an environment problem. Your shell session has a PATH, environment variables, a home directory and maybe an activated virtualenv. systemd services get a minimal environment: no .bashrc, no user PATH, no virtualenv activation.

Fixes:

• Use full paths in the script itself, not just in the service file
• Set WorkingDirectory properly
• If you need environment variables, add Environment=VARNAME=value or an EnvironmentFile= (below) to the [Service] section
• Reproduce the service's conditions by hand: sudo -u alice env -i /usr/bin/python3 /home/alice/mybot/bot.py runs it as the service user with an empty environment

A More Complete Example

A Node.js web application service with logging and environment variables, the kind of unit the AWS EC2 Deployment Guide uses as an alternative to PM2:

[Unit]
Description=My Node.js Web Application
Documentation=https://github.com/myuser/myapp
After=network.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
Type=exec
User=www-data
Group=www-data
WorkingDirectory=/var/www/myapp
ExecStart=/usr/bin/node /var/www/myapp/server.js
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp
Environment=NODE_ENV=production
Environment=PORT=3000

[Install]
WantedBy=multi-user.target

Additional directives used here:

• Group — Sets the group ID in addition to the user.
• StartLimitIntervalSec / StartLimitBurst — In [Unit]: give up after 5 starts within 5 minutes, instead of restarting forever (see the restart trap above).
• Restart=always — Restart on any exit, including clean ones. For services that should never be down.
• StandardOutput/StandardError=journal — Send stdout/stderr to the journal. That's already the default for system services; writing it out documents the intent.
• SyslogIdentifier — Tags journal entries. Without this, entries are tagged with the process name (node), which is ambiguous if you run several Node apps.
• Environment — Sets environment variables for the process. For secrets, use EnvironmentFile instead.

The app listens on port 3000 and a reverse proxy in front handles port 80/443 and TLS; Nginx Web Server Setup covers that part.

Environment Files: For Secrets and Config

Don't put secrets in the service file: unit files are normally world-readable, and Environment= values are visible to any local user through systemctl show. Use an environment file:

DATABASE_URL=postgres://user:password@localhost/mydb
API_KEY=super_secret_key_here
NODE_ENV=production

Then reference it in your service file:

EnvironmentFile=/etc/myapp/env

Make sure the file is readable only by root. systemd reads it as root before switching to the service user, so the service user doesn't need access to the file itself:

sudo chmod 600 /etc/myapp/env
sudo chown root:root /etc/myapp/env

The variables still end up in the process's environment, where the service user and root can read them, so this keeps secrets out of world-readable files rather than hiding them from the app.

Running Multiple Instances

Template units let you run the same service with different parameters. Name the file with an @ symbol (myapp@.service), and the %i specifier gets replaced with the instance name:

[Unit]
Description=My App Instance %i

[Service]
Type=exec
ExecStart=/usr/bin/python3 /opt/myapp/app.py --config /etc/myapp/%i.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

The %i gets replaced with whatever you put after the @:

sudo systemctl start myapp@production
sudo systemctl start myapp@staging

# Now you have two instances running with different configs

Timers: Better Than Cron (Sometimes)

systemd timers are an alternative to cron. A timer unit triggers a service unit on a schedule (see systemd.timer(5)). Two files required:

[Unit]
Description=Daily backup script

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh

[Unit]
Description=Run backup daily

[Timer]
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer (not the service):

sudo systemctl enable --now backup.timer
sudo systemctl list-timers # See when it will run next

# Check a calendar expression before using it
systemd-analyze calendar "*-*-* 03:00:00"

The advantages over cron: with Persistent=true, if the system was off at 3 AM, systemd runs the job soon after the next boot, where plain cron would just skip it. Each run's output lands in the journal under backup.service, and a failed run shows up as a failed unit in systemctl --failed.

Restricting What a Service Can Touch

Running as a non-root User= is the first step. systemd can also hide or freeze parts of the filesystem from the service, so a bug or a compromised dependency can do less damage. These directives from systemd.exec(5) suit the Node.js app above, which only needs to read its code and write its own state:

NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
StateDirectory=myapp

• NoNewPrivileges=yes — The process and its children can never gain privileges, for example through a SUID binary.
• PrivateTmp=yes — The service gets its own empty /tmp and /var/tmp, invisible to other processes.
• ProtectSystem=strict — The whole filesystem becomes read-only for the service, apart from API filesystems such as /proc and /dev.
• ProtectHome=yes — /home, /root and /run/user appear empty. Don't use this for a service whose code lives in a home directory, like the bot example; ProtectHome=read-only is the alternative there.
• StateDirectory=myapp — Creates /var/lib/myapp, owned by the service user, and keeps it writable despite ProtectSystem=strict. The path is passed to the process as $STATE_DIRECTORY. If the app writes anywhere else, list that path in ReadWritePaths=.

Run sudo systemctl daemon-reload and restart the service, then check the journal for "Read-only file system" or "Permission denied" errors, which mean the app writes somewhere you haven't allowed. systemd-analyze security myapp scores the unit's exposure and lists which settings are still open; treat it as a checklist rather than a target, because some services legitimately need what it flags.

Services Without Root

If you don't have sudo on the machine, or the service belongs to one user anyway, run it under your own per-user systemd instance. Put the unit in ~/.config/systemd/user/, leave out User=, use WantedBy=default.target in [Install], and add --user to every command:

systemctl --user daemon-reload
systemctl --user enable --now mybot
journalctl --user -u mybot -f

# Keep your user services running after you log out, and start them at boot
loginctl enable-linger "$USER"
loginctl show-user "$USER" --property=Linger   # Linger=yes

Without lingering, the per-user instance (and every service in it) stops when your last session ends. Whether you can enable it for your own account without sudo depends on the system's polkit policy; if it's refused, an administrator has to run the command for you. loginctl disable-linger turns it off again.

Troubleshooting Checklist

1. Did systemd load your latest file? sudo systemctl daemon-reload after every edit, then systemctl cat NAME to see exactly what it loaded, including drop-ins.
2. Is the file valid? systemd-analyze verify /etc/systemd/system/NAME.service reports unknown or misspelled directives and missing executables, without starting anything.
3. What state is it in, and why? systemctl status NAME: look at Active: and the status= code on the process line.
4. What did it print? journalctl -u NAME -b for this boot, -f to follow, -p err for errors only.
5. Is it stuck in a restart loop? Check Restart=, RestartSec= and the start limits above; systemctl reset-failed NAME after fixing.
6. Does it start too early? If it needs the network or another service at start-up, add the right Wants=/After= pair, for example network-online.target, or the database's unit.
7. Does it work by hand but not as a service? Run it as the service user with an empty environment (see above), and check file permissions for that user.
8. Did sandboxing break it? Comment out the restriction directives one at a time, reloading each time, to find the one the app trips over, then allow the specific path with ReadWritePaths= instead of dropping the restriction.
9. Is it slow to start, or timing out? systemd-analyze blame lists how long each unit took at boot, and systemd-analyze critical-chain NAME.service shows what it waited for.

Removing the Example Units

If you created the example units only to practise, stop and disable them, delete the files, and reload so systemd forgets them:

sudo systemctl disable --now mybot.service myapp.service backup.timer
sudo systemctl stop 'myapp@*.service'
sudo rm /etc/systemd/system/mybot.service /etc/systemd/system/myapp.service \
        /etc/systemd/system/myapp@.service /etc/systemd/system/backup.service \
        /etc/systemd/system/backup.timer
sudo systemctl daemon-reload
sudo systemctl reset-failed

# Remove the environment file with the test secrets, and any state directory
sudo rm -rf /etc/myapp /var/lib/myapp

systemctl status mybot should then report Unit mybot.service could not be found. For overrides made with systemctl edit, sudo systemctl revert NAME removes the drop-ins and restores the packaged unit. Old journal entries for the removed units age out under the journal's normal size and time limits.

Quick Reference

The templates at the top of the page cover the unit files. Commands:

sudo systemctl daemon-reload # After editing files
sudo systemctl start NAME # Start now
sudo systemctl stop NAME # Stop now
sudo systemctl restart NAME # Restart
sudo systemctl enable NAME # Start on boot
sudo systemctl disable NAME # Don't start on boot
sudo systemctl status NAME # Check status
sudo journalctl -u NAME # View logs
sudo journalctl -u NAME -f # Follow logs live
systemd-analyze verify FILE # Check a unit file for errors

Next Steps

• Nginx Web Server Setup: put a reverse proxy with TLS in front of the app unit.
• Understanding Linux Permissions: the directory and file modes a service user needs.
• Locking Down SSH: socket activation and journal-based Fail2Ban in practice.
• Prometheus Grafana Monitoring: watch the services once they're running.