Debugging Workflow#

Start broad, narrow down. Most problems fall into five categories: service not running, resource exhaustion, full disk, network failure, or kernel issue. Work through them in order: service, resources, network, kernel logs.

Services: systemctl and journalctl#

When a service is misbehaving, start with its status:

systemctl status nginx

This shows whether the service is active, its PID, its last few log lines, and how long it has been running. If the service keeps restarting, the uptime will be suspiciously short.

Unit Types#

systemd manages the entire system through “units,” each representing a resource or service. The most common types:

• service: Daemons and long-running processes (nginx, postgresql, your application).
• timer: Scheduled execution, replacing cron. More flexible and better integrated with logging.
• socket: Network sockets that trigger service activation on connection. Enables lazy startup and zero-downtime restarts.
• target: Groups of units that represent system states (multi-user.target, graphical.target). Analogous to SysV runlevels.
• mount: Filesystem mount points managed by systemd.
• path: Watches filesystem paths and activates units when changes occur.

Unit files live in three locations, in order of precedence: /etc/systemd/system/ (local admin overrides), /run/systemd/system/ (runtime, non-persistent), and /usr/lib/systemd/system/ (package-installed defaults). Always put custom units in /etc/systemd/system/.