--- title: "systemd Service Management: Units, Timers, Journal, and Socket Activation" description: "Comprehensive guide to systemd service management covering unit files, restart policies, service types, resource controls, timer units, journal logging, socket activation, and common debugging techniques." url: https://agent-zone.ai/knowledge/infrastructure/systemd-service-management/ section: knowledge date: 2026-02-21 categories: ["infrastructure"] tags: ["linux","systemd","services","timers","journal","logging","cgroups"] skills: ["systemd-management","service-configuration","linux-administration"] tools: ["systemctl","journalctl","systemd-analyze"] levels: ["intermediate"] word_count: 1337 formats: json: https://agent-zone.ai/knowledge/infrastructure/systemd-service-management/index.json html: https://agent-zone.ai/knowledge/infrastructure/systemd-service-management/?format=html api: https://api.agent-zone.ai/api/v1/knowledge/search?q=systemd+Service+Management%3A+Units%2C+Timers%2C+Journal%2C+and+Socket+Activation --- ## Unit Types systemd manages the entire system through "units," each representing a resource or service. The most common types: - **service**: Daemons and long-running processes (nginx, postgresql, your application). - **timer**: Scheduled execution, replacing cron. More flexible and better integrated with logging. - **socket**: Network sockets that trigger service activation on connection. Enables lazy startup and zero-downtime restarts. - **target**: Groups of units that represent system states (multi-user.target, graphical.target). Analogous to SysV runlevels. - **mount**: Filesystem mount points managed by systemd. - **path**: Watches filesystem paths and activates units when changes occur. Unit files live in three locations, in order of precedence: `/etc/systemd/system/` (local admin overrides), `/run/systemd/system/` (runtime, non-persistent), and `/usr/lib/systemd/system/` (package-installed defaults). Always put custom units in `/etc/systemd/system/`. ## Creating a Service Unit A service unit file has three sections: ```ini # /etc/systemd/system/myapp.service [Unit] Description=My Application Server Documentation=https://docs.example.com/myapp After=network-online.target postgresql.service Wants=network-online.target Requires=postgresql.service [Service] Type=simple User=myapp Group=myapp WorkingDirectory=/opt/myapp Environment=NODE_ENV=production EnvironmentFile=/opt/myapp/.env ExecStartPre=/opt/myapp/bin/check-config ExecStart=/opt/myapp/bin/server --port 8080 ExecStop=/bin/kill -SIGTERM $MAINPID Restart=on-failure RestartSec=5 StartLimitBurst=5 StartLimitIntervalSec=60 StandardOutput=journal StandardError=journal SyslogIdentifier=myapp [Install] WantedBy=multi-user.target ``` **[Unit] section**: `After` controls startup ordering (start after these units). `Wants` declares a weak dependency (start these if available, but do not fail if they are missing). `Requires` declares a hard dependency (fail if this unit is not available). **[Service] section**: `ExecStart` is the main process command. `ExecStartPre` runs before the main process (validation, migrations). `User` and `Group` set the process identity. `WorkingDirectory` sets the current directory. `Environment` and `EnvironmentFile` set environment variables. **[Install] section**: `WantedBy` determines which target pulls this service in when enabled. `multi-user.target` means the service starts during normal multi-user boot. ## Restart Policies Restart behavior is critical for production services. The policy determines what happens when the process exits: - **Restart=on-failure**: Restart only on non-zero exit codes, signals, and timeouts. The most common choice for production services -- it restarts on crashes but not on clean shutdowns (exit code 0). - **Restart=always**: Restart regardless of exit status. Use for services that should never stop (container runtimes, critical infrastructure). - **Restart=on-abnormal**: Restart on signals, timeouts, and watchdog events, but not on clean or unclean exit codes. Useful when the service might exit intentionally with a non-zero code. ```ini Restart=on-failure RestartSec=5 # wait 5 seconds between restarts StartLimitBurst=5 # max 5 restarts... StartLimitIntervalSec=60 # ...within 60 seconds ``` The `StartLimitBurst` and `StartLimitIntervalSec` parameters prevent restart loops. Without them, a service with a configuration error will restart indefinitely, consuming resources and flooding logs. When the limit is reached, systemd stops trying and marks the unit as failed. You must `systemctl reset-failed myapp` before you can start it again. ## Service Types The `Type` directive tells systemd how to determine when the service is "ready": - **simple** (default): systemd considers the service started immediately after `ExecStart` runs. The process must not fork. This is correct for most modern applications. - **forking**: The process forks into the background (traditional Unix daemon style). systemd waits for the parent to exit and tracks the child via `PIDFile`. Use this for legacy daemons. - **oneshot**: The process runs to completion and exits. systemd waits for it to finish before starting dependent units. Good for initialization scripts. - **notify**: The process sends a notification to systemd when it is ready (`sd_notify()`). This is the most accurate type -- systemd knows exactly when the service is serving traffic. Applications must be instrumented to send the notification. ```ini # For a forking daemon Type=forking PIDFile=/run/myapp/myapp.pid # For a oneshot initialization Type=oneshot RemainAfterExit=yes # consider the unit "active" even after the process exits ``` ## Resource Controls systemd uses Linux cgroups to enforce resource limits on services. These limits are hard-enforced by the kernel: ```ini [Service] # Memory limit -- OOM killer activates if exceeded MemoryMax=2G MemoryHigh=1.5G # soft limit, kernel starts reclaiming aggressively # CPU quota -- percentage of one CPU core CPUQuota=200% # allow up to 2 full cores # Maximum number of tasks (threads + processes) TasksMax=512 # I/O weight (relative priority, 1-10000, default 100) IOWeight=200 # Protect the service from OOM killer (use with caution) OOMScoreAdjust=-500 ``` `MemoryMax` is a hard ceiling. When a process exceeds it, the kernel's OOM killer terminates it. `MemoryHigh` is a soft limit -- the kernel aggressively reclaims memory from the cgroup but does not kill the process. Using both together provides a buffer zone. `TasksMax` prevents fork bombs and runaway thread creation. The default is 4915 (33% of the kernel limit), which is usually sufficient but may need increasing for applications that use many threads. ## Timer Units Timers replace cron with better logging, dependency management, and calendar expressions. A timer activates a corresponding service unit (same name with `.service` extension). ```ini # /etc/systemd/system/backup.timer [Unit] Description=Daily Database Backup [Timer] OnCalendar=*-*-* 03:00:00 # daily at 3:00 AM Persistent=true # run missed timers after reboot RandomizedDelaySec=600 # add up to 10 min random delay (avoid thundering herd) AccuracySec=1min # wake up within 1 minute of scheduled time [Install] WantedBy=timers.target ``` ```ini # /etc/systemd/system/backup.service [Unit] Description=Database Backup [Service] Type=oneshot ExecStart=/opt/scripts/backup.sh User=backup ``` Calendar expressions are flexible: ``` OnCalendar=*-*-* 03:00:00 # daily at 3 AM OnCalendar=Mon *-*-* 09:00:00 # every Monday at 9 AM OnCalendar=*-*-01 00:00:00 # first of every month at midnight OnCalendar=hourly # shorthand for *-*-* *:00:00 ``` Boot-relative timers are also available: ```ini OnBootSec=5min # 5 minutes after boot OnUnitActiveSec=1h # 1 hour after the service last ran ``` `Persistent=true` is important for backup and maintenance timers. If the system was powered off when the timer should have fired, systemd runs it immediately on next boot. Manage timers with: ```bash systemctl list-timers --all # show all timers and their next fire time systemctl start backup.timer # start the timer systemctl enable backup.timer # enable on boot ``` ## Journal: Centralized Logging systemd's journal captures stdout/stderr from all services, kernel messages, and syslog messages in a structured, indexed binary format. ```bash journalctl -u myapp # all logs for a service journalctl -u myapp -f # follow in real time journalctl -u myapp --since "1 hour ago" # time-scoped journalctl -u myapp -p err # errors and above (emerg, alert, crit, err) journalctl -u myapp -n 50 # last 50 lines journalctl -u myapp --no-pager -o json-pretty # structured JSON output ``` ### Journal Persistence By default, the journal may store logs only in memory (`/run/log/journal/`), which means logs are lost on reboot. To persist: ```bash mkdir -p /var/log/journal systemd-tmpfiles --create --prefix /var/log/journal ``` Or configure in `/etc/systemd/journald.conf`: ```ini [Journal] Storage=persistent # auto (default), persistent, volatile, none SystemMaxUse=2G # max disk usage for journal SystemKeepFree=1G # keep at least this much disk free MaxRetentionSec=30day # delete entries older than 30 days ``` Restart the journal: `systemctl restart systemd-journald`. ## Socket Activation Socket activation lets systemd listen on a socket and start the service only when a connection arrives. Benefits include: zero-downtime restarts (systemd holds the socket while the service restarts, queuing connections), lazy startup (services that rarely receive traffic start on-demand), and parallel boot (services do not wait for each other to bind ports). ```ini # /etc/systemd/system/myapp.socket [Unit] Description=My Application Socket [Socket] ListenStream=8080 Accept=no # hand the socket to the service (not per-connection) [Install] WantedBy=sockets.target ``` The service must be written to accept the socket file descriptor from systemd (file descriptor 3, or use the sd_listen_fds API). ## Common Operations Reference ```bash systemctl start myapp # start now systemctl stop myapp # stop now systemctl restart myapp # stop then start systemctl reload myapp # send SIGHUP (if supported) systemctl status myapp # show status, PID, recent logs systemctl enable myapp # start on boot systemctl disable myapp # do not start on boot systemctl enable --now myapp # enable and start immediately systemctl daemon-reload # reload unit file changes systemctl list-units --failed # show failed units systemctl reset-failed myapp # clear failed state ``` ## Debugging ```bash systemctl status myapp # first stop -- shows exit code, recent logs journalctl -u myapp -xe # recent entries with explanations systemd-analyze blame # boot time per service (find slow starters) systemd-analyze critical-chain # boot dependency chain systemd-analyze verify myapp.service # check unit file syntax ``` **Common gotcha: forgetting daemon-reload**. After editing any unit file in `/etc/systemd/system/`, you must run `systemctl daemon-reload`. Without it, systemd uses the cached version of the unit file. This is the single most common systemd mistake. **Common gotcha: ExecStart must be an absolute path**. `ExecStart=myapp` fails. It must be `ExecStart=/usr/local/bin/myapp`. systemd does not search PATH. This also applies to `ExecStartPre`, `ExecStop`, and all other Exec directives.