systemd OnFailure Alerts: Templates, OnSuccess, and What They Miss
Contents
OnFailure=notify@%n.service in a unit's [Unit] section starts an instance of the [email protected] template whenever that unit enters the failed state, with the failed unit's name as the instance and its result in $MONITOR_SERVICE_RESULT. That covers failures of a service that ran. It can't cover a timer that never fired, a host that was down, or a job that hung without a timeout, because then no unit fails. Pair it with an external heartbeat for those.
This guide covers systemd 251 and later on Linux, checked against systemd.unit(5), systemd.exec(5), and systemd.service(5) on November 16, 2026. systemd timers covers writing the timer and a basic handler; this goes further into how the hooks behave.
A Template Handler for Every Unit#
OnFailure= lists units "activated when this unit enters the 'failed' state" (systemd 201 and later). Write the handler once as a template and point every unit at it:
# /etc/systemd/system/[email protected]
[Unit]
Description=Report the failure of %i
[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure %i# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
OnFailure=notify-failure@%n.service
[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/backup
TimeoutStartSec=2hThe specifiers, from systemd.unit(5):
| Specifier | Meaning | Here |
|---|---|---|
%n |
Full unit name | backup.service |
%N |
Full name without the type suffix | backup |
%i |
In a template instance, the string between @ and the suffix |
backup.service |
%p |
In a template instance, the string before @ |
notify-failure |
%n is expanded in backup.service, so the failure starts [email protected], and %i in the handler is backup.service. The doubled suffix looks odd but is correct. Using %N instead gives [email protected] and %i of backup, which is tidier but loses the unit type when a timer, mount, or socket unit uses the same handler.
To add the hook without editing a packaged unit, use a drop-in:
sudo systemctl edit backup.service # add the two lines below, then save[Unit]
OnFailure=notify-failure@%n.serviceWhat the Handler Knows: $MONITOR_* Variables#
Since systemd 251, the ExecStart= and ExecStartPre= processes of a unit started by OnFailure= or OnSuccess= receive:
| Variable | Value |
|---|---|
$MONITOR_SERVICE_RESULT |
The failed service's result: exit-code, timeout, oom-kill, ... |
$MONITOR_EXIT_CODE |
exited, killed, or dumped |
$MONITOR_EXIT_STATUS |
The exit status number, or the signal name |
$MONITOR_INVOCATION_ID |
The failed run's invocation ID |
$MONITOR_UNIT |
The failed unit's name |
There's a condition: "when multiple services specify the same unit as their OnFailure= or OnSuccess= handler, those variables will not be passed." A plain notify-failure.service shared by ten units gets nothing; a template gets a separate instance per unit and the variables work. That's the main reason to use the template.
A handler that mails the result and the failed run's log:
#!/bin/sh
# /usr/local/bin/notify-failure: mail root a failed unit's result and its last run's log.
unit=$1
result=${MONITOR_SERVICE_RESULT:-unknown}
status=${MONITOR_EXIT_STATUS:-}
{
echo "Unit: $unit"
echo "Result: $result $status"
echo
if [ -n "$MONITOR_INVOCATION_ID" ]; then
journalctl --no-pager _SYSTEMD_INVOCATION_ID="$MONITOR_INVOCATION_ID"
else
journalctl --no-pager -u "$unit" -n 50
fi
} | mail -s "$(hostname): $unit failed ($result)" rootFiltering on _SYSTEMD_INVOCATION_ID shows only the failed run, not the previous night's. mail needs a working mail transport; to post to a chat webhook instead, replace the pipeline with a curl call.
What Counts as a Failure#
A service enters the failed state for any result other than success. The result is the $SERVICE_RESULT value from systemd.exec(5):
| Result | Cause |
|---|---|
exit-code |
The process exited non-zero |
signal |
Killed by a signal, no core dump |
core-dump |
Killed by a signal, core dumped |
timeout |
A step exceeded its timeout, such as TimeoutStartSec= |
oom-kill |
The kernel's OOM killer ended a process |
watchdog |
The watchdog deadline was missed |
start-limit-hit |
The unit hit its start rate limit |
protocol, resources |
The unit didn't follow its Type=, or a system operation failed |
Two settings change what counts. SuccessExitStatus= declares extra exit codes or signals as clean, so a job that exits 3 for "nothing to do" can be marked successful. And ExecCondition= commands that exit 1 through 254 skip the run without marking the unit failed, so a skipped run triggers nothing.
A Type=oneshot job with no TimeoutStartSec= can't time out at all: for Type=oneshot "the timeout is disabled by default," so a hung run stays activating indefinitely and OnFailure= never fires. Always set one on scheduled jobs.
OnSuccess= and OnFailureJobMode=#
OnSuccess= (systemd 249 and later) lists units activated when this unit enters the "inactive" state, which for a finished oneshot job means it succeeded. It gets the same $MONITOR_* variables. A pair of hooks covers both outcomes:
[Unit]
OnSuccess=report-run@%n.service
OnFailure=report-run@%n.serviceOnFailureJobMode= and OnSuccessJobMode= control how the handler's start job is queued. OnFailureJobMode= defaults to replace and OnSuccessJobMode= to fail. The defaults are fine for a handler that nothing else depends on.
ExecStopPost= and ExecStartPost=: Hooks Inside the Unit#
ExecStopPost= runs after every run of the service, successful or not, including runs that failed during start-up. Its processes get $SERVICE_RESULT and, when systemd identified the main process, $EXIT_CODE and $EXIT_STATUS (systemd 232). systemd.exec(5) recommends it over ExecStop= for monitoring, because ExecStop= "is only invoked for services that managed to start up correctly."
[Service]
ExecStopPost=-/usr/local/bin/record-resultThe leading - matters. Without it, a failing ExecStopPost= command turns a successful run into a failed one.
ExecStartPost= is not a failure hook. It runs only after ExecStart= succeeded (for Type=oneshot, after the last ExecStart= process exited successfully), so a failed run skips it. If an ExecStartPost= command fails without the - prefix, the unit is marked failed. Use it for a success signal only when you understand both rules; ExecStopPost= with $SERVICE_RESULT is simpler.
| Hook | Runs on success | Runs on failure | Can fail the unit | Knows the result |
|---|---|---|---|---|
OnFailure= |
No | Yes | No (separate unit) | $MONITOR_* (251+) |
OnSuccess= |
Yes | No | No (separate unit) | $MONITOR_* (251+) |
ExecStopPost= |
Yes | Yes | Yes, without - |
$SERVICE_RESULT (232+) |
ExecStartPost= |
Yes | No | Yes, without - |
No |
Restarts and Start Limits#
With Restart=on-failure and the default RestartMode=normal, a service restarts "by going through a failed/inactive state," so every failed attempt triggers OnFailure=. A job that fails five times before succeeding sends five alerts. RestartMode=direct (systemd 254) moves straight to activating during automatic restarts and skips OnSuccess= and OnFailure=, so only the final outcome triggers a hook.
Start limits end the restart loop. A unit started more than StartLimitBurst= times within StartLimitIntervalSec= isn't permitted to start again, and a unit configured with Restart= that hits the limit isn't restarted anymore; its result is start-limit-hit. The man page notes it "may still be restarted manually or from a timer or socket at a later point, after the interval has passed." Defaults come from DefaultStartLimitIntervalSec= and DefaultStartLimitBurst= in the manager configuration. systemctl reset-failed backup.service clears the counter.
For a timer-driven job, start limits rarely matter unless the timer fires faster than the burst allows. But a handler that itself fails can hit its own limit: if notify-failure@ instances fail repeatedly because mail is down, later failures may go unreported. Check the handler too.
Checking With journalctl and systemctl#
systemctl list-units --failed # everything currently failed
systemctl show backup.service -p Result -p ExecMainStatus -p ExecMainExitTimestamp
journalctl -u backup.service -n 50 # the job's recent log
journalctl -u 'notify-failure@*' # did the handler run?
systemctl status '[email protected]'The last two answer the question an alert setup has to answer: when the job failed, did the handler start, and did it succeed? Test it deliberately with a unit that exits 1:
sudo systemd-run --unit=onfailure-test \
-p [email protected] /bin/falseWhy OnFailure= Can't See Every Failure#
Every hook above runs because a service ran. When nothing runs, nothing fails:
- The timer never started.
systemctl enablewithout--now, or anOnCalendar=line that didn't parse. - The timer points at a different service, or the service was renamed.
- The host was off or suspended at the scheduled time, without
Persistent=true. - The run hung with no
TimeoutStartSec=. - The handler couldn't deliver: no mail transport, a full disk, a network outage, or the host itself went down mid-run.
These are the failures that go unnoticed longest, and an alert that lives on the same host can't report them. An external heartbeat can: the job reports each run to a service that knows the schedule, and that service alerts when a report doesn't arrive on time.
Pairing OnFailure= With an External Heartbeat#
There's no CronRadar integration for systemd and no discovery of timers; each service reports over HTTP with its own monitor key and schedule. The hooks above already know the outcome, so one template can report both:
# /etc/systemd/system/[email protected]
[Unit]
Description=Report the result of %i to CronRadar
[Service]
Type=oneshot
EnvironmentFile=/etc/cronradar/api.env
EnvironmentFile=/etc/cronradar/%i.env
ExecStart=/usr/local/bin/cronradar-report#!/bin/sh
# /usr/local/bin/cronradar-report: send one run's result, read from $MONITOR_*, to CronRadar.
report() {
path=$1
shift
curl --fail --silent --show-error --max-time 5 --output /dev/null \
--request POST --get --user "$CRONRADAR_API_KEY:" \
--data-urlencode "schedule=$CRONRADAR_SCHEDULE" \
--data-urlencode "gracePeriod=$CRONRADAR_GRACE" \
--data-urlencode "run_id=$MONITOR_INVOCATION_ID" \
"$@" "https://cron.life/api/v1/ping/$CRONRADAR_MONITOR$path" || true
}
if [ "$MONITOR_SERVICE_RESULT" = success ]; then
report "" # heartbeat: a successful run
else
report /fail --data-urlencode \
"message=systemd result ${MONITOR_SERVICE_RESULT:-unknown} ${MONITOR_EXIT_STATUS:-}"
fi
exit 0# /etc/cronradar/backup.service.env
CRONRADAR_MONITOR=nightly-backup
CRONRADAR_SCHEDULE="30 2 * * *"
CRONRADAR_GRACE=900/etc/cronradar/api.env holds CRONRADAR_API_KEY=ck_app_..., readable only by root; the service manager reads it before starting the handler. Then add both hooks to the job:
# /etc/systemd/system/backup.service.d/cronradar.conf
[Unit]
OnSuccess=cronradar-report@%n.service
OnFailure=cronradar-report@%n.serviceA success sends a plain heartbeat and a failure sends fail. Each report carries the schedule, so the first run creates the monitor and a deleted one comes back on the next run. run_id is the run's invocation ID, the same ID journalctl _SYSTEMD_INVOCATION_ID= filters on. A fail report alerts immediately with the systemd result in its message. Every case in the previous section sends nothing at all, and after the expected time plus the grace period the monitor goes from warning to critical and alerts by email, Slack, webhook, Discord, or PagerDuty.
This version reports only the outcome, so a hung run shows as a missed one. To get duration tracking and stuck-run detection, report start from ExecStartPre= and complete instead of the heartbeat, as the script in systemd timers does. CronRadar reads the schedule in the application's time zone, an IANA name in its settings (UTC when unset); set it to the zone timedatectl reports, or convert OnCalendar= times as that guide shows.
Frequently Asked Questions#
How do I get an email when a systemd service fails?#
Add OnFailure=notify-failure@%n.service to the service's [Unit] section and write a [email protected] template whose ExecStart= mails systemctl status or the journal for %i. It needs a working mail transport on the host.
What is the difference between %i and %n in OnFailure?#
%n is expanded in the failing unit to its full name, such as backup.service, and becomes the instance of the handler. Inside the handler template, %i is that instance string. So OnFailure=notify@%n.service on backup.service starts [email protected], where %i is backup.service.
Does OnFailure fire on every restart?#
With Restart= and the default RestartMode=normal, yes: each failed attempt goes through the failed state. RestartMode=direct, available since systemd 254, skips OnFailure= and OnSuccess= during automatic restarts.
Is there an OnSuccess= in systemd?#
Yes, since systemd 249. It starts the listed units when the unit enters the inactive state, which for a finished oneshot service means it succeeded. Handlers get the same $MONITOR_* variables as OnFailure= handlers on systemd 251 and later.
Why didn't OnFailure fire when my timer didn't run?#
Because nothing failed. OnFailure= reacts to a unit entering the failed state, and a timer that never fired, a powered-off host, or a hung job without a timeout never gets there. An external heartbeat monitor that expects each run catches those.
Put this guide to work
Know when the next run is missing
Connect one scheduled job, confirm a real completion, then verify the alert destination. Keep your existing scheduler in charge of running the work.