# systemd OnFailure Alerts: Templates, OnSuccess, and What They Miss

By Dan Ochoa · Published 2026-11-16 · Updated 2026-11-16
Canonical: https://cronradar.com/blog/systemd-onfailure-alerts

> Alert on failed systemd units with OnFailure= and a notify@ template: specifiers, OnSuccess=, ExecStopPost variables, restarts, start limits, and missed timers.

**`OnFailure=notify@%n.service` in a unit's `[Unit]` section starts an instance of the `notify@.service` template whenever that unit enters the failed state, with the failed unit's name as the instance and its result in `$MONITOR_SERVICE_RESULT`.** That covers failures of a service that ran. It can't cover a timer that never fired, a host that was down, or a job that hung without a timeout, because then no unit fails. Pair it with an external heartbeat for those.

This guide covers systemd 251 and later on Linux, checked against [systemd.unit(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html), [systemd.exec(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html), and [systemd.service(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html) on November 16, 2026. [systemd timers](https://cronradar.com/blog/systemd-timers) covers writing the timer and a basic handler; this goes further into how the hooks behave.

## A Template Handler for Every Unit

`OnFailure=` lists units "activated when this unit enters the 'failed' state" (systemd 201 and later). Write the handler once as a template and point every unit at it:

```ini
# /etc/systemd/system/notify-failure@.service
[Unit]
Description=Report the failure of %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure %i
```

```ini
# /etc/systemd/system/backup.service
[Unit]
Description=Nightly backup
OnFailure=notify-failure@%n.service

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/backup
TimeoutStartSec=2h
```

The specifiers, from systemd.unit(5):

| Specifier | Meaning                                                       | Here             |
| --------- | ------------------------------------------------------------- | ---------------- |
| `%n`      | Full unit name                                                | `backup.service` |
| `%N`      | Full name without the type suffix                             | `backup`         |
| `%i`      | In a template instance, the string between `@` and the suffix | `backup.service` |
| `%p`      | In a template instance, the string before `@`                 | `notify-failure` |

`%n` is expanded in `backup.service`, so the failure starts `notify-failure@backup.service.service`, and `%i` in the handler is `backup.service`. The doubled suffix looks odd but is correct. Using `%N` instead gives `notify-failure@backup.service` and `%i` of `backup`, which is tidier but loses the unit type when a timer, mount, or socket unit uses the same handler.

To add the hook without editing a packaged unit, use a drop-in:

```bash
sudo systemctl edit backup.service     # add the two lines below, then save
```

```ini
[Unit]
OnFailure=notify-failure@%n.service
```

## What the Handler Knows: $MONITOR\_\* Variables

Since systemd 251, the `ExecStart=` and `ExecStartPre=` processes of a unit started by `OnFailure=` or `OnSuccess=` receive:

| Variable                  | Value                                                                |
| ------------------------- | -------------------------------------------------------------------- |
| `$MONITOR_SERVICE_RESULT` | The failed service's result: `exit-code`, `timeout`, `oom-kill`, ... |
| `$MONITOR_EXIT_CODE`      | `exited`, `killed`, or `dumped`                                      |
| `$MONITOR_EXIT_STATUS`    | The exit status number, or the signal name                           |
| `$MONITOR_INVOCATION_ID`  | The failed run's invocation ID                                       |
| `$MONITOR_UNIT`           | The failed unit's name                                               |

There's a condition: "when multiple services specify the same unit as their `OnFailure=` or `OnSuccess=` handler, those variables will not be passed." A plain `notify-failure.service` shared by ten units gets nothing; a template gets a separate instance per unit and the variables work. That's the main reason to use the template.

A handler that mails the result and the failed run's log:

```sh
#!/bin/sh
# /usr/local/bin/notify-failure: mail root a failed unit's result and its last run's log.
unit=$1
result=${MONITOR_SERVICE_RESULT:-unknown}
status=${MONITOR_EXIT_STATUS:-}

{
  echo "Unit: $unit"
  echo "Result: $result $status"
  echo
  if [ -n "$MONITOR_INVOCATION_ID" ]; then
    journalctl --no-pager _SYSTEMD_INVOCATION_ID="$MONITOR_INVOCATION_ID"
  else
    journalctl --no-pager -u "$unit" -n 50
  fi
} | mail -s "$(hostname): $unit failed ($result)" root
```

Filtering on `_SYSTEMD_INVOCATION_ID` shows only the failed run, not the previous night's. `mail` needs a working mail transport; to post to a chat webhook instead, replace the pipeline with a `curl` call.

## What Counts as a Failure

A service enters the failed state for any result other than `success`. The result is the `$SERVICE_RESULT` value from systemd.exec(5):

| Result                  | Cause                                                            |
| ----------------------- | ---------------------------------------------------------------- |
| `exit-code`             | The process exited non-zero                                      |
| `signal`                | Killed by a signal, no core dump                                 |
| `core-dump`             | Killed by a signal, core dumped                                  |
| `timeout`               | A step exceeded its timeout, such as `TimeoutStartSec=`          |
| `oom-kill`              | The kernel's OOM killer ended a process                          |
| `watchdog`              | The watchdog deadline was missed                                 |
| `start-limit-hit`       | The unit hit its start rate limit                                |
| `protocol`, `resources` | The unit didn't follow its `Type=`, or a system operation failed |

Two settings change what counts. `SuccessExitStatus=` declares extra exit codes or signals as clean, so a job that exits 3 for "nothing to do" can be marked successful. And `ExecCondition=` commands that exit 1 through 254 skip the run without marking the unit failed, so a skipped run triggers nothing.

A `Type=oneshot` job with no `TimeoutStartSec=` can't time out at all: for `Type=oneshot` "the timeout is disabled by default," so a hung run stays `activating` indefinitely and `OnFailure=` never fires. Always set one on scheduled jobs.

## OnSuccess= and OnFailureJobMode=

`OnSuccess=` (systemd 249 and later) lists units activated when this unit enters the "inactive" state, which for a finished oneshot job means it succeeded. It gets the same `$MONITOR_*` variables. A pair of hooks covers both outcomes:

```ini
[Unit]
OnSuccess=report-run@%n.service
OnFailure=report-run@%n.service
```

`OnFailureJobMode=` and `OnSuccessJobMode=` control how the handler's start job is queued. `OnFailureJobMode=` defaults to `replace` and `OnSuccessJobMode=` to `fail`. The defaults are fine for a handler that nothing else depends on.

## ExecStopPost= and ExecStartPost=: Hooks Inside the Unit

`ExecStopPost=` runs after every run of the service, successful or not, including runs that failed during start-up. Its processes get `$SERVICE_RESULT` and, when systemd identified the main process, `$EXIT_CODE` and `$EXIT_STATUS` (systemd 232). systemd.exec(5) recommends it over `ExecStop=` for monitoring, because `ExecStop=` "is only invoked for services that managed to start up correctly."

```ini
[Service]
ExecStopPost=-/usr/local/bin/record-result
```

The leading `-` matters. Without it, a failing `ExecStopPost=` command turns a successful run into a failed one.

`ExecStartPost=` is not a failure hook. It runs only after `ExecStart=` succeeded (for `Type=oneshot`, after the last `ExecStart=` process exited successfully), so a failed run skips it. If an `ExecStartPost=` command fails without the `-` prefix, the unit is marked failed. Use it for a success signal only when you understand both rules; `ExecStopPost=` with `$SERVICE_RESULT` is simpler.

| Hook             | Runs on success | Runs on failure | Can fail the unit  | Knows the result         |
| ---------------- | --------------- | --------------- | ------------------ | ------------------------ |
| `OnFailure=`     | No              | Yes             | No (separate unit) | `$MONITOR_*` (251+)      |
| `OnSuccess=`     | Yes             | No              | No (separate unit) | `$MONITOR_*` (251+)      |
| `ExecStopPost=`  | Yes             | Yes             | Yes, without `-`   | `$SERVICE_RESULT` (232+) |
| `ExecStartPost=` | Yes             | No              | Yes, without `-`   | No                       |

## Restarts and Start Limits

With `Restart=on-failure` and the default `RestartMode=normal`, a service restarts "by going through a failed/inactive state," so every failed attempt triggers `OnFailure=`. A job that fails five times before succeeding sends five alerts. `RestartMode=direct` (systemd 254) moves straight to activating during automatic restarts and skips `OnSuccess=` and `OnFailure=`, so only the final outcome triggers a hook.

Start limits end the restart loop. A unit started more than `StartLimitBurst=` times within `StartLimitIntervalSec=` isn't permitted to start again, and a unit configured with `Restart=` that hits the limit isn't restarted anymore; its result is `start-limit-hit`. The man page notes it "may still be restarted manually or from a timer or socket at a later point, after the interval has passed." Defaults come from `DefaultStartLimitIntervalSec=` and `DefaultStartLimitBurst=` in the manager configuration. `systemctl reset-failed backup.service` clears the counter.

For a timer-driven job, start limits rarely matter unless the timer fires faster than the burst allows. But a handler that itself fails can hit its own limit: if `notify-failure@` instances fail repeatedly because mail is down, later failures may go unreported. Check the handler too.

## Checking With journalctl and systemctl

```bash
systemctl list-units --failed                         # everything currently failed
systemctl show backup.service -p Result -p ExecMainStatus -p ExecMainExitTimestamp
journalctl -u backup.service -n 50                    # the job's recent log
journalctl -u 'notify-failure@*'                      # did the handler run?
systemctl status 'notify-failure@backup.service.service'
```

The last two answer the question an alert setup has to answer: when the job failed, did the handler start, and did it succeed? Test it deliberately with a unit that exits 1:

```bash
sudo systemd-run --unit=onfailure-test \
  -p OnFailure=notify-failure@onfailure-test.service.service /bin/false
```

## Why OnFailure= Can't See Every Failure

Every hook above runs because a service ran. When nothing runs, nothing fails:

- **The timer never started.** `systemctl enable` without `--now`, or an `OnCalendar=` line that didn't parse.
- **The timer points at a different service**, or the service was renamed.
- **The host was off or suspended** at the scheduled time, without `Persistent=true`.
- **The run hung** with no `TimeoutStartSec=`.
- **The handler couldn't deliver**: no mail transport, a full disk, a network outage, or the host itself went down mid-run.

These are the failures that go unnoticed longest, and an alert that lives on the same host can't report them. An external heartbeat can: the job reports each run to a service that knows the schedule, and that service alerts when a report doesn't arrive on time.

## Pairing OnFailure= With an External Heartbeat

There's no CronRadar integration for systemd and no discovery of timers; each service reports over HTTP with its own monitor key and schedule. The hooks above already know the outcome, so one template can report both:

```ini
# /etc/systemd/system/cronradar-report@.service
[Unit]
Description=Report the result of %i to CronRadar

[Service]
Type=oneshot
EnvironmentFile=/etc/cronradar/api.env
EnvironmentFile=/etc/cronradar/%i.env
ExecStart=/usr/local/bin/cronradar-report
```

```sh
#!/bin/sh
# /usr/local/bin/cronradar-report: send one run's result, read from $MONITOR_*, to CronRadar.
report() {
  path=$1
  shift
  curl --fail --silent --show-error --max-time 5 --output /dev/null \
    --request POST --get --user "$CRONRADAR_API_KEY:" \
    --data-urlencode "schedule=$CRONRADAR_SCHEDULE" \
    --data-urlencode "gracePeriod=$CRONRADAR_GRACE" \
    --data-urlencode "run_id=$MONITOR_INVOCATION_ID" \
    "$@" "https://cron.life/api/v1/ping/$CRONRADAR_MONITOR$path" || true
}

if [ "$MONITOR_SERVICE_RESULT" = success ]; then
  report ""          # heartbeat: a successful run
else
  report /fail --data-urlencode \
    "message=systemd result ${MONITOR_SERVICE_RESULT:-unknown} ${MONITOR_EXIT_STATUS:-}"
fi
exit 0
```

```ini
# /etc/cronradar/backup.service.env
CRONRADAR_MONITOR=nightly-backup
CRONRADAR_SCHEDULE="30 2 * * *"
CRONRADAR_GRACE=900
```

`/etc/cronradar/api.env` holds `CRONRADAR_API_KEY=ck_app_...`, readable only by root; the service manager reads it before starting the handler. Then add both hooks to the job:

```ini
# /etc/systemd/system/backup.service.d/cronradar.conf
[Unit]
OnSuccess=cronradar-report@%n.service
OnFailure=cronradar-report@%n.service
```

A success sends a plain heartbeat and a failure sends `fail`. Each report carries the schedule, so the first run creates the monitor and a deleted one comes back on the next run. `run_id` is the run's invocation ID, the same ID `journalctl _SYSTEMD_INVOCATION_ID=` filters on. A `fail` report alerts immediately with the systemd result in its message. Every case in the previous section sends nothing at all, and after the expected time plus the grace period the monitor goes from `warning` to `critical` and alerts by email, Slack, webhook, Discord, or PagerDuty.

This version reports only the outcome, so a hung run shows as a missed one. To get duration tracking and stuck-run detection, report `start` from `ExecStartPre=` and `complete` instead of the heartbeat, as the script in [systemd timers](https://cronradar.com/blog/systemd-timers) does. CronRadar reads the schedule in the application's time zone, an IANA name in its settings (UTC when unset); set it to the zone `timedatectl` reports, or convert `OnCalendar=` times as that guide shows.

## Frequently Asked Questions

### How do I get an email when a systemd service fails?

Add `OnFailure=notify-failure@%n.service` to the service's `[Unit]` section and write a `notify-failure@.service` template whose `ExecStart=` mails `systemctl status` or the journal for `%i`. It needs a working mail transport on the host.

### What is the difference between %i and %n in OnFailure?

`%n` is expanded in the failing unit to its full name, such as `backup.service`, and becomes the instance of the handler. Inside the handler template, `%i` is that instance string. So `OnFailure=notify@%n.service` on `backup.service` starts `notify@backup.service.service`, where `%i` is `backup.service`.

### Does OnFailure fire on every restart?

With `Restart=` and the default `RestartMode=normal`, yes: each failed attempt goes through the failed state. `RestartMode=direct`, available since systemd 254, skips `OnFailure=` and `OnSuccess=` during automatic restarts.

### Is there an OnSuccess= in systemd?

Yes, since systemd 249. It starts the listed units when the unit enters the inactive state, which for a finished oneshot service means it succeeded. Handlers get the same `$MONITOR_*` variables as `OnFailure=` handlers on systemd 251 and later.

### Why didn't OnFailure fire when my timer didn't run?

Because nothing failed. `OnFailure=` reacts to a unit entering the failed state, and a timer that never fired, a powered-off host, or a hung job without a timeout never gets there. An external heartbeat monitor that expects each run catches those.
