Cron Job Runbook Template: What to Write Before the Alert Fires

A cron job runbook tells whoever receives the alert what the job does, who needs its result and by when, how to check each way it can fail, and exactly how to rerun one missed period safely. The last part matters most: the usual fix for a scheduled job is "run it again for yesterday," and whether that is safe depends on details only the author knows. Write them down while the job is healthy, keep the runbook next to the code, and put its link on the alert.

Why Scheduled Jobs Need Their Own Runbook#

Google's SRE book makes the general case: "thinking through and recording the best practices ahead of time in a 'playbook' produces roughly a 3x improvement in MTTR as compared to the strategy of 'winging it'" (Site Reliability Engineering, Introduction). Google says playbook; most teams say runbook. Scheduled jobs add four problems a service runbook rarely covers:

• The alert can arrive long after the cause. A run that fails at 01:15 may not get attention until morning, so the runbook must say how late is still recoverable.
• The failure is often an absence. A job that never started leaves no error, so the first checks are about the scheduler and host, not the code. How to know if a cron job ran covers what each kind of evidence proves.
• The fix is a rerun for a specific period, and the responder can't tell from the code at 3 a.m. whether a second run doubles anything.
• The real deadline is downstream: a finance import or a partner's file pickup decides when lateness becomes damage.

The Runbook Template#

The SRE workbook says playbooks "explain the severity and impact of the alert, and include debugging suggestions and possible actions to take to mitigate impact and fully resolve the alert" (The Site Reliability Workbook, On-Call). This template applies that to one scheduled job:

# Runbook: job name (use the monitor key)

Last reviewed: date, by whom
Monitor: monitor key, and the monitoring application it reports to

## What it does and who depends on it

- What it does, in one sentence someone outside the team understands
- Who uses the result: teams, systems, customers
- Deadline: when the result must exist, and what breaks if it's late

## Schedule and environment

- Schedule, in the scheduler's time zone and in UTC
- Usual duration, and the timeout that stops a hung run
- Where it runs: host, cluster, or scheduler; the user; the lock
- How it's deployed, and where the schedule is defined

## Owner and escalation

- Owning team and where to reach them
- Who to page out of hours, and when to escalate
- Who to tell when the deadline will be missed

## When the alert fires

For each case: first checks, likely causes, how to confirm.

### Missed or late (or never ran, for a new job)

### Failed

### Stuck: started, never finished

### Wrong output: completed, but the result is wrong

### Disappeared: removed from the scheduler

## Rerun one occurrence

- Safe to rerun? Why, and until when
- The exact command, with the period as an argument
- How to tell whether a run is already in progress

## Backfill and skip

- How to run several missed periods, and in what order
- Whether a period may be skipped, and who has to agree

## Don't

- Actions that make it worse

## After recovery

- Checks that prove the data is right, not only that the job exited 0

## Links

- Logs, dashboards, repository, monitor

## Changelog

- Date: what changed

What Each Section Has to Answer#

The deadline, in business terms. "Finance's ledger import loads the file at 06:00 UTC" tells a responder at 04:00 whether to wake someone. A monitoring grace period doesn't: it says how late a check-in may be before an alert, not when anyone is hurt. The CronRadar Scheduled Job Methodology uses this finish-by deadline to set a job's priority class and where its alerts go. The workbook's freshness objectives take this form, such as "The oldest data is no older than Y [seconds, days, minutes]" (Data Processing Pipelines).

One section per failure mode. Each looks different from outside, so the first question differs:

Under each, write commands, not advice: the journalctl line, the query that counts rows.

The rerun. Say why a second run for the same period changes nothing and until when that holds; idempotent scheduled jobs shows the techniques that make it true. Make the period an argument. A job that works out "yesterday" from the clock processes the wrong day when someone reruns it after midnight, so give it a --date option that defaults to yesterday. Backfills are usually manual, because cron doesn't make up runs missed while the host was down (missed runs after downtime): one period per run, oldest first, never in parallel.

What not to do, including the lock. With flock, the lock file always exists and is harmless; the lock belongs to whichever process holds the file open. Deleting the file clears nothing and can let two runs overlap (prevent overlapping cron jobs explains why). A PID file is the opposite case: remove it only after confirming its process is gone.

A Filled-In Example: Nightly Invoice Export#

# Runbook: invoice-export

Last reviewed: 2026-10-20, Priya Nair (billing platform)
Monitor: invoice-export, CronRadar application billing-prod

## What it does and who depends on it

Exports the previous UTC day's finalized invoices from the billing database
to /exports/invoices-YYYY-MM-DD.csv on the finance SFTP host.

- Used by: finance's ledger import, which loads the file at 06:00 UTC
- Deadline: a correct file by 05:45 UTC. Late or missing delays that
  morning's revenue reconciliation; on the 1st it delays month-end close.

## Schedule and environment

- Schedule: `15 1 * * *` (01:15 UTC) on batch-02 (Debian, clock in UTC), user billing
- Usual duration 4 to 12 minutes; month-end up to 25
- Timeout 2 hours: exit 124, or 137 if it had to be killed
- Lock: /var/lib/billing/invoice-export.lock (flock)
- Deadline check: invoice-export-deadline runs at 05:30 and fails if
  yesterday's file is missing
- Deployed by the billing-jobs pipeline from deploy/crontab in the repo

## Owner and escalation

- Owner: billing platform team, #billing-platform
- Out of hours: billing on-call, paged through PagerDuty
- File will miss 05:45: tell finance ops in #finance-close

## When the alert fires

Acknowledge the incident in CronRadar when you start; that stops the daily
reminder. A successful run, scheduled or manual, closes the incident.

### Missed or late: no start by 01:30

1. Did batch-02 reboot overnight? `uptime`
2. Did cron start it? `journalctl -u cron --since "01:00" | grep invoice`
3. Is the entry installed? `crontab -u billing -l | grep invoice`

Usual causes: host down at 01:15, a deploy replaced the crontab, cron
stopped. Confirmed when cron's log has no invoice-export line today.
Fix the cause, then rerun yesterday.

### Failed

The alert shows the job's failure message, which includes the exit code.

- 124 or 137: hit the timeout; check the billing replica for lock waits
- 2: database connection failed; check the replica and its credentials
- 3: the file's total didn't match billing's; billing hasn't closed
  yesterday. Ask billing on-call before rerunning.

### Stuck: started, never finished

With the timeout in place, this almost always means batch-02 died mid-run,
and the 05:30 deadline check fails before CronRadar flags it. See whether
anything holds the lock: `lslocks | grep invoice-export`.

### Wrong output

Finance reports missing invoices, or the after-recovery checks fail. Usual
cause: invoices finalized after 01:15. Rerun once billing confirms the day
is closed.

### Disappeared

Doesn't apply: this job reports over HTTP. A deleted crontab entry shows
up as missed.

## Rerun one occurrence

Safe to rerun: yes. The export reads finalized invoices for one date and
writes the file under a temporary name, then renames it, so a second run
replaces it with the same content. After 06:00 the ledger import has
already loaded that file name: ask finance ops to reload it.

    sudo -u billing /opt/billing/bin/rerun-invoice-export.sh 2026-10-23

The argument is the invoice date, normally yesterday in UTC. The script
refuses to start while another run holds the lock.

## Backfill and skip

- Several days: one date per run, oldest first
- Finish before 01:15: a manual run still holding the lock then makes
  that night's scheduled run skip.
- Never skip a day; finance expects a file for every date, even an empty one.

## Don't

- Don't run export-invoices directly; use the script or the crontab.
- Don't delete the lock file to clear a stuck run; find the holder with
  lslocks and decide whether to stop it.
- Don't edit a CSV on the SFTP host.

## After recovery

- The file's row count equals billing's invoice count for the date, plus
  the header, and its total matches the billing dashboard
- The monitor is healthy and the recovery message arrived

## Links

- Logs: https://logs.example.com/billing?job=invoice-export
- Code: https://github.com/example/billing/tree/main/jobs/invoice-export

## Changelog

- 2026-10-20: added the 05:30 deadline check; reviewed every section

Why the deadline check. CronRadar's stuck detection works per job, not per run. Every check-in moves the expected time to the job's next scheduled occurrence, and a start with no finish is flagged only when that occurrence plus the grace period passes in silence. For this job, a run that dies at 01:20 is flagged at 01:30 the next night, or never if the next run starts normally. The timeout turns a hang into a fail, but a host that dies sends nothing. When the deadline matters more than the run, a small job that checks the output before the deadline, with its own monitor, catches it.

Wrong output, detected. CronRadar can alert on this case if the job reports a bounded outcome with its completion, such as invoices_exported >= 1, after you configure that expectation on the monitor. If you add one, the rerun script must report the same value: once a key is configured, a completion that omits it fails the check (HTTP API).

Run a Missed Day by Hand, Under the Same Lock#

The rerun script validates the date, takes the scheduled run's lock, and reports to the same monitor with its own run ID. It needs util-linux flock(1) and GNU coreutils date and timeout, as on Debian, Ubuntu, Fedora, and RHEL:

#!/bin/sh
# /opt/billing/bin/rerun-invoice-export.sh 2026-10-23
day=${1:?usage: rerun-invoice-export.sh YYYY-MM-DD}

# GNU date prints a real date back unchanged; a typo or "yesterday" doesn't match.
if [ "$(date -u -d "$day" +%F 2>/dev/null)" != "$day" ]; then
  echo "not a YYYY-MM-DD date: $day" >&2
  exit 64
fi

# The scheduled crontab entry locks the same file, so only one copy runs.
exec 9>/var/lib/billing/invoice-export.lock
if ! flock -n 9; then
  echo "an invoice export is already running; see Stuck in the runbook" >&2
  exit 75
fi

# sudo starts with a clean environment, so read the key from the job's own file.
CRONRADAR_API_KEY=$(cat /etc/billing/cronradar.key 2>/dev/null)
url="https://cron.life/api/v1/ping/invoice-export"
run_id="manual-$(date -u -d "$day" +%Y%m%d)-$(date +%s)"

report() {
  event=$1
  shift
  curl --fail --silent --show-error --max-time 5 --output /dev/null \
    --request POST --get --user "$CRONRADAR_API_KEY:" \
    --data-urlencode "schedule=15 1 * * *" \
    --data-urlencode "gracePeriod=900" \
    --data-urlencode "run_id=$run_id" \
    "$@" "$url/$event" || true
}

report start
timeout -k 1m 2h /opt/billing/bin/export-invoices --date "$day"
rc=$?
if [ "$rc" -eq 0 ]; then
  report complete
else
  report fail --data-urlencode "message=manual run for $day exited with code $rc"
fi
exit "$rc"

flock -n fails at once instead of waiting, and timeout exits 124 when it stops the command, or 137 if it had to kill it. The key comes from a file because sudo's env_reset, on by default, drops the caller's variables (sudoers). Each report gives up after five seconds and ignores errors, so a monitoring outage can't block the rerun, and the script exits with the export's own status. schedule and gracePeriod apply only if the monitor no longer exists, in which case this check-in recreates it.

To CronRadar, a manual run is a run like any other. Its check-ins move the expected time to the job's next scheduled occurrence, and a successful complete resolves the open incident and sends the recovery message. The manual-20261023-... run ID tells it apart from scheduled runs in the job's history. This is why backfills should finish before 01:15: a manual run that still holds the lock then makes that night's scheduled run skip, and if it completes within the grace period, its check-ins make the skip look like an on-time run.

Store It in the Repository, Next to the Job#

A runbook's most important facts, the rerun command, the lock path, the arguments, change with the code. In the repository they change in the same pull request, get reviewed with it, and keep their history; a wiki page changes only when someone remembers. Put it beside the job (jobs/invoice-export/RUNBOOK.md), name it after the monitor key, and give the owning team a CODEOWNERS entry for that directory: GitHub then automatically requests their review on pull requests that change those files (GitHub Docs). Link alerts to its path on the default branch, keep secrets out, and link a wiki page for any part that people outside engineering maintain.

Link the Runbook From the Alert#

PagerDuty's alerting principles are blunt: "Provide clear steps to resolve the problem, or link to a run book. Alerts with neither of these things are useless" (PagerDuty Incident Response).

In CronRadar, each monitor's settings have an Owner or team field and three links: Runbook URL, Logs URL, and Repository URL. An application can set a default owner and runbook URL that its monitors inherit unless they set their own, so point the default at an index of the service's runbooks and override it on jobs with their own page.

Email, Slack, and Discord alerts link to the job's page in CronRadar, where signed-in members get buttons for the runbook, logs, and repository. To put the runbook link in the chat message itself, send alerts to a webhook and post from there. CronRadar stores the link and never fetches the page behind it, so a private repository or wiki works as long as responders can open it. It has no escalation policies or on-call schedules; PagerDuty does that part, and cron job alerting covers choosing destinations.

There is no CronRadar SDK or auto-discovery for crontab, so each job reports with its own HTTP check-in, as in the script. The Disappeared section is for jobs discovered by the Hangfire integration, and by CronRadar.Quartz 0.1.0 when all of the scheduler's triggers are repeating simple triggers: when one leaves the registry, CronRadar pauses its monitor and marks it as no longer discovered, keeping its history, and sends no alert. Look for that after deploys.

Keep It Current#

"Details in playbooks go out of date at the same rate as production environment changes," the SRE workbook warns (Maintaining Playbooks). Four habits keep a job's runbook true:

• Change it with the job. A pull request that changes the schedule, host, arguments, or lock updates the runbook and its Last reviewed line.
• Update it after every alert. On-call engineers "should update the playbook with fresh information when the corresponding page fires" (On-Call).
• Test the rerun in staging. Run it for a past date, run it again, and compare the output. Have staging report to its own CronRadar application and key, so practice doesn't touch production monitors.
• Practice on someone else. In the spirit of Google's "Wheel of Misfortune" (Disaster Role Playing), break the job in staging and have someone other than its owner fix it from the alert and the runbook alone. Every question they ask is a missing line.

When a section becomes the same command every time, the workbook's advice applies: "If your playbooks are a deterministic list of commands that the on-call engineer runs every time a particular alert fires, we recommend implementing automation" (Maintaining Playbooks). Move that step into the job.

Frequently Asked Questions#

What should a cron job runbook include?#

Purpose, consumers, and a deadline in business terms; schedule, time zone, host, and lock; owner and escalation; first checks for each failure mode; the exact rerun command with its date argument; what not to do; and the data checks that prove recovery. Date it and keep a changelog.

Where should runbooks live?#

In the repository next to the job, so it changes and is reviewed in the same pull request as the code. Link alerts to the file's path on the default branch, and link out to a wiki for parts that people outside engineering maintain.

How do I make a scheduled job safe to rerun?#

Make a second run for the same period change nothing: claim the period before working, write with upserts, write files under a temporary name and rename them, and give outside calls an idempotency key. Take the period as an argument instead of computing "yesterday" from the clock, and run manual reruns under the scheduled run's lock.

How often should runbooks be reviewed?#

Whenever the job changes, after every alert it produces, and on a fixed cycle for jobs that rarely change, so a stale host name turns up in a drill rather than an incident. The review date at the top tells readers how much to trust it.

How do I link a runbook to an alert?#

Put the URL where the alert carries it. In CronRadar, set the monitor's Runbook URL or the application's default; webhook payloads and PagerDuty custom details include it, and email, Slack, and Discord alerts link to the job's page, which shows it. CronRadar stores only the link, so the runbook can stay private.