Monitor Cron Jobs With Prometheus: Textfile, Pushgateway, Alerts

To monitor cron jobs with Prometheus, have every run record a last-success timestamp, through node_exporter's textfile collector on the job's host or the Pushgateway for jobs tied to no host, and alert when time() minus that timestamp exceeds what the job's schedule allows. Add the run's duration and exit code, a rule for series that disappear, and an always-firing Watchdog alert for the monitoring stack itself. Prometheus never learns the cron schedule, so each threshold is a fixed age you choose by hand; on Kubernetes, kube-state-metrics fills part of that gap. Versions checked October 25, 2026: Prometheus 3.15, node_exporter 1.12, Pushgateway 1.11, kube-state-metrics 2.20, Alertmanager 0.34.

Cron monitoring vs uptime monitoring maps which tools catch which scheduled-job failures; this guide goes deep on the Prometheus side.

Record a Last-Success Timestamp, Plus Duration and Exit Code#

Prometheus's instrumentation guide is direct: "The key metric of a batch job is the last time it succeeded." It also suggests the overall runtime and the last completion time, all as gauges (Prometheus). For a cron job, three gauges:

A cron job exits after each run, so there's nothing to scrape in between; a gauge left behind by the last run is what Prometheus reads. Record times in seconds, Prometheus's base unit (naming), so the age is time() minus the gauge. Name the job in a label of your own, cron_job here: Prometheus attaches job and instance to scraped series and renames clashing labels to exported_job and exported_instance (scrape_config).

On the Job's Host, Write a Textfile for node_exporter#

For batch jobs tied to a machine, Prometheus recommends node_exporter's textfile collector over the Pushgateway (Prometheus). On every scrape, node_exporter 1.12 parses each file ending in .prom in the directories named by --collector.textfile.directory (textfile.go). Upstream, the flag has no default. Debian's prometheus-node-exporter package (1.9.0 in Debian 13) sets it to /var/lib/prometheus/node-exporter, where files must be readable by the prometheus user (README.textfile).

Four rules for the files:

• Text exposition format: \n line endings, and \, ", and newlines escaped in label values (exposition formats). Job names of letters, digits, and dashes need no escaping.
• No timestamps. A sample with its own timestamp makes node_exporter 1.12 skip the whole file.
• Write, then rename. A scrape can arrive mid-write, which is why node_exporter's README writes a temporary file and moves it into place. rename(2) replaces the target atomically only within one filesystem (rename(2)), so keep the temporary file in the same directory.
• Permissions. The job's account needs write access to the directory, node_exporter's account read access to the file. mktemp creates files only their owner can read (GNU coreutils), so set the mode before renaming.

A file that fails to parse is dropped and node_textfile_scrape_error becomes 1, which is worth its own alert. This wrapper runs any command and writes the three gauges:

#!/bin/sh
# /usr/local/bin/cron-metrics: run a command, record its result for node_exporter.
# Usage: cron-metrics <job-name> <command> [args...]
dir=/var/lib/prometheus/node-exporter
name=$1
shift

# Write a temporary name the collector ignores, then rename it into place.
write_prom() {
  tmp="$dir/.$1.prom.$$"
  cat >"$tmp" && chmod 0644 "$tmp" && mv -f "$tmp" "$dir/$1.prom" || rm -f "$tmp"
}

start=$(date +%s)
"$@"
rc=$?
end=$(date +%s)

write_prom "$name" <<EOF
# TYPE cron_job_last_duration_seconds gauge
cron_job_last_duration_seconds{cron_job="$name"} $((end - start))
# TYPE cron_job_last_exit_code gauge
cron_job_last_exit_code{cron_job="$name"} $rc
EOF

# Only a successful run moves the success timestamp.
if [ "$rc" -eq 0 ]; then
  write_prom "$name-success" <<EOF
# TYPE cron_job_last_success_timestamp_seconds gauge
cron_job_last_success_timestamp_seconds{cron_job="$name"} $end
EOF
fi

exit "$rc"

The crontab entry becomes 30 2 * * * /usr/local/bin/cron-metrics nightly-backup /opt/backup.sh. The success timestamp has its own file, so a failed run leaves it alone. A failed metrics write only prints an error, and the wrapper exits with the job's own status. If the wrapper itself is killed or the host reboots mid-run, nothing is written; the age rule catches that later. Shell script best practices covers exit codes, traps, and time limits for the job itself.

Off-Host Jobs Push to the Pushgateway, With Care#

Prometheus "only recommend[s] using the Pushgateway in certain limited cases," usually "capturing the outcome of a service-level batch job," one not tied to a machine (When to use the Pushgateway). The same page lists the costs: a single point of failure and bottleneck, no per-target up metric, and pushed series kept until you delete them. That last property suits a last-success timestamp: a job that stops running leaves its old timestamp behind, and the age rule fires.

#!/bin/sh
# /usr/local/bin/cron-push: run a command, push its result to the Pushgateway.
# Usage: cron-push <job-name> <command> [args...]
gateway=http://pushgateway.internal:9091
name=$1
shift

start=$(date +%s)
"$@"
rc=$?
end=$(date +%s)

{
  echo "# TYPE cron_job_last_duration_seconds gauge"
  echo "cron_job_last_duration_seconds $((end - start))"
  echo "# TYPE cron_job_last_exit_code gauge"
  echo "cron_job_last_exit_code $rc"
  if [ "$rc" -eq 0 ]; then
    echo "# TYPE cron_job_last_success_timestamp_seconds gauge"
    echo "cron_job_last_success_timestamp_seconds $end"
  fi
} | curl --fail --silent --show-error --max-time 5 --output /dev/null \
  --data-binary @- "$gateway/metrics/job/cron/cron_job/$name" || true

exit "$rc"

What matters, per the Pushgateway 1.11 README:

• POST, not PUT. curl --data-binary sends a POST, which replaces only the metrics named in the push. PUT replaces the whole group, so a failed run pushed with PUT would delete the success timestamp. Python's push_to_gateway sends PUT and pushadd_to_gateway POST, both with a 30-second default timeout worth lowering (client_python 0.26); Go's batch-job example calls Add, not Push, for this reason (client_golang 1.25).
• The URL path is the grouping key. job="cron" (always first) and cron_job="nightly-billing" identify the group and overwrite those labels in the body. Leave instance out, so moving the job to another host doesn't strand an old group.
• Set honor_labels: true on the scrape job, or the pushed job becomes exported_job.
• Persistence is off by default. Without --persistence.file, a restart loses everything; with it, a push can wait up to --persistence.interval, 5 minutes by default, before reaching disk (main.go).
• Delete retired groups, curl -X DELETE http://pushgateway.internal:9091/metrics/job/cron/cron_job/nightly-billing, or the retired job's age alert never stops.
• Pushed timestamps are rejected. Each group gets push_time_seconds, its last successful push, which doubles as a last-run signal.

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']

Alert on Age, Failure, and Missing Series#

Both wrappers produce the same metric names, so one rules file serves both. Load it through rule_files in prometheus.yml:

groups:
  - name: cron-jobs
    rules:
      # One per job: 30 2 * * *, finishes by 03:10 UTC.
      - alert: CronJobNotSucceeding
        expr: time() - cron_job_last_success_timestamp_seconds{cron_job="nightly-backup"} > 26 * 3600
        keep_firing_for: 10m
        labels:
          severity: page
        annotations:
          summary: '{{ $labels.cron_job }} last succeeded {{ $value | humanizeDuration }} ago'

      - alert: CronJobFailed
        expr: cron_job_last_exit_code != 0
        labels:
          severity: page
        annotations:
          summary: '{{ $labels.cron_job }} on {{ $labels.instance }} exited with code {{ $value }}'

      # Seen in the past week, gone now: file removed, exporter or host down, gateway wiped.
      - alert: CronJobMetricsVanished
        expr: |
          max_over_time(cron_job_last_success_timestamp_seconds[7d])
            unless cron_job_last_success_timestamp_seconds
        for: 15m
        labels:
          severity: page

      # A job that must exist, including one that has never succeeded.
      - alert: CronJobSuccessMissing
        expr: absent_over_time(cron_job_last_success_timestamp_seconds{cron_job="nightly-backup"}[1h])
        labels:
          severity: page

The age threshold is the schedule's interval plus the longest normal run plus slack. For 30 2 * * * finishing by 03:10, 26 hours alerts by about 05:10 on the morning of a missed run. Prometheus's alerting guide suggests "at least enough time for 2 full runs of the batch job," so one failure doesn't page: 10 hours for a job that runs every 4 hours and takes 1 (alerting practices). Use one run when a single miss matters; choosing a grace period shows how to size the slack from measured run times. To skip one rule per job, have the wrapper also write a cron_job_max_age_seconds gauge and alert on time() - cron_job_last_success_timestamp_seconds > cron_job_max_age_seconds.

for and keep_firing_for. The threshold already holds the tolerance, so for on the age rule only adds delay. But a failed scrape makes series stale at once ("A failed scrape is the same as an empty scrape," scrape.go), stale series return nothing (staleness), and the alert would resolve and refire. keep_firing_for: 10m rides out the gap (alerting rules). Put for on the presence rules, where a restart shouldn't page.

Missing series are the quiet failure. When the success series is gone (file deleted, host or exporter down, Pushgateway restarted without persistence), the age expression returns nothing, and nothing fires. The vanished rule needs no job names but stops firing seven days later, and it never sees a job that never reported. For jobs that must exist, name them: absent_over_time() returns 1 when a selector had no samples in its window, and absent() with a for clause does the same (functions).

The failure rule fires on the first evaluation after a scrape sees a non-zero exit code and stays firing until a successful run writes 0.

A Fixed Age Can't Follow a Cron Schedule#

A job on 0 6 * * 1-5, weekdays at 06:00 UTC, breaks a 25-hour threshold every Saturday, and a 73-hour one reports a missed Tuesday run on Thursday. PromQL's day_of_week() and hour() return the evaluation time's weekday and hour in UTC (functions), so you can write the schedule in by hand:

- alert: CronJobNotSucceeding
  # 0 6 * * 1-5, finishes by 06:30 UTC. Quiet Saturday, Sunday, and Monday before 07:00.
  expr: |
    (time() - cron_job_last_success_timestamp_seconds{cron_job="weekday-report"} > 25 * 3600)
      unless on() (day_of_week() == 6 or day_of_week() == 0 or (day_of_week() == 1 and hour() < 7))
  keep_firing_for: 10m
  labels:
    severity: page

It works, and it shows the cost: the crontab line rewritten in PromQL, once per irregular schedule, and still approximate. A missed Friday run alerts Friday morning, then resolves at midnight UTC as if fixed. A job on the 1st of each month needs a threshold above 31 days, so a missed March 1 run after a 28-day February goes unreported until March 4. And both functions use UTC, so a schedule that follows daylight saving drifts against the rule twice a year.

Kubernetes CronJobs: kube-state-metrics Knows the Schedule#

On Kubernetes, the exporter reads the CronJob object, schedule included. kube-state-metrics 2.20 computes kube_cronjob_next_schedule_time from the schedule and timeZone as the next run after the last scheduled one, and omits it while the CronJob is suspended (cronjob.go). With the stable kube_cronjob_status_last_successful_time (CronJob metrics), that gives schedule-aware rules:

- alert: KubeCronJobRunNotStarted
  expr: time() - kube_cronjob_next_schedule_time > 600
  labels:
    severity: page
  annotations:
    summary: '{{ $labels.namespace }}/{{ $labels.cronjob }}: no Job yet for the run due {{ $value | humanizeDuration }} ago'

- alert: KubeCronJobLatestRunNotSucceeded
  # The latest scheduled run has not succeeded within an hour of its scheduled time.
  expr: |
    kube_cronjob_status_last_successful_time < kube_cronjob_status_last_schedule_time
      and time() - kube_cronjob_status_last_schedule_time > 3600
  labels:
    severity: page

For failed runs, alert on kube_job_failed{condition="true"}, not kube_job_status_failed, which counts failed Pods, including ones a retry replaced, until the Job fails for a known reason (job.go). Deleted objects drop off the metrics endpoint (kube-state-metrics), so a deleted CronJob goes quiet like a deleted textfile. Kubernetes CronJob monitoring covers events, history limits, and run-time alerts.

Watch the Watcher With an Always-Firing Alert#

Every rule above runs inside the stack it watches. If Prometheus stops, nothing is evaluated; if Alertmanager stops or loses its route out, alerts go nowhere. Prometheus's guide asks for alerts on its servers, Alertmanagers, and Pushgateways, preferably as one end-to-end test (metamonitoring). The usual form is kube-prometheus 0.19's Watchdog, whose expression is vector(1), so it always fires (general.libsonnet). Alertmanager repeats it to an external service that alerts when it stops arriving (Watchdog runbook): a dead man's switch for your monitoring.

# Prometheus rule
- alert: Watchdog
  expr: vector(1)
  labels:
    severity: none

# alertmanager.yml: merge this route and receiver into your existing config
route:
  routes:
    - matchers:
        - alertname="Watchdog"
      receiver: external-heartbeat
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 1m

receivers:
  - name: external-heartbeat
    webhook_configs:
      - url_file: /etc/alertmanager/heartbeat-url
        send_resolved: false

Alertmanager checks repeat_interval only at each group_interval, rounding up to a multiple (route), so this sends one notification a minute. url_file keeps the receiving URL, often a secret, out of the config (webhook_config). The Watchdog covers Prometheus down, Alertmanager down, a broken route, and lost egress. It doesn't cover a single exporter or Pushgateway, which the vanished-series and up == 0 rules handle, or a selector typo that matches nothing and never fires; promtool test rules catches that (unit testing rules).

What the DIY Setup Costs Compared With a Hosted Heartbeat Monitor#

Prometheus wins when it's already running and someone owns it: no new dependency, data in your stack, duration trends beside every other graph. It's weakest at the expectation itself, wherever cron is irregular, and every alert path runs through infrastructure that can fail along with the job. A hang is slow to surface either way unless the job enforces its own time limit.

Running CronRadar Alongside Prometheus#

CronRadar is one hosted option. A job checks in to https://cron.life with a heartbeat, or with start, complete, and fail. CronRadar reads the five-field cron schedule in UTC, expects each check-in at the next scheduled occurrence, and waits a grace period (90 seconds by default, 60 minimum) before a late run goes from warning to critical. After Friday's run of 0 6 * * 1-5, the next expected run is Monday at 06:00, so the weekend needs no rule. A fail alerts immediately, a start with no finish is flagged as stuck, and a complete sent over HTTP can carry up to five numeric or boolean outcomes; a missing or violated expectation opens an incident. Alerts go to email, Slack, webhook, Discord, or PagerDuty.

Its limits: schedules are UTC only, and when both day fields are restricted it expects a run only on dates matching both, while Linux cron runs on either. Stuck detection is per job, not per run: it fires only once the job's next expected run plus grace passes with no other check-in, the next run's start resets it, and there's no maximum-runtime setting. There's no CronRadar SDK or auto-discovery for shell scripts or crontab; each job reports with its own HTTP check-in. It collects no metrics streams, so dashboards and long-term trends stay in Prometheus. Cron monitoring tools compared covers other hosted services.

Both can run from one crontab entry. This wrapper sends CronRadar check-ins around the cron-metrics script above; each check-in gives up after five seconds and ignores errors, and the job's exit code passes through:

#!/bin/sh
# /usr/local/bin/cron-run: CronRadar check-ins around cron-metrics.
# Usage: cron-run <job-name> '<schedule in UTC>' <command> [args...]
name=$1
schedule=$2
shift 2
run_id="$name-$(date +%s)-$$"
# The application's API key, alone in a file only the job's account can read
CRONRADAR_API_KEY=$(cat /etc/cronradar/api-key) || CRONRADAR_API_KEY=

report() {
  event=$1
  shift
  curl --fail --silent --show-error --max-time 5 --output /dev/null \
    --request POST --get --user "$CRONRADAR_API_KEY:" \
    --data-urlencode "schedule=$schedule" \
    --data-urlencode "gracePeriod=900" \
    --data-urlencode "run_id=$run_id" \
    "$@" "https://cron.life/api/v1/ping/$name/$event" || true
}

report start
/usr/local/bin/cron-metrics "$name" "$@"
rc=$?
if [ "$rc" -eq 0 ]; then
  report complete
else
  report fail --data-urlencode "message=$name exited with code $rc"
fi
exit "$rc"

The entry is 30 2 * * * /usr/local/bin/cron-run nightly-backup '30 2 * * *' /opt/backup.sh on a UTC server; elsewhere, pass the UTC time. The schedule and grace period apply only when a check-in creates the monitor; change them later in the monitor's settings. Sending them every time recreates a deleted monitor on the next run.

Frequently Asked Questions#

Can Prometheus monitor cron jobs?#

Yes, indirectly. Each run leaves a last-success timestamp in a textfile or the Pushgateway, and a rule fires when it gets too old. Prometheus doesn't know the cron schedule, so you pick each threshold; on Kubernetes, kube-state-metrics exports the next scheduled time.

Should I use Pushgateway for cron jobs?#

Only for service-level jobs not tied to one machine; for jobs on a scraped host, Prometheus recommends the textfile collector. With the Pushgateway, push with POST so a failed run keeps the last success, set honor_labels: true, enable persistence, and delete retired groups.

How do I alert when a cron job hasn't run in Prometheus?#

Alert on time() - cron_job_last_success_timestamp_seconds > threshold, with the threshold covering one schedule interval, the job's longest run, and some slack. Add an absent() or vanished-series rule, because the age expression can't fire once its series is gone.

What is the node_exporter textfile collector?#

A node_exporter collector that exposes metrics from .prom files in the directory set by --collector.textfile.directory, alongside the host's other metrics. Batch jobs write results there in the Prometheus text format, to a temporary file that is then renamed into place.

Who monitors Prometheus itself?#

Something outside it. An always-firing Watchdog alert, repeated by Alertmanager to an external service that alerts when the notifications stop, covers Prometheus, Alertmanager, and the path out of your network with one signal.