Push vs Pull Monitoring for Scheduled Jobs: Heartbeats vs Scraping

Pull monitoring means the monitor connects to a target and reads its state, as Prometheus does when it scrapes a /metrics endpoint; push monitoring means the target sends its state to the monitor, as a cron job does when it sends a heartbeat on each run. For long-running services the choice matters little: Prometheus's FAQ calls pulling "slightly better than pushing" but says it "should not be considered a major point" (Prometheus FAQ). For scheduled jobs it decides what you can see. A batch job that isn't running has nothing to scrape, so pull setups add a long-lived holder for the last result, and push setups need a monitor that knows the schedule well enough to notice a report that never came. Versions checked October 25, 2026: Prometheus 3.15, Pushgateway 1.11, node_exporter 1.12, Nagios Core 4.5, Zabbix 7.4.

Pull Asks, Push Reports#

Pull. The monitor holds a list of targets and contacts each on its own interval. Prometheus scrapes HTTP endpoints, Nagios Core's active checks "poll" a device or service (Nagios), and an uptime checker requests a URL. Because the monitor starts every exchange, a failed connection is information in itself: each Prometheus scrape records up, 1 if the target was reachable and 0 if the scrape failed (jobs and instances). That is the advantage the FAQ names: "You can more easily and reliably tell if a target is down."

Push. The monitored thing starts the exchange. A job sends a check-in when it starts, finishes, or fails. A Nagios passive check is "initiated and performed by external applications" and its result submitted to Nagios; Zabbix trapper items "accept incoming data instead of querying for it" (Zabbix 7.4). Push alone records only what arrives; alerting on a run that never happened also takes an expectation of when the next report is due. A dead man's switch is exactly that: push plus absence detection.

Direction is separate from vantage point. Google's SRE book calls white-box monitoring "based on metrics exposed by the internals of the system" and black-box "testing externally visible behavior as a user would see it" (Monitoring Distributed Systems). An uptime probe is pull and black-box; a heartbeat is push and white-box, the job reporting on itself.

Batch Jobs Leave Pull Nothing to Scrape#

Prometheus's instrumentation guide says batch jobs "do not run continuously, which makes scraping them difficult" (instrumentation). A nightly job that runs for ten minutes is a target ten minutes a day; between runs there is a failed scrape or no target at all. Scraping it during the run still pays off for jobs that take "more than a few minutes," the same guide says, for resource use and latency.

Pushing straight into Prometheus doesn't change that. Prometheus 3.15 accepts OTLP metrics when started with --web.enable-otlp-receiver, which is off by default (OpenTelemetry guide). But a query returns a series only while its newest sample is younger than the lookback period, 5 minutes by default (staleness). A last-success timestamp pushed at 02:40 is gone from instant queries by 02:45, and an age rule like the one below has nothing to compare.

So Prometheus setups put a long-lived process between the job and the scraper:

• node_exporter's textfile collector. The job writes a .prom file, and node_exporter serves its contents with the host's other metrics on every scrape. Its README calls it the option "for metrics that are tied to a machine" (node_exporter 1.12).
• The Pushgateway. The job pushes over HTTP to a gateway that keeps the last push for each group and serves it on every scrape. Its README calls it "a metrics cache" that exists to "make the metrics of your ephemeral job scrapable at any time" (Pushgateway 1.11).

Both convert a push from a short-lived process into a pull from a long-lived one. Prometheus recommends the Pushgateway only "in certain limited cases," usually "capturing the outcome of a service-level batch job," and the textfile collector for jobs tied to a machine (When to use the Pushgateway). Monitor cron jobs with Prometheus has full wrappers and rules for both. The smallest textfile write, in the directory Debian's package configures:

#!/bin/sh
# /opt/backup-textfile.sh: on success, publish the time for node_exporter.
dir=/var/lib/prometheus/node-exporter

/opt/backup.sh
rc=$?
if [ "$rc" -eq 0 ]; then
  # Write a name the collector ignores, then rename it into place in one step.
  tmp="$dir/.nightly-backup-success.prom.$$"
  {
    echo '# TYPE cron_job_last_success_timestamp_seconds gauge'
    echo "cron_job_last_success_timestamp_seconds{cron_job=\"nightly-backup\"} $(date +%s)"
  } >"$tmp" && chmod 0644 "$tmp" && mv -f "$tmp" "$dir/nightly-backup-success.prom" || rm -f "$tmp"
fi
exit "$rc"

The temporary file stays in the same directory because rename(2) is atomic only within one filesystem.

A third pull option reads the scheduler's state instead of the job's: the scheduler runs continuously and knows the schedule. On Kubernetes, kube-state-metrics 2.20 exports each CronJob's next scheduled time and last successful time (CronJob metrics); Kubernetes CronJob monitoring covers the rules.

Detecting Absence Takes an Expected Schedule#

Neither direction alerts on a missing run by itself. Julius Volz's case for pull applies to push just as well: "if your monitoring system doesn't know what the world should look like and which monitored service instances should be there, how would it be able to tell when an instance just never reports in" (Prometheus blog, 2016).

The classic tools write that expectation as a fixed age:

• Nagios Core 4 compares the age of a passive service's last result with its freshness_threshold and, when the result is stale, runs the service's check_command. The documentation's own example is a nightly backup with freshness_threshold 93600, 26 hours, and a command that returns "CRITICAL: Results of backup job were not reported!" (freshness checking).
• Zabbix 7.4 pairs a trapper item, fed by zabbix_sender, with a trigger on nodata(/host/key,sec), which returns 1 when no data arrived during the period. The period can't be under 30 seconds (history functions).
• Prometheus compares a last-success timestamp with the rule's evaluation time:

time() - cron_job_last_success_timestamp_seconds{cron_job="nightly-backup"} > 26 * 3600

A fixed age fits a fixed interval and misfires on anything irregular: a weekday job breaks a 25-hour threshold every Saturday, as the Prometheus guide works through. A monitor that reads the cron expression computes the next expected run and adds a grace period, so a weekday job's Friday report sets Monday as the next deadline. Choosing a grace period shows how to size that window from measured run times.

Clocks matter too. The Prometheus rule compares a timestamp from the job host's clock with the server's time(), so skew between them shifts every age, and Zabbix warns that nodata() "may not work properly if there are time differences between Zabbix server, proxy and agent." A monitor that judges lateness by when reports arrive uses only its own clock.

What Each Model Detects#

For one nightly job, set up as described above:

Where Each Model Fails#

Pull Watches the Holder, Not the Job#

In a hybrid, up describes node_exporter or the Pushgateway. node_exporter can stay up for a year while the job's file never changes; only the age rule notices. When the host itself goes down, its series go stale, the age expression returns nothing, and only an up == 0 or vanished-series rule fires.

The Pushgateway trades away more. Prometheus lists the pitfalls of using it for general metrics collection: with many instances behind one gateway it "becomes both a single point of failure and a potential bottleneck"; "You lose Prometheus's automatic instance health monitoring via the up metric"; and it "never forgets series pushed to it and will expose them to Prometheus forever unless those series are manually deleted via the Pushgateway's API" (Prometheus). For a last-success timestamp that memory helps, since the stale value is what makes the age rule fire; series labeled by instance outlive their hosts.

Push Can't Tell a Dead Job From a Dead Network#

A pushed report has to leave the job's host and be accepted. An outbound firewall rule, DNS, a proxy the job's environment doesn't know about, a revoked credential, or a rate limit can each block it, and to the monitor each looks like a run that didn't happen. Pull at least reports that a target was unreachable; a push monitor knows only that nothing arrived. A wrapper should ignore check-in errors so monitoring can't fail the job, and with curl's --show-error the reason still lands in the job's stderr.

Push also spreads configuration across jobs. Each one needs the monitor's address, a credential, and its schedule, where pull keeps the target list in one place. Volz counts this against push: "If the monitoring system needs to know the desired state of the world anyway, then a push-based approach actually requires more configuration in total."

A Job That Never Reported Is Invisible to Both#

An age rule needs a series. A job that has never succeeded has none, so the expression returns nothing and nothing fires; Prometheus needs a separate absent() rule naming the job (functions). A push monitor has the same gap when the job's first report creates its monitor: no report, no monitor. In both, the fix is to declare the expectation before the first run.

Both Need the Monitor Watched#

Every alert depends on the monitoring stack being up. Prometheus's alerting guide asks for alerts on "Prometheus servers, Alertmanagers, PushGateways, and other monitoring infrastructure," and says external black-box monitoring "serves as a fallback in case internal systems completely fail" (alerting practices). The usual form is an always-firing Watchdog alert sent to an outside heartbeat monitor: push monitoring for the pull system. A hosted heartbeat monitor sits outside your failure domain, so your outage doesn't silence it, but its own availability is in the vendor's hands.

Network Direction and Security#

Push needs only outbound connections from the job's host: a heartbeat is one HTTPS request, and nothing listens on the host. Pull needs the monitor to reach every target. node_exporter listens on port 9100 by default (node_exporter), so the scraper needs a route and an inbound rule to each host. Across NAT or a firewall, Prometheus recommends running the server on the same network as its targets, or PushProx, which tunnels scrapes "while still following the pull model" (Prometheus, PushProx).

The exposed surface flips with the direction:

• Pull exposes reads. Anyone who can reach /metrics can read it. node_exporter and the Pushgateway take TLS and basic-auth settings from a --web.config.file (exporter-toolkit).
• Push exposes writes. A receiver accepts data from whoever can reach it. Prometheus ships its OTLP receiver disabled "because Prometheus can work without any authentication." A Pushgateway without that config file lets any client that reaches it overwrite or delete any group. Zabbix trapper items take an "Allowed hosts" list.
• Push to a hosted service puts a credential on every job. Load it from the environment or a file only the job's account can read, with one key per environment.

Scale and Cardinality#

Who opens the connection barely affects scale; in Volz's words, "it doesn't matter who initiates the TCP connection over which metrics are then transferred." He sees one asymmetry: it's "slightly more likely for a push-based approach to accidentally bring down your monitoring," because a rogue job can push data no central scrape config asked for.

For batch jobs, what grows is series and requests:

• Label the job, never the run. A run ID, start time, or hostname in a label creates a new series every run, and in a Pushgateway grouping key a new group every run that is never forgotten. Prometheus warns against labels holding "unbounded sets of values" (naming).
• Very frequent jobs stop being batch jobs. Prometheus suggests converting jobs that run "more often than every 15 minutes" into daemons, which can be scraped like any service.
• Hosted push monitors meter requests. A heartbeat is one request per run, start and complete are two, and a report rejected for rate limiting isn't recorded, so it can surface as a late or missed run.

Choosing a Model#

The models combine: the Prometheus guide's combined wrapper writes metrics and sends check-ins from one crontab entry.

Where CronRadar Fits#

CronRadar is a push monitor: a hosted heartbeat service that never scrapes. A job sends check-ins to https://cron.life, either a single heartbeat on success or start, complete, and fail. A heartbeat alone catches missed runs but records no duration and can't flag a stuck run. There's no CronRadar SDK or auto-discovery for shell scripts or crontab, so each job reports with its own HTTP check-in:

#!/bin/sh
url="https://cron.life/api/v1/ping/nightly-backup"
run_id="nightly-backup-$(date +%s)-$$"

report() {
  event=$1
  shift
  curl --fail --silent --show-error --max-time 5 --output /dev/null \
    --request POST --get --user "$CRONRADAR_API_KEY:" \
    --data-urlencode "schedule=30 2 * * *" \
    --data-urlencode "gracePeriod=900" \
    --data-urlencode "run_id=$run_id" \
    "$@" "$url/$event" || true
}

report start
/opt/backup.sh
rc=$?
if [ "$rc" -eq 0 ]; then
  report complete
else
  report fail --data-urlencode "message=backup.sh exited with code $rc"
fi
exit "$rc"

Each report gives up after five seconds and ignores errors, and the script exits with the backup's own code. The schedule and grace period apply when a check-in creates the monitor; after that, change them in the monitor's settings. Sending them on every run recreates a deleted monitor on the next one.

CronRadar reads the five-field schedule in UTC, and every check-in, start included, moves the expected time to the next scheduled occurrence, so Monday's 02:30 start sets Tuesday 02:30. If Tuesday's run never reports, the overdue check, which runs once a minute, marks the monitor warning once 02:30 passes. At the first check after 02:45, the run is both past its 15-minute grace and at least two minutes late, so the monitor turns critical and an incident opens. A fail report skips the grace and opens an incident immediately. Alerts go to email, Slack, webhook, Discord, or PagerDuty.

Its limits:

• It knows only what the job sends. No metrics streams, so trends and dashboards stay in Prometheus. A complete sent over HTTP can carry up to five numeric or boolean outcomes, such as records_processed, checked against expectations you configure.
• Schedules are five-field cron in UTC. When both day fields are restricted, CronRadar expects a run only on dates matching both, while Linux cron runs on either.
• Stuck detection is per job, not per run. A start with no finish turns critical as stuck once the next expected run plus grace passes with no other check-in; the next run's start resets that, and there's no maximum-runtime setting, so put a timeout inside the job to turn a hang into a fail.
• A monitor created by its first check-in can't flag a job that never runs. One created ahead of time, in the dashboard, by the Hangfire extension reading the scheduler's registry, or by the Quartz.NET 3.x extension for a job on a repeating simple trigger, stays pending and sends a never-executed alert once its first expected run plus grace passes.
• Rate limits apply per API key: 60 requests a minute and 1,000 an hour, as documented. A once-a-minute job sending start and complete uses 120 an hour.

Frequently Asked Questions#

What is the difference between push and pull monitoring?#

In pull monitoring the monitor connects to each target on its own interval, so a failed connection is itself a signal. In push monitoring the target sends data to the monitor, which needs only outbound access from the target but must know when a report is due to notice one missing. Prometheus scraping is pull; a cron heartbeat, a Nagios passive check, and a Zabbix trapper item are push.

Should I use the Prometheus Pushgateway for cron jobs?#

Only for service-level jobs not tied to one machine, the case Prometheus recommends it for; for jobs on a host Prometheus already scrapes, use node_exporter's textfile collector. With the Pushgateway, turn on persistence and delete retired jobs' groups, because it keeps every pushed series until you do.

What is heartbeat monitoring?#

Push monitoring with absence detection, also called a dead man's switch. Each run sends a check-in, and the monitor alerts when one hasn't arrived a grace period after it was due. Because the alert comes from the missing signal, it fires even when the job's host is down.

Can Prometheus detect a cron job that didn't run?#

Yes, if each successful run leaves a last-success timestamp in a textfile or the Pushgateway and a rule alerts when time() minus it passes a threshold. Outside Kubernetes, where kube-state-metrics exports the schedule, that threshold is a fixed age, and the rule can't fire once the series is gone, so add an absent() rule for jobs that must exist.

Which is better for batch jobs, push or pull?#

The job itself can't be scraped between runs, so it has to hand its result to something that stays up. The real choice is where that lands: a holder that Prometheus scrapes, keeping everything in your metrics stack, or a heartbeat monitor that reads the schedule and runs outside your infrastructure. The two combine: metrics for trends, a heartbeat for the missed-run alert.