# The CronRadar Scheduled Job Methodology Version 1.0 · Published 2026-10-31 · Updated 2026-10-31 Canonical: https://cronradar.com/methodology > Eleven rules for ranking scheduled jobs P1 to P3, routing their alerts, and building them for one server or a shared platform, each with the reason behind it. Every scheduled job is monitored. What differs from job to job is how much protection it gets, how loudly a failure is raised, and how the job must be built. One question decides all three: what does a failed run cost? ## 1. Rank every job **R1. Monitor every job, then rank it.** The rank decides how much a job gets, never whether it is watched. Why: cron launches are "fire-and-forget" ([SRE book, ch. 24](https://sre.google/sre-book/distributed-periodic-scheduling/)), so an unwatched job fails silently. **R2. Rank by five questions.** Can a run be skipped? Can it safely run twice? Does it have a finish-by deadline? What depends on it? Can it succeed with a wrong result? **R3. Assign the first class that fits.** | Class | When | Severity | Alerts go to | | ----- | -------------------------------------------------------------------------------------------------------- | -------- | ----------------------------- | | P1 | A run can't be skipped and has a finish-by deadline, or a failure loses money, customer data, or backups | Critical | PagerDuty and the owning team | | P2 | A missed or late run degrades something used the same day | Error | Slack or email | | P3 | Everything else | Warning | A webhook that opens a ticket | ## 2. Escalate by class **R4. Only P1 pages.** Why: PagerDuty treats Error like Critical, as high urgency ([Dynamic Notifications](https://docs.pagerduty.com/incident-management/incidents/dynamic-notifications)), so a P2 job sent there would wake someone. **R5. Leave escalation to the paging tool.** CronRadar sends the first alert and a daily reminder while a job stays overdue; PagerDuty's escalation policy handles levels, timeouts, and on-call. ## 3. Build for your scale **R6. One host:** every job checks in, overlap is handled on purpose, and catch-up after downtime is a choice, not a default. **R7. A scheduler inside the app:** exactly one instance schedules, and monitoring runs outside the app, because a check inside it stops with it. **R8. A scheduler service:** one leader that only launches, and each run identified by its scheduled time. A job that must not run twice skips when in doubt; a job that must not be skipped runs at least once with an idempotency key. **R9. A shared platform:** an owner for every job, jittered start times, quotas per team, retries capped by error type, and data dependencies triggered by events, not clock offsets. Why: each scale fails in its own way. Google, Slack, and Airbnb replaced cron on one host because one machine was the whole failure domain; Netflix replaced a single-leader scheduler when it stopped scaling. The [evidence](https://cronradar.com/methodology/evidence) traces every rule above to published sources. ## 4. Write it down **R10. Give every job a contract:** owner, class, schedule in UTC, overlap and missed-run policy, finish-by deadline, retry cap, run identity, grace period, alert destinations, and runbook. **R11. Review the ranking:** reconcile the inventory against monitors, re-rank a job when what depends on it changes, and review after every P1 incident. [Applying the methodology](https://cronradar.com/methodology/apply) covers each rule in CronRadar step by step. ## Applying the CronRadar Scheduled Job Methodology Canonical: https://cronradar.com/methodology/apply This page applies the rules of the [CronRadar Scheduled Job Methodology](https://cronradar.com/methodology). The research each rule rests on is on the [evidence page](https://cronradar.com/methodology/evidence). ### Rank Every Job P1, P2, or P3 CronRadar recommends monitoring every scheduled job and ranking each one. The rank decides how much protection and how loud an alert a job gets, never whether it is monitored. Evidence: cron records nothing about its launches, and Google's SRE book says owners "can (and should!)" monitor their jobs, including "independent monitoring of the effect of cron jobs." Our classes are named P1, P2, and P3 rather than critical, high, and low. "Critical" already names a CronRadar status and severity, a PagerDuty severity, a Control-M job attribute, and an HCL keyword. The numbering follows PagerDuty's priority convention, in which P1 is the most important; Temporal and Control-M's Service Priority also put 1 highest. #### Five Questions Decide the Class 1. **Can a run be skipped?** The SRE book's example: "a garbage collection cron job scheduled to run every five minutes may be able to skip one launch, but a payroll cron job scheduled to run once a month should not be skipped." A run can be skipped when the next run covers its work. 2. **Can it safely run twice?** Some jobs, such as "a process that sends out an email newsletter to a wide distribution, should not be launched more than once" (SRE ch. 24). [Idempotent scheduled jobs](https://cronradar.com/blog/idempotent-scheduled-jobs) covers how to make a second run harmless. 3. **Does it have a finish-by deadline?** A provider's cutoff, the start of a business day, or the time another job reads its output. For a job that feeds another, the deadline is the latest time its consumer can start. HCL computes this backward: "if the critical job deadline is 19:00 and the estimated duration of the critical job is 30 minutes, the critical job will not finish by the deadline unless it has started by 18:30. If the immediate predecessor of the critical job has an estimated duration of 20 minutes, it must start at latest by 18.10" ([Planning critical jobs](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgdefiningcriticaljobs.html)). 4. **What depends on it downstream?** HCL: "The critical path is the chain of dependencies, leading to the critical job, that is most at risk of causing the deadline to be missed at any given time" ([Processing and monitoring critical jobs](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgcritjobproc.html)). Control-M's SLA Management job "maps out the critical path" of a workflow ([SLA Management Job](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/SLA_Management_Job_parameters.htm)). 5. **Can it succeed with a wrong result?** An exit status of 0 says the command didn't fail, not that the output is right. Control-M treats "Job finished too quickly" as an SLA condition ([SLA Management Job](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/SLA_Management_Job_parameters.htm)). CronRadar's outcome checks test up to five reported values per run. [Silent success](https://cronradar.com/blog/scheduled-job-silent-success) covers where these runs come from. #### The Rule Our rule assigns the first class whose condition a job meets: | Class | Condition | CronRadar severity | Examples | | ----- | ----------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------- | | P1 | A run cannot be skipped and has a finish-by deadline; or a failure loses money, customer-visible data, or backups | Critical | Payroll export, billing run, nightly backup | | P2 | A skipped or late run degrades something people or downstream jobs use the same day | Error | Search reindex, daily report, cache warming | | P3 | Everything else, typically idempotent cleanups that can be skipped | Warning | Session cleanup, temporary file purge | Questions 1 and 3 test the first P1 condition. Questions 2 and 5 find failures that don't look like failures, a second run and a run that exits 0 with a wrong result; either counts toward the second P1 condition when it loses money, customer-visible data, or backups. Question 2 also decides how the job must be built (see "Architecture Rules for Each Scale"). Question 4 finds the people and jobs that use the result, which is the test for P2, and gives a job that feeds another the finish-by deadline of its consumer. In CronRadar, a monitor whose severity is blank inherits its application's default, then the workspace's, which starts as Error. A job with no severity on its monitor or application, in a workspace whose default was never changed, therefore alerts as Error, our P2. [Billing, renewal, and email jobs](https://cronradar.com/blog/business-critical-job-monitoring) covers the most common P1 jobs. ### Escalate by Class CronRadar recommends one detection path for every class and a different route for each class. #### Detection Is the Same for Every Class CronRadar applies these steps to every monitor, whatever its severity: 1. **`warning`:** the expected check-in time has passed and the monitor is within its grace period. 2. **`critical`:** the grace period has passed and the check-in is at least 2 minutes late, checked once a minute. A `fail` check-in makes the monitor `critical` at once, and so does a completed run that violates an outcome check, with the result `outcomefailed`. 3. **First alert:** an InitialFailure alert. A monitor that never reported sends NeverExecuted, and a violated outcome check sends OutcomeFailure. 4. **Reminders:** a ContinuedFailure alert every 24 hours while the monitor stays overdue, with no check-in arriving, and the incident is unacknowledged. Every check-in, `fail` included, moves the expected time to the next occurrence, so an open incident gets no reminder while the job keeps checking in on schedule: a job that reports `fail` or violates an outcome check on every run gets one alert, not daily reminders. Acknowledging the incident in CronRadar stops the reminders, not the monitoring. 5. **Recovery:** a Recovered alert when a successful run resolves the incident. Severity does not change any of this timing. The grace default depends on how a monitor is created; [Choosing a grace period](https://cronradar.com/blog/cron-grace-period) covers how to size it. #### Routing Differs by Class Our rule: only P1 routes to PagerDuty. P2 goes to Slack or email, and P3 to a webhook that opens a ticket. | | P1 | P2 | P3 | | --------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | | CronRadar severity | Critical | Error | Warning | | Alert destinations | PagerDuty, plus the owning team's Slack or email | Slack or email | A webhook to a handler that opens a ticket | | Who acts, and when | The on-call responder, now | The owner, the same working day | The owner, in normal planning | | After the first alert | PagerDuty's escalation policy: levels, timeouts, repeats | CronRadar's daily reminder while the monitor stays overdue and unacknowledged | The ticket, and CronRadar's daily reminder while the monitor stays overdue and unacknowledged | These are the Page, Notify, and Ticket tiers in [cron job alerting](https://cronradar.com/blog/cron-job-alerting). Evidence: PagerDuty's incident scale also separates a high-urgency page (SEV-3), a low-urgency page (SEV-4), and a ticket (SEV-5), and places "Cron failure (not impacting event & notification pipeline)" at SEV-4. Our rule differs: such a job is P2 or P3 and is notified by Slack, email, or a ticket, not paged. A ticket handler gets CronRadar's webhook, which carries the alert type, severity, owner, and the monitor's runbook, logs, and repository URLs. The `monitor.still_down` event repeats every 24 hours while the monitor stays overdue and unacknowledged, so the handler should update the existing ticket, not open a new one. #### The PagerDuty Urgency Caveat CronRadar sends each monitor's severity to PagerDuty as the event severity: `critical`, `error`, or `warning`. A recovery is sent as `info` and resolves the PagerDuty alert. PagerDuty then decides urgency: - On a service set to dynamic notifications, `critical` and `error` each "Uses high-urgency notification rules and escalates if not acknowledged", and `warning` "Uses low-urgency notification rules and cannot escalate" ([Dynamic Notifications](https://docs.pagerduty.com/incident-management/incidents/dynamic-notifications)). - Otherwise the service's own setting applies, and "all services initially default to High-urgency notifications, escalate as needed" ([Configurable Service Settings](https://docs.pagerduty.com/incident-management/services/configurable-service-settings)). So a P2 job, which carries Error, would page at high urgency on a service set to dynamic notifications or on a service left at its default urgency. That is why only P1 routes to PagerDuty. A P1 job carries Critical and pages at high urgency on both. #### What CronRadar Leaves to PagerDuty CronRadar has no escalation levels, escalation timers, or on-call schedules, and its reminder cadence is the same for every class. For P1, PagerDuty's escalation policy supplies the levels, the timeout (30 minutes by default), and up to nine repeats. ### Architecture Rules for Each Scale CronRadar recommends the rules below. Each names the research it rests on. #### L1 Rules: One Host 1. **Every job checks in.** Evidence: cron launches are "fire-and-forget", so nothing records a run that didn't happen. 2. **Overlap is handled on purpose:** `flock` for cron, or the systemd timer's built-in skip. Evidence: Slack's scripts that "take longer than their recurrence"; systemd never starts a second instance. [Preventing overlapping cron jobs](https://cronradar.com/blog/prevent-overlapping-cron-jobs) has the commands. 3. **Catch-up is a decision.** cron makes up nothing, anacron only daily or slower jobs, and systemd only with `Persistent=true`. Choose per job, using question 1. 4. **The API key lives in an environment file, never in the crontab line.** This rule rests on CronRadar's own behavior, not on the [research](https://cronradar.com/methodology/evidence): CronRadar check-ins use HTTP Basic authentication, which keeps the key out of "shell history, proxy logs, or copied URLs" ([CronRadar HTTP API](https://docs.cronradar.com/api-reference)). A crontab line with no key on it: ```text # /etc/cron.d/nightly-export (host clock in UTC; CronRadar evaluates schedules in UTC) 0 2 * * * exporter /usr/bin/flock -n /run/lock/nightly-export.lock /opt/jobs/run-nightly-export.sh ``` The wrapper it runs: ```sh #!/bin/sh # /opt/jobs/run-nightly-export.sh # The env file defines CRONRADAR_API_KEY; it is owned by the job's user, mode 0600. # If it is missing, the check-ins fail and the job still runs. if [ -r /etc/cronradar/nightly-export.env ]; then . /etc/cronradar/nightly-export.env; fi ping="https://cron.life/api/v1/ping/nightly-export" run_id="nightly-export-$(date -u +%F)" report() { event=$1 shift curl --fail --silent --show-error --max-time 5 --request POST \ --user "$CRONRADAR_API_KEY:" --get "$ping/$event" \ --data-urlencode "run_id=$run_id" "$@" >/dev/null || true } report start --data-urlencode "schedule=0 2 * * *" if /opt/jobs/nightly-export; then report complete else status=$? report fail --data-urlencode "message=nightly-export exited with code $status" exit "$status" fi ``` A check-in failure never changes the job's result: each report times out after 5 seconds and is ignored if it fails, and the wrapper exits with the job's own status. With `flock -n`, an occurrence skipped because the previous run still holds the lock sends no check-in. CronRadar alerts only if the previous run is still running, with a `start` but no `complete` or `fail`, when the skipped occurrence's grace period and the 2-minute threshold pass, and it records that alert as a stuck run. If the previous run completes first, the skipped occurrence raises no alert. #### L2 Rules: A Scheduler Inside the Application 1. **Exactly one instance schedules.** Use the framework's own option: Laravel's `onOneServer`, a single Celery beat, or a clustered Quartz.NET job store. Evidence: Laravel's report generated "three times" on three servers; Celery's "duplicate tasks". [Distributed cron](https://cronradar.com/blog/distributed-cron) compares the options. 2. **Jobs are idempotent or keyed by run.** Evidence: Hangfire says "there’s no reliable way to prevent multiple executions of the same background job" without transactions in the job itself. 3. **Monitoring runs outside the application.** Evidence: our L2 inference that an in-process check stops with its process. CronRadar evaluates schedules on its own servers, so an application that stops produces a missed-run alert. The CronRadar.Hangfire and CronRadar.Quartz extensions discover recurring jobs and create their monitors, so a discovered job that never runs raises NeverExecuted. CronRadar.Quartz 0.1.0 monitors only jobs on repeating simple triggers: it sends a cron trigger's six-field expression unchanged, and CronRadar rejects it until the pending 0.1.1. #### L3 Rules: A Scheduler Service With One Leader 1. **One leader, and it only launches.** Evidence: Google, Slack, Chronos, and Meson all hand execution to another system. 2. **Each run is identified by its scheduled time.** Evidence: Google's launch names, Slack's row per execution, and Kubernetes' scheduled-timestamp annotation. In CronRadar, send that identity as `run_id` on the start and terminal check-ins; an `Idempotency-Key` header makes CronRadar ignore a duplicate check-in. 3. **Choose skip or at-least-once per class.** A P1 job that must not run twice fails closed: when the scheduler cannot tell whether a launch happened, it skips, the missed check-in alerts, and a person completes the occurrence before its finish-by deadline. A job that must not be skipped runs at least once, with an idempotency key built from its scheduled time so a second run does nothing. Evidence: Google's preference for skipping and its note that "In case of a skipped launch, cron job owners can take action that appropriately matches the nature of the cron job"; the at-least-once designs of Maestro, Dropbox ATF, and Cloud Scheduler. #### L4 Rules: A Shared Platform 1. **An inventory names an owner for every job.** Evidence: Google's job owners who monitor their own jobs, Dropbox's lambda owners, and Maestro's ownership management and lineage. [Scheduled job inventory](https://cronradar.com/blog/scheduled-job-inventory) shows how to collect one. 2. **Start times are jittered or hashed.** Evidence: Google's `?` field, Netflix's midnight peak, and the jitter settings in Temporal, Cadence, systemd, and EventBridge Scheduler. A CronRadar monitor compares check-ins with the job's cron schedule, so the grace period must cover the jitter. 3. **Each team has isolation and quotas.** Evidence: Dropbox ATF's dedicated queues, clusters, and quotas; Meta Async's per-use-case quotas; FOQS's per-namespace rate limits; Slack's isolated queue; Cadence's rate limits. 4. **Retries are bounded and split by error type.** Evidence: Maestro's 2, 10, and 0 defaults; Dropbox's infinite retry loops; the caps in EventBridge Scheduler and Cloud Scheduler. A CronRadar `fail` check-in opens an incident at once, so a job that retries reports `fail` after its last attempt. [Retrying failed cron jobs](https://cronradar.com/blog/cron-job-retries) shows a wrapper. 5. **The platform team owns the scheduler; job owners own their jobs and alerts.** Evidence: Google's "Cron job owners can (and should!) monitor their cron jobs" and Dropbox's "lambda owners own all aspects". In CronRadar, a monitor with no owner at any level names the workspace in its alerts. 6. **Data dependencies use event triggers, not clock offsets.** Evidence: Maestro: "the daily workflow should process the data when the data partition is ready, not always at midnight"; Chronos "supports the definition of jobs triggered by the completion of other jobs". [ETL job monitoring](https://cronradar.com/blog/etl-job-monitoring) covers ordering steps by data. ### The Per-Job Contract CronRadar recommends one contract card per job, kept with the job's code and in the inventory. The card records the overlap, missed-run, finish-by deadline, and retry behaviors, which the research shows schedulers handle differently, so no default decides them silently. | Field | What to record | Stored by CronRadar | | ------------------ | ------------------------------------------------------------------------ | --------------------------------------------- | | Owner | A person or rotation who can change the job | Yes: Owner | | Priority class | P1, P2, or P3 | Yes, as Severity: Critical, Error, or Warning | | Schedule | Five-field cron expression, in UTC | Yes | | Overlap policy | Run both, skip, queue, or replace, in the scheduler's own terms | No: a scheduler setting | | Missed-run policy | Nothing, once, or every missed occurrence | No: a scheduler setting | | Finish-by deadline | The clock time the result must exist | No | | Retry policy | Attempts per error type, with a cap | No: a scheduler or wrapper setting | | Run identity | Job name plus scheduled time, used as the idempotency key | Per check-in: `run_id` and `Idempotency-Key` | | Check-in type | A heartbeat, or `start` plus `complete` or `fail` for jobs that can hang | Set by how the job reports | | Outcome checks | Up to five numeric or boolean expectations | Yes | | Grace period | Seconds; minimum 60; the default depends on how the monitor is created | Yes | | Destinations | From the routing table for its class | Yes: alert destinations | | Runbook URL | Steps to rerun one occurrence safely, and by when | Yes: Runbook URL | CronRadar monitors also store an environment and logs and repository URLs. [Scheduled job runbooks](https://cronradar.com/blog/scheduled-job-runbooks) has a runbook template. An example card: ```yaml job: payroll-export owner: payroll-oncall priority: P1 # CronRadar severity: Critical schedule: '0 6 25 * *' # UTC overlap: skip # Kubernetes Forbid, Temporal Skip missed_run: once # systemd Persistent=true finish_by: '12:00 UTC on the 25th' # the payroll provider's cutoff retries: # bounded, by error type platform: 3 user: 0 timeout: 0 run_id: 'payroll-export-2026-10' # job plus scheduled month check_ins: [start, complete, fail] outcome_checks: - 'employees_missing = 0' grace_seconds: 1800 destinations: [pagerduty, slack] runbook_url: 'https://git.example.com/payroll/runbooks/payroll-export.md' ``` ### Review the Ranking CronRadar recommends three reviews: 1. **Reconcile the inventory against monitors on a schedule.** A job with no monitor is unprotected, and a monitor with no job is noise. Evidence: cron launches are "fire-and-forget", and Google's job owners "can (and should!)" monitor their jobs; the reconciliation tools below are CronRadar's own. CronRadar's `GET /sync/status` lists an application's monitors, and a sync in `reconcile` mode pauses monitors whose jobs disappeared from the scheduler. [Scheduled job inventory](https://cronradar.com/blog/scheduled-job-inventory) shows the comparison. 2. **Re-rank a job when what depends on it changes.** A new consumer with a deadline can move a P3 job to P1. Evidence: HCL "dynamically recalculates the critical start times within the critical network" while the plan runs; Maestro's lineage shows "who depends on whom". 3. **Review after every P1 incident.** Change the class, severity, destinations, and grace period together. Evidence: PagerDuty's guidance on severity during an incident: "just assume the highest and review during a postmortem" ([Severity Levels](https://response.pagerduty.com/before/severity_levels/)). ### What CronRadar Covers and What It Does Not Package versions are those published when CronRadar last checked them, on September 20, 2026. | Methodology element | CronRadar today | Where not supported, use | | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Every job monitored, with auto-discovery | Supported for .NET: CronRadar.Hangfire 0.1.0 discovers recurring jobs, and CronRadar.Quartz 0.1.0 discovers only jobs on repeating simple triggers until 0.1.1; CronRadar 1.0.13 reports from other .NET code. Hangfire 0.1.0 can report a failure before Hangfire finishes retrying. Other languages and schedulers use HTTP check-ins, with no auto-discovery. | For jobs that report over HTTP, create the monitor before the first run, so a job that never runs raises NeverExecuted. | | Priority class as severity, owner, and routing | Supported. Severity, owner, and alert destinations per monitor, inherited from the application, then the workspace. Severity appears in Slack, Discord, webhook, and PagerDuty alerts and sets the PagerDuty event severity; it does not change timing or routing. | Not needed. | | Late, missed, failed, stuck, and wrong-outcome runs | Supported, with limits: a stuck run is detected only when the job's next scheduled occurrence plus grace passes, and there is no maximum-run-time setting. Outcome values and run IDs are sent over HTTP; CronRadar 1.0.13 for .NET cannot send them. | For hangs, a timeout in the job or scheduler: Kubernetes `activeDeadlineSeconds`, systemd `TimeoutStartSec=` for oneshot services, HCL `maxdur`. | | Escalation levels and on-call | Not supported; out of product scope. | PagerDuty escalation policies and schedules. | | Reminder cadence by class | Not supported. Every class gets one reminder per 24 hours while the monitor stays overdue and the incident is unacknowledged; a job that keeps checking in on schedule while failing gets no reminder. | For P1, PagerDuty's escalation timeout and repeats. | | Finish-by deadline separate from the schedule | Not supported. | A grace period sized so a missed start alerts with time to rerun before the deadline, or the scheduler's own deadline: HCL `deadline`, Control-M `Complete By`, Airflow Deadline Alerts. | | Dependencies between jobs and critical paths | Not supported. | A scheduler that models dependencies, such as Airflow, Cadence, Control-M, or HCL Workload Automation (sources below); record each job's consumers in the inventory. | Dependency sources: Airflow tasks "have upstream and downstream dependencies set between them" ([Tasks](https://airflow.apache.org/docs/apache-airflow/3.3.2/core-concepts/tasks.html)); in Cadence, "users can define parent-child dependencies" ([Announcing Cadence 1.0](https://www.uber.com/us/en/blog/announcing-cadence/)); Control-M and HCL Workload Automation map critical paths, as cited under question 4. ### Frequently Asked Questions #### How do I decide whether a scheduled job is P1, P2, or P3? A job is P1 if a run cannot be skipped and has a finish-by deadline, or if a failure loses money, customer-visible data, or backups. It is P2 if a skipped or late run degrades something people or downstream jobs use the same day. Everything else is P3. In CronRadar, P1 maps to Critical severity, P2 to Error, and P3 to Warning. #### Which scheduled job failures should page someone? Only P1 jobs. CronRadar recommends PagerDuty plus the owning team's channel for P1, Slack or email for P2, and a webhook that opens a ticket for P3. On a PagerDuty service with dynamic notifications, both the critical and error severities page at high urgency, so a P2 job sent there would page like a P1. #### Is it better for a scheduled job to skip a run or run twice? The sources disagree. Google's SRE book prefers skipping because a skipped launch is easier to recover from than a double launch. Netflix Maestro, Dropbox ATF, Google Cloud Scheduler, and EventBridge Scheduler deliver at least once; Maestro deduplicates triggers, and Dropbox ATF and Cloud Scheduler require idempotent work. CronRadar recommends deciding per job: a job that must not run twice skips when in doubt and alerts, and a job that must not be skipped runs at least once with an idempotency key. #### What does a deadline mean in a job scheduler? It depends on the tool. A start-by deadline, such as Kubernetes startingDeadlineSeconds, is the latest a late run may start. A finish-by deadline, such as HCL Workload Automation's deadline or Control-M's Complete By, is when the result must exist. A maximum run time, such as Kubernetes activeDeadlineSeconds or HCL maxdur, limits how long a run may take. #### Does CronRadar escalate alerts or manage on-call? No. CronRadar sends a first alert, a reminder every 24 hours while the monitor stays overdue and the incident is unacknowledged, and a recovery alert, the same for every severity. Escalation levels, escalation timeouts, and on-call schedules belong in PagerDuty, which CronRadar sends alerts to with the monitor's severity. ## Evidence for the CronRadar Scheduled Job Methodology Canonical: https://cronradar.com/methodology/evidence This page holds the research behind the [CronRadar Scheduled Job Methodology](https://cronradar.com/methodology). It states only what the cited sources say; where they disagree, it says so, and any inference is marked as ours. ### Terms | Term | Meaning on this page | | ------------------ | ----------------------------------------------------------------------------------------------------- | | Job | One recurring unit of work with one schedule. | | Occurrence | One scheduled time of a job, such as 02:00 UTC on October 31. | | Run | The execution of a job for one occurrence, including its retry attempts. | | Overlap | An occurrence comes due while the previous run of the same job is still running. | | Missed run | An occurrence that passed while the scheduler or its host was down. | | Start-by deadline | The latest time a late run may still start. | | Finish-by deadline | The clock time by which a run's result must exist. | | Maximum run time | How long one run, or one attempt, may take. | | Priority class | CronRadar's P1, P2, or P3 rank for a job, defined in [rule R3](https://cronradar.com/methodology). | | Severity | The Warning, Error, or Critical label on a CronRadar monitor. | | Urgency | PagerDuty's high or low notification mode for an incident. | | Escalation | Notifying the next responder when nobody acknowledges within a timeout. | | Grace period | How long past an expected check-in a CronRadar monitor stays `warning` before it can turn `critical`. | ### Four Scales of Scheduled Work The four levels are our grouping of the published systems. Each description under them is sourced. | Level | Shape | Examples in the sources | | ----- | -------------------------------------------- | -------------------------------------------------------------------------------------------- | | L1 | One host runs cron or systemd timers | cron, anacron, systemd timers | | L2 | A scheduling library inside the application | Laravel's scheduler, Celery beat, Quartz.NET, Hangfire | | L3 | A scheduler service with one active leader | Google's distributed cron, Slack's Scheduled Job Conductor, Pinterest Pinball, Netflix Meson | | L4 | One scheduling platform shared by many teams | Netflix Maestro, Dropbox ATF, Meta Async and FOQS, Uber Cadence | #### L1: One Host Running cron or systemd Timers Three of the architecture accounts reviewed describe cron on one host and why it was outgrown: Google, Slack, and Airbnb's Chronos. Airbnb's Airflow announcement names cron as a common starting point, and LinkedIn's Azkaban was itself a single-server scheduler at first. The other accounts do not describe where they started. In their words: - **Google.** "Cron's failure domain is essentially just one machine." Launches are untracked: "The cron launches are fire-and-forget, and crond makes no attempt to track these launches." The exception it names is anacron, whose "Relaunch attempts are limited to jobs that run daily or less frequently" ([Site Reliability Engineering, ch. 24](https://sre.google/sre-book/distributed-periodic-scheduling/#reliability-perspective)). - **Slack, 2023.** "There was one node with a copy of all the scripts to run and one crontab file with the schedules for all the scripts." It scaled by "moving to bigger nodes with more CPU and more RAM", and "with one box running, any issues with provisioning, rotation, or configuration would bring the service to a halt". It used flock "to ensure that only one copy of a script is running at a time"; the post adds that "there are a few scripts that take longer than their recurrence, so two copies could start running at the same time" ([Executing Cron Scripts Reliably At Scale](https://slack.engineering/executing-cron-scripts-reliably-at-scale/)). - **Airbnb, 2013.** "we were relying on a single instance with cron to kick off our hourly, daily and weekly ETL jobs". It wanted "a system that allowed retries" and "quick insights into which jobs failed and which ones succeeded" ([Chronos: A Replacement for Cron](https://medium.com/airbnb-engineering/chronos-a-replacement-for-cron-f05d7d986a9d)). - **Airbnb, 2015.** The Airflow announcement, describing companies in general: "There’s always the good old cron scheduler to get started" ([Airflow: a workflow management platform](https://medium.com/airbnb-engineering/airflow-a-workflow-management-platform-46318b977fd8)). - **LinkedIn.** Azkaban, a scheduler rather than cron, was "Initially a single server solution" ([Azkaban 3.0 Documentation](https://azkaban.github.io/azkaban/docs/latest/)). systemd 262 timers, the other common L1 scheduler, define the behaviors this page compares across tools: - **Overlap.** When the service "is already active at the time the timer elapses it is not restarted, but simply left running. There is no concept of spawning new service instances in this case" ([systemd.timer](https://www.freedesktop.org/software/systemd/man/262/systemd.timer.html)). - **Missed runs.** `Persistent=` "Defaults to false". When true, the service "is triggered immediately if it would have been triggered at least once during the time when the timer was inactive" (same page). - **Spreading.** `RandomizedDelaySec=` "Defaults to 0" and exists "to prevent them from firing all at the same time" (same page). - **Failure hook.** `OnFailure=` names units "that are activated when this unit enters the "failed" state" ([systemd.unit](https://www.freedesktop.org/software/systemd/man/262/systemd.unit.html)). - **Run-time limit.** `RuntimeMaxSec=` "does not have any effect on Type=oneshot services", and for `Type=oneshot` the start timeout "is disabled by default" ([systemd.service](https://www.freedesktop.org/software/systemd/man/262/systemd.service.html)). #### L2: A Scheduling Library Inside the Application None of the architecture accounts reviewed covers L2. The libraries' documentation addresses duplicate runs: - **Laravel 13.** "If the task scheduler is running on three worker servers, the scheduled task will run on all three servers and generate the report three times." With `onOneServer`, "The first server to obtain the task will secure an atomic lock on the job to prevent other servers from running the same task at the same time" ([Task Scheduling](https://laravel.com/docs/13.x/scheduling#running-tasks-on-one-server)). - **Celery (documentation for tag v5.6.3).** "You have to ensure only a single scheduler is running for a schedule at a time, otherwise you’d end up with duplicate tasks" ([Periodic Tasks](https://docs.celeryq.dev/en/v5.6.3/userguide/periodic-tasks.html)). - **Quartz.NET 3.x.** Clustering is the remedy: with a clustered job store, "Each firing runs on exactly one node". The same page warns: "Never start (`scheduler.Start()`) a non-clustered instance against the same set of tables that any other started instance uses" ([Configuration Reference](https://www.quartz-scheduler.net/documentation/quartz-3.x/configuration/reference.html#clustering)). - **Hangfire.** On duplicate execution of one background job: "there’s no reliable way to prevent multiple executions of the same background job other than by using transactions in background job method itself" ([Concurrency & Rate Limiting](https://docs.hangfire.io/en/latest/background-processing/throttling.html)). Our inference: a scheduler that runs inside the application stops when the application's process stops, and any check running in the same process stops with it. [Distributed cron](https://cronradar.com/blog/distributed-cron) compares the ways to run an in-app schedule on several instances. #### L3: A Scheduler Service With One Leader Four published systems run one active scheduler. Google and Slack keep standby replicas that take over; Meson was "a single leader architecture with high availability"; Pinball's single master recovers by re-reading its persisted state. - **Google.** Its distributed cron "uses a single leader job, which is the only replica that can modify the shared state, as well as the only replica that can launch cron jobs." Fast re-election keeps it "well within a generally tolerable one-minute failover time" ([SRE ch. 24, The Use of Paxos](https://sre.google/sre-book/distributed-periodic-scheduling/#the-use-of-paxos)). A leader that loses leadership "must immediately stop interacting with the datacenter scheduler" ([SRE ch. 24, The leader](https://sre.google/sre-book/distributed-periodic-scheduling/#the-leader)). - **Slack.** "we use Kubernetes Leader Election to designate one pod to the scheduling and have the other pods in standby mode so one of them can quickly take over if needed." It also "implemented logic to prevent the node from going down at the top of a minute when possible" ([Slack, 2023](https://slack.engineering/executing-cron-scripts-reliably-at-scale/)). - **Pinterest Pinball.** The master "runs in a single thread", and "each state update gets synchronously persisted before master replies to the client. This way if the master crashes, it can recover by reading the state from the storage" ([ARCHITECTURE.rst](https://github.com/pinterest/pinball/blob/c54a206cf6e3dbadb056c189f741d75828c02f98/ARCHITECTURE.rst)). - **Netflix Meson.** "Meson was based on a single leader architecture with high availability" ([Orchestrating Data/ML Workflows at Scale With Netflix Maestro](https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c)). Two practices recur at L3. **The scheduler launches the work, and something else runs it.** At Google, "Launching a job in a datacenter then effectively turns into sending one or more RPCs to the datacenter scheduler" ([SRE ch. 24](https://sre.google/sre-book/distributed-periodic-scheduling/#extended-infrastructure)). Slack offloads "almost all of the memory and CPU intensive work of actually running the scripts to Slack’s Job Queue" ([Slack, 2023](https://slack.engineering/executing-cron-scripts-reliably-at-scale/)). Chronos "runs on top of Mesos" ([Chronos](https://medium.com/airbnb-engineering/chronos-a-replacement-for-cron-f05d7d986a9d)), and Meson "delegates the actual resource scheduling to Mesos" ([Meson](https://netflixtechblog.com/meson-workflow-orchestration-for-netflix-recommendations-fc932625c1d9)). **Each run gets an identity from its scheduled time.** Google: "Simply identifying the cron job is not enough: we should also uniquely identify the particular launch using the start time" ([SRE ch. 24, The leader](https://sre.google/sre-book/distributed-periodic-scheduling/#the-leader)), and the job names it creates in the datacenter scheduler must "include the particular scheduled launch time" ([SRE ch. 24, Resolving partial failures](https://sre.google/sre-book/distributed-periodic-scheduling/#resolving-partial-failures)). Slack records "each job execution as a new row in a table" and checks for an active run before starting another ([Slack, 2023](https://slack.engineering/executing-cron-scripts-reliably-at-scale/)). "Starting with Kubernetes v1.32", each Job a CronJob creates carries `batch.kubernetes.io/cronjob-scheduled-timestamp` ([CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#job-creation)). Every source that addresses repeats either requires the work to tolerate them or deduplicates launches itself: - **Kubernetes.** "there are certain circumstances where two Jobs might be created, or no Job might be created", so "the Jobs that you define should be idempotent" ([CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#job-creation)). - **Google.** After a failover, either "All operations on external systems, which we may need to continue upon re-election, must be idempotent" or their state must be looked up ([SRE ch. 24](https://sre.google/sre-book/distributed-periodic-scheduling/#resolving-partial-failures)). - **Google Cloud Scheduler.** "Your targets should be idempotent" ([About Cloud Scheduler](https://docs.cloud.google.com/scheduler/docs/overview)). - **Netflix Maestro.** "Maestro engine service will deduplicate the triggering requests to achieve an exact-once guarantee when scheduling workflows" ([Maestro, 2022](https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c)). - **Dropbox ATF.** "A single task on a lambda can be executed multiple times within the ATF system. Developers should ensure that their lambda logic and correctness of task execution in clients are not affected by this" ([How we designed Dropbox ATF](https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox)). #### L4: A Platform Shared by Many Teams Dropbox ATF and Meta Async schedule one-time or delayed work rather than recurring jobs ("Currently, ATF is a system for one-time task scheduling"). They are included for what they show about priority, isolation, retries, and ownership. **Peak load becomes a design problem.** Google: "When people think of a "daily cron job," they commonly configure this job to run at midnight." Its fix was a `?` crontab field: "the cron system is given the freedom to choose the value. Users choose this value by hashing the cron job configuration over the given time range". Load stayed "very spiky" ([SRE ch. 24, Running Large Cron](https://sre.google/sre-book/distributed-periodic-scheduling/#running-large-cron)). Netflix reported "Slowness during peak traffic moments like 12 AM UTC" because "a lot of our workflows are run around midnight UTC" ([Maestro, 2022](https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c)). Meta's time shifting "moves the computing lift from peak hours to off-peak hours" ([Async](https://engineering.fb.com/2020/08/17/production-engineering/async/)). Schedulers offer spreading settings: - Temporal jitter: "a random offset between zero and the maximum jitter is added to each Action time" ([Schedule](https://docs.temporal.io/schedule)). - Cadence jitter "spreads load when many schedules fire at the same wall-clock time (for example, midnight)" ([Schedules](https://cadenceworkflow.io/docs/concepts/schedules/)). - systemd `RandomizedDelaySec=`, described under L1. - EventBridge Scheduler flexible time windows: "Setting a flexible time window improves the reliability of your schedule by dispersing your target invocations" ([flexible time windows](https://docs.aws.amazon.com/scheduler/latest/UserGuide/managing-schedule-flexible-time-windows.html)). [Spreading midnight cron load](https://cronradar.com/blog/cron-midnight-load-spreading) covers how to apply them. **A single leader stops scaling.** Meson "schedules around 70 thousands of workflows and half a million jobs per day", and Netflix "had to vertically scale the system to keep up and were approaching AWS instance type limits". In its replacement, Maestro, "All services in the Maestro system are stateless and can be horizontally scaled out", requests pass through distributed queues, CockroachDB holds state, and the target is "a strict SLO of less than 1 minute of scheduler introduced delay" ([Maestro, 2022](https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c)). **Priority and isolation become first-class.** - **Dropbox ATF** gives every pair of lambda and priority its own SQS queue, and "Isolation of lambdas is achieved through dedicated worker clusters, dedicated queues, and dedicated per-lambda scheduling quotas" ([Dropbox ATF](https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox)). - **Meta Async** first "had three priorities, and inside each priority queue, it was first come, first served", until "large use cases would dominate the service and smaller use cases would have to wait in the queue". It moved to delay tolerance, "how long each request can be delayed", with a queue per use case "ordered by the deadline of the job execution", and quotas: "When use cases went above the established quota, their execution would be throttled and an alert was sent to the engineers" ([Async](https://engineering.fb.com/2020/08/17/production-engineering/async/)). - **Meta FOQS.** "Priority (a user-specified 32-bit integer): A lower number signifies a higher priority", and "each namespace has a rate limit (computed as enqueues per minute)" ([FOQS](https://engineering.fb.com/2021/02/22/production-engineering/foqs-scaling-a-distributed-priority-queue/)). - **Slack.** Its cron scripts run as jobs isolated on their own queue in the Job Queue ([Slack, 2023](https://slack.engineering/executing-cron-scripts-reliably-at-scale/)), whose relay "respects the rate limits configured in Consul" ([Scaling Slack’s Job Queue](https://slack.engineering/scaling-slacks-job-queue/)). - **Uber Cadence** has "built-in rate limits, hot shard detection, version and per-domain scale tracking", and Uber invested in "better traffic isolation to avoid noisy neighbor issues" ([Announcing Cadence 1.0](https://www.uber.com/us/en/blog/announcing-cadence/)). **Retries are bounded and split by error type.** - **Netflix Maestro's** default retry policy allows 2 retries for user errors, 10 for platform errors, and 0 for timeouts ([Defaults.java](https://github.com/Netflix/maestro/blob/7da0cac2bf9438b398e369d59cad5865571e73f3/maestro-common/src/main/java/com/netflix/maestro/models/Defaults.java#L94-L104)). - **Dropbox ATF.** Client code marks a task "successfully completed, fatally terminated, or retriable". Still, "some tasks get stuck in infinite retry loops due to occasional bugs in lambda logic. This requires manual intervention from the ATF framework owners in some cases" ([Dropbox ATF](https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox)). - **Slack's earlier job queue** retried "a configured number of times until it eventually succeeds or moves onto a list of permanently failed jobs which is manually inspected and repaired" ([Scaling Slack’s Job Queue](https://slack.engineering/scaling-slacks-job-queue/)). - **EventBridge Scheduler.** `MaximumRetryAttempts` accepts 0 to 185 and `MaximumEventAgeInSeconds` 60 to 86400 ([RetryPolicy](https://docs.aws.amazon.com/scheduler/latest/APIReference/API_RetryPolicy.html)). A schedule with a dead-letter queue sends it the event once retries are exhausted ([dead-letter queue](https://docs.aws.amazon.com/scheduler/latest/UserGuide/configuring-schedule-dlq.html)). - **Google Cloud Scheduler.** "The default value of retryCount is zero" ([REST reference](https://docs.cloud.google.com/scheduler/docs/reference/rest/v1/projects.locations.jobs#RetryConfig)), and "Values greater than 5 and negative values are not allowed" ([Retry jobs](https://docs.cloud.google.com/scheduler/docs/configuring/retry-jobs)). **Ownership is split between the platform and the job owners.** Google: "Cron job owners can (and should!) monitor their cron jobs; for example, an owner might have the cron service expose state for its managed cron jobs, or set up independent monitoring of the effect of cron jobs" ([SRE ch. 24](https://sre.google/sre-book/distributed-periodic-scheduling/#cron-jobs-and-idempotency)). Dropbox: "lambda owners own all aspects of their lambdas’ operations" ([Dropbox ATF](https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox)). Maestro lists "Workflow definition and ownership management", and its signal lineage "helps the upstream and downstream workflow owners to see who depends on whom" ([Maestro, 2022](https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c)). **Published alerting stops short of per-job escalation.** - **Azkaban** can "Send failure emails after the first failure is detected", and for SLA rules, "If duration threshold is exceeded, then an alert email can be set or the flow or job can be auto killed" ([Azkaban 3.0](https://azkaban.github.io/azkaban/docs/latest/)). - **Pinball** emails a warning when a workflow has more running instances than its `max_running_instances` setting ([schedule.py](https://github.com/pinterest/pinball/blob/c54a206cf6e3dbadb056c189f741d75828c02f98/pinball/scheduler/schedule.py#L266-L294)). - **Meta Async** alerts engineers when a use case exceeds its quota (above). - **Slack**, rolling out its Kafka-backed job queue, "enqueued heartbeat canaries for every Kafka partition every minute" and "monitored and alerted on the end-to-end flow and timing for these heartbeat canaries" ([Scaling Slack’s Job Queue](https://slack.engineering/scaling-slacks-job-queue/)). - **Control-M.** An SLA Management job can act when "The service is predicted to exceed the SLA deadline": notify its Alerts tool, send mail, or open a BMC Remedy ticket ([SLA Management Job](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/SLA_Management_Job_parameters.htm)). - **PagerDuty's incident response guide** lists "Cron failure (not impacting event & notification pipeline)" under SEV-4, whose response is a "Low-Urgency page to service team". SEV-3 gets a "High-Urgency page to service team" and SEV-5 a "JIRA ticket" ([Severity Levels](https://response.pagerduty.com/before/severity_levels/)). None of the scheduler accounts or scheduler documentation reviewed defines escalation tiers for individual scheduled jobs. PagerDuty's guide gives cron failures a response level, but as part of a general incident scale, not a scheduler feature. ### The Same Behaviors Under Different Names Schedulers give the same behavior different names, and their defaults differ. We compiled these tables from each tool's documentation or source code. #### Overlap: A Run Is Due While the Previous One Is Still Running | Behavior | The new run | Names | Default in | | -------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | | Run both | Starts alongside the running one | Kubernetes `Allow`; Temporal `AllowAll`; Cadence `Concurrent`; HCL `parallel`; Maestro `PARALLEL`; Pinball `START_NEW`; Azkaban "Run Concurrently" | Kubernetes, HCL | | Skip | Does not start | Kubernetes `Forbid`; Temporal `Skip`; Cadence `SkipNew`; Maestro `FIRST_ONLY`; Pinball `SKIP`; Azkaban "Skip Execution"; HCL `donotstart`; systemd timers, always | Temporal, Cadence, Pinball, systemd | | Queue | Starts after the running one ends | Temporal `BufferOne`, `BufferAll`; Cadence `Buffer`; HCL `enqueue`; Maestro `SEQUENTIAL`; Pinball `DELAY`, `DELAY_UNTIL_SUCCESS`; Google Cloud Scheduler, always | Maestro, Cloud Scheduler | | Replace | Stops the running one, then starts | Kubernetes `Replace`; Temporal `CancelOther`, `TerminateOther`; Cadence `CancelPrevious`, `TerminatePrevious`; Maestro `LAST_ONLY`; Pinball `ABORT_RUNNING` | None of these | Qualifications: - Kubernetes `Forbid` can still start the skipped run late: "when the previous Job run finishes, .spec.startingDeadlineSeconds is still taken into account and may result in a new Job run." - HCL `donotstart`: "If, for any reason, the job stream cannot start within four minutes, the next instance does not start at all." - Cadence's older `CronSchedule` option is fixed: "always skip if previous run is active". - Cloud Scheduler: "A scheduled start time will be delayed if the previous execution has not ended when its scheduled time occurs." - Azkaban has a third option, Pipeline, which holds each job of the new execution until the same job of the previous execution has finished. Its default is not stated on the cited page. Sources: Kubernetes [concurrency policy](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#concurrency-policy); Temporal [Overlap Policy](https://docs.temporal.io/schedule#overlap-policy); Cadence [Schedules](https://cadenceworkflow.io/docs/concepts/schedules/); HCL Workload Automation 10.2.8 [onoverlap](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgonoverlap.html); Maestro [RunStrategy.java](https://github.com/Netflix/maestro/blob/7da0cac2bf9438b398e369d59cad5865571e73f3/maestro-common/src/main/java/com/netflix/maestro/models/definition/RunStrategy.java#L46-L74) and its `SEQUENTIAL` default in [Defaults.java](https://github.com/Netflix/maestro/blob/7da0cac2bf9438b398e369d59cad5865571e73f3/maestro-common/src/main/java/com/netflix/maestro/models/Defaults.java#L110-L111); Pinball [overrun_policy.py](https://github.com/pinterest/pinball/blob/c54a206cf6e3dbadb056c189f741d75828c02f98/pinball/scheduler/overrun_policy.py#L26-L42) and its `SKIP` default in [schedule.py](https://github.com/pinterest/pinball/blob/c54a206cf6e3dbadb056c189f741d75828c02f98/pinball/scheduler/schedule.py#L140-L152); [Azkaban 3.0](https://azkaban.github.io/azkaban/docs/latest/); [systemd.timer](https://www.freedesktop.org/software/systemd/man/262/systemd.timer.html); Cloud Scheduler [REST reference](https://docs.cloud.google.com/scheduler/docs/reference/rest/v1/projects.locations.jobs). #### Missed Runs: The Scheduler Was Down When a Run Was Due | Tool | By default | Setting that changes it | | ------------------ | ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------- | | cron | Nothing is made up; launches are "fire-and-forget" | anacron, for jobs that run daily or less often | | systemd timer | Nothing is made up; `Persistent=` defaults to false | `Persistent=true`: one activation if at least one was missed | | Kubernetes CronJob | A late run can still start; with `startingDeadlineSeconds` unset, none starts after more than 100 missed schedules | `startingDeadlineSeconds`: runs later than the deadline are skipped and treated as failed | | Temporal Schedule | Missed actions are taken unless over one year late | Catchup Window, minimum ten seconds | | Cadence Schedule | Missed fires within the catch-up window, default one year; after a pause, missed fires are dropped | Catch-up window; `--catch_up_policy` `Skip`, `One`, or `All` on unpause | | Apache Airflow 3.3 | A run for the latest interval only (`catchup_by_default=False`) | `catchup=True`: a run for every missed interval | | Control-M | Not made up (`RetroactiveJob` defaults to false) | Retroactive ordering: after two days down, "an additional two times" | Sources: [SRE ch. 24](https://sre.google/sre-book/distributed-periodic-scheduling/#reliability-perspective); [systemd.timer](https://www.freedesktop.org/software/systemd/man/262/systemd.timer.html); Kubernetes [CronJob limitations](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-job-limitations) and [starting deadline](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#starting-deadline); Temporal [Catchup Window](https://docs.temporal.io/schedule#catchup-window); Cadence [Schedules](https://cadenceworkflow.io/docs/concepts/schedules/); Airflow [Catchup](https://airflow.apache.org/docs/apache-airflow/3.3.2/core-concepts/dag-run.html#catchup); Control-M [Job Properties](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/API_CodeRef_JobProperties.htm) and [Scheduling](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/Job_scheduling.htm). [Missed runs after downtime](https://cronradar.com/blog/missed-runs-after-downtime) covers more schedulers and how to catch up safely. #### Deadlines: Three Different Meanings | Meaning | Names | When it passes | | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | Start-by deadline | Kubernetes `startingDeadlineSeconds`; HCL `until` on a job; Temporal and Cadence Catchup Window | The occurrence does not run; Kubernetes "treats them as failed Jobs" | | Finish-by deadline | HCL `deadline`; HCL `until` on a job stream; Control-M `Complete By` and `Complete In`; Airflow Deadline Alerts | HCL marks the job late and logs a message; Control-M runs its SLA actions; Airflow runs a callback | | Maximum run time | HCL `maxdur`; Kubernetes Job `activeDeadlineSeconds`; systemd `RuntimeMaxSec=`; Cloud Scheduler `attemptDeadline`, per attempt; Azkaban SLA duration threshold | The run is killed or failed; HCL `maxdur` can also continue | The definitions, in each tool's words: - **Kubernetes:** `startingDeadlineSeconds` "defines a deadline (in whole seconds) for starting the Job, if that Job misses its scheduled time for any reason" ([starting deadline](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#starting-deadline)). "Once a Job reaches activeDeadlineSeconds, all of its running Pods are terminated and the Job status will become type: Failed with reason: DeadlineExceeded" ([Job](https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup)). - **HCL Workload Automation 10.2.8:** `until` "specifies the latest time a job stream must be completed or the latest time a job can be launched"; under its default `onuntil` action, `suppr`, "The job or job stream and any dependent job or job stream do not run" ([until](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgchdccjcf.html)). `deadline` "Specifies the time within which a job or job stream must complete" ([deadline](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgdeadline.html)). `maxdur` "Specifies the maximum length of time a job can run", with a Kill or Continue action ([maxdur](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgmaxdur.html)). - **Control-M:** `Complete By` "Determines when the service must complete, in days and hours, after the job is checked in", and `Complete In` "in hours and minutes, after the service runs (is ordered)" ([SLA Management Job](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/SLA_Management_Job_parameters.htm)). - **Apache Airflow:** "Deadline Alerts are new in Airflow 3.1 and should be considered experimental" ([Deadline Alerts](https://airflow.apache.org/docs/apache-airflow/3.3.2/howto/deadline-alerts.html)). They replace SLAs, which Airflow 3 "Deprecated and removed" ([Upgrading to Airflow 3](https://airflow.apache.org/docs/apache-airflow/3.3.2/installation/upgrading_to_airflow3.html)). Airflow's migration guide explains the difference: under an SLA, "If the Dag run never finishes, the SLA is never checked", while a Deadline Alert is stored when the run starts and checked periodically by the scheduler loop ([Migrating from SLA to Deadline Alerts](https://airflow.apache.org/docs/apache-airflow/3.3.2/howto/sla-to-deadlines.html)). - **Google Cloud Scheduler:** `attemptDeadline` is "The deadline for job attempts"; for HTTP targets "the default is 3 minutes" ([REST reference](https://docs.cloud.google.com/scheduler/docs/reference/rest/v1/projects.locations.jobs)). - **systemd:** `RuntimeMaxSec=` "Configures a maximum time for the service to run" but has no effect on `Type=oneshot` services ([systemd.service](https://www.freedesktop.org/software/systemd/man/262/systemd.service.html)). #### Priority Numbers Run in Opposite Directions | Tool | Field | Values | Highest | Default | | ----------------------- | ------------------------------------- | -------------------------------- | --------------------- | ---------------------------------------------- | | Temporal | Task queue priority key | 1 to 5 | 1 | 3 | | Control-M | Service Priority (SLA Management job) | 1 to 5 | 1 | Not stated | | PagerDuty | Incident priority | Usually P1 to P5 | P1 | Per account | | Meta FOQS | Item priority | 32-bit integer | Lowest number | Not stated | | Uber | Service tier, in the Cadence 1.0 post | T0 to T5 | T0, "(most critical)" | Not stated | | Apache Airflow | `priority_weight` | Any integer | Largest number | 1 | | HCL Workload Automation | `priority` | 0 to 99, `hi`, `go` | `go` | 10 | | Control-M | Job Priority (Automation API) | AA to 99 | 99 | AA | | Kubernetes | PriorityClass `value` | Up to 1 billion for user classes | Largest number | 0, unless a PriorityClass sets `globalDefault` | Sources: Temporal "A lower value implies higher priority, so 1 is the highest priority level" ([Task Queue Priority and Fairness](https://docs.temporal.io/develop/task-queue-priority-fairness)); Control-M "1 represents the highest priority and 5 represents the lowest" ([SLA Management Job](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/SLA_Management_Job_parameters.htm)) and "Values range from AA (lowest) to 99 (highest)" ([Job Properties](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/API_CodeRef_JobProperties.htm)); PagerDuty "Typically, the lower the number, the higher its impact (i.e., P1 is more important than P5)" ([Incident Priority](https://docs.pagerduty.com/incident-management/incidents/incident-priority)); [FOQS](https://engineering.fb.com/2021/02/22/production-engineering/foqs-scaling-a-distributed-priority-queue/); [Announcing Cadence 1.0](https://www.uber.com/us/en/blog/announcing-cadence/); Airflow "larger numbers mean higher priority" ([Priority Weights](https://airflow.apache.org/docs/apache-airflow/3.3.2/administration-and-deployment/priority-weight.html)); HCL, where "A priority of 0 prevents the job or job stream from launching" ([priority](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgchdeaecg.html)); Kubernetes "The higher the value, the higher the priority" and "If there is no PriorityClass with globalDefault set, the priority of Pods with no priorityClassName is zero" ([Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)). The word "critical" also has several meanings across these tools: - **CronRadar:** `critical` is a monitor health status, and Critical is a monitor severity. - **PagerDuty:** `critical` is an event severity ([Dynamic Notifications](https://docs.pagerduty.com/incident-management/incidents/dynamic-notifications)). - **Control-M:** "A critical job is a job that has a higher priority to reserve resources in order to run" ([Job Properties](https://documents.bmc.com/supportu/controlm-saas/en-US/Documentation/API_CodeRef_JobProperties.htm)). - **HCL:** the `critical` keyword "Specifies that the job is mission-critical and must be processed accordingly" ([critical](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgcriticaljob.html)). #### Escalation Lives in the Paging Tool Among the sources reviewed, only PagerDuty defines escalation as ordered levels with timeouts and repeats. "An escalation policy is an ordered set of rules (also called levels)." "The default escalation timeout is 30 minutes but can be adjusted." "An escalation policy can be repeated up to nine times" ([Escalation Policy Basics](https://docs.pagerduty.com/incident-management/schedules/escalation-policies)). Low-urgency incidents "will not automatically move to higher escalation levels if no one acknowledges it" ([Configurable Service Settings](https://docs.pagerduty.com/incident-management/services/configurable-service-settings)). The schedulers end at a single response to a failure: - **A failure hook:** systemd `OnFailure=`; Airflow Deadline Alert callbacks. - **An automatic pause:** Temporal Pause-on-failure, where a failed or timed-out execution "causes the Schedule to automatically pause" ([Schedule](https://docs.temporal.io/schedule#pause-on-failure)); Airflow `max_consecutive_failed_dag_runs_per_dag`, "(experimental)", which defaults to 0, "meaning that the dags are never paused out by default" ([Configuration Reference](https://airflow.apache.org/docs/apache-airflow/3.3.2/configurations-ref.html#max-consecutive-failed-dag-runs-per-dag)). - **A dead-letter queue:** EventBridge Scheduler. - **An action on lateness:** Control-M SLA actions; HCL `onlate`, whose "supported action is kill" ([onlate](https://help.hcl-software.com/workloadautomation/v1028/distr/src_ref/awsrgonlate.html)); Azkaban SLA emails and kills. ### Where the Sources Disagree **Skip a run, or risk running it twice?** Google: "In general, we favor skipping launches rather than risking double launches, as much as the infrastructure allows", because "recovering from a skipped launch is more tenable than recovering from a double launch", so "we prefer to "fail closed" to avoid systemically creating bad state" ([SRE ch. 24](https://sre.google/sre-book/distributed-periodic-scheduling/#cron-jobs-and-idempotency)). Maestro's time-based scheduling "provides an at-least-once scheduling guarantee" and deduplicates in the engine ([Maestro, 2022](https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c)); Dropbox ATF "guarantees that a task is executed at least once after being scheduled" ([Dropbox ATF](https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox)); Cloud Scheduler "is designed to provide "at least once" delivery" ([About Cloud Scheduler](https://docs.cloud.google.com/scheduler/docs/overview)); EventBridge Scheduler "provides at-least-once event delivery to targets" ([What is EventBridge Scheduler?](https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html)). FOQS lets each use case choose: "FOQS supports both at least once and at most once delivery semantics" ([FOQS](https://engineering.fb.com/2021/02/22/production-engineering/foqs-scaling-a-distributed-priority-queue/)). Kubernetes creates a Job "approximately once per execution time of its schedule", because "two Jobs might be created, or no Job might be created". The same section says that with `startingDeadlineSeconds` unset or large and `concurrencyPolicy` set to `Allow`, "the Jobs will always run at least once", and that a CronJob with "more than 100 missed schedules" does not start the Job ([CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#job-creation)). **The defaults conflict.** Our inference: a job moved between schedulers changes behavior unless its settings are explicit. For overlap, Kubernetes and HCL run both by default; Temporal, Cadence, Pinball, and systemd skip; Maestro and Cloud Scheduler queue. For missed runs, cron and systemd make up nothing, Airflow makes up the latest interval, and Temporal and Cadence make up runs up to a year late. **Consistency or availability?** Google's leader waits for a Paxos quorum: "the actual cron job launch does not proceed until it receives confirmation that the Paxos quorum has received the launch notification" ([SRE ch. 24, The Leader](https://sre.google/sre-book/distributed-periodic-scheduling/#the-leader)). Slack's 2017 job queue redesign chose "A bias towards availability over consistency", which "does create a small risk of lost jobs in the event that a broker host dies unexpectedly before replicating" ([Scaling Slack’s Job Queue](https://slack.engineering/scaling-slacks-job-queue/)). **Is one leader enough?** Slack chose one on purpose: "we decided that synchronizing the nodes would be more of a headache than a help", because "the pods can switch leaders very quickly, making downtime unlikely in practice" ([Slack, 2023](https://slack.engineering/executing-cron-scripts-reliably-at-scale/)). Netflix outgrew Meson's single leader at about 70,000 workflows a day. Our reading of the difference: Google's and Slack's leaders only launch work, while Meson's scheduler "manages the launch, flow control and runtime of the various workflows" ([Meson](https://netflixtechblog.com/meson-workflow-orchestration-for-netflix-recommendations-fc932625c1d9)), so each job asked more of it. ## How to cite CronRadar. The CronRadar Scheduled Job Methodology, version 1.0 (2026). https://cronradar.com/methodology ## Changelog - 1.0 (2026-10-31): First publication.