Evidence for the CronRadar Scheduled Job Methodology

This page holds the research behind the CronRadar Scheduled Job Methodology. It states only what the cited sources say; where they disagree, it says so, and any inference is marked as ours.

Terms#

Four Scales of Scheduled Work#

The four levels are our grouping of the published systems. Each description under them is sourced.

L1: One Host Running cron or systemd Timers#

Three of the architecture accounts reviewed describe cron on one host and why it was outgrown: Google, Slack, and Airbnb's Chronos. Airbnb's Airflow announcement names cron as a common starting point, and LinkedIn's Azkaban was itself a single-server scheduler at first. The other accounts do not describe where they started. In their words:

• Google. "Cron's failure domain is essentially just one machine." Launches are untracked: "The cron launches are fire-and-forget, and crond makes no attempt to track these launches." The exception it names is anacron, whose "Relaunch attempts are limited to jobs that run daily or less frequently" (Site Reliability Engineering, ch. 24).
• Slack, 2023. "There was one node with a copy of all the scripts to run and one crontab file with the schedules for all the scripts." It scaled by "moving to bigger nodes with more CPU and more RAM", and "with one box running, any issues with provisioning, rotation, or configuration would bring the service to a halt". It used flock "to ensure that only one copy of a script is running at a time"; the post adds that "there are a few scripts that take longer than their recurrence, so two copies could start running at the same time" (Executing Cron Scripts Reliably At Scale).
• Airbnb, 2013. "we were relying on a single instance with cron to kick off our hourly, daily and weekly ETL jobs". It wanted "a system that allowed retries" and "quick insights into which jobs failed and which ones succeeded" (Chronos: A Replacement for Cron).
• Airbnb, 2015. The Airflow announcement, describing companies in general: "There’s always the good old cron scheduler to get started" (Airflow: a workflow management platform).
• LinkedIn. Azkaban, a scheduler rather than cron, was "Initially a single server solution" (Azkaban 3.0 Documentation).

systemd 262 timers, the other common L1 scheduler, define the behaviors this page compares across tools:

• Overlap. When the service "is already active at the time the timer elapses it is not restarted, but simply left running. There is no concept of spawning new service instances in this case" (systemd.timer).
• Missed runs. Persistent= "Defaults to false". When true, the service "is triggered immediately if it would have been triggered at least once during the time when the timer was inactive" (same page).
• Spreading. RandomizedDelaySec= "Defaults to 0" and exists "to prevent them from firing all at the same time" (same page).
• Failure hook. OnFailure= names units "that are activated when this unit enters the "failed" state" (systemd.unit).
• Run-time limit. RuntimeMaxSec= "does not have any effect on Type=oneshot services", and for Type=oneshot the start timeout "is disabled by default" (systemd.service).

L2: A Scheduling Library Inside the Application#

None of the architecture accounts reviewed covers L2. The libraries' documentation addresses duplicate runs:

• Laravel 13. "If the task scheduler is running on three worker servers, the scheduled task will run on all three servers and generate the report three times." With onOneServer, "The first server to obtain the task will secure an atomic lock on the job to prevent other servers from running the same task at the same time" (Task Scheduling).
• Celery (documentation for tag v5.6.3). "You have to ensure only a single scheduler is running for a schedule at a time, otherwise you’d end up with duplicate tasks" (Periodic Tasks).
• Quartz.NET 3.x. Clustering is the remedy: with a clustered job store, "Each firing runs on exactly one node". The same page warns: "Never start (scheduler.Start()) a non-clustered instance against the same set of tables that any other started instance uses" (Configuration Reference).
• Hangfire. On duplicate execution of one background job: "there’s no reliable way to prevent multiple executions of the same background job other than by using transactions in background job method itself" (Concurrency & Rate Limiting).

Our inference: a scheduler that runs inside the application stops when the application's process stops, and any check running in the same process stops with it. Distributed cron compares the ways to run an in-app schedule on several instances.

L3: A Scheduler Service With One Leader#

Four published systems run one active scheduler. Google and Slack keep standby replicas that take over; Meson was "a single leader architecture with high availability"; Pinball's single master recovers by re-reading its persisted state.

• Google. Its distributed cron "uses a single leader job, which is the only replica that can modify the shared state, as well as the only replica that can launch cron jobs." Fast re-election keeps it "well within a generally tolerable one-minute failover time" (SRE ch. 24, The Use of Paxos). A leader that loses leadership "must immediately stop interacting with the datacenter scheduler" (SRE ch. 24, The leader).
• Slack. "we use Kubernetes Leader Election to designate one pod to the scheduling and have the other pods in standby mode so one of them can quickly take over if needed." It also "implemented logic to prevent the node from going down at the top of a minute when possible" (Slack, 2023).
• Pinterest Pinball. The master "runs in a single thread", and "each state update gets synchronously persisted before master replies to the client. This way if the master crashes, it can recover by reading the state from the storage" (ARCHITECTURE.rst).
• Netflix Meson. "Meson was based on a single leader architecture with high availability" (Orchestrating Data/ML Workflows at Scale With Netflix Maestro).

Two practices recur at L3.

The scheduler launches the work, and something else runs it. At Google, "Launching a job in a datacenter then effectively turns into sending one or more RPCs to the datacenter scheduler" (SRE ch. 24). Slack offloads "almost all of the memory and CPU intensive work of actually running the scripts to Slack’s Job Queue" (Slack, 2023). Chronos "runs on top of Mesos" (Chronos), and Meson "delegates the actual resource scheduling to Mesos" (Meson).

Each run gets an identity from its scheduled time. Google: "Simply identifying the cron job is not enough: we should also uniquely identify the particular launch using the start time" (SRE ch. 24, The leader), and the job names it creates in the datacenter scheduler must "include the particular scheduled launch time" (SRE ch. 24, Resolving partial failures). Slack records "each job execution as a new row in a table" and checks for an active run before starting another (Slack, 2023). "Starting with Kubernetes v1.32", each Job a CronJob creates carries batch.kubernetes.io/cronjob-scheduled-timestamp (CronJob).

Every source that addresses repeats either requires the work to tolerate them or deduplicates launches itself:

• Kubernetes. "there are certain circumstances where two Jobs might be created, or no Job might be created", so "the Jobs that you define should be idempotent" (CronJob).
• Google. After a failover, either "All operations on external systems, which we may need to continue upon re-election, must be idempotent" or their state must be looked up (SRE ch. 24).
• Google Cloud Scheduler. "Your targets should be idempotent" (About Cloud Scheduler).
• Netflix Maestro. "Maestro engine service will deduplicate the triggering requests to achieve an exact-once guarantee when scheduling workflows" (Maestro, 2022).
• Dropbox ATF. "A single task on a lambda can be executed multiple times within the ATF system. Developers should ensure that their lambda logic and correctness of task execution in clients are not affected by this" (How we designed Dropbox ATF).

L4: A Platform Shared by Many Teams#

Dropbox ATF and Meta Async schedule one-time or delayed work rather than recurring jobs ("Currently, ATF is a system for one-time task scheduling"). They are included for what they show about priority, isolation, retries, and ownership.

Peak load becomes a design problem. Google: "When people think of a "daily cron job," they commonly configure this job to run at midnight." Its fix was a ? crontab field: "the cron system is given the freedom to choose the value. Users choose this value by hashing the cron job configuration over the given time range". Load stayed "very spiky" (SRE ch. 24, Running Large Cron). Netflix reported "Slowness during peak traffic moments like 12 AM UTC" because "a lot of our workflows are run around midnight UTC" (Maestro, 2022). Meta's time shifting "moves the computing lift from peak hours to off-peak hours" (Async). Schedulers offer spreading settings:

• Temporal jitter: "a random offset between zero and the maximum jitter is added to each Action time" (Schedule).
• Cadence jitter "spreads load when many schedules fire at the same wall-clock time (for example, midnight)" (Schedules).
• systemd RandomizedDelaySec=, described under L1.
• EventBridge Scheduler flexible time windows: "Setting a flexible time window improves the reliability of your schedule by dispersing your target invocations" (flexible time windows).

Spreading midnight cron load covers how to apply them.

A single leader stops scaling. Meson "schedules around 70 thousands of workflows and half a million jobs per day", and Netflix "had to vertically scale the system to keep up and were approaching AWS instance type limits". In its replacement, Maestro, "All services in the Maestro system are stateless and can be horizontally scaled out", requests pass through distributed queues, CockroachDB holds state, and the target is "a strict SLO of less than 1 minute of scheduler introduced delay" (Maestro, 2022).

Priority and isolation become first-class.

• Dropbox ATF gives every pair of lambda and priority its own SQS queue, and "Isolation of lambdas is achieved through dedicated worker clusters, dedicated queues, and dedicated per-lambda scheduling quotas" (Dropbox ATF).
• Meta Async first "had three priorities, and inside each priority queue, it was first come, first served", until "large use cases would dominate the service and smaller use cases would have to wait in the queue". It moved to delay tolerance, "how long each request can be delayed", with a queue per use case "ordered by the deadline of the job execution", and quotas: "When use cases went above the established quota, their execution would be throttled and an alert was sent to the engineers" (Async).
• Meta FOQS. "Priority (a user-specified 32-bit integer): A lower number signifies a higher priority", and "each namespace has a rate limit (computed as enqueues per minute)" (FOQS).
• Slack. Its cron scripts run as jobs isolated on their own queue in the Job Queue (Slack, 2023), whose relay "respects the rate limits configured in Consul" (Scaling Slack’s Job Queue).
• Uber Cadence has "built-in rate limits, hot shard detection, version and per-domain scale tracking", and Uber invested in "better traffic isolation to avoid noisy neighbor issues" (Announcing Cadence 1.0).

Retries are bounded and split by error type.

• Netflix Maestro's default retry policy allows 2 retries for user errors, 10 for platform errors, and 0 for timeouts (Defaults.java).
• Dropbox ATF. Client code marks a task "successfully completed, fatally terminated, or retriable". Still, "some tasks get stuck in infinite retry loops due to occasional bugs in lambda logic. This requires manual intervention from the ATF framework owners in some cases" (Dropbox ATF).
• Slack's earlier job queue retried "a configured number of times until it eventually succeeds or moves onto a list of permanently failed jobs which is manually inspected and repaired" (Scaling Slack’s Job Queue).
• EventBridge Scheduler. MaximumRetryAttempts accepts 0 to 185 and MaximumEventAgeInSeconds 60 to 86400 (RetryPolicy). A schedule with a dead-letter queue sends it the event once retries are exhausted (dead-letter queue).
• Google Cloud Scheduler. "The default value of retryCount is zero" (REST reference), and "Values greater than 5 and negative values are not allowed" (Retry jobs).

Ownership is split between the platform and the job owners. Google: "Cron job owners can (and should!) monitor their cron jobs; for example, an owner might have the cron service expose state for its managed cron jobs, or set up independent monitoring of the effect of cron jobs" (SRE ch. 24). Dropbox: "lambda owners own all aspects of their lambdas’ operations" (Dropbox ATF). Maestro lists "Workflow definition and ownership management", and its signal lineage "helps the upstream and downstream workflow owners to see who depends on whom" (Maestro, 2022).

Published alerting stops short of per-job escalation.

• Azkaban can "Send failure emails after the first failure is detected", and for SLA rules, "If duration threshold is exceeded, then an alert email can be set or the flow or job can be auto killed" (Azkaban 3.0).
• Pinball emails a warning when a workflow has more running instances than its max_running_instances setting (schedule.py).
• Meta Async alerts engineers when a use case exceeds its quota (above).
• Slack, rolling out its Kafka-backed job queue, "enqueued heartbeat canaries for every Kafka partition every minute" and "monitored and alerted on the end-to-end flow and timing for these heartbeat canaries" (Scaling Slack’s Job Queue).
• Control-M. An SLA Management job can act when "The service is predicted to exceed the SLA deadline": notify its Alerts tool, send mail, or open a BMC Remedy ticket (SLA Management Job).
• PagerDuty's incident response guide lists "Cron failure (not impacting event & notification pipeline)" under SEV-4, whose response is a "Low-Urgency page to service team". SEV-3 gets a "High-Urgency page to service team" and SEV-5 a "JIRA ticket" (Severity Levels).

None of the scheduler accounts or scheduler documentation reviewed defines escalation tiers for individual scheduled jobs. PagerDuty's guide gives cron failures a response level, but as part of a general incident scale, not a scheduler feature.

The Same Behaviors Under Different Names#

Schedulers give the same behavior different names, and their defaults differ. We compiled these tables from each tool's documentation or source code.

Overlap: A Run Is Due While the Previous One Is Still Running#

Qualifications:

• Kubernetes Forbid can still start the skipped run late: "when the previous Job run finishes, .spec.startingDeadlineSeconds is still taken into account and may result in a new Job run."
• HCL donotstart: "If, for any reason, the job stream cannot start within four minutes, the next instance does not start at all."
• Cadence's older CronSchedule option is fixed: "always skip if previous run is active".
• Cloud Scheduler: "A scheduled start time will be delayed if the previous execution has not ended when its scheduled time occurs."
• Azkaban has a third option, Pipeline, which holds each job of the new execution until the same job of the previous execution has finished. Its default is not stated on the cited page.

Sources: Kubernetes concurrency policy; Temporal Overlap Policy; Cadence Schedules; HCL Workload Automation 10.2.8 onoverlap; Maestro RunStrategy.java and its SEQUENTIAL default in Defaults.java; Pinball overrun_policy.py and its SKIP default in schedule.py; Azkaban 3.0; systemd.timer; Cloud Scheduler REST reference.

Missed Runs: The Scheduler Was Down When a Run Was Due#

Sources: SRE ch. 24; systemd.timer; Kubernetes CronJob limitations and starting deadline; Temporal Catchup Window; Cadence Schedules; Airflow Catchup; Control-M Job Properties and Scheduling. Missed runs after downtime covers more schedulers and how to catch up safely.

Deadlines: Three Different Meanings#

The definitions, in each tool's words:

• Kubernetes: startingDeadlineSeconds "defines a deadline (in whole seconds) for starting the Job, if that Job misses its scheduled time for any reason" (starting deadline). "Once a Job reaches activeDeadlineSeconds, all of its running Pods are terminated and the Job status will become type: Failed with reason: DeadlineExceeded" (Job).
• HCL Workload Automation 10.2.8: until "specifies the latest time a job stream must be completed or the latest time a job can be launched"; under its default onuntil action, suppr, "The job or job stream and any dependent job or job stream do not run" (until). deadline "Specifies the time within which a job or job stream must complete" (deadline). maxdur "Specifies the maximum length of time a job can run", with a Kill or Continue action (maxdur).
• Control-M: Complete By "Determines when the service must complete, in days and hours, after the job is checked in", and Complete In "in hours and minutes, after the service runs (is ordered)" (SLA Management Job).
• Apache Airflow: "Deadline Alerts are new in Airflow 3.1 and should be considered experimental" (Deadline Alerts). They replace SLAs, which Airflow 3 "Deprecated and removed" (Upgrading to Airflow 3). Airflow's migration guide explains the difference: under an SLA, "If the Dag run never finishes, the SLA is never checked", while a Deadline Alert is stored when the run starts and checked periodically by the scheduler loop (Migrating from SLA to Deadline Alerts).
• Google Cloud Scheduler: attemptDeadline is "The deadline for job attempts"; for HTTP targets "the default is 3 minutes" (REST reference).
• systemd: RuntimeMaxSec= "Configures a maximum time for the service to run" but has no effect on Type=oneshot services (systemd.service).

Priority Numbers Run in Opposite Directions#

Sources: Temporal "A lower value implies higher priority, so 1 is the highest priority level" (Task Queue Priority and Fairness); Control-M "1 represents the highest priority and 5 represents the lowest" (SLA Management Job) and "Values range from AA (lowest) to 99 (highest)" (Job Properties); PagerDuty "Typically, the lower the number, the higher its impact (i.e., P1 is more important than P5)" (Incident Priority); FOQS; Announcing Cadence 1.0; Airflow "larger numbers mean higher priority" (Priority Weights); HCL, where "A priority of 0 prevents the job or job stream from launching" (priority); Kubernetes "The higher the value, the higher the priority" and "If there is no PriorityClass with globalDefault set, the priority of Pods with no priorityClassName is zero" (Pod Priority and Preemption).

The word "critical" also has several meanings across these tools:

• CronRadar: critical is a monitor health status, and Critical is a monitor severity.
• PagerDuty: critical is an event severity (Dynamic Notifications).
• Control-M: "A critical job is a job that has a higher priority to reserve resources in order to run" (Job Properties).
• HCL: the critical keyword "Specifies that the job is mission-critical and must be processed accordingly" (critical).

Escalation Lives in the Paging Tool#

Among the sources reviewed, only PagerDuty defines escalation as ordered levels with timeouts and repeats. "An escalation policy is an ordered set of rules (also called levels)." "The default escalation timeout is 30 minutes but can be adjusted." "An escalation policy can be repeated up to nine times" (Escalation Policy Basics). Low-urgency incidents "will not automatically move to higher escalation levels if no one acknowledges it" (Configurable Service Settings).

The schedulers end at a single response to a failure:

• A failure hook: systemd OnFailure=; Airflow Deadline Alert callbacks.
• An automatic pause: Temporal Pause-on-failure, where a failed or timed-out execution "causes the Schedule to automatically pause" (Schedule); Airflow max_consecutive_failed_dag_runs_per_dag, "(experimental)", which defaults to 0, "meaning that the dags are never paused out by default" (Configuration Reference).
• A dead-letter queue: EventBridge Scheduler.
• An action on lateness: Control-M SLA actions; HCL onlate, whose "supported action is kill" (onlate); Azkaban SLA emails and kills.

Where the Sources Disagree#

Skip a run, or risk running it twice? Google: "In general, we favor skipping launches rather than risking double launches, as much as the infrastructure allows", because "recovering from a skipped launch is more tenable than recovering from a double launch", so "we prefer to "fail closed" to avoid systemically creating bad state" (SRE ch. 24). Maestro's time-based scheduling "provides an at-least-once scheduling guarantee" and deduplicates in the engine (Maestro, 2022); Dropbox ATF "guarantees that a task is executed at least once after being scheduled" (Dropbox ATF); Cloud Scheduler "is designed to provide "at least once" delivery" (About Cloud Scheduler); EventBridge Scheduler "provides at-least-once event delivery to targets" (What is EventBridge Scheduler?). FOQS lets each use case choose: "FOQS supports both at least once and at most once delivery semantics" (FOQS). Kubernetes creates a Job "approximately once per execution time of its schedule", because "two Jobs might be created, or no Job might be created". The same section says that with startingDeadlineSeconds unset or large and concurrencyPolicy set to Allow, "the Jobs will always run at least once", and that a CronJob with "more than 100 missed schedules" does not start the Job (CronJob).

The defaults conflict. Our inference: a job moved between schedulers changes behavior unless its settings are explicit. For overlap, Kubernetes and HCL run both by default; Temporal, Cadence, Pinball, and systemd skip; Maestro and Cloud Scheduler queue. For missed runs, cron and systemd make up nothing, Airflow makes up the latest interval, and Temporal and Cadence make up runs up to a year late.

Consistency or availability? Google's leader waits for a Paxos quorum: "the actual cron job launch does not proceed until it receives confirmation that the Paxos quorum has received the launch notification" (SRE ch. 24, The Leader). Slack's 2017 job queue redesign chose "A bias towards availability over consistency", which "does create a small risk of lost jobs in the event that a broker host dies unexpectedly before replicating" (Scaling Slack’s Job Queue).

Is one leader enough? Slack chose one on purpose: "we decided that synchronizing the nodes would be more of a headache than a help", because "the pods can switch leaders very quickly, making downtime unlikely in practice" (Slack, 2023). Netflix outgrew Meson's single leader at about 70,000 workflows a day. Our reading of the difference: Google's and Slack's leaders only launch work, while Meson's scheduler "manages the launch, flow control and runtime of the various workflows" (Meson), so each job asked more of it.