# Distributed Cron: Running Scheduled Jobs Once Across Many Servers By Dan Ochoa · Published 2026-10-25 · Updated 2026-10-25 Canonical: https://cronradar.com/blog/distributed-cron > Run each scheduled job once across many servers and keep it running when one dies: one scheduler, leader election, ShedLock, clustering, queues, managed cron. **To run a scheduled job once across many servers, make exactly one thing decide that each occurrence is due: one scheduler process your orchestrator restarts, a leader elected through a lease, a lock or claim every node competes for, a clustered scheduler with shared storage, or a managed scheduler outside your app.** Then make the job safe to run twice, because each of these can still run an occurrence twice, or not at all, in some failure. A crontab copied to every server runs the job once per server; a crontab on one server stops with that server. _Figure: Copied to three replicas, the 0 2 * * * schedule runs the 02:00 job three times. When each replica must first hold the lease as leader, or win the lock, only one run starts and the two extra arrows are gone, though an unfenced lease or an expired lock can still let a second run through._ Behavior is checked against client-go v0.37.1 (Kubernetes 1.37), ShedLock 7.10.1, Laravel 13.38, Quartz.NET 3.22, Hangfire 1.8.25, db-scheduler 16.12.0, and RedBeat 2.4.2 on October 24, 2026. ## Running Once and Surviving a Node Failure Pull in Opposite Directions You want **at most one run per occurrence** and **a run even when the usual node is gone**. From outside, a dead node looks like one that is paused or cut off from the network. Keeping the schedule alive means another node takes over after some silence, and if the first was only paused, both run. Guaranteeing one run means nobody takes over until the first node is known to be gone, and occurrences are missed meanwhile. Every mechanism below settles this with a timeout, such as a lease duration or a lock's maximum hold: short ones risk duplicates, long ones risk gaps. Which risk a job can absorb depends on its class; the [CronRadar Scheduled Job Methodology's architecture rules](https://cronradar.com/methodology) say when a job should skip when in doubt and when it should run at least once with an idempotency key. A lock released when the job finishes stops two runs _at once_, not two runs of _one occurrence_: a node whose timer fires just after another node's short run ended takes the free lock and runs it again. [Preventing overlapping cron jobs](https://cronradar.com/blog/prevent-overlapping-cron-jobs) covers lock mechanics and fencing tokens; this post covers the architecture around them. ## Option 1: One Scheduler Process, Restarted by the Orchestrator Run the schedule in exactly one process, such as one scheduler replica beside your web replicas, and let the orchestrator restart it. On Kubernetes, a Deployment with `replicas: 1` is not always one copy: - **Rolling updates overlap.** For one replica, the default 25% `maxSurge` rounds up to one and the 25% `maxUnavailable` rounds down to zero, so the new pod starts first. `strategy: Recreate` covers upgrades only; a deleted pod's "replacement will be created immediately (even if the old Pod is still in a Terminating state)" ([Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#recreate-deployment)). - **Node failures take minutes, then may overlap.** Pods stay bound to an unreachable node for 5 minutes by default (`tolerationSeconds=300`) before eviction ([taints and tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)), and the replacement may start while the original still runs on a node that only lost contact with the control plane. A one-replica StatefulSet guarantees "at most one Pod with a given identity running in a cluster" by not replacing a pod on an unreachable node until the Node object is deleted or its kubelet returns ([StatefulSet pods](https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/)): no duplicate, but a gap until something acts. An occurrence due during any gap may never run, because an in-process scheduler may compute its next run from when it starts; robfig/cron v3 does ([cron.go](https://github.com/robfig/cron/blob/v3.0.1/cron.go#L242-L247)). ## Option 2: Leader Election, Where Every Replica Waits and One Runs the Schedule Start the scheduler only in the replica holding a lease. Kubernetes elects `kube-controller-manager` and `kube-scheduler` leaders with Leases, and its docs suggest workloads define their own ([Leases](https://kubernetes.io/docs/concepts/architecture/leases/)). In Go, client-go's `leaderelection` package handles it: ```go // main.go: client-go v0.37.1, github.com/robfig/cron/v3 v3.0.1 package main import ( "context" "log" "os" "os/signal" "sync/atomic" "syscall" "time" "github.com/robfig/cron/v3" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" "k8s.io/client-go/tools/leaderelection" "k8s.io/client-go/tools/leaderelection/resourcelock" ) func main() { cfg, err := rest.InClusterConfig() if err != nil { log.Fatal(err) } podName, err := os.Hostname() // the pod name: unique per replica if err != nil { log.Fatal(err) } ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() var leading atomic.Bool stopped := make(chan struct{}) leaderelection.RunOrDie(ctx, leaderelection.LeaderElectionConfig{ Lock: &resourcelock.LeaseLock{ LeaseMeta: metav1.ObjectMeta{Name: "billing-scheduler", Namespace: "billing"}, Client: kubernetes.NewForConfigOrDie(cfg).CoordinationV1(), LockConfig: resourcelock.ResourceLockConfig{Identity: podName}, }, LeaseDuration: 15 * time.Second, RenewDeadline: 10 * time.Second, RetryPeriod: 2 * time.Second, Callbacks: leaderelection.LeaderCallbacks{ OnStartedLeading: func(leaderCtx context.Context) { leading.Store(true) defer close(stopped) c := cron.New(cron.WithLocation(time.UTC)) if _, err := c.AddFunc("0 2 * * *", func() { runInvoices(leaderCtx) }); err != nil { log.Fatal(err) } c.Start() <-leaderCtx.Done() // lease lost, or shutting down <-c.Stop().Done() // start nothing new; wait for a run in progress }, OnStoppedLeading: func() { log.Print("not leading") }, }, }) // RunOrDie returns on SIGTERM or a lost lease. Exiting lets Kubernetes // restart the pod, which rejoins the election as a standby. if leading.Load() { <-stopped } } ``` The service account needs `get`, `create`, and `update` on `leases` in `coordination.k8s.io`, the calls `LeaseLock` makes ([leaselock.go](https://github.com/kubernetes/client-go/blob/v0.37.1/tools/leaderelection/resourcelock/leaselock.go)). From the [package source](https://github.com/kubernetes/client-go/blob/v0.37.1/tools/leaderelection/leaderelection.go): - **A standby waits a full `LeaseDuration`** without seeing the lease record change before taking over. Core clients use 15 seconds, with a 10-second `RenewDeadline` and a 2-second `RetryPeriod`. - **The old leader gives up after `RenewDeadline`** and cancels `leaderCtx`. A running job stops only if it watches that context, and the five seconds between the two timeouts is the whole margin. - **There is no fencing.** The package "does not guarantee that only one client is acting as a leader." A leader frozen by a garbage-collection pause or suspended VM can wake mid-job after its successor started. - **`ReleaseOnCancel`** speeds up handover at shutdown but is safe only if the guarded code has finished before the context is canceled, so the example leaves it off. - **A handover can skip a run**, because the new leader's cron schedules from the moment it starts. [Golang cron jobs](https://cronradar.com/blog/go-cron-jobs) covers robfig/cron and gocron themselves: specs, time zones, panics, and overlap. etcd and Consul offer the same primitive. etcd's Go client holds a `Session` on a lease (60-second TTL by default), and `Election.Campaign` blocks until the caller leads ([session.go](https://github.com/etcd-io/etcd/blob/v3.7.3/client/v3/concurrency/session.go), [election.go](https://github.com/etcd-io/etcd/blob/v3.7.3/client/v3/concurrency/election.go)). Consul sessions add a lock-delay, 15 seconds by default, so a possibly still-live leader can notice it lost the lock, which the docs call "not a bulletproof method" ([sessions](https://developer.hashicorp.com/consul/docs/automate/session)). ## Option 3: Every Node Schedules, and a Shared Lock Picks One Keep the scheduler running everywhere and make each run first win a lock or claim in a shared store, so any live node can run the next occurrence. **ShedLock** 7.10.1 (Java 17+, tested with Spring 7.0 and Spring Boot 4) guards Spring `@Scheduled` methods. When the lock is held, other nodes "are not blocked, but are simply skipped" ([README](https://github.com/lukas-krecan/ShedLock/blob/shedlock-parent-7.10.1/README.md)). With the README's `shedlock` table created: ```java @Configuration @EnableScheduling @EnableSchedulerLock(defaultLockAtMostFor = "30m") class SchedulingConfig { @Bean LockProvider lockProvider(DataSource dataSource) { return new JdbcTemplateLockProvider(JdbcTemplateLockProvider.Configuration.builder() .withJdbcTemplate(new JdbcTemplate(dataSource)) .usingDbTime() // lock times from the database clock, not each node's .build()); } } @Component class InvoiceJobs { private final InvoiceService invoices; InvoiceJobs(InvoiceService invoices) { this.invoices = invoices; } @Scheduled(cron = "0 0 2 * * *", zone = "UTC") // Spring cron: seconds first @SchedulerLock(name = "nightly-invoices", lockAtMostFor = "2h", lockAtLeastFor = "5m") public void createInvoices() { LockAssert.assertLocked(); invoices.createDue(); } } ``` - **`lockAtMostFor`** releases a lock that a dead node left behind. Set it "much longer than normal execution time," because a run that outlasts it can overlap another node's. - **`lockAtLeastFor`** holds the lock after the method returns, so a node whose clock is a little behind doesn't rerun a short job's occurrence. The README warns that a clock difference beyond it can still cause a second run. - **`usingDbTime()`** takes lock times from the database server, which the README strongly recommends. ShedLock is "not a distributed scheduler": an occurrence locked by a node that then died doesn't run, and nothing catches it up. [Spring @Scheduled cron jobs](https://cronradar.com/blog/spring-scheduled-monitoring) covers the scheduler's thread pool and other ways to run a task once across replicas. **Laravel 13's `onOneServer()`** does this for `schedule:run` on every server, provided all servers share a `database`, `memcached`, `dynamodb`, or `redis` default cache ([scheduling](https://laravel.com/docs/13.x/scheduling#running-tasks-on-one-server)): ```php use Illuminate\Support\Facades\Schedule; // routes/console.php, deployed to every server Schedule::command('invoices:create') ->dailyAt('02:00') ->timezone('UTC') ->onOneServer(); ``` It's a claim on the occurrence, not a lock. The key is the task's mutex name plus the hour and minute when `schedule:run` started, it lasts 3,600 seconds, and nothing releases it when the task ends ([CacheSchedulingMutex.php](https://github.com/laravel/framework/blob/v13.38.0/src/Illuminate/Console/Scheduling/CacheSchedulingMutex.php#L43-L56)). A server that crashes mid-task takes that occurrence with it, and closures need `->name()` to qualify. For plain crontabs, a row keyed on the job and its date makes the same kind of claim; [why a cron job runs twice](https://cronradar.com/blog/cron-job-running-twice) shows one in PostgreSQL. ## Option 4: A Clustered Scheduler With Shared Storage Cluster-aware schedulers keep schedules and run state in shared storage and handle node choice and recovery themselves: - **Quartz.NET 3.22** with a clustered `AdoJobStore`: "Each firing runs on exactly one node." Peers write off a node whose last check-in is older than `clusterCheckinInterval` plus `clusterCheckinMisfireThreshold`, 15 seconds on the defaults, and re-execute its jobs flagged "requests recovery." Clocks must agree within a second, and a node at 100% CPU can miss check-ins and have its jobs recovered "by re-running" ([configuration reference](https://www.quartz-scheduler.net/documentation/quartz-3.x/configuration/reference.html#clustering)). Java Quartz 2.5 clusters the same way ([JDBC-JobStore clustering](https://www.quartz-scheduler.org/documentation/quartz-2.5.x/configuration/ConfigJDBCJobStoreClustering.html)). - **Hangfire 1.8:** every server's recurring-job scheduler takes the storage-wide `recurring-jobs:lock` before enqueuing due jobs, using storage time where available. Its source says running several "only adds support for fail-over, and does not increase the performance" ([RecurringJobScheduler.cs](https://github.com/HangfireIO/Hangfire/blob/v1.8.25/src/Hangfire.Core/Server/RecurringJobScheduler.cs#L51-L54)), and the docs say "there's no reliable way to prevent multiple executions of the same background job" ([throttling](https://docs.hangfire.io/en/latest/background-processing/throttling.html)). [Hangfire vs Quartz.NET vs BackgroundService](https://cronradar.com/blog/hangfire-vs-quartz-vs-backgroundservice) compares the two. - **db-scheduler 16.12** for Java uses one table and row locking so "one and only one scheduler-instance gets to pick and run a task-execution." An execution that misses 6 heartbeats, 5 minutes apart, is declared dead, and a recurring one is rescheduled to now. There are "no guarantees that all instants in a schedule for a `RecurringTask` will be executed" ([README](https://github.com/kagkarlsson/db-scheduler/blob/v16.12.0/README.md)). - **Celery beat** must run as a single instance, "otherwise you'd end up with duplicate tasks" ([periodic tasks](https://docs.celeryq.dev/en/v5.6.3/userguide/periodic-tasks.html)). RedBeat 2.4.2 keeps the schedule in Redis behind a lock that the active beat refreshes every tick; a second beat waits as a standby until the lock expires, 1,500 seconds by default ([config](https://github.com/sibson/redbeat/blob/v2.4.2/docs/config.rst), [schedulers.py](https://github.com/sibson/redbeat/blob/v2.4.2/redbeat/schedulers.py#L592-L595)). With `lock()`, Symfony Scheduler also keeps standby workers that take over if the active one stops; [Symfony Scheduler](https://cronradar.com/blog/symfony-scheduler) covers it. ## Option 5: The Scheduler Only Enqueues, and Workers Do the Work Let a scheduler, chosen by any option above, publish one message per occurrence for workers on any node, as Celery beat, Hangfire's recurring jobs, and Laravel's `Schedule::job()` do. The scheduler stays small and cheap to fail over, and the queue handles dead workers, at least once: an SQS message not deleted before its visibility timeout, 30 seconds by default, "becomes visible again in the queue and can be retrieved by another consumer" ([visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html)). Celery defaults the other way, acknowledging a task just before running it, so a worker crash loses that run unless you set `task_acks_late` ([configuration](https://docs.celeryq.dev/en/v5.6.3/userguide/configuration.html#task-acks-late)). Queues can also drop duplicate enqueues. An SQS FIFO queue acknowledges but doesn't deliver a message whose `MessageDeduplicationId` it accepted in the last 5 minutes ([deduplication](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagededuplicationid-property.html)), so an ID built from the job name and scheduled time collapses two schedulers' messages for one occurrence into one. ## Option 6: A Managed Scheduler Outside Your App Let a service you don't operate start a container or call your app. With a load-balanced HTTP target, whichever instance receives the request runs the job. Checked October 24, 2026: - **Kubernetes CronJob:** the controller runs in `kube-controller-manager`, leader-elected in HA control planes, and creates a Job "approximately once per execution time"; "two Jobs might be created, or no Job might be created" ([CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#job-creation)). [Kubernetes CronJob monitoring](https://cronradar.com/blog/kubernetes-cronjob-monitoring) covers missed schedules and deadlines. - **Google Cloud Scheduler** is "designed to provide 'at least once' delivery," and targets "should be idempotent." Its `X-CloudScheduler-ScheduleTime` header "remains constant across retry attempts" ([overview](https://docs.cloud.google.com/scheduler/docs/overview)). - **Amazon EventBridge Scheduler** "provides at-least-once event delivery to targets," with configurable retries ([overview](https://docs.aws.amazon.com/scheduler/latest/UserGuide/what-is-scheduler.html)). You take on a platform dependency, an endpoint that must authenticate its caller, and a target that must be up when the schedule fires. ## How the Options Compare | Option | One run per occurrence | When a node dies | Takeover timeout | Cost | | --------------------- | --------------------------------------------------------- | ------------------------------------------------------- | ------------------------------------------------ | ------------------------------------ | | One scheduler process | Yes, except during rolling updates and some node failures | Restarted; runs due meanwhile can be skipped | Pod eviction, 5 minutes by default | Lowest | | Leader election | Usually; no fencing | A standby takes over; a run due mid-handover is skipped | `LeaseDuration`, 15 s in core clients | Election code and RBAC | | Shared lock or claim | Only with a minimum hold or a per-occurrence key | Any node runs the next one; the cut-off run is lost | `lockAtMostFor`; Laravel's one-hour claim | A table or cache you already run | | Clustered scheduler | Yes per firing; recovery can rerun a cut-off job | Another node fires next and may recover the run | Quartz 15 s; db-scheduler 30 min; RedBeat 25 min | Schema or Redis, synchronized clocks | | Enqueue plus workers | As the scheduler decides; delivery is at least once | The dead worker's message is redelivered | Visibility timeout or late acknowledgement | A broker | | Managed scheduler | Approximately or at least once | Unaffected by your nodes; the target must be up | The provider's retry policy | A platform dependency | ## Clocks Drift and Deliveries Repeat, So Make the Job Idempotent Cron and every node-local scheduler fire by each host's own clock. ShedLock "assumes that clocks on the nodes are synchronized," Quartz wants them within a second, and Hangfire reads storage time where it can; client-go tolerates skew but not "arbitrary clock skew rate." Run NTP on every node and prefer database time where a library offers it. Every option also has a documented path to a second run, from unfenced leases and expired locks to Quartz recovery, at-least-once queues, and "approximately once." Key the work on the job name plus the occurrence's **scheduled** time, which a retry shares and a start time doesn't. Kubernetes 1.32+ annotates each CronJob's Job with `batch.kubernetes.io/cronjob-scheduled-timestamp`, Cloud Scheduler sends `X-CloudScheduler-ScheduleTime`, and a recovering Quartz job is scheduled for the original firing's time ([IJobExecutionContext.cs](https://github.com/quartznet/quartznet/blob/v3.22.7/src/Quartz/IJobExecutionContext.cs#L38-L48)); in a cluster that is when the original actually fired ([JobStoreSupport.cs](https://github.com/quartznet/quartznet/blob/v3.22.7/src/Quartz/Impl/AdoJobStore/JobStoreSupport.cs#L5766-L5990)), which can trail the due time, so key on the period it falls in, such as the date of a nightly run. [Idempotent scheduled jobs](https://cronradar.com/blog/idempotent-scheduled-jobs) covers claims, upserts, and side effects. ## Monitor the Run, Not the Nodes 1. **Report from the node that ran the job.** Put the check-in after winning the lease, lock, or claim, not in every node's scheduler loop. A node that skipped must send nothing, or its check-in would say the occurrence ran when the winner crashed or nobody ran it. 2. **Use one monitor per job, not per node.** Every node reports to the same key; a per-node key leaves each standby's monitor overdue. A schedule-based heartbeat then catches failover that didn't happen: a StatefulSet pod stranded on an unreachable node, a ShedLock lock held until `lockAtMostFor`, a RedBeat standby waiting out its lock, a run skipped during a leader handover. None of these raises an error, which is the case a [dead man's switch](https://cronradar.com/blog/cron-dead-mans-switch) exists for. There is no CronRadar SDK for Go, Java, or Kubernetes CronJobs and no auto-discovery for them, and the CronRadar Laravel and Celery packages aren't installable today, so each job reports with its own HTTP check-in. For the program above: ```go // invoices.go: same package as main.go package main import ( "context" "fmt" "log" "net/http" "net/url" "os" "time" ) var checkins = &http.Client{Timeout: 5 * time.Second} // report sends one check-in and ignores every error, so monitoring can't fail the job. func report(event string, params url.Values) { params.Set("schedule", "0 2 * * *") params.Set("gracePeriod", "1800") endpoint := "https://cron.life/api/v1/ping/nightly-invoices/" + event + "?" + params.Encode() req, err := http.NewRequest(http.MethodPost, endpoint, nil) if err != nil { return } req.SetBasicAuth(os.Getenv("CRONRADAR_API_KEY"), "") if resp, err := checkins.Do(req); err == nil { resp.Body.Close() } } // runInvoices runs only on the leader, so only the node doing the work reports. func runInvoices(leaderCtx context.Context) { host, _ := os.Hostname() runID := fmt.Sprintf("nightly-invoices-%s-%d", host, time.Now().Unix()) ctx, cancel := context.WithTimeout(leaderCtx, 90*time.Minute) // a hang becomes a fail defer cancel() report("start", url.Values{"run_id": {runID}}) if err := createInvoices(ctx); err != nil { log.Printf("nightly-invoices on %s: %v", host, err) report("fail", url.Values{"run_id": {runID}, "message": {"createInvoices failed on " + host}}) return } report("complete", url.Values{"run_id": {runID}}) } // createInvoices does the billing work and must return when ctx is canceled. func createInvoices(ctx context.Context) error { return ctx.Err() } ``` The run ID includes the host, so two nodes never share one, and the failure message names the node but leaves out error text, which can contain customer data. A `fail` alerts immediately. A missing run goes `critical` once it is past the grace period and at least two minutes late, so set `gracePeriod` (seconds, applied only when the first check-in creates the monitor) to cover the run time plus the slowest takeover you'd accept. Schedules are read in UTC. Every check-in, `start` included, moves the expected time to the next occurrence, so a `start` with no finish goes `critical` as stuck only once that time plus grace passes: the next night here, which is why the job has a deadline. CronRadar doesn't flag duplicates; idempotency has to cover those. ## Frequently Asked Questions ### How do I run a cron job on only one server? Schedule it in one place: a single scheduler replica, a Kubernetes CronJob, or a managed cloud scheduler. If every server keeps the schedule so any can take over, make each run win a shared lock or claim first, such as ShedLock's `@SchedulerLock` in Spring or `onOneServer()` in Laravel. Either way, make the job safe to run twice. ### What happens to cron jobs when I scale to multiple servers? Each server's cron daemon or in-process scheduler runs every job on its own, so three servers run a nightly job three times. Neither cron nor most in-process schedulers coordinate between hosts. In NestJS, every replica fires each `@Cron()` method, and the documented fix is `@OnOneInstance()` from `@nestjs/locks`; see [NestJS scheduled tasks](https://cronradar.com/blog/nestjs-scheduled-tasks). ### Is leader election needed for scheduled jobs? Usually not. A per-run lock, a clustered scheduler, or a managed scheduler gives one run per occurrence with less code. Leader election suits a replica that owns many schedules or in-memory state, and its jobs must stop when the lease is lost, because client-go's election has no fencing. ### What is ShedLock? ShedLock is a Java library for Spring, Micronaut, or plain Java that runs a scheduled method on one node at a time by taking a lock in a shared store such as a JDBC database, Redis, or MongoDB. Nodes that find the lock held skip the run. It isn't a scheduler, so lost runs aren't caught up. ### How do I make scheduled jobs highly available? Remove the single node that decides: use a clustered scheduler, a leased leader, a per-run lock on every node, or a managed scheduler. Make jobs idempotent, because failover can repeat a run, and monitor each one with a schedule-based heartbeat, because failover can also fail without an error.