Scheduled Jobs During Deploys: Lost, Killed, and Doubled Runs
Contents
A deployment breaks scheduled jobs in three ways: a run is lost because no scheduler was running at its due time, a run is killed because the old process stopped before it finished, or a run is doubled because old and new schedulers overlapped. Which one you get depends on how long your shutdown waits, whether interrupted work is redelivered, and whether two versions can schedule at once. Each framework has a setting for each, and the defaults rarely match a real deploy.
This guide covers Hangfire 1.8, Celery 5.5 and later, Kubernetes, and Laravel 12 and 13, checked November 16, 2026. For what each scheduler does about runs missed during longer outages, see missed runs after downtime.
Three Ways a Deploy Breaks a Run#
| What happened | When | Typical cause | What you see later |
|---|---|---|---|
| Lost | The due time fell between the old scheduler stopping and the new one starting | Recreate deploys, maintenance mode, slow startup | Nothing; no run, no error |
| Killed | A run was in progress when its process was stopped | Shutdown timeout shorter than the job | A partial run, maybe retried, maybe not |
| Doubled | Old and new versions both scheduled the same due time | Rolling deploys of in-process schedulers | Two runs, duplicate emails or rows |
Lost and doubled runs depend on the scheduler; killed runs depend on the worker. A system can have all three: a Celery deploy can kill a running task, briefly run two beat processes, and lose the run that was due while the new beat started.
Hangfire: Shutdown Timeout and At-Least-Once Processing#
When a Hangfire server stops, it signals its workers and waits. BackgroundJobServerOptions.ShutdownTimeout is 15 seconds by default (BackgroundProcessingServer.cs). A job that observes cancellation can stop cleanly within that window; one that doesn't is cut off when the process exits.
An interrupted Hangfire job isn't lost. Hangfire processes jobs at least once, so a job that was being processed when its server stopped is picked up again later by another server. That turns "killed" into "run twice, the first time partially," which is safe only if the job is idempotent.
Make the job observe shutdown by taking a CancellationToken, which Hangfire supplies, and size the timeouts so they nest:
using Hangfire;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHangfire(config => config
.UseSqlServerStorage(builder.Configuration.GetConnectionString("Hangfire")));
builder.Services.AddHangfireServer(options =>
{
// Wait up to 25 s for running jobs; the host and the platform must wait longer.
options.ShutdownTimeout = TimeSpan.FromSeconds(25);
});
// The generic host stops hosted services, including the Hangfire server, within this window.
builder.Services.Configure<HostOptions>(options => options.ShutdownTimeout = TimeSpan.FromSeconds(28));
var app = builder.Build();
app.Services.GetRequiredService<IRecurringJobManager>()
.AddOrUpdate<ExportJobs>("nightly-export", jobs => jobs.Export(CancellationToken.None), "0 2 * * *");
app.Run();
public class ExportJobs
{
public async Task Export(CancellationToken cancellationToken)
{
foreach (var batch in Enumerable.Range(1, 100))
{
cancellationToken.ThrowIfCancellationRequested(); // Stop between batches on shutdown.
await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
}
}
}Hangfire replaces the CancellationToken.None placeholder with a real token at run time. The order that matters is Hangfire's ShutdownTimeout < the host's HostOptions.ShutdownTimeout < the platform's kill timeout, such as Kubernetes' terminationGracePeriodSeconds (default 30 seconds). If the platform kills first, nothing inside the process gets to finish.
Hangfire doesn't double recurring runs during an overlap. The recurring job scheduler in each server coordinates through a distributed lock, so old and new servers sharing one storage create each occurrence once. A due time that passes while no server is running is handled by MisfireHandling when one returns; the default creates one catch-up run. The deploy risk Hangfire does have is code: an old server that can't load a newly renamed job type records errors against it, covered in Hangfire jobs missing after a deploy.
Celery: Warm Shutdown, acks_late, and One Beat#
A Celery worker that receives SIGTERM does a warm shutdown: "The worker will finish all currently executing tasks before it actually terminates" (workers guide). Celery itself doesn't impose a limit, but your platform does. When its grace period ends, it sends SIGKILL, and the tasks still running die with the process.
Whether a killed task runs again depends on when its message was acknowledged:
- Default (early acknowledgment): the message is acknowledged when the worker receives it, so a task killed mid-run is gone. The workers guide says running tasks are lost on a cold shutdown "unless the tasks have the
acks_lateoption set." acks_late=True: the message is acknowledged after the task finishes, so a killed task's message goes back to the broker and runs again, from the start. With the Redis broker, redelivery waits for the visibility timeout, one hour by default, so the retry can arrive long after the deploy.
Celery 5.5 added a soft shutdown: with worker_soft_shutdown_timeout set to a positive number of seconds, the worker gets a time-limited warm period before a cold shutdown. It's disabled by default. Set the platform's grace period above the longest task you're willing to wait for, and use acks_late only for tasks that are safe to run twice (Celery task retries).
Beat is the doubling risk. Celery beat must be a single process, and a rolling deploy of a beat container briefly runs two, both sending the tasks due in that window. Deploy beat with a stop-then-start strategy (Recreate in Kubernetes), and accept that a task due during the gap is sent when the new beat starts, as covered in Celery beat not running.
Kubernetes: Grace Periods, Rolling Updates, and CronJob Pods#
When Kubernetes stops a pod, it runs any preStop hook, sends SIGTERM to the containers, and sends SIGKILL when terminationGracePeriodSeconds (30 seconds by default) runs out. The preStop hook's time counts against the same grace period. So a pod's shutdown budget is one number shared by every step, and every timeout inside the application must fit in it.
Two Kubernetes behaviors cause most deploy-time scheduling problems:
- Rolling updates overlap versions. A Deployment's default
RollingUpdatestrategy starts new pods before old ones are gone (maxSurge25%). Any scheduler running inside the app, such as node-cron, aBackgroundServiceloop, APScheduler, or Celery beat, runs in both versions during the overlap and can fire twice. Run in-process schedulers in a separate single-replica Deployment withstrategy: Recreate, or guard the job with a distributed lock. - CronJob pods are separate from your deploy. Updating a CronJob's template affects Jobs created after the change; a Job already running keeps its pod and old image. But a cluster or node upgrade that drains nodes evicts running CronJob pods. Each evicted pod counts as a failure toward the Job's
backoffLimitunless apodFailurePolicyignores pods with theDisruptionTargetcondition.
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-export
spec:
schedule: '0 2 * * *'
timeZone: 'Etc/UTC'
concurrencyPolicy: Forbid
jobTemplate:
spec:
backoffLimit: 2
podFailurePolicy:
rules:
- action: Ignore # Retry evicted pods without using up backoffLimit.
onPodConditions:
- type: DisruptionTarget
template:
spec:
restartPolicy: Never
terminationGracePeriodSeconds: 120
containers:
- name: export
image: registry.example.com/export:1.4.2
args: ['--run', 'nightly']podFailurePolicy requires restartPolicy: Never. With concurrencyPolicy: Forbid, a run whose previous Job is still retrying is skipped, which is the right trade for most exports; Kubernetes CronJob monitoring covers the other fields.
Laravel: Maintenance Mode and schedule:interrupt#
Laravel's own deploy steps can stop the schedule:
- Maintenance mode. "Your application's scheduled tasks will not run when the application is in maintenance mode" (task scheduling docs). A deploy script that runs
php artisan downbefore migrations andphp artisan upat the end loses every task due in between, with no log entry. Tasks that must run anyway need->evenInMaintenanceMode(). - Sub-minute tasks. When the schedule has sub-minute tasks,
schedule:runkeeps running until the end of the minute, using the previously deployed code. Addphp artisan schedule:interruptto the deploy script after the new code is in place so in-progressschedule:runinvocations stop. - Stale overlap locks.
withoutOverlapping()uses a cache lock that expires after 24 hours by default. A run killed during a deploy can leave its lock behind, and the task is then skipped until the lock expires.php artisan schedule:clear-cacheclears scheduler locks. - Two servers during a cutover. If old and new servers both run
schedule:runfor a while, each runs every task unless the task uses->onOneServer(), which needs a shared cache (database,memcached,dynamodb, orredis) across all servers.
use Illuminate\Support\Facades\Schedule;
Schedule::command('billing:charge-renewals')
->dailyAt('02:00')
->onOneServer()
->withoutOverlapping(120) // Lock expires after 2 hours, not 24, if a run is killed.
->evenInMaintenanceMode();Scheduled tasks that dispatch queued jobs also depend on queue workers, which deploy scripts usually restart with php artisan queue:restart so workers finish their current job and exit. Laravel schedule not running covers the scheduler's other failure modes.
Make Deploys Safe for Scheduled Work#
The fixes above reduce to a short list that applies to any stack:
- Nest the timeouts. Job cancellation check interval < framework shutdown timeout < host shutdown timeout < platform kill timeout.
- Make killed runs safe to repeat. Anything that redelivers interrupted work (Hangfire,
acks_late, Job retries) runs it again from the start. - Run one scheduler. Keep in-process schedulers out of rolling, multi-replica deployments, or coordinate them with a lock.
- Know your gap. If the scheduler is down during a deploy, decide whether a due run should be caught up, and check what your scheduler actually does.
- Avoid deploying across critical due times, or move those jobs off the deploy window.
See Deploy Gaps With Start and Finish Check-Ins#
None of these failures appears in the deploy log. A lost run leaves no trace; a killed run looks like a run that stopped logging; a doubled run looks normal twice. Start and finish check-ins from each job make each case distinct, because an outside service compares what arrived with what the schedule expected:
| What happened during the deploy | Check-ins received | What CronRadar shows |
|---|---|---|
| Lost | None | warning after the expected time, then critical and an alert after the grace period |
| Killed, not retried | start with no finish |
Flagged as stuck |
| Killed, then redelivered | start, later start and complete |
Both attempts recorded, with the delay between them visible |
| Failed on the new code | start, fail |
critical with an immediate alert |
| Doubled | Two starts and two completions | Both runs recorded; no alert for an extra run |
Alerts go to email, Slack, webhook, Discord, or PagerDuty. Set each monitor's grace period longer than a normal deploy, so a run delayed by a few minutes of restart doesn't alert, but a run lost to a long maintenance window does. Give concurrent runs distinct run_ids and reuse a run's run_id across retries, so a redelivered run pairs with its own start.
How each stack reports:
- Hangfire:
CronRadar.Hangfirewithbuilder.Services.AddCronRadarHangfire();discovers recurring jobs and reports start, completion, and failure through a server filter. It reports a failure only after Hangfire's retries finish. - Celery:
cronradar-celerywithsetup_cronradar(app)discoversapp.conf.beat_scheduleentries and reports from task signals. - Laravel:
composer require cronradar/laraveldiscovers every scheduled task. - Kubernetes CronJobs and other schedulers: there's no Kubernetes integration; report over HTTP from the container's command, with a monitor key and schedule per job, or wrap the job with a base SDK (.NET, Node, Python, PHP).
A shell wrapper for a CronJob container, following the full lifecycle pattern, keeps the job's own exit code so Kubernetes still sees failures:
#!/bin/sh
run_id="nightly-export-$(date +%s)-$$"
curl --fail-with-body --silent --show-error --max-time 5 --request POST \
--user "$CRONRADAR_API_KEY:" \
--get "https://cron.life/api/v1/ping/nightly-export/start" \
--data-urlencode "schedule=0 2 * * *" \
--data-urlencode "run_id=$run_id" || true
if /app/export --run nightly; then
curl --fail-with-body --silent --show-error --max-time 5 --request POST \
--user "$CRONRADAR_API_KEY:" \
--get "https://cron.life/api/v1/ping/nightly-export/complete" \
--data-urlencode "run_id=$run_id" || true
else
exit_code=$?
curl --fail-with-body --silent --show-error --max-time 5 --request POST \
--user "$CRONRADAR_API_KEY:" \
--get "https://cron.life/api/v1/ping/nightly-export/fail" \
--data-urlencode "run_id=$run_id" \
--data-urlencode "message=Export exited with code $exit_code" || true
exit "$exit_code"
fiThe --max-time 5 and || true keep a monitoring outage from failing or stalling the job. CronRadar evaluates schedules in the application's time zone, set in its settings (UTC if unset), so match it to the CronJob's timeZone. Use a separate CronRadar application and key per environment, so a staging deploy's runs never count as production's.
Frequently Asked Questions#
What happens to a running Hangfire job during a deploy?#
The server signals cancellation and waits up to ShutdownTimeout, 15 seconds by default. A job that observes its CancellationToken can stop cleanly; one that doesn't is cut off when the process exits. Hangfire processes jobs at least once, so the interrupted job is picked up again later and runs from the start.
Does Celery lose tasks when a worker is restarted?#
On SIGTERM a worker does a warm shutdown and finishes its running tasks. If the platform kills it before they finish, tasks with default early acknowledgment are lost. With acks_late=True the message is redelivered and the task runs again, after the visibility timeout on Redis, which is one hour by default.
Why do my scheduled jobs run twice after a deploy?#
Old and new versions of an in-process scheduler overlapped. Rolling updates run both for a while, and node-cron, BackgroundService loops, APScheduler, and Celery beat each fire every due job in every running copy. Run the scheduler as a single replica with a stop-then-start deploy, or coordinate runs with a lock.
Do Laravel scheduled tasks run during maintenance mode?#
No. Scheduled tasks are skipped while the application is in maintenance mode, so tasks due during a deploy that calls php artisan down don't run. Add ->evenInMaintenanceMode() to tasks that must run anyway, and keep maintenance windows short.
How can I tell if a deploy lost a scheduled run?#
Have each job report when it starts and finishes to a service that knows its schedule. A run lost during the deploy sends nothing, so the monitor goes late and alerts after its grace period; a run killed mid-way leaves a start with no finish and is flagged as stuck.
Put this guide to work
Know when the next run is missing
Connect one scheduled job, confirm a real completion, then verify the alert destination. Keep your existing scheduler in charge of running the work.