# Scheduled Jobs During Deploys: Lost, Killed, and Doubled Runs

By Dan Ochoa · Published 2026-11-16 · Updated 2026-11-16
Canonical: https://cronradar.com/blog/cron-jobs-during-deploys

> How deploys lose, kill, or double scheduled jobs in Hangfire, Celery, Kubernetes and Laravel: shutdown timeouts, acks_late, grace periods, two schedulers.

**A deployment breaks scheduled jobs in three ways: a run is lost because no scheduler was running at its due time, a run is killed because the old process stopped before it finished, or a run is doubled because old and new schedulers overlapped.** Which one you get depends on how long your shutdown waits, whether interrupted work is redelivered, and whether two versions can schedule at once. Each framework has a setting for each, and the defaults rarely match a real deploy.

This guide covers Hangfire 1.8, Celery 5.5 and later, Kubernetes, and Laravel 12 and 13, checked November 16, 2026. For what each scheduler does about runs missed during longer outages, see [missed runs after downtime](https://cronradar.com/blog/missed-runs-after-downtime).

## Three Ways a Deploy Breaks a Run

| What happened | When                                                                          | Typical cause                                    | What you see later                      |
| ------------- | ----------------------------------------------------------------------------- | ------------------------------------------------ | --------------------------------------- |
| Lost          | The due time fell between the old scheduler stopping and the new one starting | Recreate deploys, maintenance mode, slow startup | Nothing; no run, no error               |
| Killed        | A run was in progress when its process was stopped                            | Shutdown timeout shorter than the job            | A partial run, maybe retried, maybe not |
| Doubled       | Old and new versions both scheduled the same due time                         | Rolling deploys of in-process schedulers         | Two runs, duplicate emails or rows      |

Lost and doubled runs depend on the scheduler; killed runs depend on the worker. A system can have all three: a Celery deploy can kill a running task, briefly run two beat processes, and lose the run that was due while the new beat started.

## Hangfire: Shutdown Timeout and At-Least-Once Processing

When a Hangfire server stops, it signals its workers and waits. `BackgroundJobServerOptions.ShutdownTimeout` is 15 seconds by default ([BackgroundProcessingServer.cs](https://github.com/HangfireIO/Hangfire/blob/v1.8.25/src/Hangfire.Core/Server/BackgroundProcessingServer.cs)). A job that observes cancellation can stop cleanly within that window; one that doesn't is cut off when the process exits.

An interrupted Hangfire job isn't lost. Hangfire processes jobs at least once, so a job that was being processed when its server stopped is picked up again later by another server. That turns "killed" into "run twice, the first time partially," which is safe only if the job is [idempotent](https://cronradar.com/blog/idempotent-scheduled-jobs).

Make the job observe shutdown by taking a `CancellationToken`, which Hangfire supplies, and size the timeouts so they nest:

```csharp
using Hangfire;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHangfire(config => config
    .UseSqlServerStorage(builder.Configuration.GetConnectionString("Hangfire")));

builder.Services.AddHangfireServer(options =>
{
    // Wait up to 25 s for running jobs; the host and the platform must wait longer.
    options.ShutdownTimeout = TimeSpan.FromSeconds(25);
});

// The generic host stops hosted services, including the Hangfire server, within this window.
builder.Services.Configure<HostOptions>(options => options.ShutdownTimeout = TimeSpan.FromSeconds(28));

var app = builder.Build();
app.Services.GetRequiredService<IRecurringJobManager>()
    .AddOrUpdate<ExportJobs>("nightly-export", jobs => jobs.Export(CancellationToken.None), "0 2 * * *");
app.Run();

public class ExportJobs
{
    public async Task Export(CancellationToken cancellationToken)
    {
        foreach (var batch in Enumerable.Range(1, 100))
        {
            cancellationToken.ThrowIfCancellationRequested(); // Stop between batches on shutdown.
            await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
        }
    }
}
```

Hangfire replaces the `CancellationToken.None` placeholder with a real token at run time. The order that matters is Hangfire's `ShutdownTimeout` < the host's `HostOptions.ShutdownTimeout` < the platform's kill timeout, such as Kubernetes' `terminationGracePeriodSeconds` (default 30 seconds). If the platform kills first, nothing inside the process gets to finish.

Hangfire doesn't double recurring runs during an overlap. The recurring job scheduler in each server coordinates through a distributed lock, so old and new servers sharing one storage create each occurrence once. A due time that passes while no server is running is handled by `MisfireHandling` when one returns; the default creates one catch-up run. The deploy risk Hangfire does have is code: an old server that can't load a newly renamed job type records errors against it, covered in [Hangfire jobs missing after a deploy](https://cronradar.com/blog/hangfire-jobs-missing-after-deploy).

## Celery: Warm Shutdown, acks_late, and One Beat

A Celery worker that receives SIGTERM does a warm shutdown: "The worker will finish all currently executing tasks before it actually terminates" ([workers guide](https://docs.celeryq.dev/en/stable/userguide/workers.html#stopping-the-worker)). Celery itself doesn't impose a limit, but your platform does. When its grace period ends, it sends SIGKILL, and the tasks still running die with the process.

Whether a killed task runs again depends on when its message was acknowledged:

- **Default (early acknowledgment):** the message is acknowledged when the worker receives it, so a task killed mid-run is gone. The workers guide says running tasks are lost on a cold shutdown "unless the tasks have the `acks_late` option set."
- **`acks_late=True`:** the message is acknowledged after the task finishes, so a killed task's message goes back to the broker and runs again, from the start. With the Redis broker, redelivery waits for the visibility timeout, one hour by default, so the retry can arrive long after the deploy.

Celery 5.5 added a soft shutdown: with `worker_soft_shutdown_timeout` set to a positive number of seconds, the worker gets a time-limited warm period before a cold shutdown. It's disabled by default. Set the platform's grace period above the longest task you're willing to wait for, and use `acks_late` only for tasks that are safe to run twice ([Celery task retries](https://cronradar.com/blog/celery-task-retries)).

Beat is the doubling risk. Celery beat must be a single process, and a rolling deploy of a beat container briefly runs two, both sending the tasks due in that window. Deploy beat with a stop-then-start strategy (`Recreate` in Kubernetes), and accept that a task due during the gap is sent when the new beat starts, as covered in [Celery beat not running](https://cronradar.com/blog/celery-beat-not-running).

## Kubernetes: Grace Periods, Rolling Updates, and CronJob Pods

When Kubernetes stops a pod, it runs any `preStop` hook, sends SIGTERM to the containers, and sends SIGKILL when `terminationGracePeriodSeconds` (30 seconds by default) runs out. The `preStop` hook's time counts against the same grace period. So a pod's shutdown budget is one number shared by every step, and every timeout inside the application must fit in it.

Two Kubernetes behaviors cause most deploy-time scheduling problems:

- **Rolling updates overlap versions.** A Deployment's default `RollingUpdate` strategy starts new pods before old ones are gone (`maxSurge` 25%). Any scheduler running inside the app, such as node-cron, a `BackgroundService` loop, APScheduler, or Celery beat, runs in both versions during the overlap and can fire twice. Run in-process schedulers in a separate single-replica Deployment with `strategy: Recreate`, or guard the job with a distributed lock.
- **CronJob pods are separate from your deploy.** Updating a CronJob's template affects Jobs created after the change; a Job already running keeps its pod and old image. But a cluster or node upgrade that drains nodes evicts running CronJob pods. Each evicted pod counts as a failure toward the Job's `backoffLimit` unless a `podFailurePolicy` ignores pods with the `DisruptionTarget` condition.

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-export
spec:
  schedule: '0 2 * * *'
  timeZone: 'Etc/UTC'
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 2
      podFailurePolicy:
        rules:
          - action: Ignore # Retry evicted pods without using up backoffLimit.
            onPodConditions:
              - type: DisruptionTarget
      template:
        spec:
          restartPolicy: Never
          terminationGracePeriodSeconds: 120
          containers:
            - name: export
              image: registry.example.com/export:1.4.2
              args: ['--run', 'nightly']
```

`podFailurePolicy` requires `restartPolicy: Never`. With `concurrencyPolicy: Forbid`, a run whose previous Job is still retrying is skipped, which is the right trade for most exports; [Kubernetes CronJob monitoring](https://cronradar.com/blog/kubernetes-cronjob-monitoring) covers the other fields.

## Laravel: Maintenance Mode and schedule:interrupt

Laravel's own deploy steps can stop the schedule:

- **Maintenance mode.** "Your application's scheduled tasks will not run when the application is in maintenance mode" ([task scheduling docs](https://laravel.com/docs/12.x/scheduling#maintenance-mode)). A deploy script that runs `php artisan down` before migrations and `php artisan up` at the end loses every task due in between, with no log entry. Tasks that must run anyway need `->evenInMaintenanceMode()`.
- **Sub-minute tasks.** When the schedule has sub-minute tasks, `schedule:run` keeps running until the end of the minute, using the previously deployed code. Add `php artisan schedule:interrupt` to the deploy script after the new code is in place so in-progress `schedule:run` invocations stop.
- **Stale overlap locks.** `withoutOverlapping()` uses a cache lock that expires after 24 hours by default. A run killed during a deploy can leave its lock behind, and the task is then skipped until the lock expires. `php artisan schedule:clear-cache` clears scheduler locks.
- **Two servers during a cutover.** If old and new servers both run `schedule:run` for a while, each runs every task unless the task uses `->onOneServer()`, which needs a shared cache (`database`, `memcached`, `dynamodb`, or `redis`) across all servers.

```php
use Illuminate\Support\Facades\Schedule;

Schedule::command('billing:charge-renewals')
    ->dailyAt('02:00')
    ->onOneServer()
    ->withoutOverlapping(120) // Lock expires after 2 hours, not 24, if a run is killed.
    ->evenInMaintenanceMode();
```

Scheduled tasks that dispatch queued jobs also depend on queue workers, which deploy scripts usually restart with `php artisan queue:restart` so workers finish their current job and exit. [Laravel schedule not running](https://cronradar.com/blog/laravel-schedule-not-running) covers the scheduler's other failure modes.

## Make Deploys Safe for Scheduled Work

The fixes above reduce to a short list that applies to any stack:

1. **Nest the timeouts.** Job cancellation check interval < framework shutdown timeout < host shutdown timeout < platform kill timeout.
2. **Make killed runs safe to repeat.** Anything that redelivers interrupted work (Hangfire, `acks_late`, Job retries) runs it again from the start.
3. **Run one scheduler.** Keep in-process schedulers out of rolling, multi-replica deployments, or coordinate them with a lock.
4. **Know your gap.** If the scheduler is down during a deploy, decide whether a due run should be caught up, and check what your scheduler actually does.
5. **Avoid deploying across critical due times,** or move those jobs off the deploy window.

## See Deploy Gaps With Start and Finish Check-Ins

None of these failures appears in the deploy log. A lost run leaves no trace; a killed run looks like a run that stopped logging; a doubled run looks normal twice. Start and finish check-ins from each job make each case distinct, because an outside service compares what arrived with what the schedule expected:

| What happened during the deploy | Check-ins received                    | What CronRadar shows                                                                   |
| ------------------------------- | ------------------------------------- | -------------------------------------------------------------------------------------- |
| Lost                            | None                                  | `warning` after the expected time, then `critical` and an alert after the grace period |
| Killed, not retried             | `start` with no finish                | Flagged as stuck                                                                       |
| Killed, then redelivered        | `start`, later `start` and `complete` | Both attempts recorded, with the delay between them visible                            |
| Failed on the new code          | `start`, `fail`                       | `critical` with an immediate alert                                                     |
| Doubled                         | Two starts and two completions        | Both runs recorded; no alert for an extra run                                          |

Alerts go to email, Slack, webhook, Discord, or PagerDuty. Set each monitor's grace period longer than a normal deploy, so a run delayed by a few minutes of restart doesn't alert, but a run lost to a long maintenance window does. Give concurrent runs distinct `run_id`s and reuse a run's `run_id` across retries, so a redelivered run pairs with its own start.

How each stack reports:

- **Hangfire:** `CronRadar.Hangfire` with `builder.Services.AddCronRadarHangfire();` discovers recurring jobs and reports start, completion, and failure through a server filter. It reports a failure only after Hangfire's retries finish.
- **Celery:** `cronradar-celery` with `setup_cronradar(app)` discovers `app.conf.beat_schedule` entries and reports from task signals.
- **Laravel:** `composer require cronradar/laravel` discovers every scheduled task.
- **Kubernetes CronJobs and other schedulers:** there's no Kubernetes integration; report over HTTP from the container's command, with a monitor key and schedule per job, or wrap the job with a base SDK (.NET, Node, Python, PHP).

A shell wrapper for a CronJob container, following the [full lifecycle](https://docs.cronradar.com/getting-started) pattern, keeps the job's own exit code so Kubernetes still sees failures:

```bash
#!/bin/sh
run_id="nightly-export-$(date +%s)-$$"

curl --fail-with-body --silent --show-error --max-time 5 --request POST \
  --user "$CRONRADAR_API_KEY:" \
  --get "https://cron.life/api/v1/ping/nightly-export/start" \
  --data-urlencode "schedule=0 2 * * *" \
  --data-urlencode "run_id=$run_id" || true

if /app/export --run nightly; then
  curl --fail-with-body --silent --show-error --max-time 5 --request POST \
    --user "$CRONRADAR_API_KEY:" \
    --get "https://cron.life/api/v1/ping/nightly-export/complete" \
    --data-urlencode "run_id=$run_id" || true
else
  exit_code=$?
  curl --fail-with-body --silent --show-error --max-time 5 --request POST \
    --user "$CRONRADAR_API_KEY:" \
    --get "https://cron.life/api/v1/ping/nightly-export/fail" \
    --data-urlencode "run_id=$run_id" \
    --data-urlencode "message=Export exited with code $exit_code" || true
  exit "$exit_code"
fi
```

The `--max-time 5` and `|| true` keep a monitoring outage from failing or stalling the job. CronRadar evaluates schedules in the application's time zone, set in its settings (UTC if unset), so match it to the CronJob's `timeZone`. Use a separate CronRadar application and key per environment, so a staging deploy's runs never count as production's.

## Frequently Asked Questions

### What happens to a running Hangfire job during a deploy?

The server signals cancellation and waits up to `ShutdownTimeout`, 15 seconds by default. A job that observes its `CancellationToken` can stop cleanly; one that doesn't is cut off when the process exits. Hangfire processes jobs at least once, so the interrupted job is picked up again later and runs from the start.

### Does Celery lose tasks when a worker is restarted?

On SIGTERM a worker does a warm shutdown and finishes its running tasks. If the platform kills it before they finish, tasks with default early acknowledgment are lost. With `acks_late=True` the message is redelivered and the task runs again, after the visibility timeout on Redis, which is one hour by default.

### Why do my scheduled jobs run twice after a deploy?

Old and new versions of an in-process scheduler overlapped. Rolling updates run both for a while, and node-cron, `BackgroundService` loops, APScheduler, and Celery beat each fire every due job in every running copy. Run the scheduler as a single replica with a stop-then-start deploy, or coordinate runs with a lock.

### Do Laravel scheduled tasks run during maintenance mode?

No. Scheduled tasks are skipped while the application is in maintenance mode, so tasks due during a deploy that calls `php artisan down` don't run. Add `->evenInMaintenanceMode()` to tasks that must run anyway, and keep maintenance windows short.

### How can I tell if a deploy lost a scheduled run?

Have each job report when it starts and finishes to a service that knows its schedule. A run lost during the deploy sends nothing, so the monitor goes late and alerts after its grace period; a run killed mid-way leaves a start with no finish and is flagged as stuck.
