Prevent Overlapping Cron Jobs: flock, Locks, and Concurrency Policies

To prevent overlapping cron jobs, run each job under a lock that only one run can hold at a time. On one Linux host, prefix the crontab command with flock: flock -n /var/lib/inventory/sync.lock /opt/inventory/sync.sh runs the command only if it can take the lock immediately, and otherwise exits without running it. Cron won't do this for you: cronie and Debian's cron start a new copy at every matching minute, finished or not (cronie job.c). Jobs on several hosts need a shared lock, such as a database advisory lock or a Redis key; Kubernetes CronJobs and systemd timers have overlap policies of their own.

Decide What the Next Run Should Do#

Every overlap fix implements one of four policies:

Skip is the right default when the work isn't tied to one occurrence: a sync of everything changed since its last success loses nothing to a skip, while one that processes "the last 15 minutes" loses that window. If every period must be processed, record which are done and have each run handle all pending ones; the queue becomes a safe skip. Preventing errors in scheduled automation covers making runs safe to repeat. Schedulers name these policies differently and choose different defaults; the CronRadar Scheduled Job Methodology's overlap table maps the names across Kubernetes, Temporal, Cadence, HCL Workload Automation, Netflix Maestro, Pinterest Pinball, and systemd.

One Host: flock#

flock ships in util-linux on Debian, Ubuntu, Fedora, RHEL, and Arch; everything below matches the util-linux 2.41 source and flock(1) and works in 2.33 and later. Alpine's BusyBox flock has no -w, -E, or -o (flock.c), so run apk add flock for util-linux's.

The Crontab Line#

*/15 * * * * flock -n -E 75 /var/lib/inventory/sync.lock /opt/inventory/sync.sh

• -n fails instead of waiting when the lock is held. flock creates the lock file if it's missing, and once it has the lock, runs the command and holds the lock until the command exits.
• -E 75 sets the exit status for "lock busy." The default, 1, looks exactly like the job's own failure; when the command runs, flock exits with the command's status. 75 is EX_TEMPFAIL in sysexits.h; any value from 0 to 255 your job doesn't use works.
• A skip is silent. flock prints nothing and cron mails only output, so nothing records a skip unless something reads the exit status.

For the wait policy, flock -w 300 waits up to 300 seconds, then exits with the -E code. With neither option, flock waits indefinitely, every run that arrives meanwhile waits too, and flock(2) promises no order among them. The lock is advisory, excluding only processes that ask for it, so make manual runs use the same flock command.

Locking Inside the Script#

To make a script protect itself however it's started, lock a file descriptor at the top:

#!/bin/sh
# /opt/inventory/sync.sh
exec 9>/var/lib/inventory/sync.lock
if ! flock -n 9; then
  logger -t inventory-sync "skipped: the previous run still holds the lock"
  exit 0
fi

/opt/inventory/bin/export-orders
/opt/inventory/bin/push-stock-levels

Everything after the flock line runs under the lock, which is released when the script exits and descriptor 9 closes. 9> needs write permission on the file, and logger sends the skip to syslog instead of cron mail.

Background Children Keep the Lock#

A flock lock belongs to the open file, every child inherits the descriptor, and the lock is released only when all copies close (flock(2)). If the job starts something that outlives it, such as nohup /opt/inventory/bin/watcher &, that process keeps the lock and every later run skips. flock -o closes the descriptor in the command's process, so only flock itself holds the lock until the command exits (source). In the in-script form, close it for that child: nohup /opt/inventory/bin/watcher 9>&- &. When a job keeps skipping, lslocks(8) shows which process holds the lock.

Where to Put the Lock File#

• In a directory the job's user owns, such as /var/lib/inventory/. The contents never matter, and a leftover file is harmless; the lock exists only while a process holds the file open.
• /run/lock (also /var/lock) suits root jobs. Upstream systemd creates it root-only, mode 0755 (legacy.conf.in); check ls -ld /run/lock before using it for another user.
• Not a fixed name in /tmp. Any local user can create /tmp/sync.lock first and hold its lock, and your job skips forever. Cleanup isn't the danger: systemd-tmpfiles ages /tmp out after 10 days upstream (tmp.conf) but skips any file holding a BSD lock (tmpfiles.d(5)).
• Never delete it at the end of the job. A run that opened the old file just before the delete can lock it while the next run locks a new file, and the two run side by side.
• On a local file system. flock(1) warns that it can fail outright on NFS and CIFS, depending on mount options.

PID Files Go Stale; flock Locks Don't#

The older idiom fails three ways:

# Don't use this: stale files, PID reuse, and a race.
pidfile=/var/lib/inventory/sync.pid
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then
  exit 0
fi
echo $$ > "$pidfile"
/opt/inventory/sync.sh
rm -f "$pidfile"

• Stale files. A run ended by SIGKILL, the OOM killer, or a power cut never reaches rm.
• PID reuse. Linux reuses PIDs after they wrap at pid_max (proc_sys_kernel(5)). If a stale file names a PID that now belongs to another process, kill -0 can succeed, and the job skips until someone deletes the file.
• The race. Two runs can both pass the test before either writes the file. Creating it atomically, with mkdir for example, fixes the race but not the stale file.

A flock lock has none of these problems: the kernel releases it when the last descriptor closes, which happens when the process exits for any reason, and no lock survives a reboot.

Ubuntu's run-one package (1.17 in Ubuntu 26.04; not in Debian) wraps the same mechanism: run-one /opt/inventory/sync.sh runs flock -xn on a file in ~/.cache/run-one/ named after an MD5 hash of the command line (source). The lock is per user and per exact command line, and a skip exits 1. run-this-one first kills that user's processes with a matching command line: the replace policy (run-one(1)).

systemd Timers Never Overlap, but Wait by Default#

A timer never starts a second copy of its service. From systemd.timer(5) (systemd 262): "in case the unit to activate is already active at the time the timer elapses it is not restarted, but simply left running." The slot isn't simply skipped, though. The timer waits for the service to stop (timer.c) and then schedules from its previous trigger time, which is now in the past, so it starts again at once; several missed slots collapse into one run. That's the wait policy. For a skip, set DeferReactivation=yes (systemd 257 and later, OnCalendar= timers only), which schedules the next run from when the service finished. The protection covers the unit, not the script, so running the script directly bypasses it. systemd timers covers timers in full.

Kubernetes: Forbid Skips, Then May Run Late#

concurrencyPolicy sets what the CronJob controller does when a Job is due while the previous one is still active (concurrency policy). Allow, the default, starts both. Forbid skips the new Job and records a JobAlreadyActive event. Replace deletes the running Job and creates the new one.

The docs add that under Forbid, "when the previous Job run finishes, .spec.startingDeadlineSeconds is still taken into account and may result in a new Job run." In the Kubernetes v1.37.1 controller, a finishing Job wakes the CronJob, which looks for the most recent scheduled time since the last Job it started, ignoring times more than startingDeadlineSeconds old (utils.go). The skipped time counts, so:

• With no startingDeadlineSeconds, the most recent skipped run starts as soon as the previous Job finishes; several skips still produce one late run. That's the wait policy.
• With startingDeadlineSeconds: 120, the late run starts only if the previous Job finished within 120 seconds of the skipped time. A deadline well under the interval makes Forbid a true skip. The docs warn against values below 10 seconds, and the deadline applies to on-time starts too.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: inventory-sync
spec:
  schedule: '*/15 * * * *'
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 120 # drop a skipped run instead of starting it late
  jobTemplate:
    spec:
      activeDeadlineSeconds: 1800 # a hung run can't block later runs forever
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: sync
              image: registry.example.com/inventory-sync:1.4.2
              command: ['/opt/inventory/sync.sh']

Three gaps remain. A source comment concedes it is "theoretically possible to have concurrency with Forbid" when the controller's view is stale. A Job made with kubectl create job --from=cronjob/inventory-sync is owned by the CronJob but never added to the active list the controller checks (cronjob_controllerv2.go), so a scheduled run can start beside it. And Replace deletes the old Job in the background (injection.go), so its pod gets the usual 30-second termination grace period and may still be shutting down as the new pod starts. Kubernetes CronJob monitoring covers the rest of the spec.

Many Hosts: Database and Redis Locks#

A flock lock lives in one machine's kernel, so a crontab copied to three hosts runs the job three times. The simplest fix is to schedule it on one host; why a cron job runs twice covers finding duplicate schedulers. When any host must be able to run it, take the lock in a service they all reach.

A lock stops two runs at once, not two runs of one occurrence: a host that finishes a 20-second job before another host's clock reaches that minute lets the second host run it again. When that matters, record each occurrence, for example as a row keyed on the scheduled time with a unique constraint.

Database Advisory Locks#

Sources: PostgreSQL advisory locks, MySQL locking functions, sp_getapplock.

Each lock belongs to a database session, so the job must hold one dedicated connection open for the whole run, outside any connection pool. Three things break that:

• A pooler in between. In transaction pooling, PgBouncer returns the server connection to the pool after each transaction, and its feature table marks session-level advisory locks "Never" supported in that mode (PgBouncer features). Connect the lock directly or through session pooling.
• The connection dying first. A network drop, failover, or server-side timeout ends the session and releases the lock without telling the running job. PostgreSQL's idle_session_timeout and idle_in_transaction_session_timeout, both off by default, end exactly this kind of idle connection (client settings).
• An open transaction. A driver's implicit transaction leaves the lock connection idle in a transaction all run, which those docs warn holds back vacuum; use autocommit.

Redis: SET NX PX With a Token#

Redis's SET documentation gives the single-instance pattern: NX with an expiry, a random token as the value, and a release script that deletes the key only if it still holds that token. As a crontab wrapper:

#!/bin/sh
# /opt/inventory/redis-lock-run.sh COMMAND [ARGS...]
# Set REDISCLI_AUTH in the environment if the server needs a password.
key="lock:inventory-sync"
token="$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')"

# Hold the lock for at most 30 minutes; any reply but OK means it's taken.
[ "$(redis-cli -h "$REDIS_HOST" SET "$key" "$token" NX PX 1800000)" = "OK" ] || exit 75

"$@"
rc=$?

# Delete the key only if it still holds this run's token.
redis-cli -h "$REDIS_HOST" EVAL \
  "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end" \
  1 "$key" "$token" > /dev/null
exit "$rc"

redis-cli prints plain replies when its output isn't a terminal, and an unreachable server returns no OK, so the job skips. On Redis 8.4 and later, DELEX key IFEQ token replaces the script, and SET key token IFEQ token PX 1800000 renews only your own lock.

A lease fails differently from a session lock:

• The TTL must outlast the longest run. Past it, the key expires, another host takes the lock, and the runs overlap; the token only stops the first run from deleting the second one's key. Cap the job's runtime below the TTL, or renew the lease while it runs.
• A crash leaves the key until it expires, and every host skips meanwhile.
• Failover can grant it twice. Replication is asynchronous; the distributed locks page walks through a primary crashing before the lock reaches its replica, so a second client gets the same lock.

No Lock Guarantees Mutual Exclusion#

Any lock that expires or depends on a connection can be lost while its holder keeps working: a garbage-collection pause, a stalled VM, a partition, or a clock change (Redis TTLs don't use a monotonic clock). The same Redis page advises: "You should implement fencing tokens. This is especially important for processes that can take significant time and applies to any distributed locking system." A fencing token grows with each grant, and the resource being written rejects tokens older than the highest it has seen (Martin Kleppmann). Where you don't control the writes, make the job safe to run twice and treat the lock as what makes overlap rare.

In-App Schedulers Have Their Own Settings#

• Hangfire: [DisableConcurrentExecution] makes a second run wait, then fail on timeout; Hangfire DisableConcurrentExecution shows how to skip instead.
• Quartz.NET: [DisallowConcurrentExecution] prevents concurrent runs of one JobDetail; two JobDetails sharing a class can still overlap (More about jobs).
• node-cron 4: noOverlap: true, off by default, skips a run while the previous one executes and emits execution:overlap (scheduling options).
• Celery 5.6: nothing built in; the task cookbook builds a lock from an atomic cache add with an expiry, and warns that "your tasks run-time shouldn't exceed the timeout."

Each guards only its own scheduler's runs.

Monitor the Job Without Alerting on Every Skip#

Put the monitoring inside the lock: flock runs the wrapper, and the wrapper runs the job, so a skipped run sends nothing. There is no CronRadar SDK for crontab and no auto-discovery, so each cron job reports with its own HTTP check-in:

*/15 * * * * flock -n -E 75 /var/lib/inventory/sync.lock /opt/inventory/sync-monitored.sh

#!/bin/sh
# /opt/inventory/sync-monitored.sh
# The key lives in a file only the job's account can read: cronie and Debian's cron
# copy crontab variables into the headers of the mail they send.
CRONRADAR_API_KEY=$(cat /etc/cronradar/api-key 2>/dev/null)
url="https://cron.life/api/v1/ping/inventory-sync"
run_id="inventory-sync-$(date +%s)-$$"

report() {
  event=$1
  shift
  curl --fail --silent --show-error --max-time 5 --output /dev/null \
    --request POST --get --user "$CRONRADAR_API_KEY:" \
    --data-urlencode "schedule=*/15 * * * *" \
    --data-urlencode "gracePeriod=600" \
    --data-urlencode "run_id=$run_id" \
    "$@" "$url/$event" || true
}

report start
timeout -k 1m 30m /opt/inventory/sync.sh
rc=$?
if [ "$rc" -eq 0 ]; then
  report complete
else
  report fail --data-urlencode "message=sync.sh exited with code $rc"
fi
exit "$rc"

When the 02:00 run takes 22 minutes, CronRadar sees this:

• 02:00: start arrives, and the monitor's expected time moves to the next occurrence, 02:15.
• 02:15: flock finds the lock held and exits 75 before the wrapper starts. Nothing is sent; the API has no event for a skipped run.
• 02:22: complete arrives before 02:15 plus the 10-minute grace period, so no alert fires, and the expected time moves to 02:30.

If a run is still going at 02:25 with its start as the last event, the monitor goes critical as stuck. That's the right alert: the job has overrun its interval by more than the grace you allowed, so set gracePeriod to the overrun you'll tolerate. At 02:30, timeout stops the job, the wrapper reports fail, and the lock is free for the next run.

The order matters. With flock inside the wrapper instead, every skipped run would report a start and then a fail, and since every check-in moves the expected time, those reports would keep resetting the stuck window of the run they skipped. Without timeout, a hung run holds the lock forever and the stuck alert is the only signal; CronRadar has no per-job maximum runtime, so the limit belongs in the job.

The first run creates the monitor from its schedule, and the dead man's switch explains how missed runs are detected.

Frequently Asked Questions#

How do I stop a cron job from running if it's already running?#

Put flock -n and a lock file in front of the command: */15 * * * * flock -n /var/lib/inventory/sync.lock /opt/inventory/sync.sh. If the previous run still holds the lock, the new run exits without running the command.

What does flock -n do?#

It requests the lock without waiting. If another process holds it, flock exits with status 1, or the code set with -E, without running the command; otherwise it runs the command and releases the lock when the command exits.

Is a PID file enough to prevent overlapping cron jobs?#

No. A killed run leaves the file behind, its PID can later belong to an unrelated process, and two runs can both check before either writes. The kernel releases a flock lock when the process exits, however it exits.

How do I prevent overlapping jobs across multiple servers?#

Schedule the job on one server, or take a lock every server reaches: a PostgreSQL advisory lock, MySQL GET_LOCK, SQL Server sp_getapplock, or a Redis key set with NX and an expiry. None of them guarantees exclusion through network failures or long pauses, so keep the job safe to run twice.

What is concurrencyPolicy Forbid in Kubernetes?#

It stops the CronJob controller from starting a Job while the previous one is still active. When the previous Job finishes, the controller still starts the most recent skipped run unless startingDeadlineSeconds has passed, so set that well below the interval for a true skip.