How to Retry Failed Cron Jobs Without Hiding the Failure

Cron doesn't retry a failed job, so to retry one, run the command through a wrapper that tries again a few times with growing, randomized delays, and only for errors that can clear on their own. A crontab entry is a schedule and a command, with no retry setting (crontab(5)). Cap both the attempts and the total time so the last attempt ends before the next scheduled run, and exit with the final attempt's status. Then make sure the final failure still reaches someone: a job that retries quietly can fail for days, or barely recover every hour, without anyone knowing.

This expands the bounded-retries item in preventing errors in scheduled automation. Versions and limits below were checked October 24, 2026.

Retry Only Failures That Can Clear on Their Own#

A retry bets that the next attempt sees a different world: the connection is back, the rate limit has reset, the lock is free. When nothing will change, it only delays the alert and repeats the failed attempt's side effects. Large platforms cap retries by error type, such as Netflix Maestro's defaults of 2 retries for user errors, 10 for platform errors, and none for timeouts; the CronRadar Scheduled Job Methodology collects these limits.

Retry-After holds a number of seconds or a date (RFC 9110 §10.2.3), and a 429 may carry it too (RFC 6585 §4).

Give the Job an Exit Code That Means "Try Again"#

A wrapper sees only an exit status, so the job has to classify its own failures. The conventional "try again" status is 75, EX_TEMPFAIL: sysexits(3) says the request "should be reattempted later", and systemd.exec(5) lists it as "Temporary failure; user is invited to retry". FreeBSD now discourages sysexits in general, but no other status carries that meaning. Exit 1 for everything else:

#!/bin/sh
# /opt/sync/sync.sh: exit 75 when a retry may help, 1 when it won't.
out=/var/lib/sync/orders.json
code=$(curl --silent --show-error --max-time 120 --output "$out" \
  --write-out '%{http_code}' "https://partner.example.com/v1/orders/export")
case $? in
  0) ;;
  6 | 7 | 28 | 52 | 56) exit 75 ;; # DNS, connect, timeout, empty reply, receive error
  *) exit 1 ;;
esac
case $code in
  2??) ;;
  408 | 429 | 500 | 502 | 503 | 504) exit 75 ;;
  *)
    echo "export returned HTTP $code" >&2
    exit 1
    ;;
esac
/opt/sync/load-orders "$out"

curl's own exit status says whether a response arrived at all (exit codes); %{http_code} says what it was.

Make the Job Safe to Run Again#

A retry runs the job from the top. If the failed attempt already inserted half the orders or sent an email, the retry does it again. RFC 9110 says a client "SHOULD NOT automatically retry a request with a non-idempotent method" unless it knows repeating it is safe (§9.2.2); the same goes for a job. Here, load-orders has to tolerate the same export twice. Idempotent scheduled jobs covers upserts, watermarks, and idempotency keys.

A POSIX sh Retry Wrapper With Backoff and Jitter#

#!/bin/sh
# /usr/local/bin/retry: run a command, retrying temporary failures.
# Usage: retry COMMAND [ARG...]   Exits with the command's last status.
max_attempts=${RETRY_ATTEMPTS:-4}  # runs in total, the first included
base_delay=${RETRY_BASE_DELAY:-10} # seconds; doubles after each failure
max_delay=${RETRY_MAX_DELAY:-300}  # cap on the delay before any one retry
budget=${RETRY_BUDGET:-1200}       # no retry starts after this many seconds
retry_on=${RETRY_ON:-75}           # exit statuses that mean "try again"

random_upto() { # prints a random integer from 0 to $1
  r=$(od -A n -N 2 -t u2 /dev/urandom 2>/dev/null | tr -d ' ')
  echo $((${r:-$$} % ($1 + 1)))
}

retryable() {
  for code in $retry_on; do
    [ "$1" -eq "$code" ] && return 0
  done
  return 1
}

started=$(date +%s)
attempt=1
delay=$base_delay
while :; do
  "$@"
  rc=$?
  [ "$rc" -eq 0 ] && exit 0
  if [ "$attempt" -ge "$max_attempts" ] || ! retryable "$rc"; then
    exit "$rc"
  fi
  pause=$(random_upto "$delay") # full jitter: anywhere from 0 to delay
  if [ $(($(date +%s) - started + pause)) -ge "$budget" ]; then
    echo "retry: out of time after attempt $attempt (exit $rc)" >&2
    exit "$rc"
  fi
  echo "retry: attempt $attempt of $max_attempts exited $rc; retrying in ${pause}s" >&2
  sleep "$pause"
  attempt=$((attempt + 1))
  delay=$((delay * 2))
  [ "$delay" -gt "$max_delay" ] && delay=$max_delay
done

• Attempts count runs. RETRY_ATTEMPTS=4 is the first run plus three retries.
• Backoff with jitter. The delay doubles from 10 seconds up to the 300-second cap, and each pause is a random time between zero and that delay, the "full jitter" Marc Brooker compared with other schemes on the AWS Architecture Blog. Cron needs it: every server with the same crontab starts the job in the same minute, and without jitter they all retry together too.
• Portable. POSIX sh has no $RANDOM (shell variables), so the wrapper reads /dev/urandom with od, falling back to its process ID. date +%s is standard since POSIX.1-2024 added %s to strftime, which date follows.
• A time budget. No retry starts once the budget has passed, but a started attempt runs to its end, so limit each attempt too and keep the budget plus that limit below the schedule interval.
• The last status wins. Any status not in RETRY_ON, such as 1 or 127, ends the loop at once, and the wrapper exits with the final attempt's status, so cron mail, systemd, and any monitor see the real result.

For the hourly sync:

0 * * * * RETRY_ON="75 124" /usr/local/bin/retry timeout 10m /opt/sync/sync.sh

GNU timeout kills a hung attempt and exits 124 (timeout(1), coreutils 9.11), which this line treats as retryable. The last retry starts within 20 minutes and ends within 30.

For One HTTP Request, Let curl Retry#

In curl 8.23.0, --retry N retries a timeout or HTTP 408, 429, 500, 502, 503, 504, 522, or 524. It waits one second, doubles the wait up to ten minutes, and follows Retry-After since 7.66.0 (--retry). --retry-max-time caps the total, --retry-connrefused adds refused connections, and --retry-all-errors is, in its own documentation's words, "the sledgehammer" (--retry-all-errors):

curl --fail --silent --show-error --retry 4 --retry-max-time 600 --max-time 120 \
  --output /var/lib/sync/orders.json "https://partner.example.com/v1/orders/export"

Retry Settings in Schedulers#

Kubernetes Jobs: backoffLimit and podFailurePolicy#

A CronJob creates one Job per scheduled run, and the Job retries. backoffLimit defaults to 6 (types.go), and with restartPolicy: Never the Job fails once more Pods have failed than that (job_controller.go): the first Pod plus six retries. Replacement Pods wait 10 s, 20 s, 40 s, and so on, capped at ten minutes (Job docs), so the default limit adds 10 minutes 30 seconds of back-off to seven Pods' run time. The docs for 1.36 and earlier say six minutes, but the controller has used ten since at least 1.29.

podFailurePolicy, stable since Kubernetes 1.31, stops retrying failures that won't clear. It requires restartPolicy: Never, the first matching rule wins, and exit code 0 never matches (Pod failure policy, types.go):

apiVersion: batch/v1
kind: CronJob
metadata:
  name: partner-sync
spec:
  schedule: '0 * * * *'
  jobTemplate:
    metadata:
      labels:
        app: partner-sync
    spec:
      backoffLimit: 3
      activeDeadlineSeconds: 1800
      podFailurePolicy:
        rules:
          - action: Ignore # evictions and preemptions don't use a retry
            onPodConditions:
              - type: DisruptionTarget
          - action: FailJob # anything but 75 fails the Job now
            onExitCodes:
              containerName: sync
              operator: NotIn
              values: [75]
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: sync
              image: registry.example.com/partner-sync:1.4.2
              command: ['/opt/sync/sync.sh']

The Ignore rule comes first because an evicted Pod's exit code could otherwise match FailJob. A 75 matches no rule, so it counts against backoffLimit as usual. activeDeadlineSeconds is the time budget: it covers the whole Job "no matter how many Pods are created", wins over backoffLimit, and fails the Job with reason DeadlineExceeded (Job termination).

Kubernetes doesn't tell a Pod which retry it is: the downward API exposes the Pod's name, labels, and annotations, but no failure count. Kubernetes CronJob monitoring uses the batch.kubernetes.io/job-name label as a run ID that every retry Pod shares.

systemd: Restart= on a Oneshot Service#

A timer-started Type=oneshot service can use Restart=on-failure; always and on-success are rejected for oneshot (systemd.service(5)). The defaults suit daemons: RestartSec= is 100 ms and the start limit is 5 starts in 10 seconds (systemd-system.conf(5)), so a job that fails at once uses up its retries in under a second, as systemd timers shows. Raise RestartSec= to 30 seconds and the opposite happens: five starts never fit in ten seconds, so systemd restarts a failing job indefinitely. Set your own limit. StartLimitIntervalSec= and StartLimitBurst= count every start, the timer's included (systemd.unit(5)):

# /etc/systemd/system/partner-sync.service.d/retry.conf
[Unit]
StartLimitIntervalSec=30min
StartLimitBurst=4

[Service]
RestartForceExitStatus=TEMPFAIL
RestartSec=30s
RestartSteps=2
RestartMaxDelaySec=2min

• Only exit 75 restarts. RestartForceExitStatus= restarts on the listed statuses even with Restart= left at no, but never restarts a oneshot that succeeded (service.c). To retry most failures instead, set Restart=on-failure and list the permanent statuses in RestartPreventExitStatus=.
• Growing delays. RestartSteps= and RestartMaxDelaySec= (systemd 254) grow the delay geometrically, here 30 s, 60 s, 120 s, and RestartRandomizedDelaySec= (262) adds jitter. Debian 12 and CentOS Stream 9 package systemd 252, with neither; Ubuntu 24.04 (255) and Debian 13 (257) lack only the jitter (Debian, Ubuntu, CentOS Stream).
• A hard stop. Four starts per 30 minutes is the timer's start plus three restarts. The next restart is refused with result start-limit-hit; the timer can start the unit again after the interval, so keep it shorter than the timer's period.

systemd doesn't give the job its attempt number, and each restart gets a new $INVOCATION_ID (service.c), so a reporter keyed on it sees separate runs. For attempt-aware reporting, leave restarts off and run a retry wrapper from ExecStart=.

Other Schedulers#

If CronRadar monitors your Hangfire jobs, note that NuGet's latest CronRadar.Hangfire, 0.1.0 (checked October 24, 2026), can report a failure before Hangfire finishes retrying, including cancellations. Reporting only after retries are exhausted requires the pending 0.2.1 release.

How Retries Hide Failures#

• The alert waits for the last attempt. A monitor of the final status hears nothing until the chain ends: at least 10.5 minutes for a Kubernetes Job at defaults, over four hours for Hangfire. A long chain can still be going when the next run is due.
• A later success erases the evidence. If attempt three works, the exit status is 0 and every dashboard is green. A dependency that now fails most first attempts looks healthy until the day every attempt fails.
• Retries multiply load during an outage. Every job that shares the failing dependency retries against it. Caps and jitter keep that bounded.
• Retried permanent errors look like slow transient ones. Retrying an expired credential delays its alert by the whole budget.

So record attempts, not only results. The wrapper writes a stderr line per retry, which lands in cron mail, the journal, or container logs; count those per job and alert when the share of runs needing a retry climbs, or when a run needs its last attempt. In Kubernetes, a Job that succeeded with a non-zero failed count needed a retry:

kubectl get jobs -l app=partner-sync \
  -o custom-columns=JOB:.metadata.name,SUCCEEDED:.status.succeeded,FAILED:.status.failed

Monitoring Retried Jobs With CronRadar#

CronRadar check-ins carry two fields for retries (HTTP API reference): run_id, one ID of at most 128 characters shared by every attempt of a run, and attempt, the zero-based attempt number. attempt is diagnostic only: CronRadar records it with the check-in and never retries your job. The choice that matters is when to send fail, which opens an incident and alerts immediately, with no grace period:

• Only after the last attempt. An alert always means the run failed for good. It can arrive a whole retry chain after the first failure, and a run that recovers raises nothing; its check-ins with attempt above 0 are the record that it needed help.
• After every failed attempt. You hear about the first failure at once, and when a later attempt sends complete, CronRadar records the recovery. For a job that often needs a second try, that is an alert and a recovery every time, which teaches people to ignore both. Choose it when a failed attempt can leave partial side effects, or failures are rare enough that each one is news.

This wrapper does the first by default and the second with each_failure=yes:

#!/bin/sh
# /opt/sync/sync-monitored.sh: hourly partner sync, retried and reported.
url="https://cron.life/api/v1/ping/partner-sync"
run_id="partner-sync-$(date +%s)-$$" # one ID for every attempt of this run
max_attempts=4
budget=1800     # no retry starts after 30 minutes
each_failure=no # yes: also report fail for attempts that will be retried

report() {
  event=$1
  shift
  curl --fail --silent --show-error --max-time 5 --output /dev/null \
    --request POST --get --user "$CRONRADAR_API_KEY:" \
    --data-urlencode "schedule=0 * * * *" \
    --data-urlencode "gracePeriod=600" \
    --data-urlencode "run_id=$run_id" \
    --data-urlencode "attempt=$attempt" \
    "$@" "$url/$event" || true
}

random_upto() {
  r=$(od -A n -N 2 -t u2 /dev/urandom 2>/dev/null | tr -d ' ')
  echo $((${r:-$$} % ($1 + 1)))
}

started=$(date +%s)
attempt=0 # zero-based, as CronRadar expects
delay=15
while :; do
  report start
  timeout 10m /opt/sync/sync.sh
  rc=$?
  if [ "$rc" -eq 0 ]; then
    report complete
    exit 0
  fi
  msg="sync.sh exited $rc on attempt $((attempt + 1)) of $max_attempts"
  pause=$(random_upto "$delay")
  final=no
  case $rc in 75 | 124) ;; *) final=yes ;; esac
  [ $((attempt + 1)) -ge "$max_attempts" ] && final=yes
  [ $(($(date +%s) - started + pause)) -ge "$budget" ] && final=yes
  if [ "$final" = yes ]; then
    report fail --data-urlencode "message=$msg"
    exit "$rc"
  fi
  [ "$each_failure" = yes ] && report fail --data-urlencode "message=$msg; retrying"
  sleep "$pause"
  attempt=$((attempt + 1))
  delay=$((delay * 2))
  [ "$delay" -gt 240 ] && delay=240
done

• Same guarantees as a single check-in. Each report gives up after five seconds and ignores errors, and the script exits with the job's own status. Every report carries the schedule, so the first run creates the monitor and a deleted one is recreated; schedule and gracePeriod apply only when a check-in creates the monitor, so change an existing one in the dashboard.
• The budget protects stuck detection. Each check-in moves the expected time to the next scheduled run. A chain still sleeping past that time plus the grace period, with start as its last event, would turn the monitor critical as stuck. A 30-minute budget plus a 10-minute timeout ends every run within the hour.
• Run it from cron with CRONRADAR_API_KEY in its environment: 0 * * * * /opt/sync/sync-monitored.sh.

When the scheduler retries, each attempt is a new process that can't know whether another will follow, so every failed attempt reports fail: the every-attempt policy. Keep the run together anyway. In Kubernetes, the job-name label is a run_id shared by every retry Pod. In GitHub Actions, GITHUB_RUN_ID survives re-runs and GITHUB_RUN_ATTEMPT starts at 1 (variables), so send attempt=$((GITHUB_RUN_ATTEMPT - 1)). To alert only on final failures, move the retries into the wrapper and turn the scheduler's off or down.

Frequently Asked Questions#

Does cron retry failed jobs?#

No. cronie 1.7.2, Debian's cron (3.0pl1-197 in Debian 13), and BusyBox 1.37 crond have no retry setting, so a failed run stays failed until the next scheduled time. Retry inside the command with a wrapper script, or run the job under a scheduler with retry settings, such as a Kubernetes Job or a systemd service.

How do I retry a failed cron job automatically?#

Point the crontab entry at a wrapper that reruns the command when it exits with a status you've marked as temporary, waits a growing, randomized delay between attempts, and stops after a fixed number of attempts or a time budget. Keep the whole chain shorter than the schedule interval.

How many times does a Kubernetes CronJob retry?#

Each scheduled run is a Job, and a Job retries failed Pods up to backoffLimit times, six by default. Set backoffLimit, activeDeadlineSeconds, and podFailurePolicy under jobTemplate.spec, not on the CronJob's own spec. A new scheduled run starts with a fresh limit.

Should I alert on every failed attempt?#

Usually not, for jobs that often hit transient errors: alert on the final failure and watch how many runs need retries. Alert on every attempt when a failed attempt can leave partial work behind, or the job almost never fails. With CronRadar, any fail check-in alerts immediately, so the wrapper decides which policy you get.

How do I add exponential backoff to a shell script?#

Keep a delay variable, sleep a random part of it after each failure, then double it up to a cap, as the wrapper above does. Bash has $RANDOM for the random part; POSIX sh needs od and /dev/urandom. For a single HTTP request, curl's --retry already backs off.