# Kubernetes CronJob Failure Handling: Retries, Deadlines, and Alerts

By Dan Ochoa · Published 2026-11-16 · Updated 2026-11-16
Canonical: https://cronradar.com/blog/kubernetes-cronjob-failure-handling

> How a Kubernetes CronJob fails: backoffLimit, restartPolicy, activeDeadlineSeconds, podFailurePolicy, history limits, TTL, and reporting failures over HTTP.

**A failed CronJob run is a Job whose Pods kept failing until `backoffLimit` was reached, or that ran past `activeDeadlineSeconds`, or that matched a `FailJob` rule in its `podFailurePolicy`.** Kubernetes then marks the Job `Failed`, keeps it according to `failedJobsHistoryLimit` and `ttlSecondsAfterFinished`, and starts the next run on schedule. It doesn't notify anyone. Retries, deadlines, and cleanup are yours to tune; the alert has to come from somewhere else.

This guide covers `batch/v1` Jobs created by CronJobs on Kubernetes 1.31 and later, checked against the [Job documentation](https://kubernetes.io/docs/concepts/workloads/controllers/job/) on November 16, 2026. For CronJobs that never start, see [Kubernetes CronJob not running](https://cronradar.com/blog/kubernetes-cronjob-not-running).

## How a Run Fails

A CronJob creates a Job per scheduled time; the Job creates Pods; a container's exit code decides each attempt. Exit 0 is success. Anything else, or a container killed for exceeding its memory limit, is a failed attempt, and the Job decides whether to try again.

A complete jobTemplate with every field this guide discusses:

```yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: invoice-export
spec:
  schedule: '15 3 * * *'
  timeZone: 'Etc/UTC'
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      backoffLimit: 2 # up to 3 attempts in total
      activeDeadlineSeconds: 3600 # the whole Job, all attempts
      ttlSecondsAfterFinished: 604800
      podFailurePolicy:
        rules:
          - action: FailJob # a config error won't fix itself on retry
            onExitCodes:
              containerName: export
              operator: In
              values: [2]
          - action: Ignore # node drains and preemption don't count
            onPodConditions:
              - type: DisruptionTarget
      template:
        spec:
          restartPolicy: Never # required by podFailurePolicy
          containers:
            - name: export
              image: registry.example.com/invoice-export:3.2.0
              args: ['--since', 'yesterday']
```

## restartPolicy: OnFailure or Never

The Pod template must set `restartPolicy` to `OnFailure` or `Never` for a Job. The choice changes where retries happen and what evidence survives:

|                                   | `OnFailure`                                                | `Never`                                             |
| --------------------------------- | ---------------------------------------------------------- | --------------------------------------------------- |
| A failed attempt                  | The kubelet restarts the container in the same Pod         | The Pod fails; the Job controller creates a new one |
| What counts toward `backoffLimit` | Container restarts in Pods that are `Pending` or `Running` | Pods in phase `Failed`                              |
| At the backoff limit              | The Pod is terminated                                      | Failed Pods stay until the Job is deleted           |
| Logs of failed tries              | Only the previous attempt, via `kubectl logs --previous`   | One Pod per attempt, each with its own logs         |
| `podFailurePolicy`                | Not allowed                                                | Allowed                                             |

The Job docs warn that with `OnFailure`, the Pod "will be terminated once the job backoff limit has been reached," which "can make debugging the Job's executable more difficult," and suggest `Never` when debugging ([pod backoff failure policy](https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy)). For scheduled jobs, `Never` is usually the better default: each attempt leaves a Pod to inspect, and you can use `podFailurePolicy`.

Either way, the same program can run more than once for one scheduled time. The docs say so even with `parallelism: 1`, `completions: 1`, and `restartPolicy: Never`. Make the job safe to rerun; [idempotent scheduled jobs](https://cronradar.com/blog/idempotent-scheduled-jobs) covers how.

## backoffLimit: How Many Attempts

`backoffLimit` is the number of retries before the Job is considered failed. It defaults to 6, so a failing job is attempted seven times. Failed Pods are recreated "with an exponential back-off delay (10s, 20s, 40s ...)"; the current docs cap it at ten minutes. Six retries at the default delays take over ten minutes before the Job finally fails.

When the limit is reached, the Job gets a `Failed` condition with reason `BackoffLimitExceeded`, and any running Pods are terminated. Check it:

```bash
kubectl get job invoice-export-29384955 \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
```

Choose the limit from the failure you expect. A job that calls a flaky API benefits from two or three retries. A job that fails because a file is malformed fails the same way seven times and delays the signal; set a low limit or fail it fast with `podFailurePolicy`. Retrying is also only safe if a half-finished attempt can be repeated, which [cron job retries](https://cronradar.com/blog/cron-job-retries) discusses outside Kubernetes.

## activeDeadlineSeconds: A Limit for the Whole Job

`.spec.activeDeadlineSeconds` "applies to the duration of the job, no matter how many Pods are created." When it's reached, running Pods are terminated and the Job fails with reason `DeadlineExceeded`. It "takes precedence over" `backoffLimit`: a Job still retrying stops creating Pods once the deadline passes ([Job termination and cleanup](https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup)).

Two details matter:

- **Put it on the Job spec.** The Pod spec has a field with the same name that limits one Pod, not the Job. In a CronJob, the Job spec is `spec.jobTemplate.spec`.
- **Size it to the schedule.** Under `concurrencyPolicy: Forbid`, a Job that hangs holds every later run. A deadline shorter than the interval means one hang costs one run. [Cron job timeouts](https://cronradar.com/blog/cron-job-timeout) covers the same idea for crontab jobs.

Terminated Pods get SIGTERM, then SIGKILL after `terminationGracePeriodSeconds` (30 seconds by default). A shell running as the container's PID 1 ignores SIGTERM unless it sets a trap, and doesn't pass it to the job, so expect the kill after the grace period, not a clean exit.

## podFailurePolicy: Decide by Exit Code and Cause

`podFailurePolicy` lets the Job react to why a Pod failed, by container exit code or Pod condition. It's stable since Kubernetes 1.31 and requires `restartPolicy: Never`. Rules are evaluated in order, the first match wins, and unmatched failures get the default handling ([pod failure policy](https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-failure-policy)):

| Action      | Effect                                                               |
| ----------- | -------------------------------------------------------------------- |
| `FailJob`   | Mark the Job failed and terminate its running Pods                   |
| `Ignore`    | Don't count this failure toward `backoffLimit`; create a replacement |
| `Count`     | The default handling: count it toward `backoffLimit`                 |
| `FailIndex` | For Indexed Jobs with `backoffLimitPerIndex`; fail just that index   |

The example above uses both common rules. Exit code 2, which the job uses for bad configuration, fails the Job at once instead of retrying a mistake. A `DisruptionTarget` condition, which Kubernetes sets on Pods evicted by a drain, preemption, or taint, is ignored, so a node upgrade at 03:15 doesn't burn a retry. A Job failed by a `FailJob` rule gets a `FailureTarget` condition first and `Failed` once its Pods have terminated.

This only works if the job's exit codes mean something. A script that exits 1 for every error gives the policy nothing to match; [cron job exit codes](https://cronradar.com/blog/cron-job-exit-codes) covers choosing them.

## History Limits and TTL: How Long the Evidence Lasts

A failed Job and its Pods are the record of what went wrong, and two mechanisms delete them:

- **CronJob history limits.** `successfulJobsHistoryLimit` (default 3) and `failedJobsHistoryLimit` (default 1) set how many finished Jobs the CronJob keeps. With the default, a second failure deletes the first one's Job, Pods, and logs.
- **`ttlSecondsAfterFinished`.** The TTL controller deletes a finished Job, `Complete` or `Failed`, and its Pods that many seconds after it finishes. `0` deletes it immediately. Unset means the TTL controller leaves it alone ([TTL mechanism](https://kubernetes.io/docs/concepts/workloads/controllers/job/#ttl-mechanism-for-finished-jobs)).

Whichever comes first wins. A `ttlSecondsAfterFinished: 0` copied from a one-off Job template erases each failed run the moment it fails. For scheduled work, keep several failed Jobs and a TTL measured in days, and ship container logs off the cluster if you need more than that.

## Why Nothing Alerts by Default

When a Job fails, Kubernetes updates its status, records events, and moves on. The CronJob's next run is created on schedule regardless, and `.status.lastSuccessfulTime` on the CronJob simply stops advancing. None of this pushes a notification. Events last one hour by default, so the record a person might find later is the Job object, if the history limit and TTL haven't deleted it yet.

In-cluster options exist: kube-state-metrics exposes `kube_job_failed` and Prometheus can alert on it, as the [Kubernetes CronJob monitoring](https://cronradar.com/blog/kubernetes-cronjob-monitoring) guide shows. They depend on the cluster's monitoring being up, and they don't see a run that was never created.

## Report Start, Complete, and Fail Over HTTP

Reporting from the container covers both failures and missing runs: the job tells an outside service when it starts and how it ended, and that service alerts when a report is a failure or doesn't arrive. CronRadar has no Kubernetes integration and doesn't discover CronJobs, so each CronJob reports over HTTP with its own monitor key and schedule.

Wrap the job's command in a small script and keep the job's exit code, so Kubernetes still sees the real result and `backoffLimit` and `podFailurePolicy` behave as before. Each report gives up after five seconds and ignores errors, so a monitoring outage can't fail the run:

```yaml
template:
  spec:
    restartPolicy: Never
    containers:
      - name: export
        image: registry.example.com/invoice-export:3.2.0 # needs sh and curl
        env:
          - name: CRONRADAR_API_KEY
            valueFrom:
              secretKeyRef:
                name: cronradar
                key: api-key
          # The Job's name: one per scheduled run, shared by its retries
          - name: RUN_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.labels['batch.kubernetes.io/job-name']
        command:
          - /bin/sh
          - -c
          - |
            report() {
              event=$1
              shift
              curl --fail --silent --show-error --max-time 5 --output /dev/null \
                --request POST --get --user "$CRONRADAR_API_KEY:" \
                --data-urlencode "schedule=15 3 * * *" \
                --data-urlencode "gracePeriod=1800" \
                --data-urlencode "run_id=$RUN_ID" \
                "$@" "https://cron.life/api/v1/ping/invoice-export/$event" || true
            }

            report start
            /app/invoice-export --since yesterday
            rc=$?
            if [ "$rc" -eq 0 ]; then
              report complete
            else
              report fail --data-urlencode "message=invoice-export exited with code $rc"
            fi
            exit "$rc"
```

Store the application's API key in a Secret, never in the manifest, and use a separate CronRadar application and key per cluster environment. The `run_id` is the Job's name, which every retry Pod of one scheduled run shares, so the attempts of one run group together. Because the reports carry the schedule, the first run creates the monitor; keep sending it so a deleted monitor re-creates itself.

How each failure shows up:

- **A failed attempt** sends `fail`, which alerts immediately with the exit code in the message. With `backoffLimit` above 0, that's the first failed attempt, not the final one, because the container can't tell which attempt is last. If you only want to hear about a run whose retries all failed, drop the `fail` report: the run never sends `complete`, and the monitor alerts as missed after the expected time plus the grace period.
- **A deadline or eviction** kills the container before it reports, so the last report is `start`, and CronRadar flags the run as stuck.
- **A run that never starts**, from a suspended CronJob to a quota rejection, sends nothing. After the expected time plus grace, the monitor goes from `warning` to `critical` and alerts by email, Slack, webhook, Discord, or PagerDuty.

CronRadar reads the schedule in the application's time zone, an IANA name in the application's settings (UTC when unset), so match it to the CronJob's `timeZone`.

## Frequently Asked Questions

### What is the default backoffLimit for a Kubernetes Job?

Six retries, so seven attempts in total, with an exponential delay of 10, 20, 40 seconds and so on between them. When the limit is reached, the Job fails with reason `BackoffLimitExceeded`. Set it per CronJob in `spec.jobTemplate.spec.backoffLimit`.

### Should a CronJob use restartPolicy OnFailure or Never?

`Never` is usually better for scheduled jobs. Each failed attempt gets its own Pod with its own logs, and `podFailurePolicy` requires it. `OnFailure` retries in the same Pod and terminates that Pod at the backoff limit, which can leave only the last attempt's logs.

### Does activeDeadlineSeconds include retries?

Yes. The Job-level `activeDeadlineSeconds` covers the whole Job, however many Pods it creates, and takes precedence over `backoffLimit`. When it's reached, running Pods are terminated and the Job fails with reason `DeadlineExceeded`.

### Why did my failed Job disappear?

The CronJob keeps only `failedJobsHistoryLimit` failed Jobs, 1 by default, and `ttlSecondsAfterFinished` deletes finished Jobs after that many seconds. Raise the history limit, lengthen or remove the TTL, and ship logs off the cluster.

### How do I get alerted when a Kubernetes CronJob fails?

Kubernetes records the failure but doesn't send it anywhere. Alert on `kube_job_failed` from kube-state-metrics in Prometheus, or have the container report its start and result over HTTP to a monitor outside the cluster, which also catches runs that never started.
