Idempotent Cron Jobs: Make Every Scheduled Run Safe to Repeat

An idempotent cron job is one whose second run for the same period changes nothing: no second charge, no second email, no duplicate rows. No scheduler promises a single run, so the job makes repeats harmless: it claims each run's work under a unique key before acting, writes with upserts, and gives every outside call a key the receiver deduplicates on. The examples use PostgreSQL 18, MySQL 8.4, psycopg 3.3, and stripe-python 16.0, current on October 24, 2026.

No Scheduler Runs a Job Exactly Once#

Duplicate runs come from ordinary features:

• Retries. A Kubernetes Job retries a failed pod up to 6 times by default (Job), and Hangfire 1.8 retries a failed job 10 times (Hangfire automatic retry), each from the first line.
• At-least-once schedulers. Kubernetes creates a CronJob's Job "approximately once" per scheduled time and says Jobs "should be idempotent" (CronJob). Google Cloud Scheduler is "designed to provide 'at least once' delivery" (overview).
• Overlap, clocks, and people. A slow run still going when the next starts (overlapping cron jobs), the repeated hour when clocks go back (daylight saving time), two hosts with one crontab, a manual rerun.

Why a cron job runs twice finds which one you have. Even with every source removed, a run that dies after charging a card but before recording it leaves its restarter unable to know whether the charge happened. What you can get is an exactly-once effect: the job makes every run after the first a no-op. A lock doesn't, because it stops two runs at once, not one after another. Published schedulers split on this trade-off: Google prefers to skip a launch, while Netflix Maestro, Dropbox ATF, and Cloud Scheduler deliver at least once; the CronRadar Scheduled Job Methodology compares them and says which jobs need which.

Claim the Period Before Doing the Work#

Name the Period by the Work, Not the Clock#

The claim's key is what the run is responsible for: 2026-10-23 for a nightly job that covers yesterday, 2026-10 for monthly billing. Derive it from the scheduled time or the data covered, not from when the run starts, or a 23:55 job that starts at 00:02 claims tomorrow. Since Kubernetes 1.32, a CronJob's Jobs and Pods carry the scheduled time in the batch.kubernetes.io/cronjob-scheduled-timestamp annotation (annotation), readable through the Downward API, and Cloud Scheduler sends X-CloudScheduler-ScheduleTime. Cron passes nothing, so schedule well clear of the boundary and accept the period as an argument for reruns. A date key also gives both runs in a repeated fall-back hour one claim.

A Runs Table With Three States#

In PostgreSQL, one statement claims a new period, retries a failed one, or takes over a stale claim:

CREATE TABLE job_runs (
  job         text        NOT NULL,
  period      text        NOT NULL,
  state       text        NOT NULL CHECK (state IN ('claimed', 'done', 'failed')),
  attempt     integer     NOT NULL DEFAULT 1,
  claimed_at  timestamptz NOT NULL DEFAULT now(),
  finished_at timestamptz,
  PRIMARY KEY (job, period)
);

-- Returns a row only when this run now owns the period.
INSERT INTO job_runs AS r (job, period, state)
VALUES ('nightly-invoices', '2026-10-23', 'claimed')
ON CONFLICT (job, period) DO UPDATE
  SET state = 'claimed', attempt = r.attempt + 1,
      claimed_at = now(), finished_at = NULL
  WHERE r.state = 'failed'
     OR (r.state = 'claimed' AND r.claimed_at < now() - interval '3 hours')
RETURNING attempt;

-- After the work succeeds. Set 'failed' instead when it raises.
UPDATE job_runs SET state = 'done', finished_at = now()
WHERE job = 'nightly-invoices' AND period = '2026-10-23'
  AND attempt = 1 AND state = 'claimed';

For a done period or a recent claim, RETURNING returns nothing, since it returns "only rows that were successfully inserted or updated," and the run exits 0. ON CONFLICT DO UPDATE "guarantees an atomic INSERT or UPDATE outcome... even under high concurrency" (INSERT), and a simultaneous claim waits for the other's uncommitted row (unique checks), so the claim holds across hosts and replicas.

MySQL 8.4 has no RETURNING, so its claim takes two steps, on the same table with VARCHAR keys and DATETIME(6) times:

INSERT IGNORE INTO job_runs (job, period, state)
VALUES ('nightly-invoices', '2026-10-23', 'claimed');
SELECT ROW_COUNT(); -- 1: claimed. 0: try the takeover.

The takeover is an UPDATE setting state = 'claimed' and attempt = attempt + 1 for that job and period where the row is failed or was claimed before NOW(6) - INTERVAL 3 HOUR. A ROW_COUNT() of 1 means this run owns the period; read attempt back for the finish. IGNORE also downgrades other errors, such as a NULL in a NOT NULL column (the list), and adjusts invalid values instead of failing (INSERT). A plain INSERT that treats error 1062 (ER_DUP_ENTRY) as already claimed avoids both.

Reclaim a Stale Claim Safely#

A claim with no finish means the run crashed, was killed, or is still working, and the table can't tell which:

• Make the lease longer than any run, and enforce it with a hard limit such as timeout 2h or activeDeadlineSeconds (preventing errors in scheduled automation), or a slow run is taken over mid-work.
• Fence the finish. A run whose claim was taken over matches no row on attempt; treat that as an error.
• Expect partial work. A crashed attempt may have done half the period, so the work itself must be safe to repeat, as the sections below make it.

A done period is never reclaimed automatically; a person reruns it by setting the row to failed, and dropping r.state = 'failed' makes failed runs wait for a person too. The claim commits before the work because a charge or an email can't be rolled back; when all the work stays in one database, claim, work, and finish can share one transaction.

Process Everything Since the Last Watermark#

A job that processes "the last hour" is wrong both ways: a rerun repeats that hour, and a late run leaves a gap. A watermark records how far the job has got. Each run processes everything past it and commits the new watermark with the work, so a crash rolls back both and the first run after an outage catches up.

-- Source table: orders(id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
--   customer_id bigint, total_cents bigint, created_at timestamptz DEFAULT now())
CREATE INDEX ON orders (created_at, id);

CREATE TABLE loyalty_points (
  order_id    bigint  PRIMARY KEY,
  customer_id bigint  NOT NULL,
  points      integer NOT NULL
);

CREATE TABLE job_watermarks (
  job             text        PRIMARY KEY,
  last_created_at timestamptz NOT NULL,
  last_id         bigint      NOT NULL
);
INSERT INTO job_watermarks VALUES ('loyalty-points', '1970-01-01T00:00:00Z', 0);

# loyalty_points.py (psycopg 3.3): award points once per order.
JOB = "loyalty-points"


def award_points(conn, batch=1000, max_batches=None):
    """Process every order past the watermark and return how many it processed."""
    processed = batches = 0
    while max_batches is None or batches < max_batches:
        with conn.transaction():
            # A concurrent run waits here, then reads the watermark this run commits.
            last_at, last_id = conn.execute(
                "SELECT last_created_at, last_id FROM job_watermarks WHERE job = %s FOR UPDATE",
                (JOB,),
            ).fetchone()
            rows = conn.execute(
                """
                SELECT id, customer_id, total_cents, created_at FROM orders
                WHERE (created_at, id) > (%s, %s)
                  AND created_at < now() - interval '10 minutes'
                ORDER BY created_at, id
                LIMIT %s
                """,
                (last_at, last_id, batch),
            ).fetchall()
            if not rows:
                break
            with conn.cursor() as cur:
                cur.executemany(
                    "INSERT INTO loyalty_points (order_id, customer_id, points)"
                    " VALUES (%s, %s, %s) ON CONFLICT (order_id) DO NOTHING",
                    [(order_id, customer, cents // 100) for order_id, customer, cents, _ in rows],
                )
            conn.execute(
                "UPDATE job_watermarks SET last_created_at = %s, last_id = %s WHERE job = %s",
                (rows[-1][3], rows[-1][0], JOB),
            )
        processed += len(rows)
        batches += 1
    return processed

Open the connection with autocommit=True so each conn.transaction() block commits on its own. The watermark is the pair (created_at, id), so rows sharing a timestamp aren't skipped. FOR UPDATE serializes duplicate runs: in Read Committed, the second waits, then reads the updated row (transaction isolation).

Late-Arriving Rows#

DEFAULT now() stamps a row with its transaction's start time (current date/time), so an order from a 90-second transaction commits 90 seconds old, possibly behind a watermark that has passed it. IDs have the same gap: assigned at insert, visible at commit.

• Leave a settle delay. created_at < now() - interval '10 minutes' covers every writer transaction shorter than the delay. On PostgreSQL 17 and later, ALTER ROLE app_writer SET transaction_timeout = '5min' enforces that bound (client settings).
• Use arrival time, not event time. Events recorded elsewhere can arrive hours after their own timestamp; watermark a column the database sets on insert.
• Keep the unique key. It makes rewinding the watermark to reprocess a range safe.

Upsert on the Business Key#

A table whose only unique key is a generated ID accepts every duplicate. Make the key of the thing a row represents unique, such as an account and a day, and write so a second write lands on the same row with the same value:

-- PostgreSQL; daily_usage has PRIMARY KEY (account_id, usage_date).
INSERT INTO daily_usage (account_id, usage_date, api_calls)
SELECT account_id, DATE '2026-10-23', count(*)
FROM api_requests
WHERE requested_at >= '2026-10-23T00:00:00Z' AND requested_at < '2026-10-24T00:00:00Z'
GROUP BY account_id
ON CONFLICT (account_id, usage_date) DO UPDATE SET api_calls = EXCLUDED.api_calls;

-- MySQL 8.4
INSERT INTO daily_usage (account_id, usage_date, api_calls)
VALUES (4182, '2026-10-23', 18420) AS new
ON DUPLICATE KEY UPDATE api_calls = new.api_calls;

MySQL deprecates VALUES() in favor of the row alias and advises against ON DUPLICATE KEY UPDATE on tables with several unique indexes (manual), so make the business key the primary key.

• Set, don't add. balance = balance + 50 adds again on every run. Recompute and overwrite, or record each change as a row unique on its cause, such as (account_id, source_event_id), and sum the rows.
• Prefer ON CONFLICT to MERGE. PostgreSQL's MERGE raises a uniqueness violation when a duplicate is inserted concurrently (transaction isolation); ON CONFLICT turns that race into an update.

Give Every External Side Effect a Key#

A database rolls back; a card network and a mail server don't, so anything leaving the database needs a key the receiver, or your own table, checks first. Monitoring billing, renewal, and email jobs covers what those jobs should report and how to reconcile each run.

Payments: Derive the Key From Job, Period, and Entity#

Stripe accepts an Idempotency-Key header of up to 255 characters on every v1 POST. It replays the first response for a key to retries, "including 500 errors," and returns an error if the parameters differ (idempotent requests). Derive the key from job, period, and account, such as monthly-billing/2026-10/4182. The random key stripe-python adds "when not given" (README) protects its own network retries, not a second run of your job.

Stripe may prune a key once it's 24 hours old, after which "we generate a new request," so a late catch-up would charge again. And a decline is replayed too, so retrying a declined card on purpose needs a new key.

CREATE TABLE charges (
  account_id      bigint      NOT NULL,
  period          text        NOT NULL,
  amount_cents    bigint      NOT NULL,
  idempotency_key text        NOT NULL UNIQUE,
  state           text        NOT NULL DEFAULT 'pending',
  payment_intent  text,
  created_at      timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, period)
);

# charge_accounts.py (stripe-python 16.0). Reads invoices(account_id, period,
# amount_cents) and accounts(account_id, stripe_customer, payment_method).
import os

import stripe

client = stripe.StripeClient(os.environ["STRIPE_SECRET_KEY"], max_network_retries=2)


def charge_period(conn, period):
    # Record every intended charge first. A rerun finds the same rows and keys.
    conn.execute(
        """
        INSERT INTO charges (account_id, period, amount_cents, idempotency_key)
        SELECT account_id, %(p)s, amount_cents, 'monthly-billing/' || %(p)s || '/' || account_id
        FROM invoices WHERE period = %(p)s
        ON CONFLICT (account_id, period) DO NOTHING
        """,
        {"p": period},
    )
    # Retry pending charges only while Stripe still holds their keys.
    pending = conn.execute(
        """
        SELECT c.account_id, c.amount_cents, c.idempotency_key,
               a.stripe_customer, a.payment_method
        FROM charges c JOIN accounts a USING (account_id)
        WHERE c.period = %s AND c.state = 'pending'
          AND c.created_at > now() - interval '23 hours'
        """,
        (period,),
    ).fetchall()
    for account_id, amount, key, customer, payment_method in pending:
        try:
            intent = client.v1.payment_intents.create(
                params={
                    "amount": amount,
                    "currency": "usd",
                    "customer": customer,
                    "payment_method": payment_method,
                    "automatic_payment_methods": {"enabled": True},
                    "return_url": "https://app.example.com/billing/complete",
                    "off_session": True,
                    "confirm": True,
                    "metadata": {"billing_key": key},
                },
                options={"idempotency_key": key},
            )
            state, intent_id = intent.status, intent.id
        except stripe.CardError:
            state, intent_id = "declined", None
        conn.execute(
            "UPDATE charges SET state = %s, payment_intent = %s"
            " WHERE account_id = %s AND period = %s",
            (state, intent_id, account_id, period),
        )

The parameters follow Stripe's off-session charge example; the stored amount makes a retry repeat the original request. A row still pending after 23 hours may have reached Stripe before its run died, so before charging it, list the customer's PaymentIntents (the list endpoint filters by customer and created) and look for its billing_key metadata.

Emails and Webhooks: Write to an Outbox#

Write each message as a row in the same transaction as the work that causes it, unique on a dedupe key, and let a loop send unsent rows. A rerun's insert does nothing, and a crash leaves either no row or a row for the next pass.

# send_outbox.py: send unsent rows through Resend's HTTP API. The job inserts into
# outbox(dedupe_key text PRIMARY KEY, recipient, subject, body, created_at, sent_at)
# with ON CONFLICT (dedupe_key) DO NOTHING, using keys like 'weekly-digest/2026-W43/4182'.
import os

import requests


def send_pending(conn):
    while True:
        with conn.transaction():
            row = conn.execute(
                "SELECT dedupe_key, recipient, subject, body FROM outbox"
                " WHERE sent_at IS NULL ORDER BY created_at LIMIT 1 FOR UPDATE SKIP LOCKED"
            ).fetchone()
            if row is None:
                return
            key, recipient, subject, body = row
            response = requests.post(
                "https://api.resend.com/emails",
                headers={
                    "Authorization": f"Bearer {os.environ['RESEND_API_KEY']}",
                    "Idempotency-Key": key,
                },
                json={
                    "from": "Reports <[email protected]>",
                    "to": [recipient],
                    "subject": subject,
                    "text": body,
                },
                timeout=10,
            )
            response.raise_for_status()
            conn.execute("UPDATE outbox SET sent_at = now() WHERE dedupe_key = %s", (key,))

SKIP LOCKED lets several senders share the table (SELECT). One gap remains: a sender that dies after the provider accepts a message, but before committing, sends it again on the next pass. Resend closes it for 24 hours, answering a repeated Idempotency-Key with "the same response, without actually sending the email again" (idempotency keys); without such a key, the window stays open. For webhooks, send the dedupe key as the event ID, the role of the Standard Webhooks webhook-id, so receivers can deduplicate (spec).

When a system has no key but can be searched, say for a ticket by your external ID, check before acting, and only under the period's claim, since check-then-act races between runs.

Write Files Under a Temporary Name, Then Rename#

Name output by its period, orders-2026-10-23.csv, so a rerun replaces the file instead of adding a second. Write it under a temporary name, then rename it: rename(2) replaces an existing target atomically but fails with EXDEV across file systems (rename(2)), so keep the temporary file in the target directory.

#!/bin/sh
# /opt/exports/export-orders.sh [YYYY-MM-DD], defaulting to yesterday in UTC (GNU date).
set -eu
day=${1:-$(date -u -d yesterday +%F)}
dir=/srv/exports
tmp=$(mktemp "$dir/.orders-$day.XXXXXX")
trap 'rm -f "$tmp"' EXIT

/opt/exports/bin/export-orders --date "$day" > "$tmp"
chmod 0644 "$tmp"
mv -f "$tmp" "$dir/orders-$day.csv"

A failed export exits before the rename, and the trap removes the partial file. GNU mktemp creates files readable only by their owner (mktemp), hence the chmod; make readers skip dotfiles. On Amazon S3, PutObject with If-None-Match: * fails with 412 Precondition Failed when the key exists, and among racing writes "the first write operation to finish succeeds" (conditional writes). Naming objects by a hash of their content gives identical reruns the same name.

Test It: Run Twice, Then Compare#

Run the job twice against the same state, and interrupt a run and finish it. Each must leave exactly what one clean run leaves:

# test_idempotency.py (pytest) against a disposable database. schema.sql
# drops and recreates orders and the tables above, watermark row included.
import os
from pathlib import Path

import psycopg
import pytest

from loyalty_points import award_points

DSN = os.environ["TEST_DATABASE_URL"]


@pytest.fixture
def conn():
    with psycopg.connect(DSN, autocommit=True) as conn:
        conn.execute(Path(__file__).with_name("schema.sql").read_text())
        conn.execute(
            "INSERT INTO orders (customer_id, total_cents, created_at)"
            " SELECT mod(g, 50), 1000 + g, now() - interval '1 hour'"
            " FROM generate_series(1, 2500) AS g"
        )
        yield conn


def points(conn):
    return conn.execute("SELECT order_id, points FROM loyalty_points ORDER BY 1").fetchall()


def expected(conn):
    return conn.execute("SELECT id, total_cents / 100 FROM orders ORDER BY 1").fetchall()


def test_second_run_changes_nothing(conn):
    assert award_points(conn) == 2500
    first = points(conn)
    assert award_points(conn) == 0
    assert points(conn) == first == expected(conn)


def test_interrupted_run_resumes_without_duplicates(conn):
    award_points(conn, max_batches=1)  # What a run killed after one batch leaves.
    award_points(conn)
    assert points(conn) == expected(conn)

Run two copies at once on separate connections, too: their processed counts must sum to 2,500. For external effects, swap in a fake client that records its calls, and assert that the second run made none or reused the first run's keys.

Watch for Runs That Did Nothing#

CronRadar doesn't flag duplicate runs; its failure modes are missed or late, failed, stuck, disappeared, and bad outcome. It catches what these patterns can hide: a run that exits 0 having done nothing because of an unexpected claim, left by a stuck run or a key built from the wrong date. Send check-ins only from the run holding the claim, with records_processed on complete. A skipped duplicate then stays quiet, a scheduled run that never gets the claim shows as missed after the grace period, and an expectation of records_processed >= 1 turns a run that claimed but processed nothing into an incident (silent success). There is no installable CronRadar SDK for Python today, so a Python job reports with its own HTTP check-ins.

Frequently Asked Questions#

What does idempotent mean for a cron job?#

Running it again for the same period has no further effect. The second run finds the work recorded and exits, or rewrites the same rows with the same values, without a second charge or email.

How do I stop a cron job from processing the same data twice?#

For work tied to a period, claim the period in a table keyed on job and period, and exit when the claim exists. For a stream of rows, process from a watermark committed with the work. Either way, keep a unique key on the output.

Can a scheduler guarantee a job runs exactly once?#

No. Kubernetes creates a CronJob's Job approximately once per scheduled time, and Cloud Scheduler promises at least once. A scheduler restarting a crashed run can't know how much of it finished, so exactly-once effects come from a job that makes repeats harmless.

What is a watermark in batch processing?#

A stored position, such as the timestamp and ID of the last row processed. Each run processes rows past it and advances it in the same transaction as the work, so a crash or rerun resumes in the right place. Skip the newest few minutes, since rows can commit out of timestamp order.

How do I test that a scheduled job is idempotent?#

Run it twice against the same data and assert that the second run changes nothing. Then interrupt a run and finish it, and run two copies at once; each must leave what one clean run leaves. For external calls, a fake client shows whether the second run calls again.