Python scheduler monitoring: APScheduler failures and missed runs

To monitor a Python scheduler such as APScheduler, check that its process is alive, that job processing isn't paused, and that an earlier run isn't still occupying the job's slot, then watch from outside the process for an expected run that never arrives. scheduler.running alone cannot tell you whether yesterday's backup succeeded.

This guide covers APScheduler 3.x and synchronous Python jobs. APScheduler already provides logging and execution events. Use them to explain failures inside the process, then use an external monitor to detect an expected run that never arrives. Start with the checks below; the final example connects a daily backup to CronRadar over HTTP. For Celery beat, see Celery monitoring; for django-apscheduler and other Django schedulers, see Django cron monitoring; for a script that system cron starts, see Python script cron jobs. For the lighter schedule library and its run_pending() loop, see the Python schedule library in production.

Why BackgroundScheduler stops when your script exits#

BackgroundScheduler.start() returns immediately. Its scheduler thread defaults to daemon=True, so that thread does not keep the Python process alive. Starting it at the end of a short script leaves no host for tomorrow's job. BackgroundScheduler reference.

Use BackgroundScheduler when an existing application owns the process lifetime. For a dedicated scheduling worker, BlockingScheduler.start() keeps the worker in the scheduler's main loop. Setting daemon=False changes thread lifetime; it does not provide process supervision or recovery after a crash.

Run one scheduling process for the same job set. Starting a scheduler in each web worker can duplicate executions, and APScheduler 3.x does not coordinate multiple schedulers through a shared persistent job store. The APScheduler FAQ recommends a separate scheduling process for that situation. APScheduler running twice shows how Gunicorn and Uvicorn workers, the Flask and Django reloaders, and --preload each change the number of schedulers, and what each fix costs.

What scheduler.running actually tells you#

In APScheduler 3.x, scheduler.running means its state is not STATE_STOPPED. A paused scheduler also returns True. Check scheduler.state to distinguish started-and-processing from paused. Neither value proves a particular job completed. Scheduler state reference.

Run this against your existing scheduler:

from apscheduler.schedulers.base import STATE_RUNNING

print("Processing scheduled jobs:", scheduler.state == STATE_RUNNING)
for job in scheduler.get_jobs():
    print(job.id, "next run:", getattr(job, "next_run_time", None))

A pending job can lack next_run_time before the scheduler starts; a paused job has no next run. Compare those diagnostics with the last actual completion, rather than returning a successful health check just because jobs are registered.

Overlaps and misfires need different checks#

APScheduler 3.x defaults to max_instances=1 per job. If the previous instance is still running, the executor rejects another submission for that job. Raising the limit allows overlap; it is appropriate only when the work can safely run concurrently. This limit does not prevent a separate scheduler process from starting the same work. Scheduler defaults and submission handling.

misfire_grace_time is how late a scheduled execution may begin. For example, 300 allows five minutes of lateness. It does not set a maximum job duration or override the concurrency limit. coalesce=True combines multiple due times into one execution; the bypassed times do not each produce a missed-run event. Choose these settings to match whether catching up is useful for your job. Misfires and coalescing.

Keep the listener branches separate:

The first three use JobExecutionEvent; the last uses JobSubmissionEvent. Reading event.exception before checking the type can break a listener when an overlap occurs. Event types and attributes.

import logging
from apscheduler.events import (
    EVENT_JOB_EXECUTED,
    EVENT_JOB_ERROR,
    EVENT_JOB_MISSED,
    EVENT_JOB_MAX_INSTANCES,
)

log = logging.getLogger("scheduled_jobs")

def job_listener(event):
    if event.code == EVENT_JOB_MAX_INSTANCES:
        log.warning("%s overlap: %s", event.job_id, event.scheduled_run_times)
    elif event.code == EVENT_JOB_MISSED:
        log.warning("%s missed: %s", event.job_id, event.scheduled_run_time)
    elif event.code == EVENT_JOB_ERROR:
        log.error("%s failed (%s)", event.job_id, type(event.exception).__name__)
    elif event.code == EVENT_JOB_EXECUTED:
        log.info("%s completed: %s", event.job_id, event.scheduled_run_time)

EVENT_MASK = (
    EVENT_JOB_EXECUTED | EVENT_JOB_ERROR |
    EVENT_JOB_MISSED | EVENT_JOB_MAX_INSTANCES
)

Attach this listener before starting the scheduler, as shown below. It keeps diagnostics local and avoids sending return values or exception contents to a monitoring service. It cannot emit an event after the entire process has stopped.

Monitor a scheduled Python job with CronRadar#

Create an application and an ingestion key using the getting-started guide. Store the key in the worker's CRONRADAR_API_KEY environment variable through your deployment's secret manager. An assistant's temporary management token is a different credential and cannot replace that ingestion key.

This example uses the CronRadar ingestion API directly, because Python SDK connections are temporarily unavailable; the Python integration docs carry the current notice. It uses HTTP Basic authentication, an empty password, five-second request timeouts and one run_id shared by the start and terminal event. It sends no job arguments, results or exception messages. Keep the listener above in the same file and replace the jobs import with your existing backup function.

import base64
import logging
import os
import urllib.parse
import urllib.request
import uuid
from jobs import backup_database

MONITOR_KEY = "nightly-backup"
SCHEDULE = "0 2 * * *"  # Daily at 02:00 UTC
log = logging.getLogger("scheduled_jobs")

class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None  # Never forward the Authorization header to another URL.

http = urllib.request.build_opener(NoRedirect())

def send_event(event, run_id):
    try:
        api_key = os.environ["CRONRADAR_API_KEY"]
        auth = base64.b64encode(f"{api_key}:".encode()).decode()
        query = urllib.parse.urlencode({
            "schedule": SCHEDULE,
            "gracePeriod": 300,
            "run_id": run_id,
        })
        key = urllib.parse.quote(MONITOR_KEY, safe="")
        request = urllib.request.Request(
            f"https://cron.life/api/v1/ping/{key}/{event}?{query}",
            data=b"",
            method="POST",
            headers={
                "Authorization": f"Basic {auth}",
                "User-Agent": "CronRadar-APScheduler-Guide/1.0",
            },
        )
        with http.open(request, timeout=5) as response:
            if not 200 <= response.status < 300:
                raise RuntimeError("Check-in rejected")
    except Exception as error:
        # A telemetry failure must not change the backup's result.
        log.warning("CronRadar %s not recorded (%s)", event, type(error).__name__)

def monitored_backup():
    run_id = uuid.uuid4().hex
    send_event("start", run_id)
    try:
        result = backup_database()
    except BaseException:
        send_event("fail", run_id)
        raise
    send_event("complete", run_id)
    return result

The wrapper returns the original result or re-raises the original job exception. HTTP errors and missing monitoring configuration are logged without preventing the backup from running. The calls are synchronous and can add request latency; there is no retry loop in the job.

Run the example in a dedicated APScheduler 3.x worker. This block goes at the end of the same file, after the listener and wrapper:

# Uses job_listener, EVENT_MASK, monitored_backup, MONITOR_KEY and SCHEDULE from above.
import logging

from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.triggers.cron import CronTrigger

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    scheduler = BlockingScheduler(timezone="UTC")
    scheduler.add_listener(job_listener, EVENT_MASK)
    scheduler.add_job(
        monitored_backup,
        CronTrigger.from_crontab(SCHEDULE, timezone="UTC"),
        id=MONITOR_KEY,
        replace_existing=True,
        max_instances=1,
        misfire_grace_time=300,
        coalesce=True,
    )
    try:
        scheduler.start()
    except (KeyboardInterrupt, SystemExit):
        if scheduler.running:
            scheduler.shutdown(wait=True)

The example uses APScheduler's in-memory job store and thread executor. If you add a persistent store or process executor, put the scheduled callable in an importable module and check its serialization requirements. Keep the worker's schedule and the monitor's expected schedule aligned. CronRadar's gracePeriod determines when a missing check-in is late; APScheduler's misfire_grace_time determines whether late work should execute. They are separate decisions, even though this example uses 300 seconds for both.

Verify the job, then the missing-run alert#

The first accepted check-in can create this monitor because it includes a schedule. Until a monitor exists, CronRadar cannot know that this job should run. Create it in advance through the dashboard or an assistant if you need coverage before the first scheduled execution.

Use a test application to confirm a real completion appears, a deliberately failed test job still raises its original error, and a missed scheduled run produces the expected alert. Do not send a fake success to make the dashboard look healthy. A missing check-in indicates that the expected signal did not arrive; check the worker and network to determine why. How to know if a cron job ran compares this with log-based checks, and the dead man's switch guide explains why the alert has to come from outside the job.

For other Python jobs, see Python monitoring. To have an assistant inspect your project and prepare the monitoring setup, use the agent quickstart. Keep the scheduler responsible for running work and CronRadar responsible for telling you when an expected run is missing or failed.

Frequently Asked Questions#

Why is my APScheduler job not running?#

The two most common causes in the APScheduler FAQ are a BackgroundScheduler whose script reaches its end and exits, and a scheduler running in a uWSGI worker without threads enabled, which needs the --enable-threads switch. Also check scheduler.state for a paused scheduler and the EVENT_JOB_MAX_INSTANCES event for a run rejected because the previous one is still going.

Why does APScheduler run my job twice?#

Usually because more than one scheduler is running the same jobs, for example one in each web server worker process. APScheduler 3.x has no coordination between processes, and its FAQ warns that sharing a persistent job store between processes leads to duplicate or missed executions. Run the scheduler in one dedicated process.

Is scheduler.running a good health check?#

No. It stays True while the scheduler is paused, and it says nothing about whether a particular job ran or succeeded. Use scheduler.state for the scheduler, and an external monitor that expects each job's check-in on schedule for the jobs.

Is there a CronRadar Python SDK?#

Python SDK connections are temporarily unavailable, so this guide reports to CronRadar over HTTP. That setup doesn't discover jobs automatically: give each job its own monitor key and real schedule. The Python integration docs carry the current availability notice.