Celery Task Monitoring: The Complete Guide

To monitor Celery in production, watch four things: whether tasks succeed, whether workers are alive, whether queues are backing up, and whether scheduled tasks actually run. Flower shows live task and worker state. A Prometheus exporter turns Celery's events into alerts for failed tasks, offline workers, and growing queues. An error tracker such as Sentry tells you why a task failed. A schedule-aware heartbeat monitor catches the failure none of them can see: a Celery beat task that never ran at all.

No single tool covers all four. This guide sets up each layer, starting with the failure modes that make Celery harder to monitor than it looks.

Why Celery monitoring is harder than it looks#

Tasks can disappear without a failure#

By default, Celery acknowledges a task message just before the task runs. If the worker process is then killed, by the out-of-memory killer or a KILL signal, the broker has already forgotten the message and the task doesn't run again. Celery's workers guide says that when a worker is killed, "currently executing tasks will be lost (i.e., unless the tasks have the acks_late option set)."

Late acknowledgement changes less than its name suggests:

Enabling task_reject_on_worker_lost re-queues the task instead, and the configuration reference warns that it "can cause message loops". Use it only for tasks that are safe to run twice. Celery also doesn't retry a failed task unless you ask it to with self.retry() or autoretry_for; by default, no exceptions are retried. Celery task retries covers the retry defaults, backoff and jitter, and which signal fires only when Celery gives up.

Code inside the task can only report what happens while its process is alive. An except block can't report a process that was killed, including by a hard time limit, which kills and replaces the worker process.

The beat scheduler blind spot#

Celery beat sends scheduled tasks to the queue, and Celery's docs say you have to ensure only a single scheduler is running for a schedule at a time, or tasks are duplicated. So beat is usually one process. If it stops, every periodic task stops with it, and no task fails.

Celery's monitoring events cover tasks and workers only; beat sends none. Flower and event-based exporters therefore can't tell you whether beat is running. Your workers look healthy, your queues look empty, and the hourly sync hasn't run since yesterday. Celery beat not running tasks walks through finding the cause.

Setting up Flower for real-time visibility#

Flower is a web UI for Celery that shows task progress, worker status, and task history from Celery's events.

Installation and basic setup#

pip install flower

# Flower reads options from FLOWER_-prefixed environment variables,
# which keeps the password out of the process list.
export FLOWER_BASIC_AUTH='admin:a-long-random-password'
celery -A your_project flower --port=5555

Flower can revoke tasks and change worker pools, so serve it over HTTPS and restrict who can reach it. For Docker deployments:

# docker-compose.yml
services:
  flower:
    image: mher/flower
    command: celery --broker=redis://redis:6379/0 flower
    ports:
      - '5555:5555'
    environment:
      - FLOWER_BASIC_AUTH=${FLOWER_BASIC_AUTH}
    depends_on:
      - redis

Enabling Celery events#

Flower and the Prometheus exporters below are built on Celery's event system. All three settings are disabled by default:

# celery.py
app.conf.update(
    worker_send_task_events=True,  # Task events for Flower and exporters
    task_send_sent_event=True,     # task-sent events, needed for queue wait time
    task_track_started=True,       # STARTED state in the result backend
)

Without task_send_sent_event, you can't measure how long tasks wait in the queue before a worker picks them up.

What Flower can and cannot do#

Flower is good at:

• Live task progress, history, and details
• Worker status, plus remote control of pool size, autoscaling, consumed queues, and task revocation
• An HTTP API and a Prometheus /metrics endpoint

Flower's limitations:

• No built-in alerting; you alert through Prometheus or watch the dashboard
• State is held in memory (up to max_tasks, 100,000 by default) and lost on restart unless you enable --persistent
• It sees only events sent while it is running
• It can't show whether beat is running

Flower is the best tool for debugging a live system. For alerting, you need more. See the Flower configuration reference for these options.

Prometheus metrics for production alerting#

Prometheus and Grafana add history, dashboards, and alert rules. There are two common ways to export Celery metrics.

Option 1: Flower's built-in Prometheus endpoint#

Flower serves metrics at /metrics as soon as it is running; no flag is needed. It exports worker and task metrics such as flower_worker_online, flower_events_total, and flower_task_runtime_seconds. See Flower's Prometheus integration.

Option 2: A dedicated Celery exporter#

celery-exporter runs independently of Flower and adds queue lengths and queue wait times. It reads the same events, so enable the settings above:

# docker-compose.yml
services:
  celery-exporter:
    image: danihodovic/celery-exporter
    environment:
      - CE_BROKER_URL=redis://redis:6379/0
    ports:
      - '9808:9808'

Key metrics to alert on#

These rules use celery-exporter's metric and label names:

# prometheus/alerts.yml
groups:
  - name: celery
    rules:
      # A task failed in the last 10 minutes
      - alert: CeleryTaskFailed
        expr: sum by (name) (increase(celery_task_failed_total[10m])) > 0
        labels:
          severity: warning
        annotations:
          summary: 'Celery task {{ $labels.name }} failed'

      # A worker stopped sending heartbeats
      - alert: CeleryWorkerDown
        expr: celery_worker_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: 'Celery worker {{ $labels.hostname }} is offline'

      # A queue is backing up
      - alert: CeleryQueueBacklog
        expr: sum by (queue_name) (celery_queue_length) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Celery queue {{ $labels.queue_name }} has {{ $value }} waiting tasks'

      # More than 5% of tasks are failing
      - alert: CeleryHighFailureRate
        expr: |
          sum(increase(celery_task_failed_total[10m]))
          /
          (sum(increase(celery_task_failed_total[10m])) + sum(increase(celery_task_succeeded_total[10m])))
          > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Celery task failure rate is {{ $value | humanizePercentage }}'

The exporter's repository also ships example alert rules and Grafana dashboards in its celery-mixin directory.

The metric that warns you first#

Queue wait time, from the moment a task is sent until a worker starts it, is the clearest sign that workers can't keep up. celery-exporter records it as the celery_task_queue_wait_time_bucket histogram, which requires task_send_sent_event=True. To graph or alert on the 95th percentile:

histogram_quantile(0.95, sum by (le) (rate(celery_task_queue_wait_time_bucket[5m])))

If wait time keeps rising, add workers or make tasks faster before the backlog becomes an outage.

Catching errors with Sentry#

Prometheus tells you how many tasks failed. An error tracker tells you why.

Basic setup#

pip install sentry-sdk

With the celery package installed, Sentry's Python SDK enables its Celery integration automatically and reports exceptions raised in tasks. The SDK must be initialized when the worker starts. In Django, initialize it once in settings.py; otherwise, use the celeryd_init signal:

# celery.py
import os

import sentry_sdk
from celery import Celery, signals

app = Celery("your_project")

@signals.celeryd_init.connect
def init_sentry(**kwargs):
    sentry_sdk.init(
        dsn=os.environ["SENTRY_DSN"],
        traces_sample_rate=0.1,  # Optional tracing; adjust to your volume
    )

Sentry Crons for beat tasks#

Sentry Crons can create monitors for your beat tasks automatically. Set monitor_beat_tasks=True on CeleryIntegration, and initialize the SDK in the beat process as well as the worker:

# celery.py
import os

import sentry_sdk
from celery import Celery, signals
from sentry_sdk.integrations.celery import CeleryIntegration

app = Celery("your_project")

@signals.beat_init.connect
@signals.celeryd_init.connect
def init_sentry(**kwargs):
    sentry_sdk.init(
        dsn=os.environ["SENTRY_DSN"],
        integrations=[CeleryIntegration(monitor_beat_tasks=True)],
    )

Sentry documents support for the default beat scheduler and RedBeat, for schedules it can parse as crontab. To monitor one task instead, add @sentry_sdk.monitor(monitor_slug="daily-report") below Celery's @app.task decorator. See Sentry's Celery Crons guide. Sentry Crons vs a dedicated cron monitor compares check-ins, time zones, alerts, and pricing.

Detecting when scheduled tasks don't run#

None of the tools above fully answers one question: how do you know when a task that should run every hour hasn't run at all?

Flower shows tasks that executed. Prometheus counts tasks that ran. Sentry captures errors from tasks that failed. If beat stops, or the message never reaches a worker, these tools show nothing, and nothing alerts.

Dead man's switch monitoring, also called heartbeat monitoring, covers that gap.

How dead man's switch monitoring works#

Instead of watching for failures, you watch for the absence of success:

1. Tell the monitor when the task should run, for example every hour.
2. The task checks in when it completes successfully.
3. If a check-in doesn't arrive on time, you get alerted.

The monitor doesn't need to know why the task didn't run. Beat stopped, workers died, the broker was down, or a deploy removed the schedule: you get alerted either way. The dead man's switch guide covers the pattern in more detail.

Reporting Celery tasks to CronRadar over HTTP#

A CronRadar Celery package, cronradar-celery (set up with setup_cronradar(app)), is on the way, but Celery SDK connections are temporarily unavailable. Use manual HTTP reporting for now; it does not discover scheduled tasks automatically, so each task needs its own monitor key and its real schedule. The Celery integration docs carry the current availability notice.

Store the ingestion key in CRONRADAR_API_KEY through your secret manager, using a separate CronRadar application and key for each environment. A single heartbeat after each success is enough to catch missed runs. Reporting a start, a completion, and a failure from Celery's task signals adds explicit failures and stuck-run detection without changing task code. The first report that includes a schedule creates the monitor.

Start with a beat schedule. CronRadar evaluates schedules in UTC, which is also Celery's default timezone:

# celery.py
from celery import Celery
from celery.schedules import crontab

app = Celery("your_project")
app.conf.timezone = "UTC"
app.conf.beat_schedule = {
    "send-invoices": {
        "task": "billing.tasks.send_invoices",
        "schedule": crontab(minute=0, hour=6),  # 0 6 * * *
    },
    "nightly-report": {
        "task": "reports.tasks.nightly_report",
        "schedule": crontab(minute=30, hour=2),  # 30 2 * * *
    },
}

# Import the signal handlers below when the worker starts.
app.conf.imports = ["monitoring"]

Then map each task to a monitor key and the same schedule as a cron expression:

# monitoring.py
import logging
import os
from urllib.parse import quote

import requests
from celery.signals import task_failure, task_prerun, task_success

log = logging.getLogger(__name__)

# Celery task name -> (CronRadar monitor key, cron schedule in UTC).
# Keep each schedule identical to the task's beat_schedule entry.
MONITORED_TASKS = {
    "billing.tasks.send_invoices": ("send-invoices", "0 6 * * *"),
    "reports.tasks.nightly_report": ("nightly-report", "30 2 * * *"),
}

def report(task, event, **params):
    monitor = MONITORED_TASKS.get(task.name)
    if monitor is None:
        return
    key, schedule = monitor
    try:
        response = requests.post(
            f"https://cron.life/api/v1/ping/{quote(key, safe='')}/{event}",
            params={
                "schedule": schedule,
                "run_id": task.request.id,  # The task ID stays the same across retries
                "attempt": task.request.retries,  # Zero-based retry count
                **params,
            },
            auth=(os.environ["CRONRADAR_API_KEY"], ""),
            timeout=5,
        )
        response.raise_for_status()
    except Exception as error:
        # A monitoring problem must never fail or change the task.
        log.warning("CronRadar %s for %s not recorded (%s)", event, key, type(error).__name__)

@task_prerun.connect
def on_start(sender=None, **kwargs):
    report(sender, "start")

@task_success.connect
def on_complete(sender=None, **kwargs):
    report(sender, "complete")

@task_failure.connect
def on_fail(sender=None, exception=None, **kwargs):
    # Send the exception type only: never arguments, payloads, or log output.
    report(sender, "fail", message=type(exception).__name__)

How this behaves:

• Task code is unchanged. The handlers observe; the task's own exception still propagates, and Celery's retry behavior is untouched. A retry that raises Retry sends no failure; only the final failure does.
• Monitoring errors are logged, not raised. Each request has a five-second timeout, and a missing key, network error, or rejected request only writes a warning. The calls are synchronous, so each one can add up to five seconds to the task.
• Each run is correlated. Celery's task ID becomes the run_id, shared by the start and the terminal event, and the retry count becomes attempt. See the HTTP API reference for both parameters.
• Every run of a mapped task reports, including one you start by hand with .delay(). Map only tasks that run on a schedule.
• The module must load in the worker. Celery's imports setting loads it when the worker starts.

Together these catch the failures the other layers miss. If beat or the workers stop, no start arrives and the monitor alerts once the grace period passes. If a worker process is killed mid-task, the start has no matching completion, so the run is flagged instead of passing silently. If the task raises, the failure opens an incident immediately. Alerts go to email, Slack, a webhook, Discord, or PagerDuty.

Docker and Kubernetes health checks#

Container orchestrators need to know when a worker is unhealthy.

Docker health check#

celery inspect ping asks the worker to reply through the broker, which makes it a real end-to-end check:

# Dockerfile
HEALTHCHECK --interval=30s --timeout=15s --start-period=30s --retries=3 \
  CMD celery -A your_project inspect ping -d "celery@$HOSTNAME" --timeout 10 || exit 1

The trade-off is cost and coupling. Each check starts a Python process that imports your app, and the check also fails when the broker is slow or unreachable, so one broker problem marks every worker unhealthy. The --timeout value is how long to wait for the reply; Celery's default is one second.

Kubernetes liveness probe#

A lighter check is a file the worker touches each time it sends a heartbeat:

# health.py
import time
from pathlib import Path

from celery.signals import heartbeat_sent, worker_ready

HEARTBEAT_FILE = Path("/tmp/celery_heartbeat")

@worker_ready.connect
def on_worker_ready(**kwargs):
    HEARTBEAT_FILE.touch()

@heartbeat_sent.connect
def on_heartbeat(**kwargs):
    HEARTBEAT_FILE.touch()

def is_worker_healthy(max_age_seconds=60):
    """Return True if the heartbeat file was updated recently."""
    if not HEARTBEAT_FILE.exists():
        return False
    return time.time() - HEARTBEAT_FILE.stat().st_mtime < max_age_seconds

heartbeat_sent fires only when the worker sends events: start it with -E or set worker_send_task_events=True. Celery starts its heartbeat timer only when the event dispatcher is enabled (source). Add health to app.conf.imports so the worker registers the handlers.

# Kubernetes deployment
livenessProbe:
  exec:
    command:
      - python
      - -c
      - 'from health import is_worker_healthy; raise SystemExit(0 if is_worker_healthy() else 1)'
  initialDelaySeconds: 30
  periodSeconds: 30

A liveness probe restarts a stuck worker. It doesn't tell you whether scheduled tasks ran; that still needs the check-ins above.

Production configuration checklist#

These settings support the monitoring above:

# celery.py
app.conf.update(
    # Events for Flower and Prometheus exporters
    worker_send_task_events=True,
    task_send_sent_event=True,
    task_track_started=True,

    # Time limits. The soft limit raises SoftTimeLimitExceeded inside the task;
    # the hard limit kills and replaces the worker process.
    task_soft_time_limit=3300,  # 55 minutes
    task_time_limit=3600,       # 1 hour

    # Replace pool processes periodically to contain slow memory growth
    worker_max_tasks_per_child=1000,
)

Time limits and worker_max_tasks_per_child have no default limit. Turn on task_acks_late and task_reject_on_worker_lost only for tasks that are safe to run twice, as described above. The configuration reference lists every default.

The complete monitoring stack#

For production Celery, cover all four layers:

Frequently Asked Questions#

How do I monitor Celery beat?#

Beat sends no Celery events, so Flower and event-based exporters can't see it. Monitor what beat produces instead: give each scheduled task a heartbeat monitor that expects a check-in on that task's schedule. If beat stops, the check-ins stop and the monitor alerts. Run beat under a process supervisor as well, so a crash restarts it.

Does Flower send alerts?#

No. Flower shows live task and worker state and exposes Prometheus metrics at /metrics. Scrape that endpoint, or run celery-exporter, and write alert rules in Prometheus.

Why do Celery tasks disappear without an error?#

By default, Celery acknowledges a task just before running it, so a task whose worker process is killed mid-run is not redelivered. With task_acks_late enabled, a task is acknowledged after it runs, but failed, timed-out, and lost-worker tasks are still acknowledged unless you also enable task_reject_on_worker_lost. Do that only for tasks that are safe to run twice.

How do I check that Celery workers are running?#

Run celery -A your_project inspect ping; each live worker replies with pong. For continuous monitoring, alert on celery-exporter's celery_worker_up metric or Flower's flower_worker_online.

Is there a CronRadar SDK for Celery?#

A Celery package, cronradar-celery, is coming, but Celery SDK connections are temporarily unavailable. Use the manual HTTP reporting shown above, which needs a monitor key and schedule for each task. The Celery integration docs carry the current availability notice.

What to do next#

Start with Flower to see what your Celery deployment is doing. Add Prometheus alerts for failures, offline workers, and queue backlogs, and an error tracker for the details.

Then ask: would you know if your hourly backup task stopped running? If beat stopped right now, how long until you'd notice? If you don't have a good answer, add check-ins for your critical scheduled tasks. The Celery integration page and the Celery docs show the current setup, and Django cron monitoring covers Celery beat inside Django projects.