# Monitor Scheduled LLM Jobs and AI Agent Runs: Failures That Exit 0

By Dan Ochoa · Published 2026-10-25 · Updated 2026-10-31
Canonical: https://cronradar.com/blog/monitoring-scheduled-ai-agent-runs

> How to monitor scheduled LLM and AI agent jobs: catch rate limits, truncated output, refusals, schema errors, cost spikes, and quality drops that still exit 0.

**To monitor a scheduled LLM job, report each run's start and finish to a monitor outside the job, and have the job count what it actually produced: items processed, items that failed, outputs that broke the schema, what the run cost, and a score on a small labeled test set.** A model call can return HTTP 200 with nothing usable in it: a refusal, an answer cut off at the token limit, JSON that doesn't parse, or confident answers from a model someone swapped. The exit code sees none of that. This guide is about jobs whose work is done by a model; having a coding agent add monitoring to a repository is covered in [AI agent cron monitoring setup](https://cronradar.com/blog/ai-agent-cron-monitoring-setup) and [the MCP server reference](https://cronradar.com/blog/mcp-server-scheduled-jobs). Running a coding agent itself on a schedule, with `claude -p` or `codex exec`, is covered in [scheduled coding agent runs](https://cronradar.com/blog/scheduled-coding-agent-runs).

## How an LLM Job Fails While Exiting 0

| Failure                     | What the provider returns                                          | Count it as                 |
| --------------------------- | ------------------------------------------------------------------ | --------------------------- |
| Rate limit or overload      | 429, 529 (Anthropic), or 503 (OpenAI), after the SDK's retries     | `items_failed`              |
| Spend cap or quota reached  | A 429 or 400 that no retry fixes                                   | stop the run, report `fail` |
| Truncated output            | HTTP 200, `stop_reason: "max_tokens"` or `finish_reason: "length"` | `items_failed`              |
| Refusal                     | HTTP 200, `stop_reason: "refusal"` or a `refusal` field            | `items_failed`              |
| Malformed structured output | HTTP 200, text that fails validation                               | `schema_errors`             |
| Quality regression          | HTTP 200, valid and wrong                                          | `eval_pass_rate`            |
| Cost blowup                 | HTTP 200, more tokens than last week                               | `cost_cents`                |
| Partial batch               | Some results errored or expired                                    | `items_failed`              |

Each of these exits 0 unless the job checks for it. The provider facts below were checked October 25, 2026.

### Provider Errors and Rate Limits

Anthropic's and OpenAI's Python SDKs both retry connection errors, 408, 409, 429, and 5xx responses twice by default with exponential backoff ([Anthropic SDK](https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/python#retries), [openai-python](https://github.com/openai/openai-python#retries)); Anthropic's honors `retry-after` when present ([Claude API errors](https://platform.claude.com/docs/en/api/errors)). Two details matter on a schedule. A nightly batch goes from idle to full speed at once, and both providers push back on sudden ramps: Anthropic warns of acceleration limits and notes that 60 requests per minute may be enforced as one per second ([rate limits](https://platform.claude.com/docs/en/api/rate-limits)), and OpenAI returns a `slow_down` 429 when "Your request rate increased too quickly" ([error codes](https://developers.openai.com/api/docs/guides/error-codes)). Ramp concurrency up gradually. And some errors never clear: Anthropic's tier spend cap returns a 429 with no `retry-after` that keeps failing until access resumes, a spend limit you set returns a 400 ([spend caps](https://platform.claude.com/docs/en/api/rate-limits#reaching-your-spend-cap)), and OpenAI says "Retrying billing, spend, or quota errors won't restore API access." When every item fails the same way, stop the run and report a failure.

### Timeouts and Long Requests

Both SDKs wait 10 minutes per request by default and retry a timeout twice ([Anthropic SDK](https://platform.claude.com/docs/en/cli-sdks-libraries/sdks/python#timeouts), [openai-python](https://github.com/openai/openai-python#timeouts)), so one hung call can hold a run for half an hour. Set `timeout` just above your slowest legitimate response. For long outputs, Anthropic recommends streaming or the Message Batches API, and with streaming an error can arrive after the HTTP 200 ([long requests](https://platform.claude.com/docs/en/api/errors#long-requests)). A request timeout doesn't bound the run; a deadline in the job's loop does, as [the timeout guide](https://cronradar.com/blog/cron-job-timeout) covers for whole processes.

### Truncated Output and Refusals Arrive as HTTP 200

Anthropic's `stop_reason` is `max_tokens` when the response hit your limit, `model_context_window_exceeded` when it filled the context window (treat both as truncated), and `refusal`, returned "as a normal HTTP 200 response, not an error" ([handling stop reasons](https://platform.claude.com/docs/en/build-with-claude/handling-stop-reasons)). OpenAI's Chat Completions sets `finish_reason` to `length` or `content_filter` ([reference](https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events)), and its Responses API returns `status: "incomplete"` with a reason in `incomplete_details` ([structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs)).

Reasoning makes truncation likelier. Claude's thinking counts toward `max_tokens`, so a limit sized for the answer alone "is often too small once Claude starts thinking on hard requests" ([steering thinking](https://platform.claude.com/docs/en/build-with-claude/thinking-steering-and-cost#cost-control)), and OpenAI warns that an incomplete response "might occur before any visible output tokens are produced" ([reasoning](https://developers.openai.com/api/docs/guides/reasoning)). Check the stop reason on every response before reading its content.

### Structured Output Still Needs Validation

Both providers can constrain output to a JSON schema, and neither makes validation optional. Anthropic lists refusals and `max_tokens` stops as cases where output "might not match your schema" and says to compare `enum` values case-insensitively ([structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)); OpenAI says "the model might not generate a valid response that matches the provided JSON schema" ([structured outputs](https://developers.openai.com/api/docs/guides/structured-outputs)). Validate every item, count failures as `schema_errors`, and keep bad items out of your database.

### Quality Drifts When the Model or Prompt Changes

Pin the model. "Every Claude model ID is a pinned snapshot" ([models overview](https://platform.claude.com/docs/en/models/overview)), and OpenAI's snapshots "let you lock in a specific version of the model" ([GPT-5](https://developers.openai.com/api/docs/models/gpt-5)). Retirement fails loudly: Anthropic gives at least 60 days' notice and "Requests to retired models fail" ([model deprecations](https://platform.claude.com/docs/en/about-claude/model-deprecations)), and a shut-down OpenAI model "will no longer be accessible" ([deprecations](https://developers.openai.com/api/docs/deprecations)). The quiet change is the deliberate one: someone edits the prompt or points `MODEL_ID` at a newer model, and the job keeps producing valid JSON with worse answers. A golden set catches that: 20 to 50 inputs with known answers, synthetic or scrubbed of personal data, scored on every run.

### Cost Grows Without an Error

Longer inputs, reruns, and reasoning raise spend without an error. Claude bills thinking as output tokens, and `output_tokens` "remains the inclusive, authoritative total used for billing" ([pricing](https://platform.claude.com/docs/en/build-with-claude/thinking-steering-and-cost#pricing)). `input_tokens` leaves out cached tokens, so if you cache prompts, add the cache fields at their own rates ([cache-aware limits](https://platform.claude.com/docs/en/api/rate-limits#cache-aware-itpm)). Compute each run's cost from the returned usage and stop at a hard ceiling.

## A Nightly Classification Job That Counts Its Own Results

This job classifies pending support tickets with the Anthropic Python SDK ([anthropic](https://pypi.org/project/anthropic/) 2.3.0, Python 3.10 or later) and reports to CronRadar over HTTP. There is no installable CronRadar SDK for Python today and no auto-discovery, so each job reports with its own HTTP check-ins. The model call lives in `classify()`; for another provider, that is the function to change. The `tickets` table has `triage_status` (`pending` or `done`) and `triage_attempts` columns.

```python
#!/usr/bin/env python3
# /opt/triage/triage.py: classify pending tickets, then report counts to CronRadar.
# Ticket text and model output never go to CronRadar.
import base64
import json
import os
import sqlite3
import sys
import time
import urllib.parse
import urllib.request

import anthropic

MONITOR_KEY = "ticket-triage"
CHECKIN_PARAMS = {"schedule": "30 2 * * *", "gracePeriod": 1800}
MODEL = os.environ["MODEL_ID"]  # a pinned model ID; change it on purpose
USD_PER_MTOK_IN = float(os.environ["USD_PER_MTOK_IN"])  # from the provider's price list
USD_PER_MTOK_OUT = float(os.environ["USD_PER_MTOK_OUT"])
HARD_STOP_CENTS = 1000  # the cost_cents check alerts at 500; this stops a runaway run
EVAL_FLOOR = 80  # the eval check alerts below 90; below 80, write no labels at all
TIME_BUDGET_SECONDS = 90 * 60
MAX_ATTEMPTS = 3
LABELS = ["billing", "bug", "account", "feature_request", "other"]
SCHEMA = {
    "type": "object",
    "properties": {
        "category": {"type": "string", "enum": LABELS},
        "urgent": {"type": "boolean"},
    },
    "required": ["category", "urgent"],
    "additionalProperties": False,
}
SYSTEM = "Classify the support ticket into one category and say whether it is urgent."

client = anthropic.Anthropic(timeout=60.0, max_retries=3)
spent_cents = 0.0


class StopRun(Exception):
    """Stop the whole run and report fail. The message must hold no ticket data."""


class NoAnswer(Exception):
    """The model refused, ran out of tokens, or stopped for another reason."""


class BadOutput(Exception):
    """An answer arrived but failed validation."""


def checkin(event, run_id, body=None, **params):
    """Report to CronRadar with a 5-second timeout. Errors are printed and ignored."""
    try:
        query = urllib.parse.urlencode({"run_id": run_id, **params}, quote_via=urllib.parse.quote)
        request = urllib.request.Request(
            f"https://cron.life/api/v1/ping/{MONITOR_KEY}/{event}?{query}",
            data=json.dumps(body).encode() if body is not None else None,
            method="POST",
        )
        token = base64.b64encode(f"{os.environ['CRONRADAR_API_KEY']}:".encode()).decode()
        request.add_header("Authorization", f"Basic {token}")
        if body is not None:
            request.add_header("Content-Type", "application/json")
        urllib.request.urlopen(request, timeout=5).close()
    except Exception as exc:
        print(f"CronRadar {event} check-in failed: {exc}", file=sys.stderr)


def classify(text):
    """Return (category, urgent) for one ticket."""
    global spent_cents
    response = client.messages.create(
        model=MODEL,
        max_tokens=4096,  # thinking counts toward this on models that think first
        system=SYSTEM,
        messages=[{"role": "user", "content": text}],
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
    )
    # USD per million tokens to cents: tokens * rate / 10,000.
    spent_cents += response.usage.input_tokens * USD_PER_MTOK_IN / 10_000
    spent_cents += response.usage.output_tokens * USD_PER_MTOK_OUT / 10_000
    if spent_cents > HARD_STOP_CENTS:
        raise StopRun(f"spend passed {HARD_STOP_CENTS} cents")
    if response.stop_reason != "end_turn":  # max_tokens, refusal, ...
        raise NoAnswer(response.stop_reason)
    try:
        data = json.loads(next(b.text for b in response.content if b.type == "text"))
        category = str(data["category"]).lower()  # enum casing isn't guaranteed
        urgent = data["urgent"]
    except (StopIteration, ValueError, KeyError, TypeError) as exc:
        raise BadOutput(type(exc).__name__)
    if category not in LABELS or not isinstance(urgent, bool):
        raise BadOutput("value outside the schema")
    return category, urgent


def eval_pass_rate(path="/opt/triage/golden.json"):
    """Score the golden set: synthetic tickets with known labels, no customer data."""
    with open(path) as f:
        golden = json.load(f)
    passed = 0
    for case in golden:
        try:
            passed += classify(case["text"])[0] == case["category"]
        except (NoAnswer, BadOutput, anthropic.APIError):
            pass
    return round(100 * passed / len(golden), 1)


def triage(db, deadline):
    counts = {"items_processed": 0, "items_failed": 0, "schema_errors": 0}
    api_errors_in_a_row = 0
    rows = db.execute(
        "SELECT id, body FROM tickets WHERE triage_status = 'pending' AND triage_attempts < ?",
        (MAX_ATTEMPTS,),
    ).fetchall()
    for ticket_id, body in rows:
        if time.monotonic() > deadline:
            raise StopRun(f"time budget reached after {counts['items_processed']} tickets")
        try:
            category, urgent = classify(body)
        except anthropic.APIError as exc:  # raised after the SDK's own retries
            counts["items_failed"] += 1
            api_errors_in_a_row += 1
            if api_errors_in_a_row >= 5:
                raise StopRun(f"{type(exc).__name__} on 5 tickets in a row")
            continue
        except (NoAnswer, BadOutput) as exc:
            counts["schema_errors" if isinstance(exc, BadOutput) else "items_failed"] += 1
            db.execute(
                "UPDATE tickets SET triage_attempts = triage_attempts + 1 WHERE id = ?",
                (ticket_id,),
            )
            db.commit()
            api_errors_in_a_row = 0
            continue
        api_errors_in_a_row = 0
        db.execute(
            "UPDATE tickets SET category = ?, urgent = ?, triage_status = 'done' WHERE id = ?",
            (category, urgent, ticket_id),
        )
        db.commit()
        counts["items_processed"] += 1
    return counts


def run():
    deadline = time.monotonic() + TIME_BUDGET_SECONDS
    rate = eval_pass_rate()
    if rate < EVAL_FLOOR:
        raise StopRun(f"eval pass rate {rate} is below {EVAL_FLOOR}")
    db = sqlite3.connect(os.environ["TRIAGE_DB"])
    try:
        counts = triage(db, deadline)
    finally:
        db.close()
    return {**counts, "cost_cents": round(spent_cents, 2), "eval_pass_rate": rate}


def main():
    run_id = f"{MONITOR_KEY}-{int(time.time())}-{os.getpid()}"
    checkin("start", run_id, **CHECKIN_PARAMS)
    try:
        outcomes = run()
    except BaseException as exc:
        reason = str(exc) if isinstance(exc, StopRun) else type(exc).__name__
        checkin("fail", run_id, message=f"ticket triage stopped: {reason}", **CHECKIN_PARAMS)
        raise
    checkin("complete", run_id, body={"outcomes": outcomes}, **CHECKIN_PARAMS)
    print(json.dumps(outcomes))  # counts only, for the job's own log


if __name__ == "__main__":
    main()
```

What each part does:

- **Per item.** Each response's stop reason is checked before its content is read, and every answer is validated. Failed tickets stay `pending`. A refusal, truncation, or invalid answer uses one of three attempts, so a ticket the model can't handle stops being retried; an API error uses none, so an outage doesn't burn attempts.
- **Per run.** Five API errors in a row, spend past the hard stop, the time budget, or an eval score under the floor raises `StopRun`, which reports `fail` (an immediate alert, no grace period) and exits non-zero. Unprocessed tickets wait for the next run.
- **Check-ins.** Each check-in times out after five seconds on the connection or any read ([urllib.request](https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen)) and ignores its own errors, and the job's exception is re-raised after `fail`. Carrying the schedule lets the first run create the monitor and a later run recreate a deleted one; schedule and grace apply only on creation, so change them later in the monitor's settings. CronRadar reads schedules in UTC: run this at 02:30 UTC.
- **A killed process.** If the job dies after `start` without reporting, CronRadar flags it as stuck only when the next expected run plus grace passes with no check-in: tomorrow night, for this job. The request timeout and loop deadline turn most slow runs into tonight's `fail`.

With OpenAI, the same checks read `choices[0].finish_reason` and the message's `refusal` field in Chat Completions, or `status` and `incomplete_details` in the Responses API.

## Alert on the Numbers, Not Only the Exit Code

In the monitor's settings, under Outcome checks, configure up to five expectations: a number compared with `>=`, `<=`, or `=`, or a boolean that must be true or false. The job sends matching values in the JSON body of its `complete` check-in. A missing or violated expectation marks the run outcome-failed and opens an incident that shows the observed and expected values.

| Outcome check            | Catches                                                        |
| ------------------------ | -------------------------------------------------------------- |
| `items_processed >= 200` | An empty queue, a broken query, or a run that stopped early    |
| `items_failed <= 5`      | Refusals, truncations, and API errors beyond the usual handful |
| `schema_errors <= 2`     | A prompt or model change that breaks the output format         |
| `cost_cents <= 500`      | Longer inputs, a pricier model, or reruns multiplying spend    |
| `eval_pass_rate >= 90`   | Worse answers on the golden set, while the output still parses |

Take floors from your own history and allow for quiet days ([choosing thresholds](https://cronradar.com/blog/scheduled-job-silent-success)). The two tiers are deliberate: the checks alert at 500 cents and below 90, while the job stops at 1,000 cents and writes no labels below 80, so drift brings an alert and a runaway hits a wall.

Five values is a hard limit that counts every key sent, configured or not, which is why refusals and truncations are folded into `items_failed` and token counts stay in the job's log. Keys are lowercase letters, digits, and underscores, starting with a letter; values are finite numbers or booleans. CronRadar rejects a body it can't accept, such as a sixth value or a string, with a 400 and records no completion, and since the next run's `start` arrives on schedule, that can go on without an alert. Send one completion by hand with `curl --fail-with-body` first.

## Retry Items, Not Runs

Don't retry a whole run when some items fail. The job above commits each ticket as it finishes, so a rerun after a crash or a `StopRun` picks up only what is still `pending`. Rerunning everything pays for every item twice and, if outputs leave your database, sends them twice. [Idempotent scheduled jobs](https://cronradar.com/blog/idempotent-scheduled-jobs) covers claims, upserts, and keys on external side effects.

At the scheduler level, retry only failures that can clear on their own, with backoff and jitter, and keep the alert path intact, as [retrying failed cron jobs](https://cronradar.com/blog/cron-job-retries) shows. OpenAI notes that "unsuccessful requests contribute to your per-minute limit, so continuously resending a request won't work" ([rate limits](https://developers.openai.com/api/docs/guides/rate-limits)).

## Batch APIs: Monitor the Submit and the Collect

Both providers run batches asynchronously at half the synchronous price, which suits nightly work:

| Detail     | Anthropic Message Batches                                      | OpenAI Batch API                                                     |
| ---------- | -------------------------------------------------------------- | -------------------------------------------------------------------- |
| Size limit | 100,000 requests or 256 MB                                     | 50,000 requests, 200 MB input file                                   |
| Time limit | Most finish within an hour; expire if not done within 24 hours | Completion window `24h`                                              |
| Leftovers  | Unsent requests get an `expired` result                        | Unfinished requests are cancelled; expired ones go to the error file |
| Order      | Any order; match on `custom_id`                                | May not match input order; match on `custom_id`                      |

Sources: [Anthropic batch processing](https://platform.claude.com/docs/en/build-with-claude/batch-processing), [OpenAI Batch API](https://developers.openai.com/api/docs/guides/batch). Anthropic keeps results for 29 days. Use the ticket ID as `custom_id` so a result lands on the right row whatever order it arrives in.

A batch splits one job into two, and each needs its own monitor. A nightly submit job creates the batch, records its ID in your database, and reports `requests_submitted >= 200`. An hourly collect job reads the status of every recorded batch, processes those that ended, and counts `errored`, `expired`, and `canceled` results as failed. It also reports `oldest_pending_hours`, the age of the oldest uncollected batch, checked with `oldest_pending_hours <= 26`: a batch unfinished after 24 hours expires, so a higher value means results aren't being collected. Send every configured key on every run, zeros included, because a missing key fails the check.

## Scheduled Agents Need Limits the Model Can't Override

An agent run, where the model chooses tools in a loop to triage tickets or write a report, adds three failures of its own: it can loop, take far more steps than usual, and change things outside your database. Put the limits in code, not in the prompt:

- **A step limit and a deadline.** Count model requests per run and stop at a fixed number. `max_tokens` doesn't cap a loop: "each request in the turn has its own `max_tokens`, so it doesn't bound the whole turn's spend" ([steering thinking](https://platform.claude.com/docs/en/build-with-claude/thinking-steering-and-cost#cost-control)). OWASP lists timeouts, throttling, and quotas against [unbounded consumption](https://genai.owasp.org/llmrisk/llm102025-unbounded-consumption/).
- **Loop detection.** Stop when the same tool is called with the same arguments several times in a row.
- **Read-only by default.** Give the agent read tools. Writes that are hard to undo, such as closing tickets, emailing customers, or deleting records, go through a dry run that records proposed actions for review, or through an approval step. OWASP's [excessive agency](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/) guidance is to "require a human to approve high-impact actions before they are taken" and to enforce authorization in downstream systems "rather than relying on an LLM."
- **Untrusted input.** Tickets are written by outsiders. Text that steers a model from a file or web page is what OWASP calls [indirect prompt injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/), and least privilege limits the damage.

Report the agent's own counters as outcomes, for example `tickets_handled >= 1`, `steps_used <= 60` under a hard stop at 80, `tool_errors <= 3`, `actions_applied <= 25`, and `cost_cents <= 500`. A jump in `actions_applied` is often the first sign an agent is doing something new.

## Keep Prompts and Outputs Out of Monitoring

A monitor needs counts, not content. Never send prompts, completions, ticket text, customer identifiers, or keys in a check-in, and write `fail` messages yourself: the job above sends an exception's type name, not its text, because error text can quote input. CronRadar's outcome checks accept only numbers and booleans, and a failure message is capped at 1,000 characters. Prompts and outputs belong in your own logs, under your own retention rules.

## Frequently Asked Questions

### How do I monitor a scheduled LLM job?

Send a start check-in when the job begins and a complete or fail check-in when it ends, so a missed or failed run alerts. Have the job count what it produced (items processed, items failed, schema errors, cost, an eval score) and send those with the completion, so a run that exits 0 with bad output still opens an incident.

### How do I detect truncated LLM output in a batch job?

Check the stop reason on every response before parsing it: `stop_reason == "max_tokens"` or `model_context_window_exceeded` on Claude, `finish_reason == "length"` in OpenAI Chat Completions, or `status == "incomplete"` in OpenAI's Responses API. Count truncated items as failures instead of saving them, alert when the count passes a threshold, and leave room in the token limit for reasoning.

### How do I alert when an AI job's quality drops?

Score a golden set of 20 to 50 inputs with known answers at the start of every run, using the same model and prompt, and report the pass rate as a number from 0 to 100. Alert below a threshold such as 90, and have the job refuse to write results below a lower floor.

### How do I keep a scheduled agent from running up costs?

Compute cost from the token usage each response returns, stop the run in code at a hard ceiling, and cap model requests per run, because a per-request `max_tokens` doesn't limit a loop. Report `cost_cents` with each completion and alert well below the hard stop; provider spend limits are a backstop, not a per-job budget.

### Should a scheduled job retry on rate limits?

Yes, a bounded number of times, with backoff and jitter, honoring `retry-after` when it's present; the official Python SDKs already retry 429s twice by default. Don't retry spend-cap, billing, or quota errors, which fail until someone acts, and stop the run when every request fails the same way. Retry the failed items, not the whole run.