Monitor Scheduled LLM Jobs and AI Agent Runs: Failures That Exit 0

To monitor a scheduled LLM job, report each run's start and finish to a monitor outside the job, and have the job count what it actually produced: items processed, items that failed, outputs that broke the schema, what the run cost, and a score on a small labeled test set. A model call can return HTTP 200 with nothing usable in it: a refusal, an answer cut off at the token limit, JSON that doesn't parse, or confident answers from a model someone swapped. The exit code sees none of that. This guide is about jobs whose work is done by a model; having a coding agent add monitoring to a repository is covered in AI agent cron monitoring setup and the MCP server reference. Running a coding agent itself on a schedule, with claude -p or codex exec, is covered in scheduled coding agent runs.

How an LLM Job Fails While Exiting 0#

Each of these exits 0 unless the job checks for it. The provider facts below were checked October 25, 2026.

Provider Errors and Rate Limits#

Anthropic's and OpenAI's Python SDKs both retry connection errors, 408, 409, 429, and 5xx responses twice by default with exponential backoff (Anthropic SDK, openai-python); Anthropic's honors retry-after when present (Claude API errors). Two details matter on a schedule. A nightly batch goes from idle to full speed at once, and both providers push back on sudden ramps: Anthropic warns of acceleration limits and notes that 60 requests per minute may be enforced as one per second (rate limits), and OpenAI returns a slow_down 429 when "Your request rate increased too quickly" (error codes). Ramp concurrency up gradually. And some errors never clear: Anthropic's tier spend cap returns a 429 with no retry-after that keeps failing until access resumes, a spend limit you set returns a 400 (spend caps), and OpenAI says "Retrying billing, spend, or quota errors won't restore API access." When every item fails the same way, stop the run and report a failure.

Timeouts and Long Requests#

Both SDKs wait 10 minutes per request by default and retry a timeout twice (Anthropic SDK, openai-python), so one hung call can hold a run for half an hour. Set timeout just above your slowest legitimate response. For long outputs, Anthropic recommends streaming or the Message Batches API, and with streaming an error can arrive after the HTTP 200 (long requests). A request timeout doesn't bound the run; a deadline in the job's loop does, as the timeout guide covers for whole processes.

Truncated Output and Refusals Arrive as HTTP 200#

Anthropic's stop_reason is max_tokens when the response hit your limit, model_context_window_exceeded when it filled the context window (treat both as truncated), and refusal, returned "as a normal HTTP 200 response, not an error" (handling stop reasons). OpenAI's Chat Completions sets finish_reason to length or content_filter (reference), and its Responses API returns status: "incomplete" with a reason in incomplete_details (structured outputs).

Reasoning makes truncation likelier. Claude's thinking counts toward max_tokens, so a limit sized for the answer alone "is often too small once Claude starts thinking on hard requests" (steering thinking), and OpenAI warns that an incomplete response "might occur before any visible output tokens are produced" (reasoning). Check the stop reason on every response before reading its content.

Structured Output Still Needs Validation#

Both providers can constrain output to a JSON schema, and neither makes validation optional. Anthropic lists refusals and max_tokens stops as cases where output "might not match your schema" and says to compare enum values case-insensitively (structured outputs); OpenAI says "the model might not generate a valid response that matches the provided JSON schema" (structured outputs). Validate every item, count failures as schema_errors, and keep bad items out of your database.

Quality Drifts When the Model or Prompt Changes#

Pin the model. "Every Claude model ID is a pinned snapshot" (models overview), and OpenAI's snapshots "let you lock in a specific version of the model" (GPT-5). Retirement fails loudly: Anthropic gives at least 60 days' notice and "Requests to retired models fail" (model deprecations), and a shut-down OpenAI model "will no longer be accessible" (deprecations). The quiet change is the deliberate one: someone edits the prompt or points MODEL_ID at a newer model, and the job keeps producing valid JSON with worse answers. A golden set catches that: 20 to 50 inputs with known answers, synthetic or scrubbed of personal data, scored on every run.

Cost Grows Without an Error#

Longer inputs, reruns, and reasoning raise spend without an error. Claude bills thinking as output tokens, and output_tokens "remains the inclusive, authoritative total used for billing" (pricing). input_tokens leaves out cached tokens, so if you cache prompts, add the cache fields at their own rates (cache-aware limits). Compute each run's cost from the returned usage and stop at a hard ceiling.

A Nightly Classification Job That Counts Its Own Results#

This job classifies pending support tickets with the Anthropic Python SDK (anthropic 2.3.0, Python 3.10 or later) and reports to CronRadar over HTTP. There is no installable CronRadar SDK for Python today and no auto-discovery, so each job reports with its own HTTP check-ins. The model call lives in classify(); for another provider, that is the function to change. The tickets table has triage_status (pending or done) and triage_attempts columns.

#!/usr/bin/env python3
# /opt/triage/triage.py: classify pending tickets, then report counts to CronRadar.
# Ticket text and model output never go to CronRadar.
import base64
import json
import os
import sqlite3
import sys
import time
import urllib.parse
import urllib.request

import anthropic

MONITOR_KEY = "ticket-triage"
CHECKIN_PARAMS = {"schedule": "30 2 * * *", "gracePeriod": 1800}
MODEL = os.environ["MODEL_ID"]  # a pinned model ID; change it on purpose
USD_PER_MTOK_IN = float(os.environ["USD_PER_MTOK_IN"])  # from the provider's price list
USD_PER_MTOK_OUT = float(os.environ["USD_PER_MTOK_OUT"])
HARD_STOP_CENTS = 1000  # the cost_cents check alerts at 500; this stops a runaway run
EVAL_FLOOR = 80  # the eval check alerts below 90; below 80, write no labels at all
TIME_BUDGET_SECONDS = 90 * 60
MAX_ATTEMPTS = 3
LABELS = ["billing", "bug", "account", "feature_request", "other"]
SCHEMA = {
    "type": "object",
    "properties": {
        "category": {"type": "string", "enum": LABELS},
        "urgent": {"type": "boolean"},
    },
    "required": ["category", "urgent"],
    "additionalProperties": False,
}
SYSTEM = "Classify the support ticket into one category and say whether it is urgent."

client = anthropic.Anthropic(timeout=60.0, max_retries=3)
spent_cents = 0.0


class StopRun(Exception):
    """Stop the whole run and report fail. The message must hold no ticket data."""


class NoAnswer(Exception):
    """The model refused, ran out of tokens, or stopped for another reason."""


class BadOutput(Exception):
    """An answer arrived but failed validation."""


def checkin(event, run_id, body=None, **params):
    """Report to CronRadar with a 5-second timeout. Errors are printed and ignored."""
    try:
        query = urllib.parse.urlencode({"run_id": run_id, **params}, quote_via=urllib.parse.quote)
        request = urllib.request.Request(
            f"https://cron.life/api/v1/ping/{MONITOR_KEY}/{event}?{query}",
            data=json.dumps(body).encode() if body is not None else None,
            method="POST",
        )
        token = base64.b64encode(f"{os.environ['CRONRADAR_API_KEY']}:".encode()).decode()
        request.add_header("Authorization", f"Basic {token}")
        if body is not None:
            request.add_header("Content-Type", "application/json")
        urllib.request.urlopen(request, timeout=5).close()
    except Exception as exc:
        print(f"CronRadar {event} check-in failed: {exc}", file=sys.stderr)


def classify(text):
    """Return (category, urgent) for one ticket."""
    global spent_cents
    response = client.messages.create(
        model=MODEL,
        max_tokens=4096,  # thinking counts toward this on models that think first
        system=SYSTEM,
        messages=[{"role": "user", "content": text}],
        output_config={"format": {"type": "json_schema", "schema": SCHEMA}},
    )
    # USD per million tokens to cents: tokens * rate / 10,000.
    spent_cents += response.usage.input_tokens * USD_PER_MTOK_IN / 10_000
    spent_cents += response.usage.output_tokens * USD_PER_MTOK_OUT / 10_000
    if spent_cents > HARD_STOP_CENTS:
        raise StopRun(f"spend passed {HARD_STOP_CENTS} cents")
    if response.stop_reason != "end_turn":  # max_tokens, refusal, ...
        raise NoAnswer(response.stop_reason)
    try:
        data = json.loads(next(b.text for b in response.content if b.type == "text"))
        category = str(data["category"]).lower()  # enum casing isn't guaranteed
        urgent = data["urgent"]
    except (StopIteration, ValueError, KeyError, TypeError) as exc:
        raise BadOutput(type(exc).__name__)
    if category not in LABELS or not isinstance(urgent, bool):
        raise BadOutput("value outside the schema")
    return category, urgent


def eval_pass_rate(path="/opt/triage/golden.json"):
    """Score the golden set: synthetic tickets with known labels, no customer data."""
    with open(path) as f:
        golden = json.load(f)
    passed = 0
    for case in golden:
        try:
            passed += classify(case["text"])[0] == case["category"]
        except (NoAnswer, BadOutput, anthropic.APIError):
            pass
    return round(100 * passed / len(golden), 1)


def triage(db, deadline):
    counts = {"items_processed": 0, "items_failed": 0, "schema_errors": 0}
    api_errors_in_a_row = 0
    rows = db.execute(
        "SELECT id, body FROM tickets WHERE triage_status = 'pending' AND triage_attempts < ?",
        (MAX_ATTEMPTS,),
    ).fetchall()
    for ticket_id, body in rows:
        if time.monotonic() > deadline:
            raise StopRun(f"time budget reached after {counts['items_processed']} tickets")
        try:
            category, urgent = classify(body)
        except anthropic.APIError as exc:  # raised after the SDK's own retries
            counts["items_failed"] += 1
            api_errors_in_a_row += 1
            if api_errors_in_a_row >= 5:
                raise StopRun(f"{type(exc).__name__} on 5 tickets in a row")
            continue
        except (NoAnswer, BadOutput) as exc:
            counts["schema_errors" if isinstance(exc, BadOutput) else "items_failed"] += 1
            db.execute(
                "UPDATE tickets SET triage_attempts = triage_attempts + 1 WHERE id = ?",
                (ticket_id,),
            )
            db.commit()
            api_errors_in_a_row = 0
            continue
        api_errors_in_a_row = 0
        db.execute(
            "UPDATE tickets SET category = ?, urgent = ?, triage_status = 'done' WHERE id = ?",
            (category, urgent, ticket_id),
        )
        db.commit()
        counts["items_processed"] += 1
    return counts


def run():
    deadline = time.monotonic() + TIME_BUDGET_SECONDS
    rate = eval_pass_rate()
    if rate < EVAL_FLOOR:
        raise StopRun(f"eval pass rate {rate} is below {EVAL_FLOOR}")
    db = sqlite3.connect(os.environ["TRIAGE_DB"])
    try:
        counts = triage(db, deadline)
    finally:
        db.close()
    return {**counts, "cost_cents": round(spent_cents, 2), "eval_pass_rate": rate}


def main():
    run_id = f"{MONITOR_KEY}-{int(time.time())}-{os.getpid()}"
    checkin("start", run_id, **CHECKIN_PARAMS)
    try:
        outcomes = run()
    except BaseException as exc:
        reason = str(exc) if isinstance(exc, StopRun) else type(exc).__name__
        checkin("fail", run_id, message=f"ticket triage stopped: {reason}", **CHECKIN_PARAMS)
        raise
    checkin("complete", run_id, body={"outcomes": outcomes}, **CHECKIN_PARAMS)
    print(json.dumps(outcomes))  # counts only, for the job's own log


if __name__ == "__main__":
    main()

What each part does:

• Per item. Each response's stop reason is checked before its content is read, and every answer is validated. Failed tickets stay pending. A refusal, truncation, or invalid answer uses one of three attempts, so a ticket the model can't handle stops being retried; an API error uses none, so an outage doesn't burn attempts.
• Per run. Five API errors in a row, spend past the hard stop, the time budget, or an eval score under the floor raises StopRun, which reports fail (an immediate alert, no grace period) and exits non-zero. Unprocessed tickets wait for the next run.
• Check-ins. Each check-in times out after five seconds on the connection or any read (urllib.request) and ignores its own errors, and the job's exception is re-raised after fail. Carrying the schedule lets the first run create the monitor and a later run recreate a deleted one; schedule and grace apply only on creation, so change them later in the monitor's settings. CronRadar reads schedules in UTC: run this at 02:30 UTC.
• A killed process. If the job dies after start without reporting, CronRadar flags it as stuck only when the next expected run plus grace passes with no check-in: tomorrow night, for this job. The request timeout and loop deadline turn most slow runs into tonight's fail.

With OpenAI, the same checks read choices[0].finish_reason and the message's refusal field in Chat Completions, or status and incomplete_details in the Responses API.

Alert on the Numbers, Not Only the Exit Code#

In the monitor's settings, under Outcome checks, configure up to five expectations: a number compared with >=, <=, or =, or a boolean that must be true or false. The job sends matching values in the JSON body of its complete check-in. A missing or violated expectation marks the run outcome-failed and opens an incident that shows the observed and expected values.

Take floors from your own history and allow for quiet days (choosing thresholds). The two tiers are deliberate: the checks alert at 500 cents and below 90, while the job stops at 1,000 cents and writes no labels below 80, so drift brings an alert and a runaway hits a wall.

Five values is a hard limit that counts every key sent, configured or not, which is why refusals and truncations are folded into items_failed and token counts stay in the job's log. Keys are lowercase letters, digits, and underscores, starting with a letter; values are finite numbers or booleans. CronRadar rejects a body it can't accept, such as a sixth value or a string, with a 400 and records no completion, and since the next run's start arrives on schedule, that can go on without an alert. Send one completion by hand with curl --fail-with-body first.

Retry Items, Not Runs#

Don't retry a whole run when some items fail. The job above commits each ticket as it finishes, so a rerun after a crash or a StopRun picks up only what is still pending. Rerunning everything pays for every item twice and, if outputs leave your database, sends them twice. Idempotent scheduled jobs covers claims, upserts, and keys on external side effects.

At the scheduler level, retry only failures that can clear on their own, with backoff and jitter, and keep the alert path intact, as retrying failed cron jobs shows. OpenAI notes that "unsuccessful requests contribute to your per-minute limit, so continuously resending a request won't work" (rate limits).

Batch APIs: Monitor the Submit and the Collect#

Both providers run batches asynchronously at half the synchronous price, which suits nightly work:

Sources: Anthropic batch processing, OpenAI Batch API. Anthropic keeps results for 29 days. Use the ticket ID as custom_id so a result lands on the right row whatever order it arrives in.

A batch splits one job into two, and each needs its own monitor. A nightly submit job creates the batch, records its ID in your database, and reports requests_submitted >= 200. An hourly collect job reads the status of every recorded batch, processes those that ended, and counts errored, expired, and canceled results as failed. It also reports oldest_pending_hours, the age of the oldest uncollected batch, checked with oldest_pending_hours <= 26: a batch unfinished after 24 hours expires, so a higher value means results aren't being collected. Send every configured key on every run, zeros included, because a missing key fails the check.

Scheduled Agents Need Limits the Model Can't Override#

An agent run, where the model chooses tools in a loop to triage tickets or write a report, adds three failures of its own: it can loop, take far more steps than usual, and change things outside your database. Put the limits in code, not in the prompt:

• A step limit and a deadline. Count model requests per run and stop at a fixed number. max_tokens doesn't cap a loop: "each request in the turn has its own max_tokens, so it doesn't bound the whole turn's spend" (steering thinking). OWASP lists timeouts, throttling, and quotas against unbounded consumption.
• Loop detection. Stop when the same tool is called with the same arguments several times in a row.
• Read-only by default. Give the agent read tools. Writes that are hard to undo, such as closing tickets, emailing customers, or deleting records, go through a dry run that records proposed actions for review, or through an approval step. OWASP's excessive agency guidance is to "require a human to approve high-impact actions before they are taken" and to enforce authorization in downstream systems "rather than relying on an LLM."
• Untrusted input. Tickets are written by outsiders. Text that steers a model from a file or web page is what OWASP calls indirect prompt injection, and least privilege limits the damage.

Report the agent's own counters as outcomes, for example tickets_handled >= 1, steps_used <= 60 under a hard stop at 80, tool_errors <= 3, actions_applied <= 25, and cost_cents <= 500. A jump in actions_applied is often the first sign an agent is doing something new.

Keep Prompts and Outputs Out of Monitoring#

A monitor needs counts, not content. Never send prompts, completions, ticket text, customer identifiers, or keys in a check-in, and write fail messages yourself: the job above sends an exception's type name, not its text, because error text can quote input. CronRadar's outcome checks accept only numbers and booleans, and a failure message is capped at 1,000 characters. Prompts and outputs belong in your own logs, under your own retention rules.

Frequently Asked Questions#

How do I monitor a scheduled LLM job?#

Send a start check-in when the job begins and a complete or fail check-in when it ends, so a missed or failed run alerts. Have the job count what it produced (items processed, items failed, schema errors, cost, an eval score) and send those with the completion, so a run that exits 0 with bad output still opens an incident.

How do I detect truncated LLM output in a batch job?#

Check the stop reason on every response before parsing it: stop_reason == "max_tokens" or model_context_window_exceeded on Claude, finish_reason == "length" in OpenAI Chat Completions, or status == "incomplete" in OpenAI's Responses API. Count truncated items as failures instead of saving them, alert when the count passes a threshold, and leave room in the token limit for reasoning.

How do I alert when an AI job's quality drops?#

Score a golden set of 20 to 50 inputs with known answers at the start of every run, using the same model and prompt, and report the pass rate as a number from 0 to 100. Alert below a threshold such as 90, and have the job refuse to write results below a lower floor.

How do I keep a scheduled agent from running up costs?#

Compute cost from the token usage each response returns, stop the run in code at a hard ceiling, and cap model requests per run, because a per-request max_tokens doesn't limit a loop. Report cost_cents with each completion and alert well below the hard stop; provider spend limits are a backstop, not a per-job budget.

Should a scheduled job retry on rate limits?#

Yes, a bounded number of times, with backoff and jitter, honoring retry-after when it's present; the official Python SDKs already retry 429s twice by default. Don't retry spend-cap, billing, or quota errors, which fail until someone acts, and stop the run when every request fails the same way. Retry the failed items, not the whole run.