Monitoring & Evals
Catch the bad answer before your customer does.
A monitor points a judge model at a sample of your live traffic and asks one question per exchange: does this response violate your instructions? Findings come back minutes-old with severity and reasoning — and the worst can disable the offending key on their own.
Animated diagram: requests from models and coding agents flow into monitors, which judge a sample of the traffic. Most exchanges pass; flagged ones become findings, send a notification, or disable the offending key, depending on severity.
Findings always log. Notifications join at the severity you pick. Disabling a key is opt-in.
Three decisions, then it watches.
01
Sample
Pick a scope — the whole org, one team, one member, one key — and a sample rate. Sampling is deterministic, so the same request always makes the same decision, and conditions can narrow a monitor to certain models or to completions long enough to be worth reading.
02
Judge
Write the rubric in plain English — “flag responses that cite documents we don’t have” beats any regex. A judge model you choose reviews each sampled exchange seconds after it completes, on your own provider keys at zero markup, and files what it finds: hallucination, policy violation, mistake, prompt injection, PII.
03
Act
Every flag lands as a finding with severity, categories, and the judge’s reasoning. Opt a monitor into email for new findings, and into disabling the offending key above a severity you set. Re-enabling is one click on the API Keys page.
Built to be pointed at production — and left there.
Never in the request path
Judging runs seconds after the response, off the hot path. A monitor cannot add latency, block a request, or touch a stream mid-flight.
Judge spend you control
Sample rate per monitor, a cheap judge model, ~4k tokens per judgment, and a daily budget that auto-pauses the monitor until midnight UTC. At 100k requests a month with 10% sampling, a mini-class judge costs roughly $2–10 of your own provider spend.
Alert storms collapse
Notifications dedupe through a ledger: at most one email per cooldown window, with repeat findings batched into a count instead of a fresh email each.
A hardened judge
Captured content is presented to the judge as data, and verdicts arrive through a forced tool call. Still — treat auto-disable as a circuit breaker for severe cases, not a content filter.
Zero retention stays the default
Monitors read captured traffic, and capture is explicit opt-in per key, team, or member — encrypted before it’s stored, retention-capped by plan, deletable in one click.
Spend that shows its work
Judge calls route through the gateway like any other traffic: logged with source=monitor, counted against budgets, visible in analytics next to the traffic they reviewed.
Evals
Flag it, fix it, prove the fix.
A finding is half the job. Open the flagged transcript, hit Add to dataset, and the failure becomes a test case — a snapshot that outlives capture retention. Run the dataset against any model or prompt version and grade every response:
exact_matchcontainsregexllm_rubricEvery run shows its worst-case cost before it starts and carries a spend cap. Prompts are versioned, so you can change one line, run again, and diff the text next to the scores. And because the gateway routes 13 providers, the useful experiment is cheap to ask: does the smaller model score the same on your traffic? When it does, switching is a config change — no code deploy.
Datasets, runs, and graders in the docsSame rubric, same 128 real conversations. Switch with a config change.
Opt-in message capture starts on Starter with 7-day retention. Monitors and eval runs are included on Team and Enterprise, with 30- and 90-day retention. Capture is off for every scope until you turn it on — here’s exactly what gets stored.
Compare plansPut a hard limit on your AI spend this week. Flat subscription, zero token markup.
Start for Free