Platform
Evals
Datasets from real traffic, graded runs, and versioned prompts.
Evals close the loop the gateway opens: capture what your AI actually did, mark what it should have done, and measure any model or prompt against it — with real numbers instead of vibes. The punchline for a spend-control platform: prove a cheaper model scores the same on your traffic, then switch.
Datasets
A dataset is a set of inputs with reference outputs. Build one from real traffic — open a transcript and hit Add to dataset — or author items by hand. Items are snapshots: retention sweeps and purges of captured messages never touch them.
Runs and graders
A run executes every item against one model (optionally with a pinned prompt version) and grades each response:
| Grader | Best for |
|---|---|
| exact_match | Deterministic answers — classifications, extractions |
| contains / regex | Required phrases, formats, refusal markers |
| llm_rubric | Open-ended quality — a judge model scores 0–100 against your rubric |
Every run shows a worst-case cost estimate before it starts, carries a spend cap, and executes through the gateway on your own keys (source=eval in logs). Runs survive worker restarts and resume without re-running completed items.
Prompt versions
Prompts are versioned system texts with {{variables}} that dataset items fill per example. Pin a run to a version, change the prompt, run again, and diff both the text and the scores. The playground’s compare mode fans one input out to multiple models side by side — save the best answer straight into a dataset.