Benchmark engines on real calls

Build evidence from the calls your agent actually made, fit every decision engine on it, and score them on calls none of them saw

The decision engines learn a table, state → move, from evidence. Until now that evidence was the Arena’s simulated episodes. This page is about the calls your agent really made: turning them into evidence, fitting every engine on the same calls, and scoring each one on calls it never saw, beside what your agent does today.

Offline scores on logged calls rank candidates; they do not prove a lift. The moves and states are a tagger’s reading of the transcript, today’s propensities are estimated from the log, and no log can show how a customer would have answered a move nobody made. Only a canary against a control group proves a lift. Every report says this, and so does every verdict.

1. Build an evidence set

From Data lab → Engines → Benchmark on real calls → Build a set from real calls, or:

levers9 engine evidence-build acme --source volume -n 20000 --tagger judge
levers9 engine evidence-sets acme

option	what it does
--source registry	the calls the world holds in the Arena (the default)
--source volume	the full export on the call-logs volume, <tenant>/<tenant>_calls.jsonl.gz unless --path says otherwise
--tagger rule	cue phrases, no model: free and instant, for seeing the pipeline work, not for deciding anything
--tagger judge	the Arena’s judge (vLLM or Anthropic) reads each call once and names every agent turn’s move, and the customer’s stage, objection and warmth before it
--tagger model:ID	a typed-decision model from the Models registry, asked one choice question per agent turn (“which of the eight moves was this?”) about the transcript up to that turn. A model with a serving Decision API is asked over HTTP (POST {endpoint}/v1/systemone) and the build stays a CPU job; a model without one (a Lev on laya-lab, a Laya or Kev Hub id) is loaded inside the build job, which then runs on 2 × RTX PRO 6000 with the weights’ volume mounted
-n	how many calls to sample, uniformly (up to 100,000)
--max-turns	agent turns per call kept, 8 by default: the opening, where calls are won or lost

What happens to each call:

• Scrubbed first. Only the turns’ text, the outcome and a hash of the call id survive. Phone numbers are removed before a tagger or the file sees a turn; nothing else of the row is kept.
• Tagged. Each agent turn gets one of the eight moves. The state (stage:trust:objection) is read from what the customer had said before the turn, so a move never leaks into the state it is chosen from. Whatever a model tagger leaves out falls back to the rule reading and is counted in the set’s tagger.stats. The rule tagger reads no warmth, so its states all sit in the middle trust bucket.
• Priced by the outcome. The platform’s label is folded by the tenant’s outcome map (the importer’s table, overridden by what you saved on the world’s Outcomes card), so viewing_booked is the win. Rewards are 0 per turn and the last kept turn carries the outcome’s terminal value. Calls with no usable outcome are left out.
• Propensities estimated. No log records how likely the agent was to make its move, so it is estimated per state from how often it did: propensity_source: "empirical".
• Split by call. 80% fit, 20% held out, by a hash of the call id and the seed: rebuild with the same seed and every call lands on the same side. Preference pairs are built from the fit calls only.

The set is written to the datasets volume at rl/<tenant>/evidence-<tagger>-<time>.json, in the same shape as every other evidence document (transitions gain a split field), and listed by GET /api/rl/evidence-sets.

2. Run the benchmark

Pick the set and the engines on the card and press Run the benchmark, or:

levers9 engine benchmark --evidence SET_ID --engines all
levers9 engine benchmark --evidence SET_ID --engines q-learning,monte-carlo -o report.json

It runs as a CPU job in your workspace, never inside the dashboard. Each engine that learns from a log (transitions or preference pairs) is fitted on the fit calls with its default knobs and scored on the held-out calls. REINFORCE, PPO, the online learners, model-based learning and R-max are listed as simulator only: they learn by acting, and a log cannot say what would have happened after a move nobody made. Their measurement is each engine’s Run against a world’s simulator.

Your own engine joins the same leaderboard. It is fitted on the set’s fit calls, here or with --remote, and its table is scored like the built-ins:

levers9 engine benchmark --evidence SET_ID --file app.py --engines none

from levers9.decision import Evidence, benchmark, wait, benchmark_report, leaderboard

fit_calls = Evidence.from_set("SET_ID").part("fit")
result = my_engine.run(fit_calls)
job = benchmark("SET_ID", "all", tables={"mine": result})
done = wait(job["id"])
print(leaderboard(benchmark_report(done["result"]["benchmark_id"])["report"]))

3. Read the leaderboard

Every row is scored on the held-out calls three ways, each with a 95% bootstrap interval over calls:

estimator	what it is	how it fails
PDWIS (the estimate)	per-decision weighted importance sampling: each held-out call reweighted by how likely the engine was to make the moves it made	only as good as the overlap: a call where the engine would have done anything else counts for nothing
DM	the direct method, by fitted-Q evaluation on the held-out calls	exactly as wrong as the model, and it guesses wherever the log never tried the engine’s move
DR	weighted doubly robust: PDWIS with the fitted model as a control variate	right if either the weights or the model is

• ESS is the effective sample size: how many held-out calls the importance-sampling estimate really rests on. A deterministic table that disagrees with your agent often has a small one.
• vs today and vs random are paired differences: the same resampled calls on both sides.
• A state the engine’s table does not know is left to today’s agent, which is what serving does.

The verdict reads:

verdict	when
Ahead of today on held-out calls	the PDWIS interval on the lift is above zero and the fitted model agrees
Behind today, do not ship it	the PDWIS interval on the lift is below zero
No detectable difference from today	the interval straddles zero
The estimators disagree	importance sampling says ahead, the model does not
Can’t tell from these calls	fewer than 30 effective calls (or 1% of the held-out calls): the model’s guess is shown, and named as a guess

Each row opens into its three estimates, the reasons, and the table it decided, in the world’s own words. An engine that comes out ahead is a candidate for a canary under Release, with a control group; that is what proves it.

The API

call	what it does
POST /api/rl/evidence-sets	build a set: {env, source, path?, tagger: {kind, model_id?}, n, seed, max_turns}; returns the job
GET /api/rl/evidence-sets?env=	sets, newest first, plus evidence files found on the volume
GET /api/rl/evidence-sets/{id}/document	the file itself
POST /api/rl/engines/benchmark	{evidence_set, engines: "all" or names, seed, tables?, bootstrap}; returns the job
GET /api/rl/jobs/{id}	the job, with report (partial while it runs) and log_tail (its last 200 lines)
GET /api/rl/engines/benchmarks	finished benchmarks; /{id} one in full; /{id}/policy/{engine} one table

evidence_set also takes sim:<world> (that world’s simulated episodes) and vol:<path> (an evidence file on the datasets volume that no record points at).

If the dashboard restarts while a benchmark runs, the job’s output is lost with it and the benchmark has to be run again. A build keeps going on the platform and its file still lands on the volume, where the list picks it up.