> ## Documentation Index
> Fetch the complete documentation index at: https://levers9.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark engines on real calls

> Build evidence from the calls your agent actually made, fit every decision engine on it, and score them on calls none of them saw

The decision engines learn a table, `state → move`, from evidence. Until now that evidence was the Arena's
simulated episodes. This page is about the calls your agent really made: turning them into evidence, fitting
every engine on the same calls, and scoring each one on calls it never saw, beside what your agent does today.

> [!WARNING]
>
> Offline scores on logged calls **rank candidates; they do not prove a lift.** The moves and states are a
> tagger's reading of the transcript, today's propensities are estimated from the log, and no log can show how a
> customer would have answered a move nobody made. Only a canary against a control group proves a lift. Every
> report says this, and so does every verdict.

## 1. Build an evidence set

From **Data lab → Engines → Benchmark on real calls → Build a set from real calls**, or:

```bash theme={null}
levers9 engine evidence-build acme --source volume -n 20000 --tagger judge
levers9 engine evidence-sets acme
```

| option | what it does |
| - | - |
| `--source registry` | the calls the world holds in the Arena (the default) |
| `--source volume` | the full export on the `call-logs` volume, `<tenant>/<tenant>_calls.jsonl.gz` unless `--path` says otherwise |
| `--tagger rule` | cue phrases, no model: free and instant, for seeing the pipeline work, not for deciding anything |
| `--tagger judge` | the Arena's judge (vLLM or Anthropic) reads each call once and names every agent turn's move, and the customer's stage, objection and warmth before it |
| `--tagger model:ID` | a typed-decision model from the Models registry, asked one choice question per agent turn ("which of the eight moves was this?") about the transcript up to that turn. A model with a serving Decision API is asked over HTTP (`POST {endpoint}/v1/systemone`) and the build stays a CPU job; a model without one (a Lev on `laya-lab`, a Laya or Kev Hub id) is loaded inside the build job, which then runs on 2 × RTX PRO 6000 with the weights' volume mounted |
| `-n` | how many calls to sample, uniformly (up to 100,000) |
| `--max-turns` | agent turns per call kept, 8 by default: the opening, where calls are won or lost |

What happens to each call:

* **Scrubbed first.** Only the turns' text, the outcome and a hash of the call id survive. Phone numbers are
  removed before a tagger or the file sees a turn; nothing else of the row is kept.
* **Tagged.** Each agent turn gets one of the eight moves. The state (`stage:trust:objection`) is read from what
  the customer had said *before* the turn, so a move never leaks into the state it is chosen from. Whatever a
  model tagger leaves out falls back to the rule reading and is counted in the set's `tagger.stats`. The rule
  tagger reads no warmth, so its states all sit in the middle trust bucket.
* **Priced by the outcome.** The platform's label is folded by the tenant's outcome map (the importer's table,
  overridden by what you saved on the world's Outcomes card), so `viewing_booked` is the win. Rewards are 0 per
  turn and the last kept turn carries the outcome's terminal value. Calls with no usable outcome are left out.
* **Propensities estimated.** No log records how likely the agent was to make its move, so it is estimated per
  state from how often it did: `propensity_source: "empirical"`.
* **Split by call.** 80% fit, 20% held out, by a hash of the call id and the seed: rebuild with the same seed and
  every call lands on the same side. Preference pairs are built from the fit calls only.

The set is written to the `datasets` volume at `rl/<tenant>/evidence-<tagger>-<time>.json`, in the same shape as
every other evidence document (`transitions` gain a `split` field), and listed by `GET /api/rl/evidence-sets`.

## 2. Run the benchmark

Pick the set and the engines on the card and press **Run the benchmark**, or:

```bash theme={null}
levers9 engine benchmark --evidence SET_ID --engines all
levers9 engine benchmark --evidence SET_ID --engines q-learning,monte-carlo -o report.json
```

It runs as a CPU job in your workspace, never inside the dashboard. Each engine that learns from a log
(transitions or preference pairs) is fitted on the fit calls with its default knobs and scored on the held-out
calls. REINFORCE, PPO, the online learners, model-based learning and R-max are listed as **simulator only**: they
learn by acting, and a log cannot say what would have happened after a move nobody made. Their measurement is
each engine's Run against a world's simulator.

Your own engine joins the same leaderboard. It is fitted on the set's fit calls, here or with `--remote`, and its
table is scored like the built-ins:

```bash theme={null}
levers9 engine benchmark --evidence SET_ID --file app.py --engines none
```

```python theme={null}
from levers9.decision import Evidence, benchmark, wait, benchmark_report, leaderboard

fit_calls = Evidence.from_set("SET_ID").part("fit")
result = my_engine.run(fit_calls)
job = benchmark("SET_ID", "all", tables={"mine": result})
done = wait(job["id"])
print(leaderboard(benchmark_report(done["result"]["benchmark_id"])["report"]))
```

## 3. Read the leaderboard

Every row is scored on the held-out calls three ways, each with a 95% bootstrap interval over calls:

| estimator | what it is | how it fails |
| - | - | - |
| **PDWIS** (the estimate) | per-decision weighted importance sampling: each held-out call reweighted by how likely the engine was to make the moves it made | only as good as the overlap: a call where the engine would have done anything else counts for nothing |
| **DM** | the direct method, by fitted-Q evaluation on the held-out calls | exactly as wrong as the model, and it guesses wherever the log never tried the engine's move |
| **DR** | weighted doubly robust: PDWIS with the fitted model as a control variate | right if either the weights or the model is |

* **ESS** is the effective sample size: how many held-out calls the importance-sampling estimate really rests
  on. A deterministic table that disagrees with your agent often has a small one.
* **vs today** and **vs random** are paired differences: the same resampled calls on both sides.
* A state the engine's table does not know is left to today's agent, which is what serving does.

The verdict reads:

| verdict | when |
| - | - |
| Ahead of today on held-out calls | the PDWIS interval on the lift is above zero and the fitted model agrees |
| Behind today, do not ship it | the PDWIS interval on the lift is below zero |
| No detectable difference from today | the interval straddles zero |
| The estimators disagree | importance sampling says ahead, the model does not |
| Can't tell from these calls | fewer than 30 effective calls (or 1% of the held-out calls): the model's guess is shown, and named as a guess |

Each row opens into its three estimates, the reasons, and the table it decided, in the world's own words. An
engine that comes out ahead is a candidate for a canary under **Release**, with a control group; that is what
proves it.

## The API

| call | what it does |
| - | - |
| `POST /api/rl/evidence-sets` | build a set: `{env, source, path?, tagger: {kind, model_id?}, n, seed, max_turns}`; returns the job |
| `GET /api/rl/evidence-sets?env=` | sets, newest first, plus evidence files found on the volume |
| `GET /api/rl/evidence-sets/{id}/document` | the file itself |
| `POST /api/rl/engines/benchmark` | `{evidence_set, engines: "all" or names, seed, tables?, bootstrap}`; returns the job |
| `GET /api/rl/jobs/{id}` | the job, with `report` (partial while it runs) and `log_tail` (its last 200 lines) |
| `GET /api/rl/engines/benchmarks` | finished benchmarks; `/{id}` one in full; `/{id}/policy/{engine}` one table |

`evidence_set` also takes `sim:<world>` (that world's simulated episodes) and `vol:<path>` (an evidence file on
the `datasets` volume that no record points at).

> [!NOTE]
>
> If the dashboard restarts while a benchmark runs, the job's output is lost with it and the benchmark has to be
> run again. A build keeps going on the platform and its file still lands on the volume, where the list picks it up.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.