Call logs

Getting real calls in: webhook, dataset import, paste; how each turn is scored

In the game this is mission 1, Bring calls in: the environment the agent is learning from.

Real conversations are the ground truth. The Call logs tab lists them per tenant, each with its turns, the judge’s score for every agent reply, rubric violations, and the outcome.

Three ways in

Webhook

Open Webhook for your call platform. It shows the URL and the secret for this tenant. Configure the telephony or agent system to POST each finished call:

curl -X POST https://levers9-api.uselevers.com/api/rl/calls/webhook/<tenant> \
  -H "X-Levers-Webhook-Key: <secret>" -H "Content-Type: application/json" \
  -d '{"call_id": "CALL-0142", "started_at": "2026-09-06T10:12:00Z",
       "customer": {"name": "أبو محمد", "city": "Riyadh", "phone": "+9665xxxxxxx"},
       "agent": {"id": "agent-7"}, "outcome": "follow_up",
       "turns": [{"role": "customer", "text": "..."}, {"role": "agent", "text": "..."}]}'

turns may be replaced by a transcript string of customer: … / agent: … lines. Phone numbers are masked on arrival. The path accepts the tenant’s app id, its id, or a culture pack name.

Send the secret in the X-Levers-Webhook-Key header, not in the URL: a ?key= ends up in proxy logs, is logged as a warning, and will stop working (see the Arena guide, which also covers rotating a key).

From a dataset

Import from dataset takes an stt dataset from the Data Lab: rows are grouped by call id, and the human transcripts and roles become turns. Transcribe recordings, annotate, then import: that is the audio route.

Paste

Paste a transcript for a quick look. Useful in a demo; not how you get to thousands.

A platform export

Most call platforms export a CSV before they will POST a webhook. lab/rl/import_call_logs.py turns one into scored calls:

cd lab/rl
python import_call_logs.py export.csv --tenant acme --limit 2000 --replace-imported

It expects ;-delimited rows with the transcript as Assistant: / User: lines, and maps each tenant’s own outcome vocabulary onto the Arena’s four; a label with no mapping is reported rather than silently becoming unknown. Calls that reached a voicemail menu instead of a person are dropped, and the sample is stratified by outcome so a rare but valuable ending does not vanish. --dry-run reports what would arrive and writes nothing; --replace-imported resizes a previous import instead of duplicating it.

Seed demo data fills the tenant with a synthetic corpus (marked mock) so the map and the review queue have something to show; Delete mock calls removes it again. Start over goes further, for the day real calls replace the demo: it removes the mock calls and everything reviewed, trained or measured on top of the calls (episodes, picks, evaluations, scenarios, engine runs), and keeps the real calls and the trained versions. It asks for the pack id back, because picks cannot be got back. From a script: POST /api/rl/packs/<pack>/reset {"confirm": "<pack>"}, with "versions": true to drop trained versions too (a fresh v1 baseline is made so the world can still run an episode).

Outcomes: the platform’s labels

A call platform has its own words for how a call ended (one says Hot Lead, Warm Lead, Not Interested…; another Booked Visit, Requested Info, Disqualified…). The Arena keeps four endings, because its simulator and the reward’s terminal value are defined on them: viewing booked, follow-up, dropped, unknown. Each call shows the platform’s label beside the Arena’s ending, and Environment → Outcomes lists every label found in the world’s real calls with its count and what it means here (won, follow-up, lost, ignore). The importer’s fold stands until you change one; then every call with that label folds your way, the scenario boards gain a Won column (the share of a scenario’s moments in a call labelled won), and calls that arrive later with that label fold the same way. Insights’ rl:calls carries the label, grade, sentiment and how the call ended as columns. From a script: GET /api/rl/packs/<pack>/outcomes, PUT /api/rl/packs/<pack>/outcomes {"mapping": {"Cold Lead": "lost"}}.

Scoring

Every agent turn is scored on arrival against the tenant’s culture pack: each rubric dimension 1 to 5, weighted to 0 to 100. The score on arrival is triage by cue phrases, which is quick and needs no model, and on real calls it is close to flat: two thousand real Arabic calls scored 51 whether the lead was hot or lost, and the compliance guardrail never fired. Rescore all calls is where stored calls meet a model. With a judge configured (Environment → Judge: vLLM endpoint, model, and the workspace token for an endpoint behind the gateway; or PUT /api/rl/settings; the header badge turns from amber to green and names the judge, next to what the agent and the customer run on) it scores every agent turn with the call so far as context, keeps the judge’s reason per dimension, and counts a guardrail dimension at 1 or 2 as a violation. Calls already judged by that model against the current rubric are skipped, so a second rescore is free; a call with a turn the model failed on (unreachable, or an answer that was not JSON) is judged again, and only that turn goes to the model. A rubric edit makes every call stale again. The button is at the bottom of Environment → Culture pack (always, with the judge it will use), and again on the Judge card right after you save a judge; the run shows at the top of the world’s page while it lasts. Jobs survive a restart of the dashboard: one that was running is marked interrupted, and a rescore or regroup is started again on its own and picks up where it stopped. A job that was interrupted or failed stays on the world’s page with what it had reached, and Resume (or Retry) starts the same work again explicitly; Dismiss takes it off. From a script: POST /api/rl/jobs/<id>/resume. Without a judge it is the cue rescore. A reviewer can overrule either score while comparing.

Reading a call

Open a call for the whole conversation with per-turn scores, the weakest moments marked, Similar calls (nearest by situation), and Review this moment which sends one turn to Compare. Ignore hides a call that should not count (a test, a wrong number).

Tenants

The switch in the RL header picks the tenant: it drives the culture pack, the webhook key and every filter on Call logs and Scenarios. Tenants are declared in lab/rl/tenants.yaml (a name, the pack, the app ids the webhook accepts).