RL Arena

Listen, teach, ship: post-training a call agent with human feedback

The Arena is a working preview. The game loop, the scoring and the reply API are real; the learning underneath is prompt-level while the maths of the reinforcement-learning step (the reward model and the policy update) is being worked out. Use the numbers to explore the loop, not to make decisions yet.

Worlds: one per tenant

The Arena opens on a list of worlds. A world is one tenant’s arena: its real calls, an agent with a job, customers who answer, a reward that scores every reply, and its own versions of the agent. Where a culture pack has tenants, each tenant is its own world, so a property-sales tenant and a rent-split qualification tenant have separate calls, versions, scores and objectives rather than being two tabs of one arena. Create a world makes another, cloning an existing pack’s rubric, guardrails and personas so it can run an episode immediately, or starting from a generic rubric you then edit under Environment.

Inside a world, three features. The Arena path is the loop: calls in, hot spots, duel, train, evaluate. Environment is the reward and the world’s dynamics: the rubric and guardrails the judge scores with, the objective, the judge, the outcomes map, and the simulated customers. Policies is every version of the agent: where it came from, its checkpoint, what it runs on, what v1 starts from, and where one is written by hand. In RL terms the Environment is r(s, a) and the Policies are π(a | s); the API still calls a world an environment (/api/rl/environments).

Every card names the four parts that make it a decision process before anyone plays:

part	what it is here
Agent	the policy: the prompt (and later the weights) that writes the next reply
State	the dialogue and the stage, which the agent sees; trust, the drawn objection and the persona, which it does not
Action	one reply in the pack’s language, constrained by the guardrails
Reward	the judge’s weighted rubric score, 0 to 100 per agent turn

The model behind a world

Model on an environment writes the decision process down and lets you change it. It is a finite-horizon, partially observed process: the agent sees the conversation, while trust, the objection and the persona are hidden, which is what makes it a POMDP rather than a plain MDP.

• Transition: the customer answers from the persona and the stage, then trust moves with the reply just scored, trust ← clip(trust + (r − 50)/100 × 0.6, 0, 1). The stage advances one step. Seeded, so two versions meet the same customers.
• Horizon H: agent turns per episode, four by default.
• Discount γ: 1.0 by default, which is well defined because the horizon is finite.
• Return G = Σ γᵗ·rₜ + γᴴ·terminal − c·violations: the one number an optimiser would maximise. The terminal reward is what the call was worth in the end, the part a per-turn rubric never captures: a reply can score well and still lose the customer. The violation penalty c is charged per guardrail breach.

Changing the objective recomputes the return of existing episodes; their per-turn rewards are untouched.

The decision engines below learn exactly this return. What ships to a live agent today is still the preference-based step, a contextual bandit rather than a policy gradient, because the engines are not yet reachable from the Arena’s pages. So there is still no credit assignment across turns in production: a reply that sets up a booking two turns later gets no credit for it. The return above is what a value function is trained to predict, which is why it is defined first.

Decision engines

An engine takes what happened and returns a move for every state: stage:trust:objection in, one of the eight moves out. That mapping is the decision engine, in the form the Arena can render, compare and promote.

Fourteen are registered, across three kinds of evidence. Which kind matters more than which algorithm, because it decides whether an engine can run at all on a given pack:

Learns from	Engines	What that means
Stored episodes	Q-learning, SARSA, Monte Carlo, TD(λ), value iteration, policy iteration	Offline and cheap. Cannot try a move nobody ever tried
Preference pairs	DPO, Bradley-Terry reward model	Offline. A pair means one move beat another from a state, decided by judged return
A live simulator	REINFORCE, PPO, online Q-learning, online SARSA, model-based, R-max	Explores, and costs one real episode per simulated call

An engine that is handed evidence it cannot use refuses before it starts, saying which kind it needed and how to get some: rather than returning an empty result or failing somewhere inside an algorithm.

Two engines re-solve the whole model every time they learn something: R-max and model-based learning. That is correct on the small gridworlds they were written for and quadratic in states here, where a pack has 176. Both estimate their cost first and refuse with the number and the knob that would change it, because an engine that cannot finish must not look like one still running. R-max also takes its own discount, since its optimistic value is R_max/(1−γ) and the Arena’s γ is deliberately 1.

Preferences, at two levels

The Arena already learns from human preferences, and these engines do too, and they are not the same preferences. A person picking between two replies is choosing text, and those pairs ship to a GPU job for a real DPO weight update on the language model. The engines here work one level up, over the eight discrete moves, where a preference means one move beat another from a state and the evidence is the judged return.

Nothing in a stored human preference records which move produced the winning reply, so the two are kept apart rather than merged: the move-level engine decides what to do, the text-level DPO decides how to say it.

Adding one

An engine is one file. It declares what it learns from and its hyperparameters as data, then implements fit. Because the parameters are data rather than a function signature, the API lists them and a form can be drawn for an engine nothing has heard of, so a new algorithm needs no change to the API or the web app.

The loop

The RL Arena is where the call agent gets better, and it is played rather than administered: reinforcement learning from human feedback as seven missions. Open Data Lab → RL Arena in the dashboard; the Play tab is the game board and its Home guide ticks the missions off as you go. A longer walkthrough for a first-time reader is in the repository at docs/rl-tutorial.md.

The seven missions, and the RL behind each

#	in the game	what you do	in RL terms
1	Bring calls in	real conversations arrive from your call platform, a dataset or the demo corpus; every reply is scored	episodes and a reward signal from the real environment
2	Find the hot spots	thousands of moments are grouped into a few dozen situations, weakest first, on a map	clustering states, prioritising where the policy is weak
3	Duel	one moment, two hidden and shuffled replies; pick the better, tick why, +10 XP	pairwise human preferences, the training signal of RLHF
4	Level up (train)	your picks become a new, numbered version of the agent	a policy update: prompt-level now, DPO with LoRA for weights
5	Boss fight (evaluate)	new vs old on the same customers and seeds: win rate, radar, regressions	held-out evaluation with fixed seeds
6	Crown the champion (deploy)	only a version that won goes live; this is the deploy step, there is no separate button	promotion gate
7	Connect (serve)	your call platform asks the champion for its next reply; every reply comes back as a live call	the deployed policy, with online data collection

Every mission card shows its plain name under the game name (train, evaluate, deploy, serve), and clicking the card runs it: training and deploying are missions 4 and 6 on the Play tab, not separate screens.

Level is the agent’s version number, XP is ten per pick, “in a row” counts picks without leaving the Duel (the flame lights at 3, roars at 5 and 10, with a banner at each), the day streak counts days in a row with at least one pick (it breaks after a day without one, so the hub nudges you to duel today), and a rule violation is a reply that used a phrase the culture pack forbids.

The loop

1. Listen. Real calls arrive through a webhook, a dataset or paste. Every agent reply is scored on the market’s rubric by a judge, and thousands of moments are grouped into a few dozen scenarios so you see where the agent struggles.
2. Teach. For one moment, two drafts side by side. Pick the better one and tick why. Twenty picks take ten minutes; each is a lesson. See Compare & approve.
3. Ship. Train turns the picks into a new version. Evaluate runs it against the old one on the same customers. Make it live only when it wins.

The tabs

The Arena is played, not administered. Play shows the agent’s level (its version), XP (one pick is 10 XP), a streak, and six missions in order with a ring each; the big Play button starts whichever mission is next.

tab	what it is for
Play	level, XP, the six missions: bring calls in, find the hot spots, duel, level up, boss fight, crown the champion
Calls	real calls in, each turn scored; webhook setup
Hot spots	the map of situations across all calls, ranked by where the agent is weakest
Duel	one moment, two replies, pick the better one with 1 / 2 or the arrow keys, tick why, Enter for the next. Opened from a hot spot or a call it works through that set; opened on its own it works through the review queue, the forty weakest real replies from the top hot spots. The header says which moment of how many, from which call and turn and hot spot; Up next lists them and jumps to any. Simulated customers only while a world has no calls
Boss fight	the new version against the old one on the same customers
Arena (advanced)	simulated customers talk to a policy; watch, fork, replay
Versions (advanced)	policy versions with lineage; make one live
Culture pack (advanced)	the rubric, guardrails and personas for this market
Compare (classic), Details (advanced)	the earlier form-based Compare tab and the numbers-first Home

Connect: the inference layer

Mission 7 opens the Connect dialog: the endpoint, the key, an example, what the champion has served (replies, average score, violations), and a box to talk to the live agent as a customer. The call platform posts the conversation so far and gets the crowned version’s next reply, judged and rule-checked:

curl -X POST https://levers9-api.uselevers.com/api/rl/agent/<tenant>/reply \
  -H "X-Levers-Webhook-Key: <secret>" -H "Content-Type: application/json" \
  -d '{"call_id": "CALL-0142", "turns": [{"role": "customer", "text": "..."}]}'
# {"reply": "...", "policy": {"name": "v2 · trained", "version": 2}, "judge": {"total": 76, ...}, "violations": [], "latency_ms": 1400}

Every exchange is filed as a live call (source live) under the tenant, so production traffic feeds Calls, Hot spots and the next Duel. GET /api/rl/agent/<tenant>/stats returns what has been served, by version.

Send the key in the X-Levers-Webhook-Key header, never in the URL. A key in the URL (?key=) ends up in load balancer and proxy logs. It still works for now, and every request that uses it is logged (without the key) so the dashboard’s operator can find the sender; once nobody uses it, LEVERS9_RL_WEBHOOK_QUERY_KEY=false turns it off. A key that has been in a URL can be replaced with POST /api/rl/webhook-info/rotate {"pack": "<pack>"}, by an admin of the workspace signed in to the dashboard. The old key stops working at once, so tell the call platforms that use it first.

Who can use the Arena

Everything in the Arena (its calls, versions, judge, webhook keys, the Decision API, Models and Compare) lives in one workspace. When the dashboard’s operator sets LEVERS9_ARENA_WORKSPACES to that workspace’s id, only its members get the Arena: anyone else gets a 403 that names the workspace to switch to, the Lab hides the Arena for them (GET /auth/me says "arena": false), and Insights doesn’t list the Arena’s tables. The call webhook and the agent routes aren’t affected: a call platform has no session, and they keep checking the pack key. With the setting empty, or sign-in off, everyone gets the Arena, as before.

Jobs are filed under the workspace that started them. GET /api/rl/jobs lists your workspace’s own, and a job of another workspace is a 404. Jobs from before that are filed under the Arena’s workspace when the dashboard starts.

Endpoints the Arena calls

A version’s endpoint and the judge’s are addresses you type, and the dashboard calls them from inside the cluster. So they’re checked, when you save them and again when they’re called: a deployment on the platform’s own gateway (…/asgi/<name>/v<N> or …/endpoint/<name>/v<N>), OpenAI and Anthropic, hosts the operator lists in LEVERS9_PROVIDER_HOSTS, and any other host over https whose addresses are all public pass. A private, loopback, link-local or cluster address doesn’t, and neither does the controlplane’s own API. Right now the dashboard only logs what it would refuse (LEVERS9_EGRESS_GUARD=observe); with enforce a refused address is a 422 that says why, and nothing is sent to it.

A version you point at another host leaves its token behind unless you give a new one with it. Moving between two deployments on the same host, or between two addresses of the platform’s gateway, keeps it.

Vocabulary

word	meaning
episode	one conversation: turns, a score per reply, an outcome (viewing booked / follow-up / dropped)
reward	the score of one reply, 0 to 100, from the rubric
judge	whatever produces scores: cue-based rules until a model key is configured, then a model reading the whole conversation. Reviewers can overrule it
preference	a reviewer’s pick between two replies for one situation, plus the reasons ticked
policy version	one agent version; keeps its parent and the number of preferences it learned from
scenario	a group of similar situations across many calls, found automatically
snapshot	anything frozen: a preference version (TSV on the volume), a policy version, a forked conversation
tenant	one client; its own culture pack, webhook key and filters

The Arena tab

Simulated customers with personas, moods and hidden needs talk to a policy. A customer’s trust moves with each judged reply and decides the outcome. Start an episode, watch it turn by turn with the score and the rubric reasons for each reply, and fork it at any turn to continue with another version and see where the two diverge. Real calls can be turned into episodes too, so the synthetic customer continues from where the real one hung up.

Without model keys the arena runs a deterministic mock: the customer follows a staged script and reply quality depends on which rubric skills the policy’s prompt carries, so a trained version visibly improves. With OPENAI_API_KEY or ANTHROPIC_API_KEY in the dashboard API’s environment, or a vLLM endpoint, policies and the judge use real models; the UI shows which is active.