Decision models

Register a Hugging Face decision model, run it over real calls as a GPU job, and read its report live on the Decision Engine page

The Decision Engine page (Data lab, then Decision Engine, at /lab/engines) has two sections:

• Models: trained decision models. You do not fit them; you measure them.
• Algorithms: the decision engines that learn a state to move table from a world’s evidence (Bring your own engine).

A decision model is a classifier that answers typed questions about a state. There are three families, and each run loads or calls the model with its family’s own backend:

family	backend	models	what it is
typed-decision	laya	convaiinnovations/laya-multilingual	typed-question classifier, a probability for every option
kev	kev (kev.serve.Server, installed from git)	jaredpalmer/kev-0.8b, kev-4b, kev-9b	decision models on Qwen3.5 with the System One API shape, Apache-2.0
jev	typesafe-sdk	jev (TypeSafe’s current public model, jev-latest)	TypeSafe’s hosted System One API; external: see External models

Jev is the one model that is not loaded on your cards: it is called over the internet, in a CPU job, with the workspace secret TYPESAFE_API_KEY. It is a third-party service hosted outside the Kingdom, so it never receives real customer calls unless the tenant has allowed it, with a reference to the client’s written permission (External models).

import laya

agent = laya.load("convaiinnovations/laya-multilingual")
out = agent.predict({"transcript": ["agent: ...", "customer: ..."]}, questions, max_len=2048)
out["answers"]["outcome"]   # {"choice": "booked", "probabilities": {...}, "answer_confidence": 0.62}

Kev answers the same question in the same shape (its stated confidence is called confidence), so a run reports the same numbers for both. One difference to keep in mind: Kev was trained on states of up to 384 tokens, and a call is about two thousand. A run passes the whole call as the state and the report says how long the calls were and what share went past 384 tokens, so a weak Kev number reads as what it is: a model outside its training length.

Lev: our own fine-tuned decision model

A Lev is Levers’ own decision model: a fine-tuned checkpoint in a folder on a volume, registered in your workspace as lev-<tenant>-<version> (for example lev-acme-v1). Its base (Laya or Kev) is shown beside it. No Lev ships with the dashboard: each one is registered per workspace from its checkpoint folder, and how well it does is whatever your own runs measure on your own held-out calls.

A Lev comes with the exact question it was trained on, and a run asks that question, never the workspace default. The question is read from the checkpoint’s rl_agent_config.json when its training block records it, either as training.questions ({"acme": {"question", "options", "labels", "won"}}) or as training.question plus training.options; the job reads it again at run time, so the checkpoint is the authority. A checkpoint whose config records no question can have its training text pinned when you register it (questions on POST /api/rl/models). A Lev is asked only about the tenants it was trained for.

A Lev can also know its held-out set (held_out at registration: dataset, size, seed). Run it on anything else (another seed, another size) and the report warns that the sample may contain calls it was trained on; the Run panel has a button that puts the held-out set back.

A run asks the model about a sample of real calls and compares every answer with the label the platform already wrote on the call. It tells you, in about a minute of GPU time, whether a model is worth using, worth fine-tuning, or neither.

The platform’s label was written by a model, not a person. What a run measures is agreement, not accuracy. It is still the fastest honest answer to “is this model any use on my calls?”.

For labels a person has confirmed, build a gold set. The decision dataset kind in the Data Lab is made for it: reviewers answer your questions row by row and mark the events behind each answer (Decision gold sets).

Register a model

Laya multilingual, Kev 0.8B, 4B and 9B, and Jev are registered already. To add another, open Add a model in the Models section:

• Hugging Face Hub: paste the id (owner/name). The family is kev for jaredpalmer/kev-* and Laya otherwise, unless you pick one. The id is checked on the Hub before it is stored, so a typo fails at once and not after a GPU has started. A private or gated repo needs the cluster’s HF_TOKEN to see it.
• Checkpoint on a volume (Lev): the volume (laya-lab), the checkpoint folder and the tenant it was trained for. The folder is read: rl_agent_config.json makes it a Laya fine-tune, head.pt a Kev one. It is registered as the next lev-<tenant>-v<N> unless you give a version.

A base model is asked your workspace’s own outcome question for each world, built from that world’s data when you run it: “How did the call in transcript end?”, with one option per platform label found on the world’s real calls (the 20 most common), and the labels its Outcomes card calls won counted as wins. For a world whose calls are labelled Booked, Hot Lead, Warm Lead and Not Interested, the options are booked, hot_lead, warm_lead and not_interested. It is also asked the customer’s overall sentiment (positive, neutral, negative). A call whose label maps to no option is left out and counted. A world with fewer than two labels on its real calls has no question until it has, or until you pin one (questions on POST /api/rl/models).

Pick a dataset

The Run panel lists what a run can read, built each time you open it:

• Call-log exports on the call-logs volume, one folder per tenant. An undated file (acme/acme_calls.jsonl.gz) shows as ACME · full history; a dated one (acme/acme_calls_20260929.jsonl.gz) as ACME · export of 29 Sep 2026, so a later week sits beside the history under its own name. Until a run has counted a file, the picker shows its file name; the first run that reads it counts it, and the count shows afterwards.
• Arena calls: every world’s calls in the Arena registry, with their labels counted now.

The sample is stratified by default: an equal share per label, and what a small label cannot fill goes to the others. Untick it for a uniform draw that keeps the export’s own mix. The same seed draws the same calls.

Only the transcript text reaches the model: one string per turn (Laya keeps the end of a call longer than its window; Kev is given the whole call). Phone numbers and every other field of an export row stay on the volume, and a run’s stored answers carry no transcript.

Run it and watch the report

Run starts a headless levers9 job in your workspace (lab/rl/model_run.py, app lab-models) on two RTX PRO 6000 cards by default. It keeps going if you close the page, and a restart of the dashboard picks it up again from its platform task. The report fills in as it labels, polled every second:

1. Stages: load and sample, load the model, label calls, report.
2. Progress: calls done, calls a second, time left.
3. Agreement so far against chance (one over the number of options), won against not won, sentiment.
4. Confusion: rows are the platform’s label, columns the model’s answer, growing as calls are labelled. A model that answers the same thing for everything shows as one bright column.
5. Confidence against correctness: how often calls in each band of stated confidence actually agreed, with the 85% review threshold: below it a call would go to a person.
6. Calibration: one temperature fitted on half the calls and tested on the other half. A fitted temperature far above 1 means the model is better calibrated only because it admits it does not know.
7. Verdict: not usable without training (under 1.5 times chance), weak: fine-tune before use (under 60%), or promising: calibrate on the gold set.

Full log opens the job container’s own output, which prints the same report as aligned tables.

Finished runs are kept under Past runs with the same report. The per-call answers (call id, the platform’s label, the model’s choice and probabilities) are written to the model-runs volume as <model>/<run>/answers.jsonl.gz.

For scale: Laya labels a few thousand calls in about a minute on an RTX PRO 6000, roughly 13 ms a call.

Compare models

Compare models, above the Run panel, puts two or more models on the same calls and one leaderboard. Pick the models, pick the dataset, and run:

• The same sample for every model. The calls are drawn the way a run draws them (grouped by label, the same seed), inside each job, and each job reports a fingerprint of the call ids so the page can check they match. With a Lev in the comparison, its question is what the calls are grouped by and, unless you pick otherwise, the dataset, size and seed are its recorded held-out set, so the draw is exactly the calls it never saw in training. If the sample is anything else, the comparison warns that the Lev may have been trained on some of these calls.
• Each model is asked the question it was trained on. A Lev asks its own wording; a base model asks the workspace’s outcome question for the world. Each answer is scored against that model’s own mapping of the platform label, so the numbers compare; when wordings differ the report says so.
• Where it runs. Laya, Kev and every Lev run one after another in one GPU job (two RTX PRO 6000 cards by default, for the reason under Which card); Jev runs in its own CPU job, up to 8 requests in flight, each retried with backoff.

The leaderboard fills in while the jobs label. Per model:

column	what it is
Outcome agreement	answers that match the platform’s label, on the balanced (stratified) sample; a bar with the chance line on it
Real mix	the same agreement weighted to the label mix of the whole dataset, so a label that is rare in real life counts as rarely
Won vs not	whether the model and the platform agree on won against not won
Calibration	calibration error as shipped, then after one fitted temperature
Over 85%	the share of calls the model is at least 85% sure of (they would skip review), and how often those are right
p50 / p95	per-call latency: GPU time per call for a model on the cards, the round trip for Jev
Per 1,000 calls	cost: GPU seconds (and dollars when LEVERS9_COMPARE_GPU_HOUR_USD is set), or Jev’s input tokens (and dollars when LEVERS9_JEV_PRICE_PER_M_INPUT is set)
Verdict	the same three verdicts as a run

Below it: recall by label (of the calls under each label, the share each model got right) and, when both jobs have finished, how often each two models give the same answer (the same option when they share options, else the same side of won against not won).

Finished comparisons are kept under Past comparisons (the leaderboard, never a transcript). The per-call answers are on the model-runs volume as compare/<id>/<model>.jsonl.gz.

The scripted dataset, Decide page scenarios, holds the made-up calls from the Decide page in English and Saudi Arabic, each with the ending a person would give it. It is small (eight calls), needs no permission to send to an external model, and is how Jev can be tried before a tenant allows it.

From a terminal

levers9 model ls
levers9 model add jaredpalmer/kev-4b
levers9 model add --volume laya-lab --path models/acme-ft --tenant acme      # a Lev: lev-acme-v1
levers9 model datasets
levers9 model run laya-multilingual --dataset acme --n 5000       # follows the job and prints the verdict
levers9 model run laya-multilingual --dataset arena:acme --no-wait
levers9 model runs
levers9 model compare --models lev-acme-v1,kev-4b,laya-multilingual,jev --dataset acme   # a Lev alone defaults to its held-out set
levers9 model compare --models kev-4b,jev --dataset scripted:decide
levers9 model compares
levers9 model external-allow --tenant acme --reference "signed letter ACME-2026-14"

A run or comparison started here is the same job the page draws, so it shows up there live.

The API

call	what it does
GET /api/rl/models	the registry, with each model’s questions, run count and last result
POST /api/rl/models	register {hf_id, family?, label?, questions?}, or a Lev {volume, path, tenant, version?, questions?, held_out?}; 422 when the Hub does not have it or the folder is not a checkpoint
DELETE /api/rl/models/{id}	forget a model; its runs are kept
GET /api/rl/datasets	the datasets a run can read
POST /api/rl/models/{id}/run	{dataset, n, seed, stratify, gpu?, gpu_count?}; answers a ledger job of kind model-run
GET /api/rl/jobs/{job}	the job, with its live report, the last 200 log_tail lines, task_id and container_id
GET /api/rl/models/runs, GET /api/rl/models/runs/{run}	finished runs and one run’s full report
POST /api/rl/models/compare	{models: [ids], dataset?, n?, seed?, stratify?, tenant?}; answers a ledger job of kind model-compare whose report.models fills per model as it labels; 422 from the external-model guard
GET /api/rl/models/compares, GET /api/rl/models/compares/{id}	finished comparisons and one comparison’s leaderboard, recall grid and pairwise agreement
GET /api/rl/external-models?tenant=, PUT /api/rl/external-models	a tenant’s external-model setting; see External models

Which card

The card and the count default to LEVERS9_MODEL_RUN_GPU (RTXPro6000) and LEVERS9_MODEL_RUN_GPU_COUNT (2) on the dashboard API. Two cards, not one, on purpose: on the attached RTX PRO 6000 pool a one-card job can land on a one-card node that never starts it, and it sits PENDING while its container reads RUNNING. The model needs one card; the second keeps the job off that node. See Multi-GPU for how cards are chosen.