Decision gold sets

Build and annotate a human gold set for a decision model: answers to typed questions, the events that caused them, counterfactuals, and the review of a logged decision

A decision model answers typed questions (yes or no, one of a few options, a level on a scale) and gives each answer a probability. It does not write text. It is the kind of model served on the Decision API. A gold set is a set of examples whose labels a person has confirmed: you measure a model against it, and with enough rows you train on it.

The decision dataset kind (shown as Decision model) is where that gold set is made. Its rows are about leads: a lead is a customer being followed up, such as someone asking about a property or a policyholder whose insurance is due for renewal. A row is one decision point of a lead: the moment something had to be decided about it. It holds:

• the lead’s journey: its past events, oldest first;
• the new call;
• optionally a logged decision: the action an engine took at that moment, with its propensity (the probability the engine gave that action) and its outcome (whether the lead then succeeded). The engine is whatever picks the next action for a lead: a policy, a set of rules, or the Decision API’s decision step.

A reviewer does four things with a row:

1. answers each question;
2. marks the events that caused each answer;
3. says how an answer would change without an event;
4. reviews the logged decision.

The lifecycle is the one every dataset has: import, optional pre-label, annotate, export.

How it differs from the language-model kind

	llm (Language model)	decision (Decision model)
a row	a conversation	one decision point of a lead: its journey, the new call, optionally a logged decision
what a person writes	the target reply, as free text	one option per typed question, and the events that caused it
pre-label	a language model drafts the reply	a Decision API drafts every answer
what leaves	TSV	TSV, the gold set as JSON lines, the questions as a schema
the model it is for	one that writes the next reply	one that scores fixed options

Choose decision when the model picks from fixed options: where a lead stands, whether price is still an objection, which action to take next. Choose llm when the model writes.

The questions

The questions are System One questions: the same JSON the Decision API takes in questions. Each one has an id, a type, the question in words and its options.

type	the answer	its options
choice	one of a few options	criteria, as {key: description}, two or more
noul	yes or no	true and false
score	a level on an ordered scale	criteria, a list of levels from lowest to highest; the options are 0, 1, 2 and so on

A questions file for a property sales lead:

{
  "stage": {
    "type": "choice",
    "question": "Where does this lead stand after the call?",
    "criteria": {
      "ready_to_reserve": "asked to reserve, to pay a deposit, or for a written offer",
      "wants_viewing": "asked for a viewing or agreed to one",
      "needs_information": "asked for details and committed to nothing",
      "not_interested": "said no, or asked not to be called again"
    }
  },
  "price_objection": {
    "type": "noul",
    "question": "Is price still an open objection for this lead?"
  },
  "urgency": {
    "type": "score",
    "question": "How soon does the lead want to move?",
    "criteria": ["no date in mind", "within a few months", "within this month"]
  }
}

The words of the question go in question. instructions, the name the Decision API’s own examples use, is read the same way, so a questions file you already send to the Decision API works as it is.

Write each description as the test a reviewer applies, not as a restatement of the key. Two reviewers who read “asked to reserve, to pay a deposit, or for a written offer” agree more often than two who read “ready”.

The questions are part of the dataset’s config, as config.task.questions, and a decision dataset cannot be built without them.

The actions

A row may carry a decision the engine logged. The actions are what that decision chose from: a JSON object of {key: description}, with no_action among them. They are optional, stored as config.task.actions, and they are what a reviewer picks from when saying what they would have done.

{
  "send_offer": "send a written price offer for the unit they viewed",
  "hand_to_rep": "hand the lead to a human sales rep",
  "call_again": "make an automated follow-up call",
  "no_action": "do nothing for now"
}

The record format

The import reads JSON lines (one record per line), records printed over several lines one after another, or a file holding a JSON array of records. A webhook takes the same record as its body. One complete record, spread over several lines here for reading:

{
  "id": "lead-2041-d3",
  "lead_id": "lead-2041",
  "decision_time": "2026-09-24T16:30:00+03:00",
  "segments": [
    {"role": "agent", "text": "Good afternoon. I am calling about the two-bedroom unit you viewed last week."},
    {"role": "customer", "text": "Yes, we liked it. The price was the problem, but your colleague sent a new payment plan."},
    {"role": "agent", "text": "Does the new plan work for you?"},
    {"role": "customer", "text": "It does. Please send the offer in writing so I can show my brother."}
  ],
  "journey": [
    {"id": "ev-1", "time": "2026-09-10T11:02:00+03:00", "source": "CRM", "type": "stage",
     "text": "Inbound lead from the website, asked about two-bedroom units", "provenance": "confirmed"},
    {"id": "ev-2", "time": "2026-09-14T18:40:00+03:00", "source": "AI call", "type": "label",
     "text": "First call: price objection, asked for a lower down payment", "provenance": "model"},
    {"id": "ev-3", "time": "2026-09-20T10:00:00+03:00", "source": "calendar", "type": "booking",
     "text": "Viewing attended, unit B-204", "provenance": "confirmed"},
    {"id": "ev-4", "time": "2026-09-22T09:15:00+03:00", "source": "call center", "type": "message",
     "text": "Rep sent a revised payment plan: 10% down, 24 months"},
    {"id": "ev-5", "time": "2026-09-25T12:00:00+03:00", "source": "payments", "type": "payment",
     "text": "Deposit received for unit B-204", "provenance": "confirmed"}
  ],
  "decision": {"action": "send_offer", "propensity": 0.9333, "policy": "engine-v3",
               "outcome": null, "time": "2026-09-24T16:31:00+03:00"},
  "labels": {"stage": "ready_to_reserve"},
  "causes": {"stage": ["ev-3", "ev-4"]}
}

field	what it is
id	the row’s id, unique in the dataset. Without one the row is named after its lead
call_id or lead_id	the lead. Every decision point of one lead carries the same value
decision_time	when the decision was made, in ISO 8601. An event at or after it is not history
segments	the new input: the call as [{role, text}]. turns (a list of "agent: ..." and "customer: ..." lines, as the Decision API takes a call) or context or text (the call as plain text, one turn per line) are read too
journey	the history: the lead’s events, oldest first, each {id, time, source, type, text} with an optional provenance. events is read as the same field. An event without an id is given one (ev-1, ev-2, and so on). An empty list is a lead with no history
decision	optional: the decision the engine logged, {action, propensity, policy, outcome, time}
labels	optional: answers you already have, as {field: option}
causes	optional: causes you already have, as {field: [event ids]}
questions	optional: System One questions the record brings with it. The questions set when the dataset is created take precedence

A record needs a call, a history, or both. Some of these words need a definition:

• An event is one thing that happened to the lead: a stage change in the CRM, a call, a booking, a payment, a message. Its source says where it came from and its type what it is. Both are free text; the Causality builder uses the same fields.
• Provenance says how far to trust an event’s text: model (a model wrote it), confirmed (a person checked it) or corrected (a person changed it).
• The propensity is a number between 0 and 1. An engine that sometimes tries an action other than its best one logs it with every decision (Recommendations). policy names the engine or the version of it that decided.
• The logged outcome is yes or no. Leave it empty when it was not known when the record was written.

Two things in the example are worth a second look. ev-5 is dated after the decision time, so it is not part of the history: it cannot be marked as a cause, and it is left out of the exported journey. And the labels and causes the record brings are imported as drafts, never as human labels. A row counts only once a reviewer has saved it.

Tables

A tabular source (TSV, CSV, parquet, a query) maps its columns to these roles:

role	the column holds
call_id	the lead
context	the new call, as text
journey	the events, as JSON text in the cell
decision_time	the ISO 8601 time
decision	the logged decision, as JSON text in the cell

Sources

A decision dataset reads from every source that feeds the language-model kind: a folder on the datasets volume, files dropped in the browser, S3, GCS, BigQuery, Hugging Face, a URL list, a webhook, or a query against your own Postgres or MySQL. It can also be built from a dataset you already have: a language-model dataset, or the validated rows of a speech-to-text dataset. Those give one row per call with an empty journey, which is the right start when you have calls and no record of events yet.

What a row holds

After the build, every decision point is one line of rows.tsv in the dataset’s folder on the datasets volume:

column	what it is
id	the row’s id
call_id	the lead
decision_time	when the decision was made
context	the new call: its turns as role: text, on one line
journey	the lead’s events, JSON on one line
decision	the logged decision, JSON on one line; empty when there is none
draft_output	the drafts for this row, JSON on one line
provider	who wrote the drafts
status	pending, validated or flagged
split	train, val or test. It is drawn from the lead, so every decision point of one lead lands in the same split

Create one

From the dashboard

Open Datasets, then New dataset. Pick the kind Decision model, then a source, then whether a model drafts the answers. On the last step, name the dataset and give its Questions: paste the JSON, or upload the file. The dialog lists back each question it read, with its type and its number of options, so a wrong paste shows before the build. Actions are optional: one per line, as key: description. The build runs on the cluster and the page follows it, as for every kind.

From the CLI

levers9 dataset upload ./decisions.jsonl --to leads-gold        # onto the datasets volume, in uploads/leads-gold
levers9 dataset create leads-gold --kind decision --path uploads/leads-gold \
        --questions questions.json --actions actions.json

--questions is sent as task.questions; leave it out only when the records in the folder carry their own questions. --actions is optional and is sent as task.actions. The CLI builds from a folder on the datasets volume; pick any other source in the dialog.

For a table, name the columns with --map. inspect prints the columns and a sample first:

levers9 dataset inspect uploads/renewals --kind decision
levers9 dataset create renewals-gold --kind decision --path uploads/renewals \
        --questions questions.json \
        --map call_id=PolicyNo --map context=CallText --map journey=Events \
        --map decision_time=DecidedAt --map decision=Logged

Pre-label with the Decision API

Pre-labelling is optional. The provider for this kind is decision_api, with an endpoint: the URL of a Decision API deployed in this workspace (the endpoint line of levers9 decide status). For every row it drafts:

• every answer, with the model’s confidence in it;
• the events the model read most as evidence for that answer, when the served model returns them;
• a chance per action, when the response carries one.

levers9 dataset create leads-gold --kind decision --path uploads/leads-gold \
        --questions questions.json --actions actions.json \
        --provider decision_api --endpoint https://levers9.uselevers.com/asgi/lev/v1

To draft a dataset that already exists, use Pre-process on its page, or POST /api/datasets/{id}/prelabel with {"provider": {"type": "decision_api", "endpoint": "..."}}. Only rows without a draft are drafted unless you ask to overwrite, and a draft never overwrites a human label. The provider none imports only: reviewers start from an empty row.

A draft pulls a reviewer toward the model’s answer. If the gold set will measure the model that drafted it, the agreement comes out too high. For rows that will measure a model, import without pre-labelling, or draft with a different model.

The annotation screen

Open the dataset. The row list is on the left, as for every kind (Annotation). The editor is laid out the way a row is reviewed: reading first, answering second.

Reading. The journey lists the events before the decision time, oldest first; a lead with no history says so. The new call follows it. Events at or after the decision time are listed apart and dimmed: they are not part of the history and cannot be marked as causes.

Answering. The questions come one block each, and one question is active at a time. Then the logged decision, then a rationale.

part	what the reviewer does	saved as
Answers	picks one option for each question, or None of these when no option fits	answers
not sure	marks a question they are not sure of. An answer given with it is kept and counts for half in the export	unsure
Causes	with a question active, clicks the events in the journey that caused its answer: the ones without which they would have answered differently. Each lands under the question as a chip. No event marked means the answer comes from the call alone	causes
Counterfactual	on a cause’s chip, picks what the answer would be without that event. Removing the cause removes it	counterfactual
The logged decision	reads the action the engine took and what was logged with it; sets the reviewed outcome as they find it (yes, no or unknown), whatever was logged; picks the action they would have taken. A row with no logged decision has no outcome to review	outcome, action
Rationale	writes, in a sentence or two, why. Optional	rationale

Drafts. A model’s draft is shown preselected, with its confidence and the probability it gave each option. It is not the reviewer’s answer until they confirm it: by clicking it, by pressing its number, or by validating the row, which confirms every draft left as it was. The events the draft read most are offered as suggested causes, and each action shows the draft’s chance when there is one.

Saving. Edits are saved automatically a moment after the last one, and at once when you move to another row. Validate & next moves the row from pending to validated; every question needs an answer or a not sure mark first. Flag & next marks a row that is unclear or unusable. When the API refuses a label, its sentence is shown at the top of the editor.

keys	action
1 to 9	pick that option of the active question. The number after the last option is None of these
Enter	confirm the draft of the active question
↑ ↓	move between questions
U	not sure
⌘ Enter (Ctrl on Windows)	validate and go to the next row
⌘ .	flag and go to the next row
⌘ ↑ ⌘ ↓	previous or next row

Statuses, filters, named links for outside reviewers and the NDA gate work as for every kind (Sharing).

The labels

Every save appends to labels.tsv. Nothing is ever edited in place, and each line carries the reviewer’s name. A label’s value is a string; the first four hold JSON. In them, a field is a question’s id (stage, price_objection) and an option is one of that question’s option keys.

label	value	meaning
answers	{field: option_key}	the answer to each question. "__none__" is none of these. A noul question takes "true" or "false", a score question the level’s number as a string
unsure	[field]	the questions the reviewer was not sure of
causes	{field: [event_id]}	the events that caused each answer
counterfactual	[{field, event_id, answer}]	without this event, the answer to this question would be this option
outcome	yes, no or unknown	the reviewed outcome of the logged decision
action	an action key	what the reviewer would have done
rationale	free text	why

Through the API, one save is one request:

curl -s -X POST $DASH/api/datasets/$ID/labels -H "Authorization: Bearer $TOKEN" -H 'content-type: application/json' -d '{
  "row_id": "lead-2041-d3",
  "status": "validated",
  "user": "reviewer-1",
  "labels": {
    "answers": "{\"stage\": \"ready_to_reserve\", \"price_objection\": \"false\", \"urgency\": \"1\"}",
    "unsure": "[\"urgency\"]",
    "causes": "{\"stage\": [\"ev-3\", \"ev-4\"], \"price_objection\": [\"ev-4\"]}",
    "counterfactual": "[{\"field\": \"price_objection\", \"event_id\": \"ev-4\", \"answer\": \"true\"}]",
    "outcome": "yes",
    "action": "send_offer",
    "rationale": "The revised plan removed the price objection. She asks for the offer in writing."
  }
}'

The API checks every save and answers 422 with a sentence when something does not fit: a field that is not one of the dataset’s questions, an option that is not one of that question’s, an event id that is not in the row’s journey. An event at or after the decision time cannot be a cause.

What each label is for

• Answers are what the model should say. In training they are the target for each question. In evaluation they are the yardstick: the share of rows where the model’s answer is the reviewer’s, for each question and for each option. An answer marked not sure counts for half.
• Causes tell you whether the model read the right events. A served model can return the events it weighed most for an answer (explain); compare them with the events the reviewer marked. A model that gets the answer right from the wrong events is right by luck.
• Counterfactuals become twin records: a copy of the row with one event removed, and the answer the reviewer said would follow. They test whether the answer follows its cause. Ask the model about the row and about its twin: if the answer does not change when the cause is gone, the model did not use it. What a model reads most is a clue, not proof; the twin is the test.
• Outcome and propensity are for learning what an action changes. The history up to the decision, the action and the reviewed outcome make one example of what happens after that action. Weighting each example by one over its propensity corrects for the engine having preferred some actions, and the same logs can then score a new policy before it runs. An unknown outcome is left out; it is not a failure.
• The reviewer’s action measures agreement with a person in the review lane: the decisions the engine sends to a person because they are close or doubtful (Recommendations). Where the engine’s action and the reviewer’s differ is where to look first.
• The rationale is for the people who read the disagreements. It is not a label a model learns from.

Export

Three files leave a decision dataset.

route	what it is
GET /api/datasets/{id}/export.tsv and POST /api/datasets/{id}/import	the spreadsheet round trip, as for every kind (Annotation). The label columns hold the same values as labels.tsv. The file has no quoting, so the quotes inside a JSON cell are written with a backslash in front ({\"stage\":\"qualified\"}); the import reads that form, a spreadsheet’s re-quoted form and plain JSON alike
GET /api/datasets/{id}/export.jsonl?status=validated&twins=1	the gold set: one JSON object per line
GET /api/datasets/{id}/schema.json	the questions as a decision-model schema

levers9 dataset export leads-gold --format jsonl --status validated --twins -o gold.jsonl
levers9 dataset export leads-gold --format schema -o schema.json
levers9 dataset export leads-gold --status validated -o leads-gold.tsv     # --format tsv is the default

Without -o the export goes to stdout, so it can be piped.

These are the owner’s exports, and they hold the text as it is. If the dataset’s policy is Mask, a share link’s gold set holds the masked call, journey and rationale; add ?text=masked to your own export.jsonl or export.tsv for the same file, or --text masked to levers9 dataset export.

A decision dataset is not published to a bucket: that copies recordings and a transcript manifest, and this kind has neither. Download the gold set instead.

The gold set

One line of export.jsonl for the record above, wrapped here for reading:

{"id": "lead-2041-d3", "call_id": "lead-2041", "tenant": "leads-gold",
 "decision_time": "2026-09-24T16:30:00+03:00",
 "segments": [{"role": "agent", "text": "Good afternoon. I am calling about the two-bedroom unit you viewed last week."},
              {"role": "customer", "text": "Yes, we liked it. The price was the problem, but your colleague sent a new payment plan."},
              {"role": "agent", "text": "Does the new plan work for you?"},
              {"role": "customer", "text": "It does. Please send the offer in writing so I can show my brother."}],
 "journey": [{"id": "ev-1", "time": "2026-09-10T11:02:00+03:00", "source": "CRM", "type": "stage", "text": "Inbound lead from the website, asked about two-bedroom units", "provenance": "confirmed"},
             {"id": "ev-2", "time": "2026-09-14T18:40:00+03:00", "source": "AI call", "type": "label", "text": "First call: price objection, asked for a lower down payment", "provenance": "model"},
             {"id": "ev-3", "time": "2026-09-20T10:00:00+03:00", "source": "calendar", "type": "booking", "text": "Viewing attended, unit B-204", "provenance": "confirmed"},
             {"id": "ev-4", "time": "2026-09-22T09:15:00+03:00", "source": "call center", "type": "message", "text": "Rep sent a revised payment plan: 10% down, 24 months"}],
 "labels": {"stage": "ready_to_reserve", "price_objection": "false", "urgency": "1"},
 "weights": {"urgency": 0.5},
 "cause": {"stage": ["ev-3", "ev-4"], "price_objection": ["ev-4"]},
 "decision": {"action": "send_offer", "propensity": 0.9333, "policy": "engine-v3", "outcome": "yes",
              "time": "2026-09-24T16:31:00+03:00", "logged_outcome": null},
 "reviewer_action": "send_offer",
 "rationale": "The revised plan removed the price objection. She asks for the offer in writing.",
 "split": "train", "annotators": ["reviewer-1"], "twin_of": null, "flip": null}

field	what it is
id, call_id, decision_time, segments	the row, as imported
tenant	whose data this is: the dataset’s name
journey	the events before the decision time. ev-5 is gone
labels	the reviewer’s answers, {field: option}
weights	{field: 0.5} for each answer marked not sure. A field that is not listed counts in full
cause	the events marked as causes, {field: [event ids]}
decision	the logged decision, or null when the row has none. outcome is the reviewed one (yes, no, or null when the reviewer set unknown); logged_outcome keeps the original
reviewer_action	the action the reviewer would have taken
rationale	the reviewer’s sentence
split	the row’s split
annotators	everyone who labelled the row
twin_of, flip	empty on an original record; set on a twin

status=validated keeps the rows a person validated, which is what a gold set is. A row with no human answer is never exported, whatever its status.

Twins

With twins=1 every counterfactual adds one more record: the same row with that event removed from the journey. Its twin_of is the original’s id, its flip names the event and the question, and its labels hold that one question only, with the answer the reviewer gave for the world without the event. The twin of the record above has the journey without ev-4 and:

{"id": "lead-2041-d3~cf1", "twin_of": "lead-2041-d3", "flip": {"removed": "ev-4", "field": "price_objection"},
 "labels": {"price_objection": "true"}}

Keep a record and its twin on the same side of any split you make yourself: they share everything but one event.

The schema

schema.json is the dataset’s questions in the form a decision model is configured with: every question becomes an enum field, whatever its type.

{
  "tenant": "leads-gold",
  "version": 1,
  "fields": {
    "stage": {"type": "enum", "question": "Where does this lead stand after the call?",
              "options": {"ready_to_reserve": {"description": ["asked to reserve, to pay a deposit, or for a written offer"]},
                          "wants_viewing": {"description": ["asked for a viewing or agreed to one"]},
                          "needs_information": {"description": ["asked for details and committed to nothing"]},
                          "not_interested": {"description": ["said no, or asked not to be called again"]}}},
    "price_objection": {"type": "enum", "question": "Is price still an open objection for this lead?",
                        "options": {"true": {"description": ["yes"]}, "false": {"description": ["no"]}}},
    "urgency": {"type": "enum", "question": "How soon does the lead want to move?",
                "options": {"0": {"description": ["no date in mind"]}, "1": {"description": ["within a few months"]},
                            "2": {"description": ["within this month"]}}}
  }
}

The option keys in the schema are the keys in labels, so the gold set and the schema always fit each other.

How many rows, and which

• Stratify by answer. A random draw gives you hundreds of the common answer and five of the rare one, and the rare one is where a model fails. Use the drafts, or any label you already have, to pick about the same number of rows for each option of the question you care about most. The reviewer still decides the answer.
• Include rows with no history. A first contact has an empty journey. These rows show whether the model can answer from the call alone, and no event should be marked as a cause on them.
• Include rows where the history contradicts the call. The CRM says the lead is cold and the customer asks for a viewing. An old price objection was settled by a later message. These are the rows where reading the right events changes the answer, and where a counterfactual is worth the most.
• Put two reviewers on a share of the rows, one in ten to one in five, without either seeing the other’s answers, and measure their agreement: for each question, the share of those rows where both picked the same option. labels.tsv keeps every save with the reviewer’s name, so it can be computed from that file. The simplest way to keep the second reviewer blind is a second dataset built from the same records. Low agreement on a question usually means its option descriptions are unclear: fix them before labelling the rest. No model can be expected to agree with the gold set more than two people agree with each other.
• Size sets what you can see. On 200 rows an agreement number moves by five to seven points either way by chance; on 1,000 rows by two to three. A difference smaller than that between two models is a tie.
• Keep the gold set out of training. A lead in the gold set should not be in the data a model trains on. The split column is drawn from the lead for that reason.

Next: Decision models to measure a model, the Decision API to serve one, and Recommendations for what the logged decision and its propensity are.