Personal data

Find names and numbers in a dataset’s text, choose who reads them, and measure how much the scan misses

Call transcripts are full of personal data: the customer’s name, a phone number read out digit by digit, an ID number, an IBAN. The Data Lab deals with it in three steps, all on the Anonymization card of a dataset:

1. Scan. Find the personal data in every text of the dataset.
2. Choose a policy. Decide who reads the text as it is and who reads it masked.
3. Check the scan. Have people review a random sample of rows, so you know how much the scan gets wrong and how much it misses. Until you’ve done that, nobody knows how good the masking is.

Nothing here changes the dataset. The rows and labels stay as they were written; the scan and the review write their own files beside them.

When the dataset is ready to train on, the same card writes it out with the personal data taken out, and a report beside the file that says how well the scan was checked: see Export for training.

What the scan reads

kind	text that is scanned
stt	the draft transcript, and the transcript a person wrote
tts	the sentence, and what Whisper heard when it listened back to the clip
llm	the conversation, the model’s draft, and the reply a person wrote
decision	the call, every text inside the journey, and the reviewer’s rationale and action

A label is read whenever it’s text a person wrote. A label a person picks isn’t text and isn’t read: a speaker’s role, a clip’s rating, and a decision dataset’s answers, causes and outcome.

What it finds, and how

The scan is four layers that read one text at a time, and a pass that reads the whole dataset first. Each looks in its own way. Where two of them find the same thing, the stronger one wins: a value the call is known by beats a number that proves itself with a checksum, which beats a pattern, which beats a guess from the words around it. A language model can be asked for names on top of these, as a last step: see Asking a model for names.

• Known values. If the import mapped the customer’s phone number and name (see Known values), the scan looks for exactly those in that call’s text, in every form: with or without the country code, in Arabic-Indic digits, said as words, and a name through the ways Arabic is spelled or in any letter case. This is the layer that finds a name said with no cue beside it.
• Numbers with a shape. Saudi mobile numbers however they’re written (+9665…, 009665…, 05…, with spaces or dashes, in Arabic-Indic digits), national ID and Iqama numbers, IBANs, card numbers, emails, URLs, plates, and any other run of six digits or more. A number of nine digits or more written in groups (920 000 00 01, 011...400...0000) is found too: a phone when it has the shape of a Saudi landline, a unified number (9200) or a free one (800), a number otherwise. An @handle of three characters or more is masked as [URL]. An ID and a card are checked with Luhn and an IBAN with mod 97. A ten-digit number that looks like an ID but fails the check is still reported, with a lower score, because a transcript often gets one digit wrong. A long number with a currency or a unit right after it (750000 ريال, 120000 متر) is a price or a size, and is left as it is.
• Numbers said out loud. On a call a customer dictates a phone or an ID number digit by digit. Five or more digit words in a row («صفر خمسة خمسة واحد …», zero five five one …) are a spoken number.
• Names by what’s next to them. A word is taken for a name after a vocative or an honorific (يا فهد, أستاذة نورة, أبو فهد), when someone introduces themselves (اسمي …, معك نورة من شركة …, my name is …), and for a capitalised Latin word right after a greeting (مرحبا FAHAD). An ordinary phrase that only starts the same way, like يا هلا, أنا أبي شقة or معك حق, is not a name.

Each finding becomes a typed placeholder, so the sentence still reads: [NAME], [PHONE], [ID], [IBAN], [CARD], [EMAIL], [URL], [PLATE], [NUMBER].

Names the dataset gives away

Some names can’t be found by reading one row at a time. So before a scan reads the rows, it reads the whole dataset once and learns three things.

Where a script puts the name. If one side of your calls works from a script, it says the customer’s name in the same spot of its opening, call after call. The scan finds that spot by counting: the words around it stay the same and the word in it keeps changing. Whatever stands there in a row is treated as that row’s own known name and is looked for in every text of the row. You don’t tell the scan which side is the caller. It works that out from whose opening repeats.

What people answer when asked who’s speaking. When the caller asks for a name, a short answer is taken as one.

Which words are just ordinary. A word the other side says in half a percent of the calls or more is never taken for a name by these steps. That’s what keeps a customer’s hello from being hidden as a name. A common first name is the exception: however often it’s said, it stays a name.

A name the scan knows in Latin letters, from a known_name column or from the script, is also found where the transcript writes it in Arabic. FAHAD in the greeting and «معك فهد» later in the call are the same name, and both are masked. This is a reading, not a spelling, so now and then it hides an ordinary word that happens to fit. The report tells you how often: it searches every row for another row’s name, which has no reason to be there, and counts what that finds.

All of this needs transcripts with speaker lines (Assistant: and User:, agent: and customer:, العميل: and الموظف:, SPEAKER_00:) and at least 200 conversations. A smaller dataset, or one with no speaker lines, is scanned the way it was before, and the report says so. It doesn’t learn from speech datasets, where a row is one clip. A name learned from a row is used on that row only, and a script that always says the name as two words or more isn’t found.

On a hundred thousand real sales calls, counted without anyone reading them, the cue-word layer alone found 28 in 100 of the times a scripted greeting said the customer’s name. The pass above found the rest.

Outside those cases the cue-word layer is still the weak one. A name said with no cue beside it, not mapped as a known value and not in a script’s place, isn’t found by any of these. A language model can find it: see Asking a model for names. Either way the scan has to be checked before you rely on it.

Running a scan

Anonymize on the card runs it, and so does Pre-process on the dataset page, where it follows drafting so that it reads the new text. All of it runs on the platform and sends your text to no model. A small dataset is scanned inside the dashboard and the report is there when the button comes back. A large one (more than 5,000 clips or sentences, or more than 500 conversations or decision points) is scanned on the cluster by the same pipeline that built it: the card says the scan is running and shows the report when it’s done, and you can keep annotating meanwhile.

The card has two more options, both off by default and off again every time you open the page: Also ask a model on your own cluster to find names and Also ask an outside model to find names. They’re described in Asking a model for names. Only one of them can be on.

From a terminal the scan is levers9 dataset pii scan. It follows a scan that runs on the cluster and prints the report when it’s done: rows, rows with findings, findings by type and by how each was found, what the scan learned from the dataset as a whole, and where the text went. Where the scan learned something, the report also tells the rows with somebody’s personal data from the rows where the caller only says its own name.

levers9 dataset pii scan calls-sept             # --no-wait starts a big one and comes straight back
levers9 dataset pii report calls-sept           # the last report again; --json for the API's answer
levers9 dataset pii scan calls-sept --names-model names-qwen     # also ask a model on your own cluster for names
levers9 dataset pii scan demo-calls --outside-model

--names-model and --outside-model are the two boxes on the card, and a scan takes one of them at most. --names-model asks a model you serve on levers9: the text stays on your cluster and is masked before the model reads it. --outside-model asks Anthropic or OpenAI, which is sent the rows as they are, so the command prints one warning line first. Leave it off for real customer data. Both are described in Asking a model for names. With neither, the command asks no model.

Over the API the scan is POST /api/datasets/{id}/pii/scan (POST /api/datasets/{id}/anonymize is the same route under its first name), and GET /api/datasets/{id}/pii returns the last report and how the latest scan is going. {"runner": "job"} sends a scan to the cluster whatever the size (--on-cluster from a terminal).

Asking a model for names

The layers above all miss one kind of name: a name said with nothing beside it. «فهد، وش عندكم؟» holds a name to anyone who reads it, and to none of the rules. A language model reads it the way a person does. A scan can ask one, in one of two ways.

	a model on your own cluster	an outside model
what it is	a language model you serve on levers9, from your Endpoints page	Anthropic or OpenAI, whichever the dashboard has a key for
what it’s sent	each text after the scan has masked what it found, so [PHONE], [ID] and [NAME] stand where the numbers and names already caught were	the text of your rows as it is
where the text goes	nowhere new: to your own endpoint, through the platform’s own gateway	out of the platform
real customer data	yes	no

A model on your own cluster

On the card, tick Also ask a model on your own cluster to find names, pick one of the models you’re running, and scan. The text stays on your cluster.

From a terminal it’s --names-model with the endpoint’s name as the Endpoints page lists it, or its URL:

levers9 dataset pii scan calls-sept --names-model names-qwen
levers9 dataset pii scan calls-sept --names-model last          # the model that read this dataset last

The command says before it asks that the text stays on your cluster, and the report it prints says so again from what the scan recorded: which model read the text, how many texts it read and gave no answer for, the names it returned, kept and dropped (and why), and the findings it added that no other layer had.

The model comes last, and here’s what happens:

1. The scan runs as it always does: known values, numbers, spoken numbers, cue words, and what the dataset gives away.
2. Each text is masked with what that found, and the masked text is what the model reads. A phone number, an ID or a name that was already found is never sent, even to your own cluster.
3. The model answers with a list of the names it sees.
4. Each answer is checked before it’s believed (see below). A name that passes is masked wherever it stands in that text. It’s also looked for in the row’s other texts the way a known name is, so a name the model reads once in the call is hidden in the draft and in what a person wrote for that row too.

The model’s findings never change another layer’s. Where the two meet, the other layer’s finding stays as it is and the model’s keeps only the part nobody else covered. So a finding marked model is exactly what the model added.

A model makes things up, and it takes a district for a person. So an answer is kept only if it stands in the text it was asked about, character for character, as a word of its own; is four words or fewer; isn’t a placeholder like [NAME]; isn’t a city, or something that starts with a word like شركة, حي or مشروع; and isn’t made of words that are never a name, of the dataset’s ordinary words or of the script’s own words. Everything dropped is counted in the report by the reason it was dropped for. A common first name is never dropped for being said often.

Picking a model. Any chat model you serve on levers9 that reads Arabic works: deploy one from the Endpoints page or with levers9 deploy --llm. The scan sends up to 4,000 characters of text and a prompt of about a page in one request, so the model needs a context of 8,000 tokens or more. Bigger models read dialect better and answer slower. We haven’t measured which model finds the most names, so start with the largest one you already run and check it.

What it costs in time. The model reads every text, so this is much slower than a plain scan: one request for each text, and more for a long one, which is read in pieces of 4,000 characters that overlap a little. A small dataset, scanned inside the dashboard, sends 8 requests at a time. A large one is scanned on the cluster 500 rows to a container, four containers at a time, so the endpoint gets about 32 requests at once however big the dataset is. Plan for a second or so a text on one GPU: minutes for a few hundred calls, hours for a hundred thousand. That figure isn’t measured yet; the card shows how far the scan is.

If the endpoint is idle (scaled to zero), a scan on the cluster waits up to ten minutes for it to start. A scan inside the dashboard waits a minute and a half and then says the model didn’t answer; send the endpoint one request first, or run the scan on the cluster: --on-cluster from a terminal, {"runner": "job"} over the API.

When a request fails. It’s tried twice more. After that the text keeps what the other layers found, and the report says how many texts got no answer. If the model answers nothing at all, the scan stops with an error and the files of the scan before it are left as they were. A wrong address or a model the endpoint doesn’t serve fails on the first request, before any row is read.

It still has to be checked. A model misses names too, and the checks above drop a few real ones: a one-word name that’s written like an ordinary word, as «علي» is like «على», is dropped. So review a sample after a scan with a model as you would after any scan. The review says which model the scan it measured asked, so you can scan once without a model and once with one, against the same reviews, and see what the model added to recall for names. Do it in that order: a plain scan made after a scan with the model keeps the model’s names, and the review says so (“asked no model, and kept the names names-qwen found on 7 Oct”), so its numbers are the model’s and not a scan without one.

What a later scan keeps

Most scans ask no model: a plain Scan again, the scan in Pre-process, the one that follows a build or a draft run under Mask, the one that choosing Mask starts, and levers9 dataset pii scan with no option. None of them gives a reader back a name a model had hidden.

Every scan reads what the scan before it found, and for each text that hasn’t changed it keeps the findings a model made, beside what it finds now. The masked text of an unchanged row is the same after the scan as before it, to the letter. This needs no model and no network, so it can’t fail, and it works the same way whether the scan runs inside the dashboard or on the cluster.

Three things to know about it:

• A text that changed keeps nothing. A draft written again or a label typed since is another text, and the places of the old findings mean nothing in it. A scan without the model reads it with the other layers only. So does a row that’s new. The card says how many rows that is: “412 rows keep names a model found on 7 Oct. 38 rows are new or changed since then and no model has read them. Scan with the model again to cover them.”
• A scan with the model starts over for the texts it reads. Its new answer replaces what a model found there before. For a text it got no answer for, it keeps what was there.
• A known value still wins. If the import now knows the name, or another layer now finds it, the finding is that layer’s and is counted once.

What a reviewer settled still comes before all of this, as it always has.

The scans that start on their own never ask a model, even on a dataset a model has read. They’d be starting a GPU endpoint nobody asked for, and failing whenever it’s asleep. Keeping the earlier findings is what protects the reader until you scan with the model again.

The dataset remembers its model. When a scan with a model on your own cluster ends well, the dataset keeps which endpoint and model it was (pii.names_model in its config and in meta.json, beside the policy). The card then says “Last read by names-qwen on 7 Oct” under the choice and picks that model first when you tick the box. That’s all it does: no scan asks a model because one is remembered. An outside model is never remembered.

From a terminal, levers9 dataset pii report and levers9 dataset pii policy print the same two things: how many rows keep a model’s names and how many no model has read, and which model read the dataset last. levers9 dataset pii scan NAME --names-model last asks that model again, which is how you cover the rows that are new or changed.

An outside model

Also ask an outside model to find names sends the text of your rows, as it is, to Anthropic or OpenAI. Leave it off for real customer data.

The request and the report

A model on an endpoint is names_model in the body of the scan:

{"names_model": {"type": "endpoint", "endpoint": "https://<your gateway>/asgi/<name>/v1/v1", "model": "<the name it serves under>"}}

endpoint is the endpoint’s URL with /v1 at the end (the base an OpenAI client would use), or just its name as the Endpoints page lists it. model can be left out for an endpoint the page lists.

The endpoint counts as your own when it’s one of the endpoints your workspace serves, as the Endpoints page lists them: you name it, or you give a URL that is that endpoint’s. Then it needs nothing else, and it’s called with your workspace’s token. Being on the platform’s gateway isn’t enough by itself: the gateway serves every workspace’s endpoints, and your token and text go to yours only. If the list of your endpoints can’t be read at that moment, the scan is refused and nothing is sent. Any other address is an outside model: without "allow_external": true it isn’t asked, nothing is sent, the scan still runs, and the report says in names_model_skipped that sending text outside was not allowed. With it, the endpoint is asked from the dashboard, reads the same masked text, and is sent no token. Because it’s asked from inside the cluster, its address is checked first: it has to be https and public, never a private, loopback or link-local address or the cluster’s own. Once the dashboard’s operator turns that check on (LEVERS9_EGRESS_GUARD=enforce) such an address is refused with a 422 that says why, and nothing is sent; until then it’s only logged. From a terminal that yes is --allow-outside, beside --names-model URL and --model NAME: the command prints one warning line first, and without the flag an address that isn’t yours is never asked.

Anthropic or OpenAI takes two fields, and both have to be true: {"use_model": true, "allow_external": true}. With use_model alone nothing is sent and the report says so. A key on the dashboard changes nothing by itself. names_model and use_model in one request is refused with 422: a scan asks one model.

The report says what happened:

key	what it says
names_model	which model read the text in this scan and where it runs: endpoint:<name> for one on your own cluster, outside:<name> for an endpoint anywhere else, anthropic or openai, or nothing
names_model_sent	what that model was sent: masked for a model on an endpoint, original for Anthropic or OpenAI, nothing when no model was asked
text_left_the_platform	true when text of the dataset left the platform in this scan: any model but one on your own cluster. false for no model and for endpoint:<name>
names_model_skipped	why a model that was asked for wasn’t used
names_by_model	for a model on an endpoint, in numbers: texts asked about, texts_failed, texts_skipped (nothing left to read once masked), requests, requests_failed, names_returned, names_kept, names_dropped by reason, and findings_added, the findings on something no other layer had found
names_by_model.carried	what this scan kept of an earlier scan’s model findings: texts, the rows they’re in, findings, and whose they are when that’s known (model, and at, when the scan that asked it ran)
names_by_model.never_read	texts no model has read as they are now, and the rows that hold one: new or changed since, or all of them when no model was ever asked

{"names_model": null, "names_model_sent": null, "text_left_the_platform": false,
 "names_by_model": {"state": "off", "where": null, "texts": 0, "texts_failed": 0, "texts_skipped": 0, "requests": 0, "requests_failed": 0,
                    "names_returned": 0, "names_kept": 0, "names_dropped": {}, "findings_added": 0,
                    "carried": {"texts": 520, "rows": 412, "findings": 610, "model": "endpoint:names-qwen", "at": "2026-10-07T09:00:00Z"},
                    "never_read": {"texts": 41, "rows": 38}}}

That’s a plain scan made after a scan with a model: it asked nobody, sent nothing anywhere, and kept 610 findings a model had made in 412 rows. text_left_the_platform is about this scan only; what an earlier scan sent is in that scan’s report. GET /api/datasets/{id}/pii adds names_model_sent and text_left_the_platform to a report made before those keys existed, read off its names_model the same way, so every report answers the question alike.

Findings a model made have source model in spans.jsonl, and the line of a text a model has read says which model (model). No name the model gave, and nothing else it wrote, is in any file.

Who reads the text: Off, Flag and Mask

A scan on its own changes what nobody reads. What a reader gets is the dataset’s policy:

policy	what it does
Off	Everyone who can open the dataset reads the text as it is. Every dataset starts here.
Flag rows with personal data	Nothing is hidden. Each row in the workspace, yours and a link’s, says how many findings it has, and the TSV export gains two columns, pii_findings and pii_types.
Mask personal data for everyone but me	Every reader except you, at the dataset’s own pages, gets the masked text and never the original.

Under Mask, these read the masked text:

• a share link: its rows, the labels it shows back, its TSV export and, for a decision dataset, its gold set;
• the manifest of a published dataset;
• the model that drafts in Pre-process: a language model, or the Decision API;
• Insights: every text column of the dataset’s table;
• a dataset built from this one (the Existing Lab dataset source), and the calls the RL Arena takes from it.

These still read the original:

• You. Signed in, you read the text as it is on the dataset page and in the label log, and your own Export TSV and gold set hold the original under every policy. Add ?text=masked to either export for the masked copy, which is the file a link gets. The card has a link for it.
• A reviewer of the sample. Someone checking the scan has to see what it missed, so the review shows the sampled rows as they are. See Who can review.

Mask never falls back to the original. The masked text is made from the findings over the text as it is now, and only when that text is the one that was read (a hash of each text is kept with its findings). A draft that was written again, or a row that was never scanned, reaches those readers as held back until the personal data scan has run. A stage that would turn those words into data stops short of it: a publish fails before it uploads anything, a row like that isn’t drafted, and it’s left out of a dataset built from this one. The card says how many rows are held back, and scanning again releases them.

Three things keep that rare:

• choosing Mask on a dataset that was never scanned starts the scan;
• a build or a draft run of a Mask dataset, started from the dashboard or its API, ends with a scan whether or not you asked for one (run from lab/ingest by hand, follow it with --pii);
• a label is read for personal data as it’s saved, by the scan’s own rules and with no outside model. So an annotator sees their label back masked right away, and it never waits for the next scan. The label is stored as it was typed.

An annotator on a link works over masked text. A [NAME] they leave in a transcript is what’s saved. An exported TSV of masked text can be imported back safely: a cell nobody edited isn’t taken for a label.

From a terminal the policy is levers9 dataset pii policy. With no value it prints the policy and how many rows are flagged and how many aren’t scanned yet. Setting Mask on a dataset that was never scanned starts the scan, and the command follows it and prints its report. levers9 dataset export with --text masked downloads the masked copy.

levers9 dataset pii policy calls-sept           # the policy, and how far the scan covers the rows
levers9 dataset pii policy calls-sept mask      # --no-wait leaves a scan it started to run
levers9 dataset export calls-sept --text masked -o calls-sept-masked.tsv

Over the API, PUT /api/datasets/{id}/pii/policy with {"policy": "off" | "flag" | "mask"} sets the policy and GET /api/datasets/{id}/pii/policy reads it. ?coverage=1 adds how many rows are flagged and how many are held back. The policy is kept in the dataset’s config as pii.policy and written into meta.json on the volume, which is where the pipeline’s containers read it.

Check the scan

Mask is as good as the scan, and a scan nobody has checked is a guess. The check is a review: people read a sample of rows as they are, with what the scan found marked, and settle each one. Two things come out of it. You get numbers for how good the scan is. And every text a person has settled is masked with what they settled from then on, not with what the scan found.

Everything below is on the card. From a terminal it’s levers9 dataset pii review:

levers9 dataset pii review draw calls-sept                 # the sample: 300 rows, or --size N; fixed until --redraw
levers9 dataset pii review status calls-sept               # how far the review is, then the table with its ranges
levers9 dataset pii review link calls-sept --for Fatima    # a review link, which shows original text

draw says so plainly when the dataset hasn’t been scanned or already has a sample. status prints the table described under What the numbers mean once enough rows are reviewed, and before that how many more it takes; --json gives the API’s answer. Above the table it says which names model the measured scan asked, and whether it kept a model’s names from an earlier scan, so two measurements can be compared. link takes --view-only and --days N, and prints one warning line before it makes the link. levers9 dataset links calls-sept lists a dataset’s links and says which of them open the review. The rows themselves are reviewed on the dataset page or through a review link.

Draw a sample

Draw a sample on the card picks rows at random from the whole dataset: 300 unless you say otherwise, and every row when the dataset has fewer. Every row has the same chance, whether the scan found something in it or not. That matters. A sample taken from the rows the scan flagged could tell you how often a finding is right, but never how much the scan missed, because a miss is a row it didn’t flag.

The sample is fixed once it’s drawn, so everyone reviews the same rows and the numbers are of one sample. The rows are reviewed in the order they were drawn, which means the first fifty reviewed are a random fifty too. Draw again draws a new sample and keeps every review already made.

A dataset has to be scanned before a sample can be drawn.

Review a row

Review the sample opens the rows one at a time. Each text of the row is shown as it is, with the scan’s findings as coloured marks that say their type. For each row you:

• take off a mark that isn’t personal data (an order number the scan took for a number to hide);
• change the type of a mark that’s right about the place and wrong about what it is;
• mark anything the scan missed: select the text and pick a type;
• confirm the row. A row with nothing to find is confirmed as it is, and so is a row with no text.

It works with the mouse or the keyboard. Arrows move through the text and Shift with an arrow selects. A letter marks the selection (N name, P phone number, I national ID, B IBAN, C card number, E email, U web address, L plate, M long number), Backspace takes off the mark the caret is in, and Ctrl or ⌘ with Enter confirms the row. Arabic runs right to left and the page handles that: what you select is what gets marked, in a line that mixes Arabic, Latin and digits too.

If a text changes while you have its row open (someone saved a label, or a draft was written again), the row isn’t saved. The page shows it as it is now and you confirm it again. A row that changes after it was reviewed is listed as needing another look.

Who can review

• You, on the dataset page.

• Someone with a review link. Review link on the card makes one per person. It’s a different kind of link from the ones in the Share dialog:

  • it shows the sampled rows as they are, names and numbers included, even when the policy is Mask;
  • it opens the sample and nothing else: no other row, no audio, no export, no import;
  • the person always accepts an agreement before they see a row. If the dataset has an agreement written, it’s that one, whether or not you require it of other links. If it has none, it’s a short default confidentiality text;
  • you can take it back on the card at any time.

  Hand a review link only to someone you’d let read those rows.

What the numbers mean

Once enough rows are reviewed, the card shows a table: one line per type, and one for all types together.

column	what it says
Precision	Of what the scan marked, how much really was personal data of that type. Low precision means the scan hides things it didn’t need to.
Recall	Of the personal data that was there, how much the scan found. Low recall means names and numbers get through.
Right	findings a reviewer kept
Wrong	findings a reviewer took off or gave another type
Missed	things a reviewer marked that the scan hadn’t

Recall is the one that matters for Mask: what the scan misses is what a link is shown.

Every number comes with a range, and the range is the honest part. The sample is a few hundred rows out of many, so 80% in the sample doesn’t mean 80% in the dataset. The range (a 95% Wilson interval) is where the true figure most likely is: if you drew and reviewed samples like this one again and again, about 95 in 100 of the ranges would hold it.

How wide the range is depends on how many findings the rate was worked out from, not on how many rows you reviewed. If the scan found 28 of 100 names that were really there, recall is 28% and the range is 20% to 37%. If it found 3 of 10, you’d also read about 30%, but the range is 11% to 60%: a sample that small can’t tell a scan that misses half the names from one that misses nine in ten. So a sample of 300 rows with only ten IBANs in it measures phone numbers well and IBANs hardly at all. If a range is too wide to act on, draw a bigger sample.

A few more things to know when you read the table:

• The numbers are of the scan, not of the review. They compare what the scan found on its own with what reviewers settled. After a new scan they’re worked out again against the same reviews, so you can see whether a change to the scan helped. The line under the table says which model, if any, that scan asked for names, and whether it kept names a model found in an earlier scan, so a scan with a model and one without can be told apart.
• A wrong type counts twice. A national ID the scan called a phone number is one wrong for “Phone number” and one missed for “National ID”. The line under the table counts it as found, because under Mask it’s hidden either way (as [PHONE] where [ID] would be right). That line is the share of personal data Mask actually hides.
• A finding counts as right when it overlaps the reviewed one and has the same type. A mark one word too wide still hides the number.
• Not measured yet. The table appears when 50 sampled rows are reviewed, or all of them when the sample is smaller. Before that the card says how many are done.
• A text that changed after its review, or one the scan hasn’t read as it is now, isn’t counted.

What a review changes right away

For a text a person has reviewed, what they settled is the truth. From the moment a row is confirmed, every reader of masked text gets those texts masked with the reviewed findings: a share link’s rows and exports, a published manifest, the model that drafts, Insights, a dataset built from this one. A name the scan missed and a reviewer marked is [NAME] for all of them. A finding the reviewer took off is no longer hidden.

A later scan doesn’t undo this. The scan writes its own findings again, the reviews stay where they are, and the reviews still win. A review holds for the text it was made on: if that text is edited afterwards, the review no longer applies to it, and the new text is held back until it’s scanned or reviewed again.

Reviewing only fixes the rows in the sample. It tells you about the rest of the dataset; it doesn’t clean it.

Over the API

route	what it does
POST /api/datasets/{id}/pii/review	draw the sample. Body: {"size": 300, "seed": 7, "redraw": false}, all optional. 409 when the dataset hasn’t been scanned, or when a sample exists and redraw isn’t set
GET /api/datasets/{id}/pii/review	the measurement: sample (size, reviewed, by whom), by_type, all and any_type with tp, fp, fn, precision, recall and their ranges (precision_ci, recall_ci), rows_with_nothing, measured with a reason when it’s false, and scan: the report version and time of the scan that was measured, names_model, the model it asked for names (empty when it asked none), and names_carried, the model findings it kept from an earlier scan (model, at, texts, findings; empty when it kept none)
GET /api/datasets/{id}/pii/review/rows	a page of the sample to review (offset, limit): each row’s texts as they are, with sha256, the scan’s findings (scan) and what was reviewed before (reviewed)
POST /api/datasets/{id}/pii/review/rows	confirm one row: {"row_id": "…", "texts": [{"field": "draft_text", "sha256": "…", "spans": [{"start": 4, "end": 9, "type": "name"}]}]}. Every text of the row has to be there
GET /api/share/{token}/pii/review	the same page, for a review link
POST /api/share/{token}/pii/review	confirm one row through a review link

Offsets count characters from the start of the text, the way Python does: an emoji is one character. A type is one of name, phone, national_id, iban, card, email, url, plate, number. A review is refused with 422 when a finding falls outside the text, two findings overlap or a type isn’t known, and with 409 (this text changed since you opened it) when the sha256 you send is no longer the text’s.

A review link is made with POST /api/datasets/{id}/shares and {"label": "Fatima", "task": "pii"}. The other routes of a link (rows, labels, files, exports, import) answer 403 for it, and the two review routes answer 403 for any other link.

Export for training

Mask decides what a reader of the dataset is shown. Training is another question: you want a file you can hand to a training run, with the personal data taken out of it for good. Export for training on the card writes that file, and a report beside it that says how the file was made and how well the scan behind it was checked.

The export is made from the findings a share link is masked with today: what a reviewer settled comes first, then what the scan found, the names a model found and the ones a later scan kept among them. So an export never shows more than a link does. It also means an export is exactly as good as the scan, which is why its report opens with whether anybody measured the scan.

From a terminal it’s levers9 dataset pii export:

levers9 dataset pii export calls-sept --require-measured -o calls-sept.jsonl    # made-up names and numbers
levers9 dataset pii export calls-sept --render mask                             # placeholders
levers9 dataset pii exports calls-sept                                          # what was made, newest first
levers9 dataset pii export-delete calls-sept deid-surrogate-20261007T101500Z.jsonl

The first thing export prints is whether the scan was measured, then the recall for names with its range, the rows it wrote and the ones it left out with why, where the file and its report are on the volume, and what the file doesn’t cover, in the report’s own sentences. It reads the review before it starts, so the measurement it copies is of the scan as it stands. -o FILE also saves the file where you say, with its report beside it (calls-sept.report.json beside calls-sept.jsonl), and never writes over a file unless you add --force. The options are the request’s: --status, --split, --leave-out-held-back, --require-measured, --min-recall name=0.98, --by point, and --new-salt, which asks before it acts. A large dataset is written on the cluster and the command follows it; --no-wait comes straight back.

Two ways to write a finding out

	Made-up names and numbers	Placeholders
in the API	"render": "surrogate", the default	"render": "mask"
what a finding becomes	a made-up value of the same kind, written the way the original was	its placeholder: [NAME], [PHONE], [ID]
«معك نورة، رقمي 0500000001»	«معك ريما، رقمي 0500047213»	«معك [NAME]، رقمي [PHONE]»
what a model learns from it	to say names and numbers the way people do, and none of them is anybody’s	to say [NAME]
pick it for	a model that talks, writes or transcribes	a model that should write placeholders itself, or a file people will read

A made-up value follows three rules:

• The same person keeps the same made-up name all through a call. Every row with the same call_id is one call, and a row with no call id is a call of its own. A number said three times in a call is the same made-up number three times, in the groups it was said in each time ([phone] stays four groups). Each word of a name has its own made-up word, so a full name, the first name alone and the family name alone are still one person. In another call the same person gets another made-up name, so two calls can’t be joined by it.
• It’s written the way the original was. Arabic-Indic digits stay Arabic-Indic. A number that was dictated digit by digit stays words. A Latin name in capitals stays in capitals, and «أبو فهد» becomes «أبو» and another name.
• It looks made up to anyone who checks. A phone is 05000 and five digits, a national ID has 000 after its first digit and a check digit that holds, an IBAN belongs to bank 00, a card starts 000000, and an email or a link is at example.com.

Two things to know before you train on made-up values. Names are drawn from a short list, about forty first names and ten family names in each script, so a model sees far less variety in names than real calls have. And a name written in Latin letters and the same name written in Arabic get two different made-up names: FAHAD in a greeting and «فهد» later in the call don’t stay one person in the export.

The salt

A made-up value isn’t stored anywhere. It’s worked out every time from four things: the dataset’s salt, the row’s call, the finding’s type and the value itself. The salt is 32 random bytes, made the first time a dataset needs one. It’s what keeps anybody from working backwards: without it, nobody can test whether a made-up name stands for a given real one.

• Where it’s kept. On the dashboard’s own disk, in a file beside its list of datasets (~/.levers9/dashboard/dataset-salts.json, readable by the dashboard’s own user and nobody else). It isn’t in the dataset’s config or its meta.json, it’s never written to the datasets volume, no route of the API returns it, and it isn’t in an export or its report. A large export is written on the cluster, and that one run is handed the salt for as long as it lasts.
• Why it’s kept. Two exports made under the same salt line up: the same person in the same call has the same made-up name in both. The report says when the salt was made (surrogates.salt_made_at), so you can tell whether two files line up by comparing that.
• A new one. {"new_salt": true} in the request makes a new salt before the export is written. The answer says so under new_salt: surrogate exports made before no longer line up with new ones. Do that when you want a release nobody can hold against an earlier one. The new salt is made when the export starts, so it stays the dataset’s salt even if that export is then refused for a row held back.
• Losing it. If the salt is lost (the dashboard’s disk is replaced, or the dataset is removed from the Data Lab and added again), nothing leaks and the exports you already made stay as good as they were. The next export gets a new salt and other made-up values, and that’s all. Removing a dataset from the Data Lab removes its salt.

An export with placeholders needs no salt, and none is made for it.

The file

An export is two files under the dataset’s exports/ folder on the datasets volume, named by the render and the time in UTC:

calls-sept/exports/deid-surrogate-20261007T101500Z.jsonl
calls-sept/exports/deid-surrogate-20261007T101500Z.report.json

The .jsonl has one line per row, in a shape a trainer of that kind reads as it is:

kind	a line
llm	{id, call_id, split, messages: [{role, content}], reply, pii}
stt	{id, call_id, split, audio_path, text, pii}
tts	{id, split, text, pii}
decision	the gold set’s record, with its texts rendered, and pii

{"id": "call-0142", "call_id": "call-0142", "split": "train", "messages": [{"role": "assistant", "content": "السلام عليكم HISHAM معك ريما"}, {"role": "user", "content": "هلا، رقمي 0500047213"}], "reply": "تمام يا HISHAM", "pii": {"findings": 4, "types": ["name", "phone"]}}

• messages are the turns of the conversation, cut at its speaker lines (Assistant: and User:, agent: and customer:, العميل: and الموظف:). The side that works for you is written as assistant: assistant, agent, AI, bot, rep, operator, advisor, consultant, and «الموظف», «المساعد», «الوكيل», «المندوب», «المستشار», «البائع», «المسوق». The other side is user: user, customer, client, caller, human, and «العميل», «المستخدم», «المتصل», «الزبون». A numbered speaker (SPEAKER_00) doesn’t say which side it is, so the first one to talk in a conversation is written as assistant and every other one as user, unless a speaker of that conversation is already the assistant by its own word. A system: line keeps the role system, and so does anything that stands before the first speaker line. A text with no speaker lines is one user message. Turns of the same speaker in a row are left as they are, not joined.
• reply is the reply a person wrote, or the model’s draft when nobody wrote one. A row with neither has no reply.
• text of a clip is the transcript a person wrote, or the draft when nobody wrote one.
• pii says how many findings were written out of that line and of which types, so you can count them or leave such rows out.
• A decision point’s record has its call (segments), every event of its history (journey) and the reviewer’s rationale and reviewer_action rendered. Everything else of it is as the gold set has it.

A row with no text to train on is left out: a clip with neither a transcript nor a draft, a conversation with no text, a decision point nobody answered. The report counts them.

You can keep only some rows: "status": "validated" for the rows a person validated, and "splits": ["train", "val"] for the splits you name.

The report beside it

The .report.json is how the file says of itself how it was made. It holds counts, names of models and sentences. It holds no value that was found, no made-up value, and nothing of the salt but when it was made.

key	what it says
measured, note, names	first, whether the scan was measured, in one sentence, and how many of the names it found. For a scan nobody checked: "measured": false, "note": "Not measured. Nobody has drawn a sample of this dataset and reviewed it yet."
render, at, file, dataset, kind, options	which render, when, and what was asked for
rows	in_dataset, written, by_split, and left_out by why: status and split (not asked for), held_back, no_text or not_answered
findings	how many findings were written out, in how many rows, by type
roles	for conversations: how many turns each speaker word was written as, and the rule for numbered speakers
surrogates	for made-up values: what “the same” is scoped to, and salt_made_at
scan	the scan the export rests on: its version and at, the names model it asked, what it carried over from an earlier scan’s model, and how many texts no model has read (never_read)
review	the review’s measurement as it stands, copied and not worked out again: measured and reason, precision and recall per type with their ranges (by_type), all types together (all), the figure that counts a value as found whatever its type (any_type), and how many sampled rows were reviewed. Who reviewed them is left out. of_this_scan says whether it measured the scan the export rests on
checks	what was asked of the measurement and how each type stood
audio_deidentified	always false
not_covered	what the export doesn’t cover, in sentences

The measurement counts only when it’s of the scan the export rests on. After a new scan, the review’s numbers are still the earlier scan’s until the review is read again, which works them out for the new one. The card does that before every export, and so does levers9 dataset pii export. Over the API, call GET /api/datasets/{id}/pii/review first. An export made in between says “Not measured” and why.

What it refuses

An export of a scan nobody measured is made. It just says so first, in the answer and in the report, and the file’s name is the same as any other’s. These are the cases where no file is written, each with a plain sentence (409):

• The dataset was never scanned. There are no findings to write it with.
• Something is writing the dataset: a build, a draft run, a scan or another export.
• A row holds text no scan has read as it is now, because a draft was written again or a label was typed since. Nobody knows what’s in it, so by default the export stops and says how many rows that is: scan again and export after. Or send "held_back": "leave_out" and the other rows are written, with the count in the report. The text of such a row is never written, and neither are the words a link is shown in its place.
• "require_measured": true and the scan isn’t measured.
• "min_recall": {"name": 0.98, "phone": 0.98} and the review’s recall for one of those types is under its number. The number is held against the lower end of the 95% range, which is the careful reading: 93 names found of 100 is 93%, but the range starts at 86%, so 0.9 refuses it. "by": "point" holds it against the estimate itself. The refusal names each type that fell short with its figure. A type with nothing to find in the sample has no recall, so it passes, and the report says so. Besides the types of a finding you can name all and any_type.
• No row is left to write after what you asked for.

A small dataset (the same sizes as the scan) is written inside the dashboard, and the answer holds the report. A large one is written on the cluster by one container that reads the rows from the volume: the answer says the export is running, the card shows it when it’s done, and a refusal that needs the rows (a row held back, nothing to write) shows up as the run’s state. Both write the same bytes for the same dataset, salt and time.

Over the API

route	what it does
POST /api/datasets/{id}/pii/export	make an export. Body, all optional: {"render": "surrogate" | "mask", "status": "validated", "splits": ["train"], "held_back": "refuse" | "leave_out", "require_measured": false, "min_recall": {"name": 0.98}, "by": "lower" | "point", "new_salt": false}. The answer opens with measured, note and names, then state, file, path, report_path, and for a small dataset rows, report (the report’s content) and download
GET /api/datasets/{id}/pii/exports	what was made, newest first: exports (each with path, render, at, rows, left_out, measured, note, names, download and report_download), state (how the latest run is going: running, done, refused, error or interrupted) and salt_made_at
DELETE /api/datasets/{id}/pii/exports/{file}	remove one export, its file and its report

All three are the owner’s. A file is downloaded through the owner’s own route for the volume, GET /api/volumes/datasets/download?path=<path>, or from the Storage page. No share link can make an export, list them or fetch one: a link’s file route opens a row’s clip and nothing else.

What an export doesn’t cover

• What the scan didn’t find. It’s still in the text. The report says how much the review found that to be, or that nobody has measured it.
• Audio. audio_path in a speech export points at the recording as it was made. A clip still says the names and numbers its text no longer has, so in a row with findings the text and the audio don’t match. If you train speech to text on this file, leave those rows out (pii.findings above zero) or train on text you don’t need the audio for.
• Ids and call ids. They’re written as they are. Don’t use a phone number or a name as one.
• What’s said around a finding. A made-up name hides the real one, but a rare fact in the text, like a job, a street or a date, can still point at a person.
• What isn’t text. For a decision dataset: event ids, times, sources and types, the answers, the causes and the logged decision. The people who answered a record are written as annotator-1, annotator-2 and so on, the same person the same within one file, so their names don’t travel with it.
• The folder. Like every file on the datasets volume, an export can be read by anyone in your workspace.

The files

Everything the scan and the review write is under the dataset’s pii/ folder on the datasets volume. None of these files holds a value that was found: they say where a finding is and what it is. An export for training is under exports/, beside that folder.

file	what it holds
pii/spans.jsonl	the scan’s findings, one line per text: the row, the field, a hash of the text, and each finding as {start, end, type, score, source}. A line of a text a names model has read as it is now also has model, which model that was. The next scan reads this file before it writes its own, to keep the findings a model made
pii/rows.masked.tsv	row_id, field, findings, types, text: each text masked with the scan’s own findings. It doesn’t include what a review settled; for a masked copy that does, use export.tsv?text=masked
pii/report.json	rows scanned, rows with a finding, counts by type, by field and by source, and when. learned says what the dataset taught the scan, in numbers only: each place a script puts the name, how many rows gave a name in an answer, how many Latin names were found in Arabic and how much of that is chance, and how many words count as ordinary. script_names counts the names the caller itself says in a fifth of all conversations, like its own: they’re masked too, and rows_with_nothing_else says how many rows hold only that, so you can tell rows with a customer’s data from rows where the caller just says who it is. A report with version below 4 has neither; scan again to get them. names_model and names_by_model say which model was asked for names and what came of it (see the request and the report). A report with version below 5 has no names_by_model: no model on an endpoint was asked, and a names_model it has is Anthropic or OpenAI, which was sent the rows as they are
pii/labels.spans.jsonl	only under Mask: the same lines as spans.jsonl, for the labels saved since the last scan
pii/known.jsonl	the phone numbers and names each call is known by, from the import. This one does hold values, and nothing but the scan reads it
pii/review/sample.json	the sampled row ids in the order drawn, the seed, when, and the version of the scan report it was drawn under
pii/review/reviews.jsonl	what reviewers settled, one line per row and text, only ever added to: {row_id, field, sha256, chars, spans: [{start, end, type}], user, at}. The newest line for a row and text is the one that counts
pii/review/report.json	the measurement, as it was last worked out

A field is a column of the row (draft_text, text, verify_text, context, draft_output), a label a person wrote (label.transcript, label.reply, label.rationale, label.action), or one text of a journey event (journey.<event id>.text). source says how the scan made a finding: known, checksum, pattern, spoken, context or model.

Insights shows three columns from all this: pii_count, pii_types and anonymized_text. An earlier version of the scan kept the values it found in pii.jsonl; the next scan of a dataset removes that file.

What Mask doesn’t cover

• What the scan doesn’t find. This is the big one. A name said with no cue is shown, and Mask hides only what was found or what a reviewer marked. Check the scan to learn how much that is. If recall is too low for your data, map the known values at import, or ask a model on your own cluster for names. A model misses names too, so check again after.
• Audio. Mask covers text. Audio still plays as recorded: the clip a link plays is the recording, speech goes to a transcription model as audio, and a text-to-speech build sends each sentence to its voice as written, because the clip is the sentence.
• What isn’t text. Ids, call ids, a clip’s path, times, speaker and role, status and split, and a journey event’s id, time, source and type are never masked. So don’t use a phone number as a call id.
• The volume. The policy governs the Lab’s own readers, not the files. Anyone in the workspace can still open rows.tsv and labels.tsv on the Storage page, mount the datasets volume in a job, or query a file as file:datasets/... in Insights.
• What already left. Calls the RL Arena took from the dataset before you chose Mask, a file published before, an export someone downloaded: the policy applies from the moment it’s set.
• The review sample, for its reviewers. A review link reads the sampled rows as they are. That’s a few hundred rows, behind an agreement, and it’s the one place besides your own pages where the original is shown.