Publishing to a bucket

Send a finished dataset to Google Cloud Storage with a service-account key held as a workspace secret

A dataset that people have annotated usually has to reach a training pipeline that lives in a cloud project. Publish copies one dataset out of the Lab into a Cloud Storage bucket: the recordings, and a manifest that names them. Nothing on the datasets volume changes, and the dataset stays annotatable afterwards.

A decision dataset is the exception. It holds no recordings and no transcript manifest, so it is not published to a bucket: download its gold set instead (Decision gold sets).

There are two kinds of destination, and the difference is only where the credential lives.

A bucket Levers operates — nothing to set up

Pick it and publish. No key, no secret, no first-time setup: the dashboard holds the credential and signs one short-lived upload URL per object, and the pipeline container uploads straight to Cloud Storage with those. A platform admin configures which buckets appear (LEVERS9_PUBLISH_DESTINATIONS on the dashboard).

This is deliberate, not just convenient. A workspace secret is readable by whoever owns that workspace — they can attach it to a job of their own and print it — so a key for a bucket holding every tenant’s training data must not live there. What the container gets instead is a ticket scoped to one dataset’s folder: it cannot write to another dataset, cannot climb out of its folder, and stops working when the run is over.

levers9 dataset publish calls-sept --to staging --only-validated

A shared bucket is one set of folders for every workspace, so a folder belongs to the first workspace that publishes to it. Publishing again from that workspace works as before. Another workspace that picks the same folder name gets a 409 saying the folder belongs to another workspace; pick another folder name with --folder (or in the Publish dialog). Folders published before owners were recorded went to the workspace that published them first, and a platform admin can hand a folder to another workspace from Admin → unowned.

The tickets are signed with the dashboard’s LEVERS9_AUTH_SECRET. A dashboard without one (a local one with sign-in off) offers no shared buckets at all, and says so if you ask for one.

Your own bucket — a key you control

For a client’s project, or anywhere Levers does not operate the bucket, the credential is a service-account key stored as a workspace secret. It is read inside the pipeline container that does the upload, never by the dashboard, and never written to the dataset’s config — the config records only which secret to read. The rest of this page is that path.

1. A service account that may write to the bucket

Give it object access to that one bucket, nothing else:

PROJECT=my-project-staging
BUCKET=my-staging-datasets

gcloud iam service-accounts create levers-lab-publisher \
  --project "$PROJECT" --display-name "Levers Lab dataset publisher"

# object read/write on this bucket only — no project-wide storage role
gcloud storage buckets add-iam-policy-binding "gs://$BUCKET" \
  --member "serviceAccount:levers-lab-publisher@$PROJECT.iam.gserviceaccount.com" \
  --role roles/storage.objectAdmin

gcloud iam service-accounts keys create /tmp/levers-lab-publisher.json \
  --iam-account "levers-lab-publisher@$PROJECT.iam.gserviceaccount.com"

objectAdmin rather than objectCreator because re-publishing overwrites the manifest. If nothing may ever be overwritten, use objectCreator and publish with a fresh folder name each time.

2. The key as a workspace secret

The whole key file is the secret’s value, under the name the pipeline reads, SERVICE_ACCOUNT_JSON:

levers9 secret create SERVICE_ACCOUNT_JSON "$(cat /tmp/levers-lab-publisher.json)"
rm /tmp/levers-lab-publisher.json

Or paste it on the dashboard’s Secrets page. Either way, delete the local file afterwards.

Publishing to more than one project? Store each key under its own name (STAGING_SA, ACME_PROD_SA) and bind it per dataset with secret_map, exactly as sources and providers do. The pipeline still asks for SERVICE_ACCOUNT_JSON; the map says which secret answers.

3. Publish

From the dataset page, Publish asks for the bucket, a folder name (the dataset’s name by default), which secret holds the key, and whether to send only the rows a human has validated.

From a terminal, with the CLI:

levers9 dataset publish calls-sept --bucket my-staging-datasets                    # everything
levers9 dataset publish calls-sept --bucket my-staging-datasets --only-validated   # signed-off rows only
levers9 dataset publish calls-sept --bucket my-staging-datasets --folder calls-september-2026
levers9 dataset publish calls-sept --bucket acme-corpora --secret ACME_PROD_SA            # another project's key

It prints each stage as the run goes, then the manifest’s path. Or run the pipeline directly from its own folder, which is the same code:

cd lab/ingest
python app.py --name calls-sept --publish --bucket my-staging-datasets --only-validated

Or against the dashboard API:

curl -X POST "$DASHBOARD/api/datasets/$DATASET_ID/publish" \
  -H "Authorization: Bearer $LEVERS9_TOKEN" -H 'Content-Type: application/json' \
  -d '{"destination": {"type": "gcs", "bucket": "my-staging-datasets", "only_validated": true},
       "secret_map": {"SERVICE_ACCOUNT_JSON": "STAGING_SA"}}'

More than one bucket: dev, staging, prod

The layout below is the same in every bucket, so a second destination is setup only — no different commands. Repeat steps 1 and 2 against that project and store its key under its own secret name:

# dev
gcloud iam service-accounts create levers-lab-publisher --project my-project-dev --display-name "Levers Lab dataset publisher"
gcloud storage buckets add-iam-policy-binding gs://my-dev-datasets --project my-project-dev \
  --member serviceAccount:[e-mail] --role roles/storage.objectAdmin
gcloud iam service-accounts keys create /tmp/dev-key.json --project my-project-dev \
  --iam-account [e-mail]

levers9 secret create DEV_SA "$(cat /tmp/dev-key.json)" && rm /tmp/dev-key.json
levers9 dataset publish calls-sept --bucket my-dev-datasets --secret DEV_SA

A project with no datasets bucket yet gets one configured the way the others are (same region, uniform bucket-level access, public access prevention) before you point a publish at it:

gcloud storage buckets create gs://my-prod-datasets --project my-project-prod \
  --location ME-CENTRAL2 --uniform-bucket-level-access --public-access-prevention

In the dashboard, the Publish dialog lists every bucket this workspace has published to before (only this workspace’s: other workspaces’ buckets and datasets aren’t listed). Picking one fills in the bucket and the secret that opened it, which is the pairing that is easy to get wrong by hand; a bucket whose secret has since been deleted is marked rather than silently offered. The list comes from the runs’ own receipts, so a bucket appears once someone has published to it and never needs registering.

Nothing distinguishes a prod bucket from a dev one except its name, and --only-validated is opt-in everywhere. Publishing without it sends model drafts too — fine for staging, worth thinking about for a corpus other people train on.

What lands in the bucket

raw/raw_audio/<folder>/<call_id>.wav      the recordings (a synthesised clip per row, for TTS)
raw/structured/<folder>/<folder>.csv      one line per row, naming the audio above

Those two prefixes are the staging bucket’s pipeline entry points (an Eventarc trigger runs the audio-pipeline workflow for objects under them), so the manifest is uploaded last: by the time anything downstream reads a line, the audio that line names is already there. The dev and prod buckets have no such trigger today, but the layout is deliberately identical so a dataset moves between them unchanged.

The manifest keeps the column names the structured folder already uses (FileName, SegmentID, SegmentStart, SegmentEnd, Speaker, GroundTruthText, ProcessedText, …), so a reader written for the datasets already in that folder reads this one too. Four columns are ours and sit at the end: Split, Status, CallID and Provider — a reader that selects by name ignores them, and a person opening the CSV can see which rows a human signed off.

For a speech dataset, one recording is uploaded once even though many rows are chunks of it; each row’s SegmentStart and SegmentEnd cut it. GroundTruthText is the transcript a person confirmed, falling back to the model’s draft when there is none — so publishing early gives drafts, and publishing again after annotation replaces them.

If the dataset’s policy is Mask, the manifest carries the masked text: [NAME], [PHONE] and the rest in place of what the scan found, or of what a reviewer settled where a row was reviewed. The recordings go as recorded, because Mask covers text and not audio. A manifest is read by a training pipeline, so a publish doesn’t write a line it can’t vouch for: if a row’s text changed since the last scan, the run stops before it uploads anything and asks you to scan again.

Publish isn’t the only way out. For a file to train on with made-up names and numbers in place of the real ones, and a report that says how well the scan was checked, see Export for training. From a terminal that’s levers9 dataset pii export.

Interrupted uploads

Uploads are resumable, which matters at corpus scale: a dataset is thousands of files and can be tens of gigabytes. Each file goes up as a GCS resumable session, and if a transfer breaks the next attempt asks the bucket how many bytes it actually holds and continues from there rather than starting the file again. Session URIs are written beside the dataset before the first byte, so a container that is killed — evicted, restarted, cancelled — resumes mid-file. If a run fails, run it again: it picks up where it stopped.

Publishing again

Publishing is safe to repeat. Audio already in the bucket is left alone (the run reports how many were skipped) and the manifest is rewritten, which is what you want after more rows are validated. Pass overwrite to send the audio again as well. Each run appends a receipt to the dataset’s meta.json — bucket, folder, counts, and when — so the dataset page can show where its rows went.

When it fails

what you see	what it means
missing secrets: SERVICE_ACCOUNT_JSON	the secret does not exist in this workspace; create it in step 2
has no key configured yet; ask a platform admin	a shared destination is listed but its key is not on the dashboard
this upload ticket is not valid (or has expired)	the run took longer than the ticket’s life; publish again, it resumes
SERVICE_ACCOUNT_JSON is not JSON	the secret holds a token or a path, not the key file’s contents
... is a authorized_user credential	that is a gcloud auth file, not a service-account key
403 does not have storage.objects.create	the binding in step 1 is missing, or on the wrong bucket
no validated rows yet	only_validated was asked for and nobody has signed a row off
rows hold text the personal-data scan has not read as it is now	the dataset’s policy is Mask and some text changed since the last scan; run the scan from the Anonymization card, then publish again

The bucket’s own README

Each dataset bucket carries a README.md describing this layout for people who open the bucket rather than the dashboard. The canonical copy is lab/ingest/BUCKET_LAYOUT.md in the repo — edit it there and upload, so the buckets and the docs cannot drift:

for P in my-project-dev my-project-staging; do
  B=$(gcloud storage ls --project $P | grep -o 'gs://my-[a-z]*-datasets/')
  gcloud storage cp lab/ingest/BUCKET_LAYOUT.md "${B}README.md" --project $P
done

Next: Annotation to get rows validated first, or Insights to check what you are about to send.