Logs

How a container’s output reaches your terminal and the dashboard, and what happens when something in between fails

Everything a job, endpoint or box prints is captured once, on the worker that runs it, and every reader gets the whole of it: the CLI that called the job, the dashboard, a stream that reconnects after a dropped connection. Nothing is shared out between readers and nothing is skipped on a reconnect.

The path

1. The worker writes every line to the container’s log file as it arrives, and numbers it. The file is the source of truth; nothing is held in memory for readers.
2. A reader tails that file from a line number. The first attach starts at line one; a reader that comes back after a drop sends the last line it saw and gets what it missed. Two readers, ten readers, each see everything.
3. The controlplane passes the line number through on AttachToContainer and JobInvoke, so the CLI and the dashboard both resume rather than replay.
4. The dashboard spools to disk and records its position next to the file. A finished container’s output stays readable after the container is gone.

Before this, the worker kept output in one in-memory buffer that handed each line to whichever reader asked first. With the CLI and the dashboard both attached, each saw half the lines (step 1, 6, 31 in one, step 16, 36, 46 in the other), and a reader that arrived late saw nothing. That is what the numbered file replaced.

Reading logs

• CLI: python app.py streams the job’s output as it runs. If the connection drops, the CLI reattaches to the same task and continues from the last line it printed.
• Dashboard: a call’s drawer shows its output live and keeps it afterwards; Full view opens the whole stream; Download gives the entire spool file, not just the last screenful.
• A long run opens at its newest part. A training job can leave hundreds of thousands of lines, so the viewer starts from the newest 64 KB and keeps the newest 3,000 lines once they are parsed, which is what paints at once. It says which of the two bounds is in force: showing the newest part of 344 KB · show everything re-reads from the first line, and showing the newest 3,000 lines · download all of it hands over the whole spool file. Neither is a gap in the capture: the spool has every line either way, and Download always gives the lot.
• Before the first line, the viewer says what the container is doing (waiting to be scheduled, running but nothing printed yet). A container that is loading its blueprint or importing a large library can be quiet for a minute or two; that is the container, not the stream.

How long logs are kept

A year, on the dashboard’s own volume, set by LEVERS9_LOG_RETENTION_DAYS. A container’s spool is removed once it is older than that, and the spool of a container still being captured is never removed however old its file looks — an always-on job that has been quiet carries the timestamp of its last line.

LEVERS9_LOG_SPOOL_MAX_MB (8 GB) is a backstop rather than a policy: if the spool grows past it, the oldest go early and the dashboard log says so, because a spool removed because the volume was filling is a different fact from one that aged out.

This used to be a count — the newest 500 spools, whatever span that covered. So how far back the logs went depended on how many containers had run rather than on time: a busy week bought a few days and a quiet one bought a month, with no way to tell which you had. The prune also ran only when the process started, so the spool was cut back at each deploy rather than ageing out, which is why logs seemed to disappear in steps.

A single container’s own spool is capped separately: past 5 MB it is trimmed to its newest 2 MB, with a line written into the file so the gap is visible rather than silent.

Keeping them longer than that

The spool is one disk attached to one pod. It survives a restart and not the loss of that disk, and a year of retention on a single disk is still a single disk. So a workspace can ask for its finished logs to be kept: Members → Keep container logs in this workspace.

With it on, a container’s output is copied to a container-logs volume in that workspace once the container finishes, and stays there after the local spool ages out — the log view falls back to the archived copy without anyone asking it to. With it off, logs live on the dashboard’s disk for the retention window above and are then gone.

It is off until a workspace admin turns it on, and turning it on creates the volume. That is storage the workspace owns and pays for, holding every container’s output indefinitely — a decision to take rather than inherit, which is why there is no deployment-wide default that does it for you.

A volume rather than a bucket of the dashboard’s own, for two reasons. The dashboard already holds a controlplane token and already writes volumes, so this needs no new credential, no IAM role and no key that could leak. And a volume belongs to a workspace, so the isolation is the storage’s own: the archived copy is fetched with the reader’s token, and a token that is not that workspace’s does not find the file.

What can go wrong

scenario	what happens	what you see
The CLI’s connection to the controlplane drops mid-job (load balancer rollout, laptop sleep, network blip)	The task keeps running; the controlplane holds it for a two-minute grace period. The CLI reattaches with the task id and the last line it printed.	A one-line warning, then the output continues from the next line. No repeats.
The controlplane restarts	Same as above for every open stream: the worker keeps writing the file, readers reattach and resume.	A short pause, then output continues.
The dashboard restarts while a container is running	The spool sees a file that is not marked finished and reattaches from its recorded line.	[reconnected to the log stream; resuming after line N] in the log, then the rest.
A reader attaches after the container has already exited, worker still up	The worker serves the file to the end and closes.	The full output, then the exit line.
A reader attaches after the container exited and the worker is gone (scaled down, node reclaimed)	The file went with the worker. The dashboard’s spool is the copy that survives, if it was attached while the container ran.	The spool’s copy, or [container gone before the stream could resume; output after line N was not captured] if the dashboard’s own stream had dropped before the end.
The dashboard is not running while a container runs	Nobody spooled it. The CLI that called it still saw everything.	”No logs captured for this container: it ran while the dashboard was offline.”
A container prints more than the worker’s per-hour line budget	Lines past the budget are dropped by the worker with one note in the log; the budget is worker.containerLogLinesPerHour (default 6000) and is the one deliberately lossy setting in this path.	Rate limit exceeded, logging at reduced rate, some logs will be dropped, once. Raise the setting for chatty training loops.
A line is torn (the reader catches a record mid-write)	The reader waits for the rest of the line; nothing is emitted until the record is complete.	Nothing; lines never come through split.
The stream between controlplane and worker drops (a tunnelled cluster, a worker pod restarting)	The controlplane’s stream ends; the CLI and the spool reattach and ask from their last line. A restarted worker has no file for containers it did not run, so those are gone.	Continues, or the “not captured” note above.
An always-on endpoint logs for days	The dashboard trims its spool to the newest few megabytes and says so; the worker’s file grows until the container ends.	[older output trimmed] at the top of the dashboard’s copy.

Where to look on a worker

Each container’s file is /var/log/worker/<container id>.log on the worker that ran it: one JSON record per line, with msg, time, container_id and task_id. levers9 worker list shows which worker.