Attach a cluster

Turn any Kubernetes cluster with GPUs into a pool of workers with one command; the cluster dials in, nothing dials into it

levers9 cluster join attaches a cluster you already run, GKE, EKS, AKS or your own, as a pool of GPU workers. It works the way levers9 login works for a person: the cluster side initiates everything. One agent runs inside the cluster, holds an outbound stream to the controlplane, reports the nodes and what they have free, and starts worker Jobs when asked. The workers carry their own inbound connections back over the same kind of stream. So there is no public node address, no firewall rule, no allow-list of controlplane addresses, and the controlplane holds no credentials for your cluster.

The shape

Both arrows are opened from inside your cluster and stay open; nothing on the controlplane side ever dials the cluster or its nodes. That is the whole difference from a configured pool:

	configured pool (provider: kubernetes)	joined cluster
who calls the cluster API	the controlplane, with a credential you give it	an agent inside, with a Kubernetes service account
who reaches the workers	the controlplane dials each worker’s public address	the worker dials out; the controlplane rides its tunnel
cloud-side setup	service account and key, authorized networks, firewall rule, image mirror, values block, redeploy	one command; mirror the image if the cluster cannot pull ours
nodes	public, sized by allocatable	private allowed, sized by what other tenants leave free
when controlplane nodes change	rerun the attach script	nothing

What the cluster needs

Three things, and only the first involves your cloud’s network settings:

	what	why
your kubectl reaches the cluster API, once	the join applies one manifest through your current context	a cluster with authorized networks (GKE) or an equivalent allow-list must include the address you run the command from, exactly as it must for any kubectl apply. Nothing else is ever added to that list: the controlplane does not call your API server
the cluster can open outbound HTTPS to the controlplane	the agent and every worker dial out and keep the stream open	private nodes are fine as long as they egress through NAT; no inbound rule, no public node address, no load balancer
the nodes can pull the worker image	Kubernetes starts the worker pod	mirror it and pass --worker-image (with --pull-secret if that registry needs one) when the cluster cannot pull ours

If you cannot reach the API server from your own machine, run levers9 cluster join … --manifest-only and apply the manifest from wherever you do have access, such as a bastion or Cloud Shell; the join completes when the agent dials in.

Upgrading from a configured pool? The controlplane egress addresses that pool required on your authorized networks are no longer used by anything and can come off the list once the agent is up.

Join

Point your kube context at the cluster, then:

levers9 cluster join my-gke --gpu RTXPro6000

What it does, and prints as it goes:

1. Registers the cluster with the controlplane, which mints a token for the agent.
2. Applies one manifest through your kube context: the namespace, a service account with the least access the agent needs, the token as a secret, and the agent Deployment.
3. Waits until the agent has dialed in and reported its nodes, and prints them.

From then on a job from your workspace that asks for that GPU type can land on the cluster:

@job(gpu="RTXPro6000", image=Blueprint(python_packages=["torch"]))
def train(): ...

Options that matter:

flag	when
--selector key=value	only some nodes are lent. key=value for one label, key=in (a,b) for several node pools at once, a bare key for “the label exists”; repeat the flag to require several. Default: every node, GPU or not, because CPU-only jobs land here too
--worker-image	the cluster cannot pull the controlplane’s worker image; give it one it can
--pull-secret	that image needs a pull secret, already created in the namespace
--flavor gke|eks|aks	detected from node labels; set it for a cluster that does not label its nodes the usual way
--manifest-only	print the manifest instead of applying it, for a cluster you reach another way

The same flow exists on the dashboard’s Clusters page: register there, then apply the manifest it shows, or run the printed command.

Who a cluster serves

A cluster you attach works for your workspace and nobody else’s. Your jobs land there by GPU type, with no pool selector, and no other workspace’s job is ever sent to it. Its workers carry a token that’s only good for your workspace, and the settings they get hold none of the platform’s own credentials: images come through the controlplane, which hands the worker a short-lived link for each image your job uses. Secrets and the LEVERS9_TOKEN your code sees are fetched for each container as it starts, the way they are on a private pool, and that token is a restricted one.

A shared cluster works for every workspace, the way the platform’s own GPU cluster does: any job asking for its card can land there. Only a platform admin can share a cluster, and only a platform admin can detach a shared one:

levers9 cluster share gke-dammam          # every workspace's jobs can land there
levers9 cluster share gke-dammam --off    # back to the workspace that attached it

levers9 cluster list shows your own clusters and the shared ones, with a Serves column. A shared cluster someone else attached shows its card, its state and what each node has free, but not its node names, namespace, image or manifest. Taking a shared cluster back takes effect with the next job. Sharing one takes effect as new workers start there. Either way, workers already running keep the token and settings they started with until they wind down, so delete their Jobs in the cluster if that can’t wait.

A workspace can attach up to five clusters (worker.attachedClusters.quotaPerWorkspace in the controlplane config; 0 means no limit). Attaching and detaching need your own token: a container’s token can’t do either.

Keeping the agent current

The worker image is pushed to your cluster’s registry by the same build that pushes ours, and each worker Job pulls the tag fresh, so workers are never behind. The agent is a Deployment and stays on the image it started with until something restarts it:

kubectl -n levers9 rollout restart deploy/levers9-agent

Nothing in flight is lost — the agent only launches and stops worker Jobs, and the ones already launched keep running while it restarts. Do it after an upgrade, or whenever levers9 cluster list shows an agent that has been up far longer than the platform’s last release.

What runs in the cluster

Exactly what the manifest contains, nothing more:

object	scope	why
ServiceAccount levers9-agent	the namespace	the identity the agent runs as; Kubernetes issues and rotates its credential
ClusterRole levers9-agent-observe	nodes and pods, read-only, cluster-wide	to report the nodes it lends and what other tenants’ pods leave free on them
Role levers9-agent-launch	Jobs and pods in the namespace	to start and stop worker Jobs
Secret levers9-agent	the namespace	the cluster token, minted by the controlplane at join
Deployment levers9-agent	the namespace	one small pod; it never listens

The agent only ever creates Jobs in its own namespace and refuses orders for any other. The worker tunnel only ever reaches the worker’s own containers and refuses any other target. Neither can be used as a way into the cluster.

Check it yourself against the service account, with kubectl auth can-i --as:

question	answer
create Jobs in the levers9 namespace	yes
create Jobs in any other namespace	no
list nodes and pods anywhere	yes, read-only
read Secrets, including its own	no
exec into a pod	no
delete anything outside the namespace, or grant itself more	no

Reading pods cluster-wide is what makes shared nodes work: the agent reports the cards other tenants leave free, so the scheduler never sends a job to a node whose GPUs are already taken.

What the controlplane learns about your cluster is exactly the nodes the selector names: their names, their cards, and their free CPU and memory. With no selector that is every node in the cluster, node pools you never meant to lend included, so name them with --selector if that matters to you.

Who talks to whom

The agent and the workers never talk to each other. Each has its own line to the controlplane and nothing else; the picture is a hub with spokes, and the hub is the controlplane.

connection	opened by	carries
agent → controlplane	the agent	node reports up; Job create and delete orders down
agent → Kubernetes API	the agent	list nodes and pods; create and delete worker Jobs in its namespace
worker → controlplane, gRPC	the worker	registration, heartbeats, container requests, results, image publishing
worker → controlplane, tunnel	the worker	container gRPC, log streams, exec and endpoint traffic, carried back down

The agent reads the worker Jobs’ outcome only to answer the controlplane; once a Job exists the agent is out of the picture for it. If the agent pod dies, running workers keep running and keep serving jobs; what stops is new placement, because the nodes’ last report goes stale after ninety seconds. Kubernetes restarts the agent Deployment on its own, so that gap is short. If a worker dies, its Job ends and the controlplane asks the agent for a new one when the next job needs the node.

Failure modes, and what covers each

what fails	what happens	what covers it
the balancer’s path to the controlplane (a node reclaimed under it)	every stream through it drops at once	the balancer targets the controlplane pod itself, not a node, so node churn never touches a stream
the controlplane restarts	every agent and worker stream drops	the controlplane pings each agent every 15 s; an agent that hears nothing for 45 s reconnects, with backoff; worker tunnels reconnect on their own; clusters are reloaded from the database at start
a caller’s stream drops mid-job (a laptop changing networks, a proxy recycling)	the controlplane can no longer tell a drop from a deliberate exit	it waits two minutes before treating the drop as a cancel; the SDK reattaches by task id and resumes the stream, or receives the result if the task finished meanwhile
the agent pod dies	node reports stop	running workers keep serving; placement pauses once the last report is 90 s old; Kubernetes restarts the agent, and its Recreate strategy means there are never two
the controlplane loses a worker’s record (a long outage)	it would place a second worker on the same node	the agent counts worker pods as node usage, so a node with a worker on it offers nothing until that pod is gone
a node is removed while a worker runs on it	the worker’s containers are lost	the dispatcher retries the task on a fresh container and the caller’s stream follows it
the worker image cannot be pulled	the worker Job pends	the pending worker times out and is retried; check the Job’s pod on the cluster, and set --worker-image or --pull-secret
the cluster token is revoked or the cluster deleted	the agent’s connect is refused	it backs off to one attempt every 30 s and logs why; cluster leave removes it
a node’s disk fills up	the kubelet reports DiskPressure and evicts the worker, and the job on it is lost	every worker keeps the image cache it leaves on the node bounded and gives it up first when the disk runs low (see Disk on the nodes); what it can’t free is the jobs’ own disk

How the nodes are sized

Every ten seconds the agent reports each node’s allocatable CPU, memory and GPUs, and what is free net of the other pods on it. A node shared with another workload offers only what that workload leaves; a node with no free GPU is not offered at all until one frees up. A worker takes every card the node has free when it is created and serves several jobs across them. The controlplane always counts its own workers on a node, pending or running, before adding another, so a node whose worker was created when fewer cards were free keeps that size until the worker winds down (a few idle minutes after its last job); the next worker then takes everything free. A job asking for several cards needs one worker with that many, so it waits for a node with that many free at once.

Two ceilings hold regardless of what the reports say: a node never gets a second worker while one of its workers is still pending (its pod has not started, so the node report cannot see it), and a cluster pool never carries more pending workers than it has nodes. The agent enforces the same ceiling on its side (no more pending worker pods than nodes) and removes a worker Job whose pod has not started within twenty minutes. Above all pools, attached or not, the scheduler holds a request when a pool already has twenty workers pending or received more than thirty in a minute (worker.provisioningGuard in the controlplane config; a negative value disables a check). A bug in the accounting above them can cost at most one idle worker per node, never a flood. How a job uses several cards, one training across them or many trainings side by side, is on Train on 1, 2, 4 or 8 cards.

Disk on the nodes

Workers keep the images they pull in a cache on the node, at /var/lib/levers9/images/cache, so the next worker there starts without pulling them again. It outlives every worker, so every worker keeps it in check:

• it’s pruned to 100 GB, the images used longest ago going first;
• anything nobody has used for seven days is removed, once an hour, even when there’s room;
• when less than 20% of the node’s disk is free, it gives up what it can until 25% is, whatever its size.

An image that a running worker on the node has mounted is never removed, and neither is anything used in the last ten minutes. The 20% is there because the kubelet’s defaults evict pods at 10% free on the node’s disk and at 15% free on the disk that holds the container images, which on GKE is the same disk. The worker is one of those pods, and so is the job running in it.

What the cache can’t free is the jobs’ own disk: a container’s scratch space and anything it writes. If a node still reports DiskPressure, the worker’s log says what is left is mounted or in use, and the fix is a bigger disk or a smaller job. The limits are worker.nodeCache in the controlplane’s config (imageCacheMaxGB, minFreePct); the controlplane hands them to the workers it starts on your cluster.

Day two

• See it. levers9 cluster list for your clusters and the shared ones, levers9 cluster nodes <name> for a cluster’s nodes and free cards; the dashboard’s Clusters page shows the same, live.
• Share it. levers9 cluster share <name> (platform admins) lets every workspace use it; --off gives it back to the workspace that attached it.
• Host network. Workers on attached clusters use the node’s network, as they always have. A platform admin can turn that off with worker.attachedClusters.hostNetwork: false: the workers then run on the pod network and reach the controlplane through their tunnel. Try it on one cluster first.
• A new worker image. Point the cluster at it with --worker-image at join, or edit the Deployment’s image; the agent creates workers from the image the controlplane names in each Job.
• Re-apply the agent. levers9 cluster manifest <name> | kubectl apply -f - reinstalls everything but the token, which stays in the secret already there.
• Leave. levers9 cluster leave <name> revokes the token, forgets the pool, and deletes the agent’s objects from the cluster. A shared cluster can only be detached by a platform admin, since other workspaces’ jobs run there. Joining again under the same name gives you a cluster that isn’t shared.

Compared with a configured pool

A pool with provider: kubernetes in the controlplane’s values (see GPUs in a GKE cluster) has the controlplane call the cluster’s API itself, with a credential you give it, and dial each worker. It still works and needs nothing inside the cluster, at the price of public nodes, a firewall rule and allow-lists that follow the controlplane’s addresses. Joining with the agent is the default for anything outside the controlplane’s own network.