Train on 1, 2, 4 or 8 cards

gpu_count is the only knob: the same training function runs on one card, on every card of a node, or as many single-card runs side by side

A job asks for cards with gpu_count. What you do with them is one of two shapes, and both are one line.

shape	when	how
many trainings, one card each	comparing settings, seeds, datasets	gpu_count=1 and fn.map([...]): each cell is its own job on its own card, all at once
one training, many cards	one run has to finish sooner	gpu_count=2 / 4 / 8 and distributed.run(fn): one process per card, gradients averaged

Many trainings, one card each

@job(gpu="RTXPro6000", gpu_count=1, ...)
def cell(lora_r: int, lr: float) -> dict:
    return finetune(lora_r=lora_r, lr=lr)

results = list(cell.map([(8, 1e-4), (16, 1e-4), (32, 1e-4), (64, 2e-4)]))

Four cells, four cards, one call; the wall clock is one cell’s. Nothing here talks to NCCL, which is why this is the shape to reach for first: on a cluster of RTX PRO 6000s a four-cell sweep finished in 92 s against 73 to 78 s for any single cell.

One training, many cards

from levers9 import distributed, job

@job(gpu="RTXPro6000", gpu_count=4, ...)
def train(steps: int = 200) -> dict:
    return distributed.run(train_one_rank, {"steps": steps})

distributed.run starts one process per card the container was given, gives each its rank, joins them into a process group, calls your function on every rank and returns what rank 0 returned. Change gpu_count to 2 or 8 and nothing else changes. With gpu_count=1 there is nothing to spawn: the function runs in place and every helper answers as a world of one, so one function covers 1x, 2x, 4x and 8x.

Inside the function, the helpers are the handful a loop needs:

from levers9 import distributed as dist

def train_one_rank(cfg: dict) -> dict:
    dev = dist.device()                      # this rank's card
    model = load_model().to(dev)
    model = dist.wrap(model)                 # DistributedDataParallel, or the model itself on one card
    for step in range(cfg["steps"]):
        audio, text = rows[(step * dist.world_size() + dist.rank()) % len(rows)]   # each rank its own clip
        loss = model(**batch(audio, text)).loss
        loss.backward(); opt.step(); opt.zero_grad()
        dist.print0(f"step {step} loss={dist.all_reduce_mean(loss).item():.4f}")  # rank 0 speaks, mean across cards
    dist.barrier()
    if not dist.is_main():
        return {}
    return evaluate_and_save(model)          # rank 0's return is what the job returns

dist.shard(rows) gives a rank its slice of a list when a static split is simpler than indexing.

The function must be a module-level function (ranks are separate processes and import it by name), and everything it needs goes in through its arguments.

A step trains world_size examples instead of one, so an effective batch of 32 on four cards is 8 per card; adjust gradient accumulation accordingly if a run is meant to be comparable to a single-card one.

torchrun and accelerate

Under torchrun or accelerate launch the process is already a rank. distributed.run notices, joins the group and calls the function; the helpers read the same RANK / WORLD_SIZE / LOCAL_RANK those launchers set. Code written for them keeps working.

What the cards need

Cards have quirks, and they live in levers9.distributed rather than in every training script.

card	quirk	what the SDK does
RTX PRO 6000 (Blackwell)	peer-to-peer transport hangs	sets NCCL_P2P_DISABLE=1, a few percent slower through shared memory
RTX PRO 6000 (Blackwell)	NCCL before 2.26 has no kernels for it; the first collective dies with Cuda failure 1 'invalid argument' after the rings connect	checks the NCCL version before the first collective and says which package to add, instead of failing inside DDP

Anything you set yourself wins over these defaults.

Sizing, measured

On four RTX PRO 6000s in one node, the same 10,170-step fine-tune:

ranks	wall clock	speedup
1	4 h 10 min	1.0x
2	2 h 16 min	1.85x
4	1 h 19 min	3.15x

3.15x on four cards is what NCCL through shared memory costs. Four single-card cells, when the work is independent, get the full 4x, which is why the sweep shape comes first above.

What a multi-card job waits for

A multi-card job needs one worker holding that many cards, so it waits for a node with that many free at the same moment. On an attached cluster the agent reports what each node has free; a node whose other cards belong to another deployment offers only what is left. To borrow a whole node, scale the neighbour down and the job starts as soon as the cards show as free.

Two limits gate more than one card per job: the controlplane’s per-stub cap (stubLimits.maxGpuCount, raised to 8 on the hosted deployment) and the workspace’s own switch:

levers9 workspace multi-gpu          # on or off?
levers9 workspace multi-gpu on       # allow gpu_count > 1 here

It is a workspace setting because several cards in one container is a whole node held by one call. It takes effect on the next call: the change clears the cached copy of the workspace that tokens carry.