Endpoints

Serve an open-weights model from a catalog, in one click, behind an OpenAI-compatible URL

Endpoints is managed inference. Pick a model, pick a GPU, deploy; the dashboard runs it with vLLM and gives you a URL that speaks the OpenAI API.

The catalog

• Featured models are open-weights checkpoints verified live against the Hugging Face Hub (Gemma, Qwen, Llama, Mistral and others), grouped by size, with a suggested GPU worked out from the parameter count.
• Search finds any transformers repo on the Hub; quantised-only formats (GGUF, MLX) are hidden because vLLM cannot serve them.
• Gated repos need a Hugging Face token: store it as a workspace secret (the Hugging Face template names it HF_TOKEN) and pick that secret when you deploy.

Create endpoint

Create endpoint (or Deploy on a catalog card) opens the journey:

1. Endpoint type: dedicated, on your GPUs, billed by GPU second while it runs and scaled to zero when idle. (A shared, per-token endpoint is not something this cluster offers.)
2. Weights: Open weights picks a model from the catalog or a Hub search. Custom weights brings your own: a Hugging Face repo (with a revision and a token secret for private repos) or a levers9 volume folder, browsed in place so you see config.json before deploying. Say whether the weights are a full model or a LoRA adapter on a base model; for an adapter you pick the base model and vLLM serves the base with the adapter applied under the endpoint’s name. That is how a version trained in the RL Arena becomes an endpoint.
3. Endpoint name, checked for availability as you type; it becomes the URL and the default served model name that clients pass as model=.
4. Resources: the GPU (suggested from the parameter count and the pools this cluster has), count, host CPU and memory, max context, keep-warm, served model name, token secret, trust remote code.

The Fit preview on the right estimates the weights in bf16 plus a third for the KV cache against the VRAM of the chosen GPUs, says whether a pool with that GPU exists here, and shows the client snippet. Deploy endpoint in the bottom bar registers it; the endpoint appears with its state, URL, and stop / start buttons. The first start downloads the weights, so it takes minutes; later starts are faster since the weights are cached on the volume.

Upload custom weights with levers9 cp ./my-finetune levers9://models/my-finetune (add --no-multipart from a laptop outside the cluster), then pick the models volume and the my-finetune folder.

from openai import OpenAI
client = OpenAI(base_url="<endpoint url>/v1", api_key="<workspace token>")
client.chat.completions.create(model="<model id>", messages=[{"role": "user", "content": "hi"}])

A cluster with no GPU pool (a laptop k3d) accepts the deploy and leaves it pending; the page says so instead of pretending.

The same thing from Python or the CLI is on LLM serving. Endpoints trained in the RL Arena (a LoRA adapter served with enable_lora) can be registered as policies there.