CoderAI's tagline, as of this week, is complete orchestration, distribution and escalation of remotizable advanced inference. That is a mouthful, and it is also a checklist. Escalation — a model that runs on your card, then on a machine you own, then on a GPU rented by the second — was the last two write-ups. This one is the other two words: several machines on a network acting as one CoderAI, and one model too large for any of them spread over their cards.

When I went to check what was already there, the answer was honest and a little embarrassing. Orchestration across machines existed as a set of one-off targets — a service_url here, a capability remote there, a host backend, the pods — each configured by hand, none of them knowing about the others. Distribution existed only inside a machine: a model over two local GPUs, yes; over two machines, nowhere. Not in llama.cpp's RPC form, not in vLLM's Ray form, not for LoRA training. So this is what was built, in the order it was built.

Another CoderAI as an engine of this one

The front already runs one engine process per GPU and knows nothing of their insides: it polls a small state endpoint, hands each engine the models it owns, and proxies requests by capability and load. A cluster node is exactly that, one network hop away. Another whole install — its own front, cards, thermal protection, admin — that this front treats as one more engine. The node needs nothing but an API token; the head needs the node's URL and that token.

The node's front answers /cluster/state from its own registry, so the head is never waiting on a busy GPU there: loaded models, VRAM summed over its engines, and the union of their capabilities — a node with an NVIDIA and a Radeon card offers both transformers and gguf, and picks the card itself when work arrives. Models are assigned to a node like to any engine, or pinned to it by name from the model page, and the head pushes each node the entries behind its assignment; a model the node's catalogue lacks is registered there in memory under a path that exists on the node, or the HuggingFace id CoderAI can recover from a cache path, or the "path on the node" you typed. Load and Unload from the head's model page reach the node too — which took a small detour, because a node's admin endpoints want a browser session and the head only holds an API token. The node now runs those two actions on the head's behalf under a session it mints and destroys itself.

The part I care most about is that the model form will not let you save nonsense any more. A Vulkan card cannot run vLLM; the Radeon engine cannot run a transformers model; a host model runs on the machine in its host block and on no engine. Pick a backend and the engines that cannot run it grey out; pin an engine and the backends it cannot run grey out; a wrong pair already selected shows a red line and Save refuses. The same table is applied again on the server, because a form is not a guarantee.

One GGUF over several machines

llama.cpp has had an RPC backend for a while: run rpc-server on a machine and its card becomes one more ggml device for a process elsewhere, so the ordinary layer split spreads a model over local and remote cards alike. llama-server --rpc does this. The Python binding CoderAI loads through never exposed it, and the bundled build did not even include the backend. So codai/backends/ggml_rpc.py does what the CLI does, through ctypes against the ggml libraries the binding ships: ask the registry for the RPC backend, fetch ggml_backend_rpc_add_server through the registry's own proc-address lookup (it is not exported), register each device the server offers.

Registration is process-wide and permanent, which is the trap. A model loaded with no device list afterwards would spread itself over every RPC device ever registered — including the one another model asked for. So from the first registration on, every load in that engine gets an explicit device list: this machine's cards first, then exactly the servers the model named. The binding's llama_model_params.devices field is annotated "unused" in its source; it is not unused by llama.cpp, and setting it from a wrapper around the load call is the whole mechanism. A model's weight-distribution ratio now reads local cards, then RPC servers in the order listed, and the automatic split counts the remote cards' free memory too.

I proved it before believing it: a CPU-only llama-cpp-python built with the RPC option in a scratch venv, an rpc-server on localhost, a 0.5B GGUF loaded with every layer assigned to RPC0, "The capital of France is" → "Paris"; then a second load without RPC servers that stayed off the RPC device entirely. Two things from that test shaped the shipping form. The RPC protocol is versioned, so the server must come from the same llama.cpp commit as the client — the build script pulls the binding's own sdist and compiles rpc-server from the tree it vendors, never from a fresh clone. And the 0.3.30 server has no memory-cap flag while newer ones do, so the manager reads --help before passing anything optional.

On the wire this is decode-friendly and load-hostile: per token only the activations between the layers on each side cross the network, which is small; loading a 20 GB slice and processing a long prompt are bound by it. 1 GbE hurts. 10 GbE is fine. The protocol is unauthenticated, so it is bound to a LAN or a WireGuard address and nothing else, and the documentation says so in the same breath as the feature.

vLLM on Ray, SGLang on nnodes

For weights that are not GGUF the servers already know how to span machines; what they need is the choreography. vLLM does tensor parallel inside a machine and pipeline parallel across them over a Ray cluster; SGLang takes --nnodes and a rendezvous address. CoderAI now starts the Ray head on the machine that took the request, runs each node's configured start command with {ray_address} filled in — the coderai-vllm pod image joins as a Ray worker when started with one environment variable — waits until Ray reports tensor × pipeline GPUs, launches, and tears it all down with the service. Per model or as defaults. The doc is blunt about the shape: tensor parallel wants an all-reduce every layer and belongs inside a machine; pipeline parallel crosses the wire once per layer boundary and is the one to use across a network.

Splitting the work, not the model

That covers language models. The question that followed was whether images, video, audio, embeddings and OCR get anything similar, and the answer has three parts of very different weight.

The cheap, large one: a request usually carries more than one unit of work — n images, a list to embed, documents to rerank, a long text to speak, a long recording to transcribe, a batch of documents to OCR. Tick Distribute on the model and the front cuts the request into parts, sends them concurrently to every engine that has the model — here and on the nodes — and puts the answers back in order. Seeds get a per-part offset so a fixed seed does not produce the same image four times. A long recording is cut at silences rather than at fixed seconds, and every segment and word timestamp is shifted back by its window's start. Spoken parts are joined with ffmpeg into the format that was asked for. A part that fails is retried once elsewhere before the request fails. A file a node generates is addressed through the head, which fetches it from the node when the client comes for it. Streaming requests are never split, and neither is srt/vtt output or a diarised transcription, because those need one machine's view of the whole.

The heavy one: a video model is several models run one after the other — a text encoder of eleven gigabytes, one or two denoising experts of ten each at 4-bit, a VAE — and what passes between them is small: prompt embeddings, a latent tensor of a few megabytes. So each part can live on a different machine and a generation becomes a relay. The head encodes the prompt where the text encoder is, runs the high-noise steps itself, hands the latents over at the expert boundary, and the low-noise machine finishes and — if the VAE is there too — returns the finished video. The far side runs the ordinary generation endpoint with a hand-off block, so model loading, acceleration presets and LoRAs happen exactly as for a normal generation there; the only new mechanism is a scheduler whose timestep table is cut so the loop resumes at step k and still finds each step's sigma pair. This one I have exercised on a fake pipeline and a real diffusers scheduler, not on real weights across two machines. It says so in the docs, and it will say something else once the second GPU box is on the 10 GbE switch.

The one I left out on purpose: tensor- or sequence-parallel inside one diffusion step across machines (xDiT and friends). Real, but built for NVLink and PCIe; over Ethernet it is an all-reduce per layer and useless. Single-machine multi-GPU parallelism is a separate item.

Training a LoRA on several machines

LoRA and QLoRA training turned out to be the best-shaped candidate of all. The trainable parameters are an adapter of a few megabytes, so synchronous data-parallel training — every machine on its own card with its share of the samples, gradients of the adapter averaged after every backward — costs a few megabytes per step on the wire. Even 1 GbE is fine. CoderAI does it with plain torch.distributed and a hand-rolled all-reduce over the adapter parameters rather than DistributedDataParallel, because PEFT adapters on a frozen diffusers base are exactly what the DDP wrapper trips over, and averaging p.grad is all DDP would have done anyway.

The machine that takes the job is rank 0. It resolves the images — a character profile exists only there — sends the same job with the images inlined to each node, broadcasts its freshly initialised adapter so every rank starts from the same point (PEFT's random init differs per process, which would otherwise quietly turn synchronous SGD into averaging unrelated adapters), and trains. Peers write into scratch that vanishes with the job; rank 0 saves; a cancel on rank 0 reaches the peers, because a peer left waiting on an all-reduce holds its card until the group times out. All five in-process trainers took the hooks — SD1.5, SDXL, Z-Image, the flow-DiT family and Wan with 4-bit — and a two-process test on one machine checks the three properties that matter: shared init, averaged gradient, interleaved samples.

The image, and what the build taught

Shipping the RPC backend meant rebuilding llama-cpp-python inside the released image, which is a CUDA runtime image with no compiler. The wheel is now built in a CUDA 13 devel stage against the same standalone CPython the image runs, for five GPU generations so the public image is not tuned to my card, and only the wheel and the rpc-server binary cross into the final layer. Four attempts. python-build-standalone records CC=clang in its sysconfig and the builder has gcc. llama.cpp's Vulkan shaders want glslc and then SPIRV-Headers, neither of which the base image's builder ever needed. And my own verification step imported the wheel, which loads libcuda.so — the driver library that exists only at run time, mounted by the NVIDIA container runtime, never inside docker build. The check is now ldd with that one library exempt, plus a symbol check for the RPC backend: the same rule the pod images learned a week ago, relearned.

The released coderai:base is a single layer with all of this in it, no configuration and no secrets — those are bind-mounted at run time and the flatten strips every key-shaped environment variable — and the newer pieces reach an installed image through coderai-docker --upgrade, which pulls the production branch in place.

What is true today: nodes, RPC, the fan-out and the training relay are tested against fakes, two processes, or a server on localhost; the RPC path has generated real tokens through a real rpc-server; nothing has yet crossed a real cable between two GPU machines. That is the next write-up.

Code: git.nexlab.net/nexlab/coderai · github.com/nextime/coderai · the guide is docs/cluster.md.