After the cluster write-up I did what I should have done first: looked at everyone else. Not to be reassured — to find the list of things they have and CoderAI did not. The survey took an hour, the list had nine entries, and the next day was spent on the five of them that a single day could close. This is the survey and the day.

Who is out there

The field sorts itself by ambition. At the top by mindshare, and it is not close: Ollama (181k stars) and Open WebUI (153k) — one machine, one user, a chat window; clustering is your problem. llama.cpp (129k) is the engine underneath half the field and ships its own rpc-server, which is the mechanism CoderAI wraps. vLLM (92k) is what production runs, and it spans machines over Ray by itself. LiteLLM (59k) routes between a hundred providers and runs nothing.

The two that matter for comparison are LocalAI (49k) and exo (47k). LocalAI is the closest thing to CoderAI in shape — one OpenAI-compatible API over text, image, audio, video and embeddings, backends as gRPC services — and this summer it grew a real distributed mode: libp2p peer discovery with a shared token, llama.cpp weight sharding, NATS-based routing that is aware of VRAM and of prefix caches. exo is the home-cluster darling: a stack of Macs finds itself over the LAN and runs a model tensor-parallel over Thunderbolt 5 RDMA. It is also, on Linux, CPU-only, with NVIDIA and AMD "under development". Below them, GPUStack (5.7k) and Xinference (9.6k) are the ones enterprises in Asia actually deploy as clusters: users, API keys, metering, Prometheus, nine accelerator vendors, Helm charts, a company behind each. NVIDIA Dynamo and llm-d are Kubernetes serving fabrics with prefill/decode disaggregation for people with racks. SkyPilot and dstack burst jobs from an on-prem cluster to the cloud, and leave the inference server to you. Petals is dead since 2024; Hugging Face archived TGI in March.

What none of them does, and I checked twice because it seemed too convenient: rent a GPU by the second from inside the inference API when a model does not fit at home, per model, with a budget. Nor does any of them fan one image, embedding, speech or OCR request out over machines, relay a video pipeline part by part, or train a LoRA across nodes. Those stay CoderAI's. The list of what they had and CoderAI did not was still nine items long.

The nine, and the five

Zero-config discovery. Tensor parallelism across machines. An ops console with metering and Prometheus. Cache-aware routing across replicas. Kubernetes-native deployment. Apple Silicon and the Chinese accelerators. A model catalogue with one-click pulls. Signed backend images. An integrations ecosystem. The last three I am not going to pretend a day fixes; Kubernetes is a deliberate no; the accelerators got an evaluation rather than code. The other five shipped in 0.2.21, and each was smaller than its reputation.

One token, mDNS, done

LocalAI and exo both win the first five minutes: the boxes find each other. CoderAI required a URL and a per-node API token typed on the head, which is fine for a datacenter and embarrassing for a living room. Now every install can announce itself as a _coderai._tcp service — name, version, port, what its engines can do — and browse for the others. Membership is one cluster token typed the same on every box. The announcement carries an HMAC fingerprint of it, never the token, so a stranger on the LAN sees that a CoderAI exists and learns nothing else; a peer with the same fingerprint is a member and, with auto-join, becomes a cluster node with no row filled in anywhere. The node accepts the shared token on its cluster endpoints like one of its own API tokens; a node listed by hand with a blank token field gets it too, so the manual and the automatic way coexist. Three announcers in one process, one with a different token, sorted themselves correctly on the first run; a stopped one vanished from the list three seconds later.

The catch is not in the code. mDNS is link-local multicast: it crosses no router, and a container on Docker's bridge network never hears it. The launcher grew a --host-network flag and the docs say so in the same sentence as the feature.

Row split

exo's headline number — 3.2× on four Macs — is tensor parallelism over RDMA. llama.cpp has had tensor parallelism across its devices for a long time under the name row split, and since CoderAI's RPC shim already makes a card on another machine one more ggml device, exposing it was a select box: layer, each device holds whole layers and only activations cross; row, every matrix over all devices at once, with an all-reduce per layer. On one PCIe bus the second is faster. On 1 GbE it is slower, and the form says so rather than letting you find out. vLLM turned out to already place tensor-parallel ranks across the Ray cluster when you ask for more than the local cards; the form used to say "one machine only" and now says "10 GbE is the floor, RDMA is the point".

Where the conversation was

LocalAI routes by prefix cache across replicas; Dynamo and llm-d make a whole discipline of it. CoderAI's engines already had per-slot affinity inside a process — a conversation kept returning to the slot whose KV cache held its opening — but the front, choosing between an engine here and a node there, picked the first holder it found. Now it keeps a small map from conversation to engine (the same key the engines use: a session header, the OpenAI user field, or a hash of the system prompt and first turn) and sends the next turn back where the cache is warm, as long as that engine is alive and still holds the model. Pins still win. And when several engines hold a model, the least busy one now takes a fresh conversation instead of the lowest-numbered.

Numbers someone else can read

GPUStack's real advantage is not a feature, it is that an operator can see what happened. CoderAI had the data — every request passes through the front, the RunPod ledger knows every dollar — and no way to get it out. GET /metrics is a Prometheus page written by hand (the text format is five lines per sample; a client library would have been the only new dependency of the day, and it was not worth one): requests and wall time per API key name, model, kind, status and the engine or node that served them; engines and nodes up, VRAM, in-flight, RPC servers, discovery, pod spend per period. The same counters are two tables on the Cluster page, and every node's log tail is one click from the head. The recovery item on the list — "automatic model failure recovery" — turned into a real bug: the entries a head pushes to a node live only in the node's memory, so a node that rebooted came back empty and stayed empty. It now gets its assignment pushed again the moment it answers.

Signed

LocalAI signs its backend images with cosign. This is a shell script and a key pair: the private half on the release machine, the public half committed as packaging/cosign.pub, every published digest signed with the version and commit as annotations. All twenty-one images on GHCR were signed the same afternoon, and the capability publish script signs whatever it pushes. cosign verify --key … ghcr.io/nextime/coderai:latest is the whole check on the user's side.

What did not get done, honestly

A Mac can be a text node today through mlx_lm.server as a remote endpoint; a real MLX node is a small pip package speaking the node protocol, two or three days, not yet started. The Chinese accelerators arrive through their vLLM forks as a service URL, and native support waits for a card to test on. A model catalogue is the one item on the list I keep postponing because a path or a HuggingFace id has never felt like the hard part. And the sentence from the last write-up is still true: nothing has crossed a real cable between two GPU machines yet.

Code: git.nexlab.net/nexlab/coderai · github.com/nextime/coderai · image ghcr.io/nextime/coderai:0.2.21 · the guide is docs/cluster.md.