Last time the point was that every model type — images, video, speech, OCR, embeddings — could run on a rented GPU through a trimmed CoderAI image, not just the language models that vLLM and llama.cpp already had server images for. Fifteen images, a capability gateway, per-model placement.
Three days later there are nineteen images, sixteen of them proven on a rented card on the current build, and the two things I most wanted have changed shape: the language models no longer need the upstream images, and a pod no longer has to be reached through somebody else's proxy. This is the write-up of that, and of what broke.
Four more images
llama.cpp as a CoderAI pod. coderai-llama is
llama-cpp-python compiled with CUDA for every GPU generation RunPod rents — Ampere
through Blackwell — on the torch-free core. A GGUF on it gets what the upstream
ggml-org/llama.cpp image cannot give: the pod's own TLS, a boot record,
seeding, local LoRA adapters, a row on the Tasks page. It is 5.3 GB. It was 11.6 GB
first, because the CUDA GGUF backend asked torch three questions about the
card — is there one, how much memory, how much free — and torch is five gigabytes.
NVML answers the same three questions from a one-megabyte package.
vLLM as a CoderAI pod. coderai-vllm is vLLM in a
venv of its own, driven by the same proxy backend the local vLLM node uses, with the
local vLLM settings — dtype, quantisation, memory utilisation — travelling to the pod.
It is 12.4 GB and I could not make it smaller: the venv is torch and its CUDA libraries,
and removing any of them but cusolver breaks import torch. So I rented a
machine to find out whether RunPod would take a 12.4 GB image at all. It does. The
refusal line is somewhere above that and below 14.5, which is more than I knew before.
The native MoE engines. coderai-engines carries ds4,
colibri and kimi-k3-in-c — the C engines that serve DeepSeek-V4, GLM-5.2 and Kimi-K3
locally. Not copied from my machine: the local ds4-server is an sm_86
build linked against CUDA 13, and the local colibri has no GPU backend at all. The image
compiles them in a CUDA 12.8 toolchain stage for sm_80 through sm_120a and ships only
the binaries, 2.4 GB in total. coderai-engines-kt does the same for
ktransformers, which I had initially refused to ship on the belief that kt-kernel was
AMX-only — it is not; it dispatches at runtime — and which turned out to need a C
compiler in the image because Triton compiles its kernels on the pod.
The engine images come with a warning the UI shows the moment you pick one: the
models they exist for are 100 GB to 1.5 TB, and a pod without a network volume
downloads all of it on every cold start. A volume_path setting points the
pod at weights already on the volume, which is the only sane way to run them.
Not through Cloudflare
Every request to a pod used to go through <pod>-8000.proxy.runpod.net,
and that hostname sits behind Cloudflare. That buys TLS for free and costs two things:
a 100-second idle limit per request, and a body-size cap. I learned the first one the
honest way — a voice clone through the proxy died with a 524 at 516 seconds
while the pod was still working on it.
RunPod also maps a raw TCP port straight to the machine. No proxy, no Cloudflare, no
cutoff — and no TLS, which would put the bearer token on the wire in clear. So the pod
now brings its own: a CA per CoderAI install, made once with openssl and kept
beside the pod registry; a short-lived certificate per pod, issued at provision time and
sent as environment exactly like the token; uvicorn on the pod started on it. This side
verifies pod connections against that CA and nothing else. No hostname check, because
the pod's public IP is not known when the certificate is issued — and it adds nothing,
since a certificate the CA did not sign fails before a name is ever compared.
That is now the default for CoderAI images. The upstream vLLM and llama.cpp images cannot take a certificate, so they stay behind the proxy, which is the only TLS they have. The same voice clone passed direct, in 218 seconds.
What a day of live tests found
Every one of the nineteen images was rebuilt with the TLS-aware boot script, and I
re-ran the whole set on rented cards before moving :latest. Sixteen passed.
The things that did not, in the order they cost me time:
vLLM's PyPI wheel is a CUDA 13 build. The first vLLM pod died at
engine start with "The NVIDIA driver on your system is too old (found version
12080)". RunPod's hosts run 12.8 drivers. vLLM publishes a +cu129
variant on its own index with torch's cu129 wheels, and CUDA 12.x minor-version
compatibility runs a 12.9 runtime on a 12.8 driver. Getting there took uv instead of
pip — pip spent an hour on sglang-kt's forty exact pins without installing a byte —
and telling uv to take the best match across indexes, because the torch index carries
stale copies of ordinary packages and first-index-wins pinned those.
The root cause was above the tail. A pod's 503 carried the last six lines of vLLM's output, and the last six lines are the wrapper's own traceback: "Engine core initialization failed. See root cause above." The worker now surfaces the lines that name an error first.
A pod terminated behind the pool's back stayed "healthy". The test harness reaps pods through the RunPod API; the engine's pool kept the handle until its next timed probe, and the next request failed with connection refused for something the pool could have known. It now drops the pod on a refused connection and retries once on a fresh one.
Two 404s at second zero. Text-to-speech on one card, OCR on another, both a 404 issued the same second the pod was declared ready, both passing unchanged a minute later. RunPod's proxy answers 404 until its route settles. The gateway now retries a fresh pod's not-found and unavailable answers for its first minute before believing them.
Docker lost 500 GB. Twice the build disk filled while
docker system df claimed 90 GB of images. The difference was overlay2 layers
belonging to old flattened images, pinned by mount records of containers the daemon no
longer knew about — invisible to every docker prune. There is now a tool
that walks the on-disk metadata the way a fresh daemon would and removes what nothing
references. It needs the daemon stopped, which is a minute of downtime and a plan.
Smaller things that were also true
Multi-GPU pods: gpu_count rents one machine with N cards, priced and
VRAM-checked as a whole, and the engine is told to shard across them. Every request
forwarded to a pod or a host is now a row on the Tasks page naming the machine it landed
on. "Runs on this machine / RunPod only / a machine of yours" is one visible choice on
the model page rather than an entry buried in a dropdown of GPU APIs. And a
host backend runs any of these images on a box you own, started and stopped
by commands you give it, with none of the renting machinery.
Where it is
CoderAI is GPLv3, on GitHub
and Nexlab GitLab.
The RunPod and remote-execution guides are in the repository and in the
documentation area.
All nineteen images are public on GHCR under ghcr.io/nextime/coderai-*.
Three of them — faceswap and the two engine images — are promoted without a pod inference behind them yet: one needs real faces, the others need a hundred gigabytes of weights on a volume. I would rather say that than let a tag imply otherwise.