Last week I wrote about renting a GPU
by the second: pin a model to backend: "runpod", CoderAI creates a pod,
serves the model on it, and destroys it when nobody is using it.
That was the easy half, and I did not notice at the time. It worked for language models, because a language model on a rented GPU is a solved shape — vLLM and llama.cpp both ship a server image that speaks the OpenAI API, so CoderAI's job was renting the machine and pointing at it.
Everything else stayed home. Image generation, video, text-to-speech, transcription,
OCR, embeddings, face swap, stem separation: all of it local, all of it queueing behind
whatever the GPU was already doing. There is no worker-stable-diffusion that
happens to expose CoderAI's image API, and there never will be, because those endpoints
are mine.
The shape of the fix
The thing that made this tractable was realising the far side did not have to be a generic server. It could be CoderAI itself, trimmed down.
So there are now ten pod images — images, video,
embeddings, stt, tts, ocr,
voice, audio, faceswap, text —
each one the same codebase with only the Python dependencies that capability needs. They
share a common core layer, so the ten of them are not ten times the download. They are
published on GHCR and public, so a pod pulls one without credentials.
In front of them sits a gateway: a piece of ASGI middleware that looks at an incoming
/v1 request, works out which capability it belongs to, and either serves it
locally or forwards it to a remote that owns that capability. From the client's side
nothing changed — it is still POST /v1/images/generations with a model name.
Three things deliberately do not get forwarded. Chat and completions, because the backend owns those already. Generated files, which are fetched back from whichever remote produced them. And pipelines, characters and environments — those are a sequence of calls, so the orchestration stays here and each step is placed by its own model's configuration. Forwarding the sequence would move the logic to a pod and place every step by that pod's catalogue, which is the opposite of the point.
Per model, not per capability
The setting I care about most is the one that says where a single model runs. You can have two video models where one is pinned local forever and the other lives on a rented card — or bursts to one only when the local GPU is busy. Per-model placement beats the capability-wide setting, so "this capability goes to the cloud" and "this particular model never leaves this machine" can both be true.
A pod also has to be able to get the model. It can pull from HuggingFace, or download a URL you name (common for GGUF files that live nowhere else), or be sent the weights from here when they exist nowhere but this disk. LoRA and QLoRA adapters travel the same way — a local adapter is content-addressed and pushed to the pod, because "adapter trained on my machine" and "model running on a rented one" should not be mutually exclusive. Optionally all of that can live on a RunPod network volume, so the second pod does not re-download what the first one already fetched.
And a pod is not a fixed catalogue. A request naming a model it has never heard of can teach it that model at runtime and have it fetch and serve it. A pod should behave like an extension of the local system, not a frozen snapshot of what I knew when I rented it.
The six failures
None of the above is the interesting part. This is: I tested it on a live account with real money and a running production instance, and it broke in six ways I want to write down, because five of them were my own reasoning being wrong rather than code being buggy.
One: a pod that served nothing. The first embeddings pod booted,
passed its health check, and answered every request with Model 'bge-m3' is not
available. Use one of: — and nothing after the colon. The pod image runs
uvicorn directly on the app, which means it never executes the startup file
where model registration lived. The config manager was still None, so the
seeded model list was read from an object that did not exist, silently. Fixed by building
the config manager in the application lifespan, which covers both entry points.
Two: a token that outlived nothing. A pod is locked with a bearer
token generated in memory. Restart the process, and the surviving pod is adopted with a
different token — so it answers 401 to everything for the rest of its paid life.
The test run reported, accurately and uselessly, that the pod served
(HTTP 401). The token is now recorded alongside the pod and adopted with it.
Three: the test that moved production. This is the one that actually embarrassed me. To test an embedding model on a pod I set the capability remote — and a capability remote is a production routing switch. It moves every request of that kind. It was still set days later, so real embedding traffic was going to a rented pod while the local GPU sat idle; I found out because the machine's owner noticed his card was quiet. The test harness now pins a single model and never touches capability routing.
Four: testing a config nobody had loaded. Writing a config file does not change where a request goes. The front pushes a reload to each engine, and an engine takes it only when it is idle — so a harness that writes and immediately tests is testing the previous configuration. That produced a whole run of false "nothing configures this to run remotely" failures. Worse, the same mechanism could have reported a remote pass for a request that ran locally, which is a test result that is not merely useless but actively misleading. There is now an endpoint that answers for the process that actually serves the request, and the harness refuses to test until it agrees.
Five: a log API that does not exist. Failed pod boots kept reporting
HTTP 400 when fetching logs, and the code politely "tried several routes" and
fell back to a console link. I finally pulled RunPod's published OpenAPI spec: twenty-three
routes, covering pods, billing, endpoints and volumes, and not one of them serves
logs. GraphQL refuses introspection. The console is a human surface. The code had
been guessing at URLs that were never real.
So the pod reports on itself. Its entrypoint now prints timestamped phases from the first instant the container exists — including the list of models it was told to serve, which is exactly the line that would have made failure number one obvious in one look — and serves that record over HTTP, readable the moment the port opens.
Six: a timeout that assumed the wrong thing. Pods kept failing with "did not expose port 8000 within 300s". The budget was written assuming the port appears once the container is running. On a rented machine the port appears once the image has been pulled, and a 7.3 GB image needs a sustained 25 MB/s to beat five minutes. The same image opened its port in 133 seconds on a machine that had the layers cached and failed repeatedly on cold ones — and each failure rented a fresh machine, which pulled the whole thing again. Three attempts, three cold pulls, same result.
It is fifteen minutes now. Being wrong by fifteen minutes on a $0.25/hour card costs six cents. Being wrong by too little cost three pods and still failed.
The one that was not about RunPod at all
Somewhere in the middle of this the disk filled. Docker reported 27 GB of images while the overlay directory held 631 GB, and my first theory — hundreds of orphaned layers — was wrong: the genuinely unreferenced directories came to about a megabyte.
The real answer was 23 layers of roughly 26 GB each, one per flattened image build from
previous sessions, whose image records had been deleted but whose layer references had
not. docker system prune -a will not touch them, which is why they had been
accumulating invisibly for weeks. Removing them by hand returned 598 GB.
My first attempt at listing them produced "1 referenced layer" and would have called
every layer an orphan — because Docker's cache-id files have no trailing
newline, so concatenating 426 of them yields one very long line. That is the entire
argument for printing the list before deleting anything.
What is actually proven
I want to be precise here, because "it works" is easy to write and hard to earn.
Embeddings are proven end to end: a real request left this machine, rented an RTX 2000 Ada, pulled the image, loaded BGE-M3, and came back with a genuine 1024-dimension vector in 68 seconds, for a fifth of a cent. Images got as far as booting, pulling the model and failing on a missing dependency that is now fixed but not yet re-tested. The rest are mid-run as I write this.
Which is the honest state of it: the mechanism is built and one capability has walked the whole path. The rest is testing, and testing is where the six failures above came from, so I expect it to keep being worth doing.
Where it is
CoderAI is GPLv3, on GitHub
and Nexlab GitLab.
The remote-execution guide and the RunPod guide are in the repository's documentation set
and in the fuller documentation area.
The capability images are public on GHCR under ghcr.io/nextime/coderai-*.
It still runs on my own hardware. It has simply got better at borrowing someone else's for the specific minutes it needs one — and at telling me, in words, what it was doing when that goes wrong.