When I first wrote about CoderAI it was an OpenAI-compatible API server that could talk to more than one GPU vendor. That was the whole pitch: NVIDIA through CUDA, AMD and Intel through Vulkan, one API in front of both.
It is a considerably stranger thing now. The single sentence version: text, images, video, speech, embeddings, document OCR and LoRA training behind one API, served by whichever runtime each model actually needs — including a GPU rented by the second when the local one is not enough.
One model, one runtime, chosen per model
The assumption I started with was that a serving stack picks a runtime and lives with it. That assumption does not survive contact with modern model releases. A 7B safetensors model wants PyTorch. A 2.78-trillion-parameter mixture-of-experts checkpoint wants an engine that keeps the dense trunk resident and streams routed experts off NVMe, because it is never going to fit anywhere else. A high-concurrency chat endpoint wants continuous batching. These are not the same program.
So the runtime became a per-model decision. CoderAI now drives seven of them:
- Transformers and llama.cpp for the ordinary cases — safetensors on CUDA, GGUF on CUDA or Vulkan.
- ds4 — antirez's from-scratch DeepSeek-V4 engine, which exposes its own OpenAI server.
- colibri — a pure-C MoE engine for GLM-5.2, DeepSeek-V4 and Kimi-K3 that streams experts from disk. CoderAI drives its C binary directly over its stdin/stdout wire protocol; no Python from that project runs at all.
- kimi-k3-in-c — Kimi-K3, 2.78 trillion parameters, on a CPU, in as little as eight gigabytes of RAM. I still find that faintly outrageous.
- ktransformers via SGLang, for CPU+GPU heterogeneous MoE.
- vLLM, in its own virtualenv because it pins a torch build that cannot coexist with anything else, appearing on the engine list as a first-class node alongside the physical cards.
CoderAI owns the whole lifecycle of each: cloning and building the C engines on first
use, downloading weights, supervising the process, sharing VRAM with the other tenants,
tearing it down. From the client side none of that is visible. It is
POST /v1/chat/completions with a model name.
Renting a GPU by the second
The newest piece, and the one I was most reluctant to build, is a RunPod backend. The reluctance was not technical. Renting cloud GPUs is the exact thing self-hosting is supposed to avoid, and a rented GPU nobody is watching bills forever.
But the local-versus-cloud framing was wrong. There is a real case where you own good hardware and still, occasionally, need a card you do not own — a model too large for the machine, or a burst of traffic that would otherwise queue for an hour. Refusing to serve that case does not make it go away; it just means it gets handled badly somewhere else.
So a model can now be pinned to backend: "runpod" and served on a rented
GPU. It appears in /v1/models like any other model, answers the same
endpoints, and shows up on the same Tasks page. The weights never touch local disk — a
cloud-only model is deliberately skipped by the download path entirely, so it can be
configured on a machine with neither the disk space nor the GPU to hold it.
Two modes. In serverless mode CoderAI is just a proxy to an endpoint RunPod autoscales. In pods mode CoderAI does the work itself: it ranks the available GPUs by price and VRAM against per-model ceilings, creates a pod, waits for it to boot, health-checks it, load-balances a pool across it, and destroys each pod a configurable time after its last request.
Two things I did not expect to need, and did:
- Capacity fallback. RunPod will advertise a GPU and then refuse the create with "there are no longer any instances available". So CoderAI ranks every matching candidate and walks down the list.
- Stuck-boot retry. Pods get created and then never boot. When that happens CoderAI dumps the pod's container log — so an out-of-memory or a bad context length is visible rather than silent — kills it, and retries on a different machine.
In live testing a cold request survived two unavailable GPUs and one failed boot before landing on a working machine, and came back with an answer nine and a half minutes later. It cost about half a cent. Wall-clock is the expensive part of a cold start, not money.
The part I care about most: never paying for a pod you forgot
Everything above is a feature. This one is an invariant, and I built it before I finished the features.
Every pod CoderAI creates is named with a stable deployment tag. A maintenance loop
lists the account's pods roughly every thirty seconds — and at startup — and
terminates any pod carrying this deployment's tag that is not in a live pool.
Crash the process, kill -9 it, restart the container mid-provision: the
orphan is found and killed within the minute. Pods from another deployment id are never
touched, and pods you created by hand in the console are never touched, because they do
not carry the tag.
On top of that there are two independent money guards, because they fail differently. A rate cap in dollars per hour refuses to start a pod that would push your burn rate over a ceiling — that bounds how fast you can lose money. A spend budget over a trailing hour, day, week or month refuses to start once the window is exhausted — that bounds the bill. Both exist per model and account-wide, on a persistent ledger.
There is also spillover: a local model can declare a cloud burst target and use it only when the local queue is full or no local GPU can serve the request. Local stays primary; the cloud is the overflow valve. That path is currently implemented for serverless targets on the direct request path — the pods target and the broker path are still to do, and I would rather say so than imply otherwise.
Reading documents properly
The other substantial addition is a document OCR subsystem, and it is deliberately not a vision-language model asked nicely to read.
A VLM will happily read a scanned page and will occasionally invent a line that was never on it. For a contract or a judgment that is not an acceptable failure mode. Real OCR — PaddleOCR, docTR, Surya — gives you a bounding box and a confidence value for every line, which means a human can check the machine's work. Only afterwards, optionally, does a text model turn that transcript into structured fields.
Those structures are defined by schemas, which are JSON documents rather than code. Supporting a new class of document is a file in a directory. There is also stamp and signature detection, which flags where a stamp or a signature appears — it locates them, it does not verify them, and I am careful to describe it that way.
Knowing who is talking
Transcription stopped being "Whisper" some time ago. The engine is now a per-model choice too — whisper.cpp, faster-whisper, CrisperWhisper for verbatim output, Wav2Vec2, Vosk on CPU, NVIDIA's NeMo Canary when you want speech translation in one pass. A model declares which languages it actually supports, so asking for one it was never trained on gets a clear rejection instead of confident nonsense.
Diarization tells you there were three speakers. It does not tell you which one is the
judge. So voiceprints can be enrolled by name, and a diarized transcript can come back
labelled with real names instead of SPEAKER_00. That is the difference
between a diarized transcript and a usable one.
Embeddings grew similarly sideways: multi-vector BGE-M3 with sparse and late-interaction vectors, shared text-image spaces, cross-encoder reranking, and — for a geolocation project — GeoCLIP and visual place recognition, which match two photographs of the same building taken years apart from different angles.
Sharing a machine without setting it on fire
Most of the unglamorous work of the last year went here. A torch-free front proxy supervises one engine subprocess per GPU, so the web UI stays responsive while a worker is busy loading forty gigabytes. Every backend is VRAM-eviction tracked, so a new load reclaims memory from an idle tenant instead of hitting an out-of-memory error. There is a host-RAM ceiling with a leak watcher, a thermal supervisor that will cooperatively pause — and if necessary SIGSTOP — an engine that is cooking a card, and a swap gate that batches requests for the same model before handing a shared GPU over, so two engines do not thrash against each other.
None of that is a feature anyone asks for. All of it is the difference between a demo and something you leave running.
Where it is
CoderAI is GPLv3. The repository carries the README and a per-engine documentation set, and there is a fuller documentation area covering installation, the engine matrix, the RunPod guide, speech, OCR, embeddings and running it behind a reverse proxy. It runs standalone; connecting it to AISBF as a broker worker is optional, and only interesting if you want to share or sell that capacity.
It still runs on my own hardware, which was the point in the first place. It now also knows how to borrow someone else's for an afternoon, and — more importantly — how to give it back.