Watching ten million documents
Watching ten million documents
Visual monitoring of a multi-model legal pipeline, and why a composition of small, self-hosted specialists beats calling a single frontier model — same quality, your data, a fraction of the cost
Abstract
Digesta turns Italian case law into structured, queryable records. The pipeline runs a dozen models at once — OCR, layout, vision, embeddings, extraction — and at corpus scale no log file tells you whether it is healthy. We describe two things that made it workable. First, a monitor that renders each stage as a live decision tree, so the work is read by shape rather than by line. Second, a design decision: instead of handing each document to a single frontier model, a composition of small, self-hosted specialist models does the same job — each trained for one stage, gated to the same quality against a larger teacher on a blind set, at a fraction of the cost, and on privileged legal documents that never leave our own infrastructure.
Keywords — pipeline observability, multi-model orchestration, self-hosted inference, vertical fine-tuning, QLoRA, data privacy, legal NLP, cost per document
Digesta turns Italian case law into structured, searchable knowledge. It starts from a pile of scanned judgments, decrees and legislative texts, and it ends with clean records a lawyer can query: who sued whom, on what grounds, with what outcome, grounded in the exact page it was read from. In between sits a pipeline, and the pipeline is never still.
At any given moment a dozen models are running across rented GPU pods. Two OCR stacks, chandra-ocr-2 and surya-ocr-2, race per page with olmOCR-2-7B held in reserve; a vision-language model resolves layout and reads the pages plain OCR mangles; bge-m3 turns passages into dense and sparse vectors; an extraction model pulls parties, claims and outcomes; a classifier decides what each document even is before any of that runs. They don't execute in a line. They retry, disagree, discard and feed each other. Tailing a log is useless here: by the time a failure surfaces as an exception, a thousand documents have already taken the wrong branch.
So the first thing we built was not a better model. It was a way to see.
▶ Watch it live — digesta.tech/alpha2run ↗
Reading the pipeline by shape
Each of the six panels is drawn as the decision tree the data actually walks. A document enters at the top, and the diagram shows every branch it can take and where it ends up: kept or discarded, read or unreadable, extracted or sent back for another pass. The numbers on each branch are live. When a stage misbehaves, you don't find a stack trace — you see a branch that should be thin go fat, or a count that should climb sit still.
The models do double duty: they run the stages and they render them. Each tree, its per-node labels and the stage summary are generated from the live counters, so a stage that is otherwise a few thousand integers states what it is doing and where the flow is pooling. That matters at this scale. Supervising ten million documents is not a dashboard problem — no table of numbers makes a stalled branch jump out — it is a layout problem, and the layout that works is the decision graph the data moves through.
The other half: cost and heat
Seeing the work is one axis. The other is the hardware doing it. CoderAI's control plane watches every GPU pod the pipeline rents — live spend against a daily cap, and for each pod its KV-cache pressure, GPU load, VRAM, temperature and health. In the snapshot below: 12 pods live, 258.5M tokens served, $10.15/hr burning against an $85/hr ceiling and a $2,000/day budget cap. A pod running hot, or a model quietly burning budget, surfaces here before it shows up on an invoice.
Six trees, one corpus
The stages are a real sequence, and naming them is the quickest way to see how many models have to cooperate before a single fact is trustworthy.
- The Choice. From roughly ten million raw documents, decide what is even in scope. Deduplicate, classify by source — Cassation, lower courts, legislation, administrative — and carve out the working subcorpus. 2,796,625 documents.
- Cataloguing. Register each document, detect its type, and route it. Passed or rejected, with a reason attached to every rejection.
- Reading. OCR and layout, the heaviest GPU stage. chandra-ocr-2 and surya-ocr-2 run per page with olmOCR-2-7B behind them; a layout pass and the vision model take the pages plain OCR mangles. Best result per page wins.
- Extraction. Pull the structured fields — parties, claims, dates, outcomes. A rules engine runs first, digesta-extraction fills the rest, and every field carries a citation back to the page span it came from. 57,863 fields from the first 779 documents.
- Chunks & Vectors. Chunk what survived and embed each passage with bge-m3 into paired dense and sparse vectors, so retrieval runs on meaning and on exact terms at once.
- Training. The loop closes. The corrected, verified output becomes a gold set that trains the next, smaller model — which then goes back to do Tree 4's job.
Tree 6 is the one that changes the economics. It takes everything the big models figured out and teaches a small one to do it alone.
Why we don't call a frontier model
The obvious way to pull fields from these documents is to hand each one to a single frontier model and let it read, reason and answer. For legal records that is a bad trade: privileged court files leave your infrastructure for someone else's API, you pay frontier prices on every page, and the thing deciding what a document says is a black box you can't inspect or hold still.
So we don't. The work is split across small, self-hosted models, each trained for one stage — the OCR ensemble, digesta-vision for layout, bge-m3 for retrieval, digesta-extraction for the fields. Extraction is the clearest case: one bounded task repeated millions of times against a fixed schema, which is exactly the shape vertical training is for. Tree 6 distils the pipeline's own agreed output into a gold set and fine-tunes a QLoRA adapter (4-bit base, one A40). The result is a model fitted to exactly one domain, running on our hardware, on documents that never leave it.
The Gate. A finished adapter isn't promoted just because training converged. It has to clear the Gate: on a held-out blind split, its field-level agreement with a larger teacher model has to meet threshold before it goes into production. The teacher sets the bar and helps build the gold set; it never sees a production document. The specialist earns its place by matching that bar on data it was never trained on.
Stack the stages up and the composition does what you would otherwise reach for a frontier model to do — at the same quality, measured against that bar, for a fraction of the cost, with the data under your own roof the whole way. No single model in the pipeline is large. The leverage is in how the small ones are arranged, and in the fact that every one of them is yours.
The bill tells the same story
The clearest proof is the spend table. The big readers and teachers are where the money goes — a general OCR reader at $1,326/month, a second at $759/month. The trained specialists, once they clear the Gate, run at a rounding error: the vision specialist at $1.15/week, the extraction QLoRA at $0.37/week — same work, a fraction of the cost and latency.
The monitor had to come first
None of the small-model economics would have been safe to lean on without the monitor. To trust a small specialist in production you have to know it isn't quietly getting things wrong at scale, and the only way to know is to watch the branches: that the discard rate held, that extraction didn't start dropping fields on some document type nobody thought to test. The Gate scores a sample before a model is promoted; the trees show the whole flow once it's running. One made the other worth trusting.
So the order wasn't model-first. We built a way to see the pipeline, used it to find where the same bounded job was being done a million times over, and only then trained something small for it. A larger teacher still has a role, but a narrow one: it sets the bar and writes the gold sets the small ones learn from, all on our own machines, never touching a production file. On this corpus that's the gap between a frontier API bill you dread and a GPU you already own — for records we can still defend field by field, back to a page that never left the building.
Built at nexlab.net. CoderAI runs the pipeline; its source is open at github.com/nextime/coderai. The Digesta monitor is live at digesta.tech/alpha2run.