The data flywheel: making small language models better, continuously

Small models are cheap enough to run on every document, if you can keep them honest at scale. Here's how.

HydraDB turns every document into a graph: find the critical facts and statements in each chunk, extract the entities and the relationships between them, then find duplicated or synonymized entities and evolve those relationships as they change over time. That runs on every chunk of every document, so the cost scales with the size of your data and how much of it you represent as a graph, not with your queries. Frontier models would do it well and we can't afford them. Small models can, at a fraction of the cost.

The catch: a small model is only as good as the harness around it. The prompt, what it reads, how the task is split, the deterministic checks that reject bad output. Change one and the same weights get better or worse. So quality is an experiment: change one thing, measure, repeat. The only thing that matters is how fast you can run that loop.

Why it's hard at scale

At a few dozen documents you read every output and fix what's wrong by lunch. At millions of calls a week, that breaks:

• You can't read everything : What you choose to read decides what you ever learn.

• You can't trust your metrics : The checks are themselves non-deterministic judges, so when a number moves it might be the model, the harness, or the measurement.

• You can't see what you never named : The worst failures are on no dashboard, because no one built one to look for them.

A wall of green numbers feels like control. If they're the wrong numbers, it's confidently wrong.

The flywheel

Count everything cheaply, read a small sample deeply. Counting says how much of each known problem you have. Reading says which problems exist, and whether you're counting the right ones.
The flywheel: four steps that feed each other

1. Count : Every call gets a cheap yes/no judge against the source. "Not sure" never passes, so every number is a floor. Validate each metric against human labels before you trust it.

2. Read, on purpose : Each cycle a reviewer reads a sample of full calls. Mostly random (random is the only sample that shows what you weren't looking for), spread across input types, a little targeted at odd cases.

3. Let reading reshape counting : The step most setups skip. Seen a failure twice? Name it, write it as a yes/no question, confirm the judge catches it, count it forever. Retire metrics that never move.

4. Assign fault, then prove the fix : Judge, harness, or model? Check the judge first: if it drifted, recalibrate. Then the harness (cheapest to change). Retrain the model last. Every fix proves itself on untuned calls, then re-check everything.

Check the judge first, then the harness, then retrain

Where this goes

Each turn sharpens the harness, gives the model better data, and tightens the checks. All three move together, so it's less a circle than a spiral that climbs.
A model that doesn't learn from its mistakes is a frozen file. The flywheel is how ours keep learning.