The biggest model is rarely the right one

Somewhere right now a frontier model with a few hundred billion parameters is doing the same thing all day: reading five fields off an invoice, formatting them as JSON, handing them back. A thousand times a day, with the full reasoning machinery of a system that was actually built to chat about quantum mechanics. It works. It works well, even. It’s just about as economical as renting an excavator to plant a single tulip.

Most of the tasks companies want to automate look like this. Not open-ended conversation, but the same narrow job, over and over, with little variation. That’s exactly what the big generalists are overqualified for. NVIDIA’s research group turned this into a sharp thesis last year: for tasks like these, small specialized models aren’t just sufficient, they’re the more suitable and more economical choice (Belcak et al., 2025). Their number: serving a 7-billion-parameter model runs ten to thirty times cheaper than a 70-to-175-billion-class one, measured in latency, energy, and FLOPs (Belcak et al., 2025). At a thousand calls a day, that’s not a rounding error.

The model is rarely the problem

Except the model is rarely the problem. In 2025 MIT studied three hundred enterprise AI projects and landed on an uncomfortable figure: 95 percent of them produced no measurable impact on the bottom line (MIT Project NANDA, 2025). The reason wasn’t that the models were too dumb. The reason was that the models were never properly built into the work. Generic tools dazzle in the demo and stall in production, because they don’t know the actual workflow and don’t learn from it (MIT Project NANDA, 2025). The money, incidentally, wasn’t where most people went looking for it either: the biggest gains sat in unglamorous back-office automation, not the sales-and-marketing pilots everyone funds. The model was capable. The environment around it wasn’t.

Focus beats size

This is where a small model gets interesting, because its strength isn’t size, it’s focus. A specialized model gets good when you give it what it needs to solve the task. Tobi Lütke coined the term context engineering for this (Lütke, 2025); Anthropic later defined it cleanly as supplying the model, at runtime, with exactly the tokens it needs for a reliable result (Anthropic, 2025). That’s more than a good prompt. It’s the right examples, the relevant knowledge, the right tool access, the state carried over from earlier steps. And it explicitly does not mean dumping in everything you have. More context doesn’t make a model more reliable, if anything the opposite. Past a certain length no model uses its context evenly, and the hit rate drops (Chroma Research, 2025). A tight, well-curated context beats a stuffed one.

The harness, and where it’s not enough

Then there’s the part almost nobody puts on the slides: the harness. The code that hands the model the task in the right order, gives it the tools, checks the output, and feeds it back when something’s off. And where that isn’t enough, targeted fine-tuning helps. This is precisely what small models are made for: cheap and fast to train on a narrow task (Belcak et al., 2025). A model that’s been taught the job will regularly beat the big generalist on that job. Not because it’s smarter. Because it knows the task.

None of this means scrapping the big models. It means using them where they’re needed. The open, hard-to-predict task gets the big model. The twenty identical routine steps around it get small, specialized ones. NVIDIA calls this a heterogeneous system, and really it’s just common sense: the right tool for the job at hand, not the most expensive one for all of them (Belcak et al., 2025).

The work is everything around it

That’s the actual point. A model doesn’t automate a business process. A model that’s been taught the job, inside a well-designed environment, does. The work isn’t in the parameter count, it’s in everything built around it so the thing does its job at quality. That’s inconvenient, because you can’t close it out with a license agreement.

Monitoring keeps you fast

And it’s never finished. A model that works on Monday can quietly be wrong by Friday, because the input data shifted or someone nudged the prompt. So continuous evaluation belongs to any automation that runs in production. Not checking every output, which would be too slow and too expensive, but scoring a sample of real traffic on an ongoing basis, as an early-warning system for drift and quality loss (LangChain, 2026). Logging alone isn’t enough. Logs tell you what happened. They don’t tell you whether it was any good.

That sounds like effort, and in the first week it is. After that it’s the opposite. A system you can watch work is one you can change without fear. One you let run blind eventually isn’t. Monitoring isn’t bureaucracy that slows you down. It’s what lets you stay fast.

The biggest model is the easy decision. The right one is work.

---

References

Anthropic (2025) Effective context engineering for AI agents. Anthropic Engineering, 29 September.

Belcak, P. et al. (2025) Small Language Models are the Future of Agentic AI. arXiv:2506.02153.

Chroma Research (2025) Context Rot: How increasing input tokens impacts LLM performance.

LangChain (2026) Why LLM observability and monitoring needs evaluations.

Lütke, T. (2025) Post on X, 19 June.

MIT Project NANDA (2025) The GenAI Divide: State of AI in Business 2025. Massachusetts Institute of Technology.