Frontier vs. Open-Weight vs. Local Models: How I Choose for the Job

I'm Tom Lang — I've built and scaled engineering organizations for three decades, from startups to 250-plus engineers and through multiple acquisitions, and I work as a fractional and interim CTO for growth-stage companies, with particular depth in regulated and integration-heavy environments.

The landscape of AI models is chaotic right now. Founders get pitched on why they absolutely must self-host their own open-weight model to "protect their IP," while their own engineers are asking for an unlimited API budget to hit the newest frontier model. Both can't be right, and usually neither is — the choice depends on your business, not on whoever pitched last.

When I make the architectural call on which model to use, I ignore the hype and look at the physics of the business. This is the concrete decision-framework companion to what "AI-first" actually means — the philosophy said AI is an accelerator; this is how you pick the engine. Here's how I actually evaluate the trade-offs between frontier models, hosted open-weight models, and fully local deployments.

The decision axes, in order of dominance

These aren't equal-weight factors on a spreadsheet. They're a hierarchy — the first one can override all the rest.

1. Data governance — the absolute dominator. Where is the data legally allowed to go? If you're processing classified data, or un-anonymized patient health information without a signed BAA, the capability of the model is irrelevant — you are legally forced into a local or strictly controlled private-cloud environment, full stop. This axis wins every argument it's in. (It's the same boundary I draw in right-sized governance: the first question is always where the data can go.)

2. Reasoning depth vs. execution width. Does the task require complex, multi-step logical deduction across messy, unstructured data — real reasoning? Or is it a highly predictable, narrow task like classifying a receipt or pulling a date out of a clean PDF — execution? The answer changes which class of model you even need.

3. Unit economics at scale. Not the cost of the prototype — the cost at ten million invocations a month. A price that's trivial in a demo can be lethal in production.

4. Latency. Are you powering a real-time voice agent that needs sub-500-millisecond responses, or running an overnight batch job that summarizes support tickets while everyone sleeps? Very different constraints.

The rule of thumb I hand founders

If you want to move fast without building massive technical debt, give your engineering team this:

Default to frontier APIs to find product-market fit. Move to open-weight only to optimize unit economics at scale. Move to local only when the law or physics forces you to.

Do not prematurely optimize for cost when you don't even know whether customers want the feature. Use the smartest, most expensive model available to prove the business case. Then, once it works and starts genuinely burning cash, you downgrade the model to improve your margins. Optimizing the model before you have a product is optimizing the wrong thing.

When each one genuinely wins

Frontier APIs (OpenAI, Anthropic, Google). They win for prototyping, complex reasoning, code generation, handling highly unstructured or ambiguous user input, and tasks with huge context windows — "read these 50 PDFs and synthesize a strategy." You're paying a premium for generalized intelligence, so use them when the cost of being wrong is high.

Open-weight models via a managed host (a Llama or Mistral model served through something like AWS Bedrock, Together, or Groq). These win for high-volume, narrow tasks. If your app extracts structured JSON from 100,000 invoices a day, frontier prices will bankrupt you — but an open-weight model, fine-tuned or dropped into a tight retrieval-augmented pipeline, gives you predictable, dramatically cheaper unit economics without you having to manage the actual GPU metal. That last part matters — which brings us to the trap.

Fully local or air-gapped (running on your own hardware or edge devices). This wins in exactly three situations: absolute data privacy (defense, strict healthcare, proprietary trading logic), zero-connectivity environments (a factory floor, a ship at sea), or ultra-low-latency edge processing (computer vision on an autonomous drone). Outside those, it's rarely the right call — which is the most expensive mistake in this whole space.

The DIY cost trap

The single most common mistake I see is what I call the DIY cost trap.

A founder looks at a $2,000-a-month frontier-API bill and panics. They tell the team to "bring it in-house" with an open-source model to save money. What happens next is predictable: the engineering team spends three months wrestling with GPU drivers, provisioning expensive instances, and trying to load-balance a 70-billion-parameter model. They end up paying something like $8,000 a month in raw cloud compute — much of it for idle GPUs — plus $150,000 a year for an ML-ops engineer just to keep the thing running.

They spent $200,000 to save $24,000.

Unless AI infrastructure is your actual core competency, do not rack your own GPUs or run your own inference servers to save a few pennies per token. You'll win the token war and lose the war on payroll and opportunity cost — which is the one that actually determines whether your company survives. (This is the model-layer version of the same discipline in keeping your cloud bill from eating your runway: the raw compute cost is never the whole cost.)

The honest goal

So when someone pitches you on self-hosting to protect your IP, or your engineers ask for an open budget on the newest frontier model, run it through the physics: Where's the data allowed to go? Does the task need reasoning or just execution? What does it cost at real scale? And how fast does it have to answer? Default to frontier to find product-market fit, move to open-weight when scale demands it, go local only when the law or physics leaves you no choice — and never rack GPUs to save pennies.

If you're trying to make this call for your product — or you've already fallen into the DIY trap and want out — that's exactly the kind of architectural decision I make for a living. Book a call and we'll pick the right engine for your actual constraints.