The real trade-off between an API model and a local one

An API model costs nothing until the bill arrives, and a local model costs everything up front: hardware, quantisation loss, engineering time, before it answers a single question. Which one is cheaper depends on volume and constraints, not on which sounds more modern.

Key takeaways

• The break-even point between API and local hosting depends on request volume and GPU utilisation, not a fixed rule of thumb. A card that sits idle most of the day never pays for itself, however cheap its per-token math looks on a spreadsheet.
• Real numbers from a registered quantisation experiment on an RTX 5070 Ti (16GB): Q4_K_M quantisation delivered 2.22x the throughput of fp16 with 0.9835 average embedding similarity, a genuine trade, not a free lunch.
• API providers get batching economies of scale that a single-GPU local deployment cannot replicate at low request volume, so a naive per-token price comparison is not comparing like with like.
• Privacy and data residency requirements can override the entire cost calculation. Some data cannot leave the building at any price, and no API pricing tier changes that.

What does an API call actually cost?

The advertised per-token price is the smallest part of the real cost. It buys elastic capacity: no idle hardware, no upgrade cycle, and access to whatever batching efficiency the provider has built into their serving stack. Yao and Zhou (2026) study exactly this kind of efficiency, showing that batched serving performance depends heavily on how fairly a scheduler shares GPU resources across requests of different sizes, an engineering problem a provider solves once across millions of customers and a single company solves, if at all, alone. That difference in scale is why per-token API pricing can undercut a naive local cost estimate even when the underlying hardware is comparable: the provider's utilisation is close to constant, while a single company's local GPU sits idle outside business hours or peak load.

The real cost that shows up later is variance. A quiet month costs little. A viral feature, a marketing push, or a seasonal spike costs whatever the new volume multiplies out to, with no ceiling unless one is deliberately engineered in. That unpredictability is the actual price of "nothing until the bill arrives": not the rate itself, but the absence of a cap on the total.

What does a local model actually cost?

Hardware first, and it is paid whether or not the card is used. A consumer GPU capable of running a useful open model, an RTX 5070 Ti with 16GB of VRAM in this case, is a fixed cost that starts amortising from day one, independent of how many requests actually arrive. A registered experiment on exactly this card measured what changes when a model is quantised to fit that budget: Q4_K_M quantisation reached 2.22x the throughput of fp16 inference, comfortably past a pre-registered 1.3x bar, while average embedding similarity between quantised and full-precision outputs held at 0.9835, against a 0.90 bar. Knoop and Holtmann (2026) describe the same trade-off across a range of consumer Blackwell cards: real cost-effectiveness for SMEs, bounded by exactly how much precision a business is willing to trade for how much throughput.

The trade is real, not free. Quantisation loss is small in this measurement but not zero, and 16GB sets a hard ceiling on model size regardless of how well quantisation works. A model that needs more context or more parameters than that budget allows is not a local option at any quantisation level, only a different, more expensive hardware conversation.

Where does privacy override the price question?

Some data cannot leave the premises under any pricing model, and that constraint decides the architecture before cost enters the conversation at all. Healthcare records, certain financial data, and anything under a contractual data-residency clause fall into this category regardless of how favourable an API's per-token rate looks. In these cases the honest framing is not "which is cheaper" but "which options are even legally or contractually available," and a local deployment, whatever its hardware ceiling, is sometimes the only one left standing once that filter is applied.

This is also the case where a masking layer changes the calculation, not the venue. Stripping identifying data before it reaches any model, local or API, can turn an otherwise-forbidden API call into a legitimate option, because what leaves the building is no longer the sensitive part. That reframing is worth doing before assuming a privacy requirement forces local hosting by default.

So which one should you pick?

There is no single answer, only a smaller set of questions worth answering in order. What is the realistic request volume, and how steady is it across the day? Does the data need to leave the building at all, with or without a masking layer in front of it? Does the task fit inside what a quantised model on the hardware actually available can deliver, measured, not assumed? A low, steady volume with no privacy constraint usually favours the API. A high, steady volume with spare GPU capacity to amortise usually favours local, once the quantisation trade-off has been measured rather than guessed. Anything in between, most real cases, tends to end up as both: API for the unpredictable long tail, local for the predictable core that justifies the hardware.

A concrete shape this takes in practice: an internal document-tagging pipeline that runs continuously against a steady inbound queue is exactly the profile that justifies the RTX 5070 Ti and the quantisation work above, because the GPU is busy most of the day and the throughput measured on it directly offsets its cost. A marketing-copy generator that fires only around occasional campaign launches, in bursts nobody can predict a month ahead, is the opposite profile: the hardware would sit idle between launches, and the API's elastic capacity is worth its per-token premium precisely because nobody is paying for idle time. The same organisation can reasonably run both at once, on the same hardware budget, once the two workloads are told apart rather than treated as one decision.

FAQ

Does buying a GPU always pay for itself eventually?

No. It pays for itself only if utilisation is high enough, often meaning steady daily volume, not occasional bursts. A card that runs one hour a day is amortising over a tiny fraction of its available capacity, and the API alternative for that same hour may already be cheaper than the electricity.

Does quantisation quality loss actually matter in production?

It depends on the task. The 0.9835 embedding similarity measured on one card and one model is a real number, not a universal one, and a task more sensitive to small output differences than embedding similarity captures could show a larger practical gap. Measure the specific task, do not assume the number transfers.

Is running some models locally and others through an API a realistic setup?

Yes, and it is common in practice once volume is uneven across use cases. A steady, high-volume internal tool justifies local hardware; an occasional, unpredictable one rarely does. Splitting by workload rather than picking one venue for everything usually reflects the actual cost structure better.

What is the simplest way to decide without building a full cost model?

Estimate daily request volume and multiply by the API's per-token price for a rough monthly bill, then compare that to a GPU's amortised monthly cost at realistic utilisation, not full-time. If the API estimate is comfortably lower, that is usually enough to decide without further modelling; if the two are close, only then does a fuller analysis, including the privacy and volatility questions above, earn its time.

References

• Jonathan Knoop, Hendrik Holtmann. Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEs, 2026. arxiv.org/abs/2601.09527
• Dayi Yao, Zijie Zhou. Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling, 2026. arxiv.org/abs/2608.02244