Open Source AI vs API: What Self-Hosting Actually Costs

Open-weight models cost pennies per million tokens compared to proprietary APIs. But self-hosting them by renting your own GPUs usually costs more than just paying per token, unless you're running well above $10,000 to $20,000 a month in volume. Below that line, the cheapest way to "go open source" is still to rent the model as a service, not the hardware underneath it.

That's a strange thing to say on a blog that spends most of its time arguing for ownership over renting. But the math only works if you're honest about what you're actually comparing.

What does GPT-6 Astra actually cost?

Start with the number that makes everyone nervous. OpenAI's GPT-6 Astra runs $10 per million input tokens and $50 per million output tokens on the standard API tier.¹² That's roughly 2.5x the price of the prior flagship, GPT-5.6 Sol.¹ Markups like this aren't unique to OpenAI, either; frontier API pricing across the board runs well above raw compute cost.

For a single chatbot reply, that's nothing. For an agentic coding workflow that reads a codebase, plans, writes, tests, and iterates, output tokens pile up fast. A five-developer team running AI-assisted coding all day can burn through real money before lunch. That's the sticker shock that sends people looking for an open-weights alternative.

How much cheaper is DeepSeek than GPT-6 Astra?

DeepSeek's V4.1 Flash prices at $0.14 per million input tokens and $0.28 per million output tokens on the official API, with cache-hit input pricing falling to about $0.0028 per million tokens.³⁴ Depending on your cache rate, that's somewhere between 35x and over 150x cheaper than Astra, per token.

The old objection to open-weight models was that you paid less and got less. That gap has narrowed hard. DeepSeek V4 Pro scored 80.6% on SWE-bench Verified, the top open-weights score, roughly matching GPT-5.5-class agentic performance, and the cheaper Flash variant landed within about 1.6 points of that.⁵ OpenRouter's own market analysis puts it plainly: open-weight models have held a consistent 3-to-6-month capability gap against closed frontier labs for over 18 months, and that gap isn't widening.⁵ The "good enough to switch" threshold keeps arriving on schedule.

So the instinct is obvious: go open source, cut your bill by two orders of magnitude. The question is how you actually consume the model, because that decision changes the math entirely.

Two different things people call "going open source"

This is where most cost comparisons get sloppy. "Open source AI" collapses two very different purchasing decisions into one phrase:

1. Token-metered API for an open-weights model. You call DeepSeek's hosted endpoint (or a provider like OpenRouter, Together, or Fireworks) exactly like you'd call OpenAI's. You pay per token. Someone else owns the GPUs, patches the drivers, and handles the multi-tenant utilization problem.
2. Renting raw GPU capacity and self-hosting the weights yourself. You spin up an H100 or A100 instance, load the model, write the serving stack, and run it as your own infrastructure. You own the uptime, the scaling, and the bill, whether or not anyone is sending it traffic.

Both get called "open source." Only one of them changes who bears the operational cost. Our earlier piece on renting tokens versus owning quantized models worked through this distinction from the hardware side; this is the version from the spend side.

What does GPU rental actually cost?

Here's where the hidden tax shows up. RunPod prices an H100 PCIe instance at $2.89 per GPU-hour and an A100 PCIe at $1.39 per GPU-hour on demand.⁶ Run that A100 continuously for a month and you're paying it whether it's serving one request or ten thousand.

DetectX modeled a single A100 generating around 1 million tokens a day and found the effective compute cost lands at $30 to $40 per million tokens, far above the API's sub-$1 rate for the same workload.⁷ A separate cost model from CheapestInference, comparing a five-developer team's AI-assisted coding workload, found self-hosting a DeepSeek V3.2-class model totaled $6,980 a month once you count compute, ops, and idle waste, versus $265 a month on a per-token API.⁸ That's a 26x cost multiple for the identical workload.⁸ The GPU rental itself was only 40% of that self-hosted total.⁸ The other 60% is the part nobody puts on the slide.

What's actually driving up the cost of self-hosting?

That other 60% has a name, and it's not exotic:

• Idle capacity. Cloud GPUs bill 24/7 regardless of utilization. A typical usage pattern of 8 hours heavy, 16 hours idle means you're paying for roughly 3x the compute you actually use.⁸
• DevOps labor. Running inference infrastructure reliably takes a person. CheapestInference estimates $2,500 to $12,500 a month depending on whether that's a fractional or full-time engineer.⁸ That's before they touch anything else.
• Multi-model overhead. Real teams don't run one model forever. Every new model version, quantization format, or serving framework upgrade is its own mini-migration, and someone has to own it.

A Reddit account from a practitioner running their own inference stack makes the same point from ground level: even a modest ~30B-parameter Qwen model, well short of frontier quality, costs about $1,400 a month for a single always-on agent instance.⁹ That's not a hobbyist expense you shrug off, and it's for one agent at a time.

We covered the utilization side of this in more detail in the piece on fixing bloat in your AI tooling stack: GPUs sitting at 5% utilization are a bill you're paying for capacity you're not using. That's exactly the trap self-hosting sets for anyone below serious scale.

When does self-hosting actually pay off?

The rough rule of thumb that keeps showing up across cost models: self-hosting starts to make financial sense once monthly API spend clears roughly $10,000 to $20,000, or once you can sustain GPU utilization above about 70%.⁸ Below that, you're paying full-time infrastructure costs to serve part-time traffic. Above it, the math flips, because ops labor and idle-capacity tax get amortized across enough tokens that the per-token cost drops below even DeepSeek's API rate. Our break-even analysis of local models versus APIs walks through this crossover in more detail.

There's a second, non-financial trigger worth naming honestly: data-sovereignty or compliance requirements that make any third-party API a non-option, cost aside.⁹ That's a real reason to self-host. It's just a different argument than "it's cheaper," and you shouldn't let the two blend together when you're building the business case.

The backdrop here isn't shrinking, either. Gartner projects worldwide AI spending will hit $2.59 trillion in 2026, up 47% year over year.¹⁰ Token cost is becoming a real line item for a lot of companies that used to treat it as a rounding error. That's exactly why this math is worth getting right instead of assuming self-hosting is automatically the frugal choice.

Open weights change who owns the tradeoff, not whether cost exists

This is the part that gets lost in the excitement about open models. The point of open weights was never that compute becomes free. It's that you stop being locked into one vendor's meter. You can choose the model, the provider, the data policy, and the point on the cost curve where you want to sit. That's a form of ownership even when you're paying per token, because you're not hostage to a single company's pricing decisions the way you are with a closed frontier model.

A platform like Remy exists in that same spirit: giving teams a way to run and route open-weights models without pretending that owning the GPU is a prerequisite for owning the decision.

The honest takeaway is less dramatic than "self-host everything" or "just use the API." The biggest lever most teams have isn't the rack in the closet. It's picking a genuinely cheap, genuinely capable open-weights model and running it through a token-metered provider until the math says otherwise.

A practical decision framework

If you're trying to figure out where your team sits on this curve, work through it in order:

1. Start on a token API, open-weights or proprietary. Don't provision hardware before you have a spend number to react to.
2. Track actual monthly spend for 60 to 90 days across your real workload, not a benchmark. Agentic coding and chat traffic burn tokens at wildly different rates.
3. Compare that number to the $10,000 to $20,000 threshold. If you're well under it, self-hosting is very likely a net cost increase once ops labor and idle time are counted honestly.⁸
4. Check whether you can sustain 70%+ GPU utilization. If your traffic is bursty or concentrated in business hours, you'll pay the idle tax regardless of the sticker price on the instance.
5. Separate cost arguments from compliance arguments. If data sovereignty is the real driver, say so, and budget for the ops labor as a compliance cost, not a savings play.
6. Re-run the comparison whenever a cheaper open-weights model ships. The DeepSeek-to-Astra gap didn't exist eighteen months ago in this form. It won't stay static either.

The cheapest AI stack isn't the one with the lowest per-token price or the one running on hardware you own. It's the one where you actually did the arithmetic before you signed the invoice.