Models, Hosting, and What It Costs to Run

This decision gets made on instinct more than any other in the field, and the instinct usually reflects what the person already owns. The arithmetic is not hard and it takes about an hour. Almost nobody does it.

Three questions hide inside one argument, and separating them dissolves most of the disagreement:

• Which model is good enough for this task?
• Where does it run?
• What does it cost per unit of work you care about?

People argue the second one first, which is the wrong order, because the answer to the third usually settles it.

Cost per task, not cost per token

Vendors price by token, so that is how everyone talks about cost. It is the wrong unit for a business decision. Nobody buys tokens. They buy quotes drafted, tickets triaged, documents summarized.

So build the number from the work up. Take one real task and measure four things. Input size, output size, model calls per complete task including retries, and how many times a day it happens. Multiply out. You now have a monthly figure that means something to a finance person and can be compared to the salary cost of the same work.

Two surprises are common when people do this for the first time. The first is that retries and failed attempts are often a third of the total, and nobody counted them. The second is that the number is usually much smaller than expected for anything human-scale. A task done a hundred times a day at a few cents each is a rounding error against the labor it replaces. The cost problem is rarely the price of a task. It is the number of tasks, or a loop nobody capped.

What self-hosting actually costs

The comparison people make is the vendor's per-token price against zero, because the hardware is already bought or already in the budget. That comparison is wrong in a specific and predictable way.

The real cost of running your own has five parts, and two of them are the ones that decide it:

1. Hardware. Capital, amortized over its useful life, which in this field is shorter than a finance team expects.
2. Power and cooling. Continuous, and significant at any real scale.
3. Utilization. The part everyone gets wrong. A machine sitting idle overnight still costs what it costs. If your work arrives in a four hour band on weekdays, you are paying for twenty four hours to use four, and your effective cost per task is six times the number on your spreadsheet.
4. People. Somebody keeps it running, patches it, and gets paged. This is usually the largest line and it is usually left off entirely.
5. Model quality. An open model you can host may be behind a frontier model at your specific task. That gap is a real cost, paid in worse output or in more human review, and it should be measured with the evals from the next chapter rather than assumed in either direction.

When self-hosting genuinely wins

It does win, and pretending otherwise is as lazy as assuming it always does. The conditions are specific:

• Volume that is high and steady. Steady matters as much as high. Continuous load is what makes utilization work.
• Data that cannot leave. A contractual or regulatory constraint, not a feeling. If this is genuinely true, it settles the question on its own and the arithmetic is a formality.
• A narrow, stable task. Smaller models do well on well-defined jobs. The gap to frontier models is widest on open-ended reasoning and narrowest on classification and extraction.
• Latency or sovereignty requirements that a shared service cannot meet.
• You already run infrastructure competently. A company with a real operations practice is in a different position from one that would be learning on this.

Where it usually loses is the middle: moderate volume, varied tasks, a team without spare operations capacity, and a wish to be seen as serious. That last motivation is more common than anyone admits and it is expensive.

Cheaper without changing where it runs

Before the hosting argument, four levers usually take a larger bite:

Route by difficulty. Most tasks in a workflow are easy. Send those to a small cheap model and reserve the expensive one for the steps that need it. Deciding which is which is an evals question, which is why the next chapter comes before this decision in practice.

Cache the repeated part. If every call sends the same long instructions or the same reference document, most vendors will let you avoid paying full price for that prefix repeatedly.

Batch what is not urgent. Work that can wait until tonight is frequently priced lower.

Stop sending what is not needed. The single most common cost problem I see is a system that stuffs everything it might need into every call. Better retrieval is a cost optimization as much as a quality one.

Keep the decision reversible

Whatever you choose, build so the model is a configuration line rather than an assumption. Anything you decide now will be wrong within a year. The companies that suffer are the ones who wrote a vendor's specific behavior into a hundred places and cannot move.

That is also the honest answer to give an executive who wants certainty here. You are not picking a permanent answer. You are picking this year's answer and making sure next year's is a small change rather than a rebuild.