How to cut LLM costs in production with model routing and fine tuning

Most teams reach for fine tuning first and routing last. I do it the other way around, and here is the reasoning, with code.

The first time a client's inference bill made someone in finance send me a Slack message, the agent in question was a support triage bot. It read a ticket, decided a category, and drafted a reply. Simple work. And it was routing every single one of those calls, including the ones that just said thanks, to the most expensive model in the catalog. If you want to reduce LLM costs in production without gutting quality, the thing that saves you is rarely a clever prompt. It is deciding which requests deserve the big model and which never did.

I embed with teams for short engagements and ship agents into production, then stay long enough to make them hold under real traffic. Cost is where that promise gets tested. A demo that costs a fraction of a cent per call feels free. Multiply it by a hundred thousand calls a day and it is a line item somebody defends in a board meeting. So let me walk through what I actually do, in the order I do it.

Where the money actually goes

Before touching anything, I get real numbers. Not the dashboard estimate. I want token counts per request, split into input and output, tagged by which feature triggered the call. Most teams have never looked at this, and the shape of it is almost always the same. A small number of request types burn most of the budget, and half of them did not need a frontier model in the first place.

Output tokens are where the real money hides. On the frontier models, generated tokens cost several times what input tokens cost, so an agent that writes long chatty replies is quietly expensive even when its prompts are lean. And long context is its own trap. Stuffing an entire knowledge base into every prompt because retrieval felt like too much work will cost you more than the retrieval system ever would have.

Two levers that reduce LLM costs in production

There are two levers that move the number in a serious way. Model routing sends each request to the cheapest model that can actually do the job. Fine tuning makes a small model good enough to be that cheap destination for a narrow task. Almost everyone tries fine tuning first because it sounds like the sophisticated answer. I think that is backwards. Routing is faster to ship, easier to reverse, and usually captures most of the savings on its own. Fine tuning is the second pass, and only where the volume justifies it.

Route first, it is the cheaper win

A GPT-4 class model or Claude Opus costs on the order of fifteen to thirty times what a small model like GPT-4o mini or Claude Haiku costs per token. That gap is the whole game. If you can prove that a cheap model handles 60 percent of your traffic at the quality bar you actually need, you have cut a huge slice of the bill before you write a single training example.

The mistake people make is routing on vibes. They pick a keyword, or worse, they ask a big model to decide which model to use, which just pays the expensive toll twice. I prefer a cheap classifier and a hard escalation path. Try the small model. If it signals low confidence, or the task carries obvious risk like a refund or a legal question, escalate. Everything else stays cheap.

def route(request):
    # cheap heuristics first, no model call needed
    if request.needs_tool_calls or request.token_estimate > 6000:
        return "large"
    if request.category in HIGH_RISK:      # refunds, legal, cancellations
        return "large"

    # try the small model, let it tell us if it is out of its depth
    result = call("small", request.prompt)
    if result.self_reported_confidence < 0.7 or result.refused:
        return escalate(request)          # retry on large, log the miss
    return result

def escalate(request):
    out = call("large", request.prompt)
    log_escalation(request.id)            # this is your training set later
    return out

Two details make this work in practice. First, log every escalation. Those logged cases are exactly the examples you will fine tune on later, so the router quietly builds your training set for free. Second, do not trust a model's self reported confidence blindly. Sample the small model's output against the large model on a few hundred real requests, measure agreement, and set the threshold from data. I usually start strict and loosen it once I can see the escalation rate settling.

Fine tuning: when it earns its keep

Fine tuning gets oversold. It will not teach a model new facts, it will not fix a broken retrieval pipeline, and it is a slow way to change behavior you could change with a prompt edit in ten seconds. What it does well is one thing: make a small, cheap model reliably match the format and judgment of a big one on a narrow, repetitive task. Classification, extraction, a house writing style, a rigid JSON shape. That is the sweet spot.

So the pattern I like is to run routing first for a few weeks, collect the escalations where the small model gave up, and use those as labeled data. You fine tune the small model on the large model's answers to precisely the cases it kept failing. Now the small model handles more of the traffic, the escalation rate drops, and the average cost per request falls again without any quality loss the user can feel.

A rough break even

Fine tuning is not free. You pay for the training run, you pay a small premium on inference for the customized model on some providers, and you pay in engineering time to build an eval set so you can prove the tuned model did not get worse. That overhead only makes sense at volume. As a rule of thumb I do not bother fine tuning a task doing a few hundred calls a day. At tens of thousands of calls a day for one stable task, it almost always pays back inside a month. Between those, do the arithmetic before you commit.

Cache the boring parts

Before I fine tune anything, I check whether I am paying to process the same tokens over and over. Both major providers now support prompt caching, where a stable prefix like your long system prompt and tool definitions gets cached and billed at a steep discount on repeat calls. If your agent sends the same two thousand token instruction block on every request, and most do, moving that block into a cached prefix is close to free money. Put the stable content at the front, the variable content at the end, and let the cache do its job.

The cheapest token is the one you never send. Retrieval, caching, and a tighter prompt beat a bigger model budget almost every time.

Semantic caching is the other layer, and it is underused. If two users ask the same question in slightly different words, you can serve a stored answer instead of generating a fresh one. I store past query embeddings in pgvector, and on a new request I check for a close match above a similarity threshold before calling any model at all. For a support bot fielding the same twenty questions all day, this alone can knock a real chunk off the bill. Keep the threshold conservative so you do not serve a stale answer to a subtly different question.

Measure cost per resolved task, not per token

Here is the trap at the end of all this. You optimize so hard for cost per token that you push work onto a model too weak for it, the answers get worse, and now a human on the support team spends five minutes cleaning up after every reply. You saved a fraction of a cent and spent a salary. The number that matters is cost per resolved task, all in, including the escalations and the human touch ups. A slightly pricier model that resolves the request on the first try is often the cheaper choice once you count everything.

So I instrument for that. Every request logs which model handled it, whether it escalated, whether a human intervened, and whether the task actually closed. That lets me see the true cost of a route, not just the sticker price of the tokens. And it turns the whole thing into a loop you can tune with evidence instead of guesses.

None of this is exotic. Route to the cheapest model that clears the bar, cache what repeats, fine tune the narrow high volume tasks once you have the data, and judge it all by cost per finished job. Do those four things and the scary invoice stops being scary. The part people miss is that it is an ordering problem, not a technology problem. Get the order right and most of the savings show up before you have trained a thing.