LLM cost management

LLM cost management is not a monthly surprise and a spreadsheet. It is knowing which app spent the tokens, capping a loop before finance sees it, and routing cheap work to a cheap model on purpose. If the only number you have is the vendor invoice, you are not managing cost. You are paying it.

This sits on the same hop as gateway architecture. Putting models in existing software, with cost on the route, is LLM integration. Cost controls belong on that path, next to logging and redaction. This post is the cost cut. It is not labor-hours ROI. Those are different dollars. Do not add them on one slide and call it a case.

We do not publish package prices. We do not invent a customer who "cut spend 40 percent."

Attribute every call to an app and a workflow

A vendor bill by API key is not attribution if twelve services share the key. The gateway should stamp:

• App or service identity.
• Workflow or route name, if you have one.
• Model and pin.
• Tokens in, tokens out, estimated dollars.
• User or tenant when it exists.

Yes/no:

• You can rank last week's spend by app without opening the vendor console.
• A new prototype cannot hide on the production key.
• Finance can see a number that matches an inventory row.

If you cannot answer "which workflow," you will cut the wrong thing. Teams will hide spend in a shared key. That is the same failure as a shared logging account.

Cap loops. Treat overage as an incident.

Retry storms and a prompt that asks for a novel will complete a novel. Cost management is a rate limit and a budget, not a stern email.

Put on the hop:

• Per-app tokens or dollars per hour and per day.
• A hard stop, not only an alert, for non-production.
• A page when production hits the cap, with a named owner who can raise it.
• Timeouts you own, so the provider's long completion cannot run unbounded.

A retry loop is an operations event. If you learn about it on the invoice, the architecture failed. The same is true of a weekend job someone pointed at the largest model "to be safe."

Route on purpose, not by whoever copied the SDK

Not every completion needs the most expensive model. Internal drafts of public copy and production customer summaries are different routes. The cost rule is simple: the route table is a budget tool.

• Low-risk, high-volume drafts go to the cheap, allowed model.
• High-risk or eval-gated work uses the model you tested, not the one on sale this week.
• Region and residency constraints beat price. A cheap endpoint in the wrong country is not a savings.

Developers will pick the smartest model if the client lets them. The gateway should not let a ticket-draft app silently move to the frontier model. That change is a change request, with a cost note.

Separate token spend from labor savings

Boards mix these up. Labor ROI is hours times loaded rate times automation share, minus implementation cost. Token spend is the run cost of the model path. A workflow can have a strong labor case and a sloppy token bill. It can also have a tiny token bill and no hours back.

Keep two lines:

• Build and run (tokens, vendors, the people who operate the gateway).
• Labor hours returned, measured, not hoped.

If you subtract an invented error-cost number to rescue a weak labor case, you are pitching. If you hide token spend inside "innovation," finance will find it.

Implementation cost in the labor calculator is your input, not a quote. Token unit prices change. Re-read the vendor terms when they do. Cost management includes noticing a price change, not only a volume change.

Review spend on the same cadence as the inventory

Operationalize or the caps go stale.

• Weekly: top apps, loops, denied-over-budget events.
• Monthly: route table versus actual model mix.
• When a new app ships: a budget on day one, even a small one.

Yes/no:

• Someone owns the spend number besides "the AI team."
• A cap change is recorded.
• Personal-account use is not counted as "savings" because it did not hit the bill. It hit leakage instead.

Cost controls on the gateway are the same engineering as redaction and deny: policy on the request, in your environment. We do not sell a magic optimizer. We put the meter where the call already has to go.

If the invoice is still your first dashboard, put the meter on the hop. Keep labor math on a separate line. Run that line on the AI ROI calculator.