Page content

AI & Clouds8 min read

AI FinOps: How an LLM Cost Audit Cuts AI Spend

Krzysztof PytelUpdated November 5, 2026

AI FinOps is the practice of tracking, allocating and cutting the cost of AI in production, from token spend to model and vendor choice, without lowering answer quality. Most teams start with an LLM cost audit: a 2-3 week review that shows where the money goes and which changes save the most.

The pressure is real. Gartner expects worldwide AI spending to reach $2.7 trillion in 2026, up 49.5% in a year, with spend on generative AI models more than doubling. And 98% of FinOps teams now manage AI spend, up from 31% two years earlier, according to the FinOps Foundation's State of FinOps 2026 survey of 1,192 practitioners.

This guide explains what AI FinOps covers, what an LLM cost audit and an AI vendor selection audit look at, which levers cut the bill fastest, and how Code & Pepper runs the work for FinTech, InsurTech and HealthTech teams.

What is AI FinOps?

AI FinOps applies FinOps (shared ownership of technology spend between engineering, finance and product) to AI workloads. It answers three questions: what each AI feature costs per request or per outcome, who owns that cost, and whether the value it creates justifies it.

The discipline grew out of cloud FinOps, but the cost model is different. The FinOps Foundation's comparison of GenAI FinOps and cloud FinOps points to four gaps:

Unit of billing

Cloud bills in vCPU-hours and GB-months. LLMs bill in tokens, and the same text produces different token counts on different models and tokenizers

Predictability

The same prompt can return answers of different length and cost. A model version switch or a small prompt change can move the bill.

Cost of failure

A failed cloud call is cheap. A failed LLM call can burn thousands of output tokens on an answer nobody can use.

Allocation

Cloud teams tag resources. AI teams have to tag each request by feature, customer or team, or the invoice stays one opaque line.

In practice, AI FinOps sits next to your LLMOps services stack: the same traces that show quality also show cost, so savings never come at the price of worse answers.

What does an LLM cost audit cover?

An LLM cost audit traces every token your product sends and receives, maps it to features and customers, and finds the spend that does not improve answers. It usually covers six areas:

  • Spend baseline - monthly cost by provider, model, feature, customer and environment, including test and staging traffic that often runs on production models.
  • Token anatomy - how much of each request is system prompt, chat history, retrieved context, tool output and generated answer. The FinOps Foundation calls context window creep the biggest hidden cost in most AI apps, because stateless APIs resend the whole conversation on every turn.
  • Output and reasoning tokens - output tokens are often priced three to five times higher than input tokens, and reasoning models add "thinking" tokens that vary run to run.
  • Model fit - which tasks run on a flagship model when a smaller or fine-tuned model would pass the same tests.
  • Caching and batching - which prompts repeat enough to cache, and which jobs (reports, back-office extraction, evals) can wait for a discounted batch run.
  • Agent loops and retries - agents multiply spend. Anthropic reports that agents use about 4× more tokens than chat, and multi-agent systems about 15×. Failed tool calls and retry loops are where budgets disappear.

The output is a unit-cost view, such as cost per resolved ticket, per processed claim or per KYC check, plus a ranked list of fixes with expected savings and quality risk for each. Without tracing, an audit is guesswork, which is why we instrument first; see our article on LLM observability for how that tracing works.

What is an AI vendor selection audit?

An AI vendor selection audit compares model providers and hosting options on your own test set, so you pick models on measured quality, cost and risk instead of benchmark headlines. Prices for the same model can differ between platforms: the FinOps Foundation notes that the same open-source model has been priced up to 30% apart across hyperscalers.

We score each option on the criteria below.

Criterion

What we check

Quality on your tasks

Pass rate on a test set built from your real cases, not public leaderboards

Total cost to a good answer

Token price plus retries, longer prompts and human review; the cheapest model per token is not always the cheapest per outcome

Latency

Time to first token and full answer at your peak load

Data location and terms

Region, data retention, training on your data, DPA and BAA availability

Lock-in

How hard it is to switch: proprietary features, prompt formats, committed-spend contracts

Commitments and capacity

Provisioned throughput, committed-use discounts and overage rules

For regulated data, the audit also covers where prompts and outputs are processed, and an AI GDPR compliance review. When no hosted provider fits, a private LLM deployment on open-weight models is part of the comparison.

Which AI FinOps levers cut costs fastest?

The biggest AI FinOps savings usually come from five levers, and most need no change to the user experience. Published figures show the ceiling; your results depend on traffic shape and quality targets.

Lever

How it works

Published saving

Prompt caching

Reuse a long, stable prompt prefix (system prompt, documents, tool definitions) instead of paying full price each call

Cache reads cost 0.1× the base input price on Anthropic's API, with a 1.25× write fee

Batch processing

Send non-urgent jobs (reports, extraction, evals) as an asynchronous batch

50% off on OpenAI's Batch API, results within 24 hours

Model routing

Send easy requests to a cheaper model and hard ones to a flagship

RouteLLM cut cost by over 85% on MT Bench while keeping 95% of GPT-4 quality

Model cascades

Try a cheap model first, escalate only when a check fails

FrugalGPT matched GPT-4 with up to 98% lower cost in research settings

Context trimming

Summarise chat history, retrieve fewer and better chunks, shorten system prompts

Depends on prompt size; often the largest lever in RAG and chat apps

Two cautions. A cache that is written more often than it is read costs more than no cache. And a cheaper model that needs more retries can cost more per good answer, which is why every change is tested against your quality set before it ships. For high-volume, narrow tasks, fine-tuned small language models can cut cost per request further.

How Code & Pepper runs an AI FinOps engagement

Our AI FinOps engagement is a fixed-scope LLM cost audit of 2–3 weeks, with an optional implementation phase. We are model-agnostic and work across OpenAI, Anthropic Claude, Google Gemini, AWS Bedrock, Vertex AI and open-weight models, so recommendations are not tied to one vendor.

  • Instrument and baseline (week 1). We connect tracing to your LLM calls (Langfuse with OpenTelemetry, or your existing tool), tag requests by feature and customer, and build the spend baseline.
  • Analyse and test (week 2). We break down token anatomy, build a test set from your real cases and run candidate models, caching and routing against it. The AI vendor selection audit happens here.
  • Report and roadmap (week 3). You get unit costs per feature, a ranked list of fixes with expected savings and quality impact, and a vendor recommendation.
  • Implement (optional). Our engineers ship the fixes, add cost alerts and budgets per feature, and hand over dashboards your finance and product teams can read.

AI FinOps FAQ

What is AI FinOps in simple terms?

AI FinOps is managing the cost of AI the way FinOps teams manage cloud cost: you measure what each AI feature costs, assign that cost to an owner, and cut waste without hurting quality. The main unit is the token, not the server hour.

How is AI FinOps different from cloud FinOps?

Cloud FinOps works with stable units such as vCPU-hours. AI FinOps works with tokens, whose count varies by model, and with outputs whose length and cost change from one request to the next. Allocation also moves from tagging resources to tagging each API request.

What is an LLM cost audit?

An LLM cost audit is a short review of how your product uses language models. It breaks spend down by feature, model and token type, tests cheaper options against your quality bar and ends with a ranked list of savings.

How long does an LLM cost audit take?

At Code & Pepper an LLM cost audit takes 2–3 weeks. Implementation of the fixes is a separate, optional phase whose length depends on how many changes you choose to ship.

What is an AI vendor selection audit?

An AI vendor selection audit compares model providers and hosting options on your own test set. It scores each one on quality, total cost per good answer, latency, data location, contract terms and lock-in risk.

How much can AI FinOps save?

Savings depend on your traffic. Published figures give the range: batch APIs cut price by 50%, cache reads can cost a tenth of normal input, and research on model routing shows cost cuts of 35–85% at 95% of flagship quality. The audit gives you an estimate for your own system.

Will cutting LLM costs make our AI worse?

It should not, if every change is tested. We run each cost fix against a test set built from your real cases and ship only those that hold quality. Ongoing LLM observability then catches any drift after release.

Do we need observability before an AI FinOps audit?

It helps but is not required. If you have no tracing, we add it in the first week of the audit, because cost per feature cannot be measured from a provider invoice alone.

Who should own AI FinOps in a company?

Ownership is shared. Engineering owns the levers (models, prompts, caching), product owns the value per feature, and finance owns budgets and forecasts. The audit sets up the dashboards and alerts all three can read.

Get control of your AI spend

If your LLM bill grows faster than usage, or nobody can say what one AI feature costs, an LLM cost audit is the fastest way to find out. In 2–3 weeks you get unit costs per feature, a tested savings plan and an AI vendor selection audit you can take to your CFO.