The problem: AI costs aren’t magic
LLM pricing is per token. Input tokens (your prompt + context) and output tokens (the response) each have a rate. A 100k-token context at $1.75/1M input costs $0.175 per call—before the model generates a single word. Scale that to hundreds of queries a day and the bill adds up fast.
Most teams don’t think about it until the invoice lands. By then, the pattern is baked in: “We just stuff everything into the prompt.”
The basics: token economics
Input vs output — Output is usually 5–10× more expensive than input. Short, focused prompts with concise answers cost less.
Context size — Bigger context = more input tokens. If you’re sending 80k tokens when 4k would do, you’re paying 20× for no gain.
Model choice — gpt-4o-mini is a fraction of gpt-4o. For retrieval, summarization, or simple Q&A, the cheaper model often suffices.
The lever that matters most: what you put in the prompt.
When RAG pays off
RAG (retrieval-augmented generation) shrinks context by fetching only relevant chunks. Instead of 80k tokens of product docs, you send 4k. Same quality of answer, 20× less input.
Rule of thumb: if your context is large and you query often, RAG pays for itself quickly. The break-even is low—a few dozen queries at scale.
We run compare_cost before committing to a workflow: “For this question, full context = X tokens, RAG = Y. At current pricing, that’s $Z vs $W per query.” No API calls, no guesswork.
A simple decision framework
Small, static context — A few KB of instructions. Full context is fine. Don’t over-engineer.
Large, growing context — Product docs, knowledge bases. RAG.
High query volume — Even small savings compound. Measure. Optimize.
Sensitive to latency — Fewer tokens = faster. RAG can help there too.
Takeaway
AI costs are predictable if you treat them as token economics. Lead with context size. Use RAG when context is large. Choose the cheapest model that meets quality. And measure before you scale.