The problem: context inflation
We have hundreds of markdown files: personas, product specs, market research, marketing copy. Throwing it all into every LLM call meant 80,000+ tokens per query. At GPT-5.2 pricing ($1.75/1M input, $14/1M output), that’s real money. And it scales with every question.
The business driver wasn’t “let’s try RAG.” It was: we’re going to ask this a lot. What does it cost?
The approach: chunk, embed, retrieve
We built a simple RAG pipeline over our product docs:
1. Chunk by section — Markdown files split at `##` and `###`, ~300–600 tokens per chunk. Sections preserve meaning; random splits don’t.
2. Local embeddings — sentence-transformers (all-MiniLM-L6-v2). No API key, no per-chunk cost. Index once, query freely.
3. Retrieve top-k — For “What is our positioning for Persona 4?” we pull 8–12 chunks, not 80k tokens.
4. Build the prompt — Question + retrieved chunks only. The LLM sees what matters.
Indexing is free. Query retrieval is free. The only paid part is the LLM call—and that call is 10–20× smaller.
The numbers
I built a `compare_cost` script that estimates both paths without making API calls. For a typical product question:
=== LLM + full context ===
input_tokens: ~80,000
output_tokens: ~400
cost: ~$0.14 per query
=== LLM + RAG ===
input_tokens: ~4,000
output_tokens: ~400
cost: ~$0.01 per query
RAG uses ~20× fewer input tokens and costs ~14× less per query.
Run it yourself: `python3 -m rag.compare_cost “What are our pricing tiers?”` — no API key needed.
Design choices that matter
- Domain filters — We can restrict retrieval to `personas` or `product` or `marketing`. Smaller context, faster and cheaper.
- Section-based chunking — Keeps “Positioning” with “Persona 4” in one chunk instead of splitting mid-paragraph.
- Local embeddings — Embedding APIs add latency and cost. For internal docs, local models are fine.
## Takeaway
RAG isn’t magic. It’s cost control. When your product context is large and you’re querying often, full context doesn’t scale. Chunk, embed, retrieve—and measure the difference. The numbers will tell you when it’s worth it.