The concept
RAG (Day 31) doesn't search whole documents. It searches chunks: smaller pieces the documents were split into ahead of time. Each chunk gets its own embedding (Day 11) and sits in the vector database (Day 30).
Chunking: how you cut documents into those pieces. It quietly decides whether retrieval finds the right passage.
Walkthrough: an 80-page benefits booklet
Too big: 1 chunk per 10 pages
- A chunk covers deductibles, pharmacy, vision and appeals all at once. Its embedding is a blurry average of everything.
- It matches lots of questions weakly, and fills the context window (Day 20) with noise.
Too small: 1 chunk per sentence
- "The deductible is $3,000." Which deductible? In-network? Family? The context is gone.
The bad cut
Say the deductible table sits on page 14:
- Column headers: In-network | Out-of-network
- Individual: $1,500 | $3,000
- Family: $3,000 | $6,000
A fixed 500-word splitter cuts right after the headers. Chunk A has the headers, chunk B has the numbers with no labels. Retrieval finds chunk B, and the model guesses which column "$3,000" belongs to.
Better chunking
- Split by structure: headings and sections, not a fixed word count.
- Keep tables whole, with their headers.
- Add overlap: repeat a sentence or two between neighboring chunks so ideas aren't sliced mid-thought.
- Attach metadata: section title, plan name, plan year, page number. Then you can filter ("2026 Gold plan only") and cite the page.
- Aim for chunks that answer one question well, often a few paragraphs.
Measure it, don't guess
Write 20–30 real member questions with the correct page for each. Run them through retrieval and check: is the right chunk in the top 3? Say 17 of 25 are. Fix the chunking, rerun, compare. Now you know it improved.
Where you see it at work
- Healthcare: benefits booklets, prior-authorization rules, clinical protocols.
- Insurance: policy wordings where exclusions sit pages away from coverage.
- Banking: fee schedules and rate tables that must never lose their headers.
- Business analysis: writing the test questions is a perfect BA task. You know what users actually ask.
In one sentence
Good RAG starts with good chunks: split by structure, keep tables whole, add overlap and metadata, and test retrieval with real questions.