Prepare and chunk source documents
Clean documents, split them into retrievable chunks, and keep the context and metadata each chunk needs.
- Compare fixed, sentence and structure-aware chunking
- Use overlap and size to keep answers intact
- Attach source, heading and permission metadata to every chunk
Search returns chunks, not whole documents, so how you cut documents decides what can be found. Before chunking, extract clean text: strip navigation menus and boilerplate, keep headings and lists, turn tables into readable rows, and remember where each piece came from.
Then choose a chunking strategy:
- Fixed-size windows of N words or tokens - simple, but can cut sentences in half.
- Sentence or paragraph packing - keeps natural units together, with uneven sizes.
- Structure-aware - split on headings and sections, keeping the heading path with each chunk.
Overlap repeats a few words between neighboring chunks so a fact that straddles a boundary still appears whole somewhere - at the cost of a bigger index.
Try it
Up to 15 XPChunk the returns policy
Try each strategy and size. Your goal: make sure the sentence that answers the test question lands inside one chunk. Notice how fixed windows with no overlap can slice it in two.
5 chunks · about 17 words each
- #1Returns policy. We want you to love what you bought. Most items can be returned within 30 days of delivery
- #2for a full refund. Items must be unused and in their original packaging. Electronics are different. Opened electronics can only
- #3be returned within 14 days, and a 10 percent restocking fee applies. Unopened electronics follow the normal 30 day window.
- #4Sale items are final sale and cannot be returned. Gift cards cannot be returned either. Refunds go back to the
- #5original payment method within 5 business days.
Test question: “Is there a fee for returning opened electronics?”
Answer sentence: “Opened electronics can only be returned within 14 days, and a 10 percent restocking fee applies.”
Chunks need context
A chunk that says “They cannot be returned either.” is useless on its own - what can’t be returned? Two fixes:
- Keep the heading path (
Returns policy > Sale items) and document title with each chunk, and include it in what you index. - Contextual retrieval (Anthropic, 2024) uses a model to write a one-line summary of where each chunk sits in its document, and prepends it before indexing. Combined with keyword search, it cut retrieval failures by 49% in their tests.
Also store metadata: source URL, section, date, language and - crucially - who may see it. You’ll need it to cite, filter and audit.
1chunk = {
2 "id": "returns#3",
3 "text": "Sale items are final sale and cannot be returned.",
4 "heading_path": ["Returns policy", "Sale items"],
5 "source": "https://help.example.com/returns",
6 "allowed_groups": ["customers", "support"],
7}
8print(" > ".join(chunk["heading_path"]) + ": " + chunk["text"])Returns policy > Sale items: Sale items are final sale and cannot be returned.
Key takeaways
Clean text first; keep structure like headings, lists and tables.
Chunk so answers stay whole: sentence or structure-aware splits, or overlap.
Store each chunk with its source, heading path and permissions.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Fixed-size chunks with overlap
The first line is size overlap; the second is a text. Split it into words and print windows of size words, each starting size - overlap words after the previous one, as N: words. Stop after the window that reaches the last word.
- Size 5, overlap 2
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Structure-aware chunks with heading paths
The input is a Markdown document. Lines starting with # start a top-level section and ## a subsection. Make one chunk per section from its non-heading lines (joined with spaces), skipping sections without text. Print each as Heading > Subheading: text (just Heading: text for top-level text).
- A help article
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…