What it means
Document chunking is the process of splitting long documents into smaller sections before they are turned into embeddings and stored for retrieval. Chunk size and boundaries strongly affect whether the system later retrieves the right information.
Real-world example
A social services department splits its 200-page benefits handbook into chunks. When the chunks were split every 500 words regardless of structure, an eligibility rule and its exceptions ended up in different chunks, and the assistant gave incomplete answers. Splitting by section headings, and keeping each section's title with its text, fixed most of these errors.
Why it matters for AI adoption
Chunking is an unglamorous design decision with a large effect on answer quality. Documents with clear structure, headings and consistent formatting are much easier to chunk well, which is one reason content quality matters for AI.
Questions to ask
- How are our documents split, and does that respect their structure (sections, tables, lists)?
- Have we tested retrieval quality with different chunking approaches?