Chunking Strategies for RAG: Sizes, Overlap, and Structure

When you embed a document for RAG, you do not embed the entire document as one vector. A 50-page SACCO policy document as a single embedding would produce a vector that represents the average meaning of the entire document, too vague to match specific questions.

Instead, you split the document into chunks, each covering a specific topic or section. Each chunk gets its own embedding. When a user asks "What is the emergency loan interest rate?", the retrieval system finds the specific chunk about emergency loans, not the entire policy document.

The challenge: chunks that are too small lose context ("12% per annum" means nothing without knowing it refers to emergency loans). Chunks that are too large dilute the embedding with unrelated information, making retrieval less precise.

The simplest approach: split text into chunks of N tokens (or characters), with optional overlap.

function chunkByTokens(
  text: string,
  chunkSize: number = 500,
  overlap: number = 50
): string[] {
  // Simple word-based approximation (1 token ~ 0.75 words)
  const words = text.split(/\s+/);
  const wordsPerChunk = Math.floor(chunkSize * 0.75);
  const overlapWords = Math.floor(overlap * 0.75);
  const chunks: string[] = [];

  let start = 0;
  while (start < words.length) {
    const end = Math.min(start + wordsPerChunk, words.length);
    chunks.push(words.slice(start, end).join(' '));
    start = end - overlapWords;
    if (start >= words.length) break;
  }

  return chunks;
}

Pros: Simple. Predictable chunk sizes. Easy to reason about costs.

Cons: Splits can land in the middle of a sentence, paragraph, or logical section. A sentence about loan requirements might be split across two chunks, making neither chunk complete.

When to use: As a starting point. If your documents are uniform in structure (like support tickets or product reviews), fixed-size chunking often works well enough.

If your documents have structure (headings, sections, numbered clauses), use that structure as chunk boundaries.

interface StructuralChunk {
  heading: string;
  content: string;
  source: string;
}

function chunkByHeadings(
  markdown: string,
  source: string
): StructuralChunk[] {
  const sections = markdown.split(/^##\s+/m);
  const chunks: StructuralChunk[] = [];

  for (const section of sections) {
    if (!section.trim()) continue;
    const lines = section.split('\n');
    const heading = lines[0].trim();
    const content = lines.slice(1).join('\n').trim();

    if (content.length < 50) continue; // skip tiny sections

    chunks.push({ heading, content, source });
  }

  return chunks;
}

// For a SACCO policy document:
// "## Emergency Loans\nInterest rate is 12% per annum..."
// becomes one chunk with heading "Emergency Loans"

Pros: Each chunk is a coherent unit of information. Retrieval is more precise because the embedding represents a complete topic.

Cons: Sections vary wildly in length. A one-sentence section produces a weak embedding. A 5,000-word section is too large for effective retrieval.

Fix: Combine short sections with their neighbors. Split long sections using fixed-size chunking within the section boundary.

Without overlap, information at chunk boundaries gets lost. If a sentence spans two chunks, neither chunk contains the complete sentence, and the embedding for each chunk may miss its meaning.

Overlap duplicates a portion of text between adjacent chunks. A 50-token overlap means the last 50 tokens of chunk N are also the first 50 tokens of chunk N+1.

How much overlap?

• 10% to 20% of chunk size is a practical default.
• For 500-token chunks, use 50 to 100 tokens of overlap.
• More overlap means better coverage but more chunks and higher storage costs.

The tradeoff: A 1,000-token document with 500-token chunks and 0 overlap produces 2 chunks. With 100-token overlap, it produces 3 chunks. The extra chunk means 50% more embeddings to generate and store, and 50% more vectors to search. For most datasets, this overhead is acceptable. For millions of documents, it matters.

Start with these defaults and adjust based on your eval results:

• Chunk size: 400 to 800 tokens. This balances precision with context. 500 tokens is a good starting point.
• Overlap: 50 to 100 tokens. Enough to cover most boundary sentences without excessive duplication.
• Top-k retrieval: 3 to 5 chunks. Start with 4 retrieved chunks per query. Increase if answers are incomplete. Decrease if irrelevant chunks are confusing the model.

When to use smaller chunks (200 to 400 tokens):

• Your documents have dense, specific information (legal clauses, technical specs, pricing tables)
• Your users ask very specific questions ("What is the penalty for early repayment?")
• You want precise retrieval and are willing to retrieve more chunks to compensate

When to use larger chunks (800 to 1,500 tokens):

• Your documents are narrative (blog posts, reports, stories)
• Your users ask broad questions ("Explain your loan products")
• Context around a fact is important for the model to generate a good answer

Store metadata alongside each chunk to improve retrieval and answer quality:

interface EnrichedChunk {
  content: string;
  embedding: number[];
  metadata: {
    source: string;       // "loan-policy-2026.pdf"
    section: string;      // "Emergency Loans"
    page: number;         // 12
    chunkIndex: number;   // 3 of 45
    dateIndexed: string;  // "2026-08-04"
  };
}

// When building the prompt, include metadata:
const context = retrievedChunks
  .map(c =>
    `[Source: ${c.metadata.source}, Section: ${c.metadata.section}]\n${c.content}`
  )
  .join('\n\n');

Metadata enables:

• Filtered retrieval. Only search chunks from a specific document or date range.
• Source citations. The model can cite "Loan Policy, Section 4.2" in its answer.
• Freshness. Prefer recently indexed chunks over stale ones.
• Debugging. When a wrong answer appears in evals, you can trace which chunks were retrieved and from which source.