What Is RAG? Retrieval Augmented Generation Explained With a Kenyan SACCO Chatbot Example

Imagine a Kenyan SACCO wants a chatbot that answers member questions: "What is the interest rate on emergency loans?" or "How long is the guarantor release period?" A vanilla LLM does not know your SACCO's specific policies. It will either refuse to answer or, worse, make up a plausible-sounding number.

You could paste the entire policy document into the prompt, but SACCO documents can run to hundreds of pages. That exceeds most context windows and costs a fortune in tokens.

RAG solves this by retrieving only the relevant paragraphs, maybe three to five chunks, and injecting just those into the prompt. The LLM sees the question plus the evidence it needs, and nothing else.

A RAG pipeline has two phases: indexing (done once) and querying (done per question).

Indexing phase:

1. Split your documents into chunks, typically 200 to 800 tokens each.
2. Generate an embedding vector for each chunk using an embedding model.
3. Store the vectors alongside the original text in a vector database (or Postgres with pgvector).

Query phase:

1. The user asks a question: "What collateral do I need for a development loan?"
2. Embed the question using the same embedding model.
3. Search the vector store for the top-k most similar chunks (typically 3 to 5).
4. Build a prompt: system instructions + retrieved chunks + the user's question.
5. Send the prompt to the LLM. It generates an answer grounded in the retrieved chunks.

The retrieval step is what makes RAG different from just prompting. The LLM sees your data at query time without needing to be retrained.

Here is a simplified RAG pipeline in TypeScript using OpenAI embeddings and Supabase with pgvector as the vector store.

Step 1: Index your documents

import { createClient } from '@supabase/supabase-js';
import OpenAI from 'openai';

const supabase = createClient(process.env.SUPABASE_URL!, process.env.SUPABASE_KEY!);
const openai = new OpenAI();

interface DocumentChunk {
  text: string;
  source: string; // e.g. "loan-policy.pdf, page 12"
}

async function indexChunks(chunks: DocumentChunk[]) {
  for (const chunk of chunks) {
    const embeddingRes = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: chunk.text,
    });

    const embedding = embeddingRes.data[0].embedding;

    await supabase.from('sacco_documents').insert({
      content: chunk.text,
      source: chunk.source,
      embedding, // pgvector column
    });
  }
}

Step 2: Query with retrieval

async function askSaccoBot(question: string): Promise<string> {
  // 1. Embed the question
  const qEmbedding = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: question,
  });

  // 2. Retrieve top 4 matching chunks
  const { data: chunks } = await supabase.rpc('match_sacco_documents', {
    query_embedding: qEmbedding.data[0].embedding,
    match_threshold: 0.7,
    match_count: 4,
  });

  // 3. Build prompt with retrieved context
  const context = chunks
    ?.map((c: { content: string; source: string }) =>
      `[Source: ${c.source}]\n${c.content}`
    )
    .join('\n\n');

  const completion = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      {
        role: 'system',
        content:
          'You are a helpful assistant for a Kenyan SACCO. ' +
          'Answer questions using ONLY the provided context. ' +
          'If the context does not contain the answer, say so. ' +
          'Cite the source document when you answer.',
      },
      {
        role: 'user',
        content: `Context:\n${context}\n\nQuestion: ${question}`,
      },
    ],
  });

  return completion.choices[0].message.content ?? 'No answer generated.';
}

The key line is in the system prompt: "Answer using ONLY the provided context." This instruction constrains the model to the retrieved evidence and makes hallucinations far less likely.

The match_sacco_documents function used above is a Postgres function that performs a cosine similarity search:

-- Run this in your Supabase SQL editor
create or replace function match_sacco_documents(
  query_embedding vector(1536),
  match_threshold float,
  match_count int
)
returns table (
  id bigint,
  content text,
  source text,
  similarity float
)
language sql stable
as $$
  select
    id,
    content,
    source,
    1 - (embedding <=> query_embedding) as similarity
  from sacco_documents
  where 1 - (embedding <=> query_embedding) > match_threshold
  order by embedding <=> query_embedding
  limit match_count;
$$;

The <=> operator is pgvector's cosine distance operator. We subtract from 1 to convert distance to similarity, where 1.0 means identical and 0.0 means completely unrelated.

This runs inside Postgres, so you get the speed of a database index without needing a separate vector search service.

RAG works well for factual Q&A over documents. It struggles with:

• Reasoning over scattered facts. If the answer requires combining information from ten different sections, retrieval might not pull all ten. You end up with partial context.
• Style and tone tasks. If you need the model to write in a specific voice or generate creative content, retrieval does not help because the task is generative, not factual.
• Rapidly changing data. RAG is only as fresh as your index. If your SACCO updates loan rates daily, your indexing pipeline needs to keep up.
• Small, stable datasets. If your entire knowledge base fits within the model's context window, you can skip retrieval and just pass everything in. Simpler is better.

For tasks where the model needs to learn a pattern it was not trained on (like a proprietary coding style or a domain-specific classification scheme), fine-tuning may be a better fit. See the RAG vs Fine-Tuning comparison for a deeper breakdown.