LLM Hallucinations: Why They Happen and How Production Apps Reduce Them

An LLM does not "know" things the way you do. It predicts the next token based on patterns in its training data. When you ask "Who is the governor of Nairobi?", the model retrieves a statistical pattern, and if the answer was well-represented in training data, it gets it right. But if you ask "What is the phone number of Mama Mboga on Tom Mboya Street?", there is no pattern to match. The model generates a plausible-looking number anyway, because generating a plausible next token is exactly what it was trained to do.

Hallucinations are more common in specific situations:

• Niche or recent topics. The model's training data has a cutoff date. Anything after that date, or anything too niche to appear often in the training set, is a hallucination risk.
• Specific numbers and dates. Precise figures (prices, statistics, phone numbers, URLs) are frequently hallucinated because the model blends similar patterns from training data.
• Multi-step reasoning. Each reasoning step is a prediction. Errors compound. By step five of a chain of thought, the model may be confidently wrong.
• Confident prompting. Ironically, telling the model "you are an expert" can increase hallucinations because the model becomes less likely to express uncertainty.

Retrieval Augmented Generation is the most effective way to reduce hallucinations for factual Q&A. Instead of relying on the model's training data, you retrieve relevant documents and inject them into the prompt.

The system prompt then instructs the model to answer only from the provided context:

const systemPrompt = `You are a customer support assistant.

Rules:
- Answer ONLY using the provided context documents.
- If the context does not contain enough information to answer,
  say "I don't have that information in our records."
- Never make up product features, prices, or policies.
- Cite the source document when answering.`;

RAG does not eliminate hallucinations entirely. The model can still misinterpret the retrieved documents or mix information from different chunks. But grounding the response in specific documents cuts hallucination rates dramatically compared to ungrounded responses.

When the model cites its sources, hallucinations become auditable. You can verify the citation against the original document.

const systemPrompt = `Answer the question using the provided documents.

For every claim in your answer, include a citation in brackets
referencing the document source. Example: "The interest rate
is 12% per annum [Loan Policy, Section 3.2]."

If no document supports a claim, do not make it.`;

In production, you can automate citation verification. After the model responds, parse the citations and check whether the referenced document actually contains the claimed information. This is a form of post-generation validation.

function verifyCitations(
  response: string,
  documents: { source: string; content: string }[]
): { claim: string; verified: boolean }[] {
  const citations = parseCitations(response);
  return citations.map(citation => {
    const doc = documents.find(d => d.source === citation.source);
    const verified = doc
      ? doc.content.toLowerCase().includes(
          citation.claim.toLowerCase().slice(0, 50)
        )
      : false;
    return { claim: citation.claim, verified };
  });
}

This is not foolproof (the model might paraphrase the source, making string matching fail), but it catches the most egregious fabrications.

The default behavior of most LLMs is to always produce an answer. This is the root cause of many hallucinations. Train the model to refuse when it does not have enough information.

In your system prompt, explicitly define refusal behavior:

const systemPrompt = `You help users with questions about our
Kenyan real estate listings.

If the user asks about:
- A property not in our database: say "I don't have information
  about that property. Please check our website directly."
- Legal advice: say "I can't provide legal advice. Please
  consult a licensed advocate."
- Price predictions: say "I don't predict future prices."

It is better to say you don't know than to guess.`;

The phrase "It is better to say you don't know than to guess" is surprisingly effective. It gives the model explicit permission to refuse, which overrides its default tendency to always answer.

For critical applications (medical, financial, legal), design the UX so that refusal is the expected path. Users should trust a "I don't have that information" response more than a confident-sounding guess.

Free-form text gives the model the most room to hallucinate. Structured output (JSON, fixed categories, multiple choice) constrains the response to valid options.

// Instead of: "What county is this property in?"
// Use a structured format:
const prompt = `Classify this property listing into one of the
following counties. Return ONLY the county name, nothing else.

Valid counties: Nairobi, Mombasa, Kisumu, Nakuru, Kiambu,
Machakos, Kajiado, Uasin Gishu

Listing: "3-bedroom apartment in Kilimani, near Yaya Centre"

County:`;

When the model can only pick from a fixed list, it cannot hallucinate a county that does not exist. Combine this with JSON mode or function calling (which enforce schema compliance) for even stronger guarantees.

For numerical outputs, define valid ranges: "Return a price estimate between 1,000,000 and 50,000,000 KES." This does not guarantee accuracy, but it prevents wildly impossible numbers.

The only way to know how often your system hallucinates is to measure it. Build a test suite of questions where you know the correct answer, and run your system against them regularly.

interface EvalCase {
  question: string;
  expectedFacts: string[]; // facts the answer must contain
  forbiddenClaims: string[]; // claims that would be hallucinations
}

const evalCases: EvalCase[] = [
  {
    question: 'What is the interest rate on emergency loans?',
    expectedFacts: ['12% per annum', 'reducing balance'],
    forbiddenClaims: ['15%', 'flat rate'],
  },
  {
    question: 'Can I get a loan without a guarantor?',
    expectedFacts: ['two guarantors required'],
    forbiddenClaims: ['no guarantor needed', 'one guarantor'],
  },
];

async function runEvals(evalCases: EvalCase[]): Promise<void> {
  for (const testCase of evalCases) {
    const answer = await askBot(testCase.question);
    const answerLower = answer.toLowerCase();

    for (const fact of testCase.expectedFacts) {
      if (!answerLower.includes(fact.toLowerCase())) {
        console.error(`MISSING FACT: "${fact}" not found in answer`);
      }
    }

    for (const claim of testCase.forbiddenClaims) {
      if (answerLower.includes(claim.toLowerCase())) {
        console.error(`HALLUCINATION: "${claim}" found in answer`);
      }
    }
  }
}

Run these evals as part of your CI pipeline. When you change the system prompt, retrieval logic, or model version, the eval suite tells you whether hallucination rates improved or regressed.