What Are Embeddings? How Text Becomes Numbers

Think of a library where every book has a GPS coordinate. Cookbooks cluster together on one shelf, programming books on another. If you know the coordinates of "JavaScript: The Good Parts," you can find nearby books that cover similar topics without reading a single page.

Embeddings work the same way, but for text. A model reads a sentence and outputs an array of numbers, typically 384 to 3,072 floats. That array is the sentence's "GPS coordinate" in a high-dimensional space. Sentences with similar meanings get similar coordinates.

For example, the sentences "M-Pesa lets you send money from your phone" and "You can transfer cash using Safaricom's mobile wallet" say roughly the same thing. Their embedding vectors will be close together, even though they share few words. Meanwhile, "The Nairobi Expressway reduced travel time to JKIA" will be far away from both, because its meaning is unrelated.

The pipeline has three steps:

1. Tokenisation. The text is broken into tokens, which are word pieces. "Embeddings" might become ["Embed", "dings"]. Each token maps to a number the model understands.
2. Model forward pass. The tokens flow through a neural network (a transformer). The network has been trained on enormous amounts of text, learning which words appear in similar contexts. Each layer refines the representation.
3. Pooling. The model outputs one vector per token. To get a single vector for the entire sentence, the outputs are combined, usually by averaging. The result is your embedding.

You do not need to understand the internals to use embeddings. Call an API, pass in text, get back an array of numbers. But knowing the pipeline helps you reason about edge cases, like why very short inputs (a single word) sometimes produce weaker embeddings than full sentences.

This TypeScript snippet embeds three sentences and compares them using cosine similarity. You need an OpenAI API key to run it.

import OpenAI from 'openai';

const openai = new OpenAI();

const sentences = [
  'M-Pesa lets you send money from your phone',
  'You can transfer cash using Safaricom mobile wallet',
  'The Nairobi Expressway reduced travel time to JKIA',
];

async function embed(text: string): Promise<number[]> {
  const res = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: text,
  });
  return res.data[0].embedding;
}

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0, magA = 0, magB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

async function main() {
  const vectors = await Promise.all(sentences.map(embed));

  console.log('M-Pesa vs Safaricom wallet:',
    cosineSimilarity(vectors[0], vectors[1]).toFixed(4));
  console.log('M-Pesa vs Expressway:',
    cosineSimilarity(vectors[0], vectors[2]).toFixed(4));
}

main();

When you run this, the first pair (both about mobile money) scores high, around 0.85 to 0.92. The second pair (mobile money vs. a highway) scores much lower, around 0.15 to 0.30. The numbers capture what your intuition already knows: the first two sentences are about the same thing.

Different models produce different vector sizes:

• text-embedding-3-small (OpenAI): 1,536 dimensions. Good balance of quality and cost.
• text-embedding-3-large (OpenAI): 3,072 dimensions. Higher quality, roughly double the storage.
• all-MiniLM-L6-v2 (open source, sentence-transformers): 384 dimensions. Runs locally on a laptop, no API key needed.
• voyage-3 (Anthropic partner): 1,024 dimensions. Strong on code and technical text.

More dimensions capture more nuance but cost more to store and slower to compare. For most applications, 1,536 dimensions from a mid-range model is a solid default. If you are running on a tight budget or want to avoid external API calls, an open-source model with 384 dimensions works surprisingly well.

The critical rule: use the same model for both indexing and querying. Vectors from different models live in different spaces and cannot be compared.

Embeddings unlock three major capabilities:

• Semantic search. Traditional keyword search fails when users phrase things differently from your documents. A search for "send money on phone" won't match a document titled "Mobile Wallet Transfer Guide" using keyword matching. With embeddings, both phrases map to nearby vectors, and the search succeeds.
• RAG (Retrieval Augmented Generation). When you want an LLM to answer questions from your own documents, you embed and store those documents, then retrieve the relevant chunks at query time. Embeddings are the retrieval engine in RAG.
• Recommendations and clustering. Embed your product descriptions or articles. Products with similar embeddings are similar in meaning. You can recommend "customers who viewed this also liked" without hand-coding rules.

If your use case is pure keyword matching and your users always use the exact terms in your data, you may not need embeddings. But the moment user language diverges from your data's language, embeddings close that gap.