Keyboard Shortcuts N Next post
P Previous post
S Save / unsave
R Read aloud
T Toggle theme
/ Focus search
Esc Close panels
🔥
Ready to read...
LLM Engineering LLM Engineering: From Zero to Production Module 1 - LLM Foundations

Embeddings & Vectors: The Math That Makes LLMs 'Understand'

Reviewed & accurate
AI Summary

What You Will Learn

INTERMEDIATE

  • The big idea: words as numbers (and why this unlocks everything)
  • Why king - man + woman ≈ queen works — vector arithmetic on meaning
  • Cosine similarity in plain English — the math behind semantic search

The Big Idea: Words as Numbers

Computers can't do math on words. They can do math on numbers. So if we want a model to "understand" language, the first step is to convert words into numbers. But not just any numbers — we need numbers that capture meaning. The word "dog" should be numerically close to "puppy" and "pup", and far from "democracy" or "calculus".

This is what embeddings are: a way to convert each word (or sentence, or paragraph, or document) into a vector of numbers — usually 256 to 4,096 dimensions — where words with similar meanings end up close together in vector space. The model learns these vectors during training, by noticing which words appear in similar contexts. "Dog" and "puppy" both appear near "bark", "leash", "pet", "cute" — so they end up with similar vectors.

Words as Vectors (Simplified to 2D for Visualization) dog[0.9, 0.1] puppy[0.85, 0.15] cat[0.7, 0.2] democracy[0.1, 0.9] vote[0.15, 0.85] election[0.2, 0.8] Animals cluster on the left, politics on the right
Embeddings place words with similar meanings near each other in high-dimensional space.

The Famous King - Man + Woman = Queen

Once words are vectors, something magical happens: vector arithmetic = meaning arithmetic. The classic example is king - man + woman ≈ queen. Here's why it works:

  1. The vector from man to woman encodes "gender direction" — maleness to femaleness.
  2. If you start at king (a male royal) and walk in the same "gender direction", you should arrive at the female equivalent.
  3. That lands you near queen in vector space.

This works for many relationships: Paris - France + Italy ≈ Rome (capital-of relation), walk - walked + swim ≈ swam (verb tense), big - bigger + small ≈ smaller (comparative). The model learns these relationships implicitly from training data. The math doesn't "know" about gender or capitals — it just learns that the offset between king/queen is the same as the offset between man/woman.

Why This Is Revolutionary
Before embeddings, search was keyword-based: 'cat' and 'feline' were completely different words, so search engines couldn't connect them. With embeddings, the model knows they're related because their vectors are close. This single insight — words can be located in meaning space — powers Google Search, recommendation engines, RAG, and most modern NLP. We'll use it constantly in this course.

Sentence & Document Embeddings

Embeddings aren't just for words. You can embed entire sentences, paragraphs, or documents into a single vector. Models like OpenAI's text-embedding-3-large, Cohere's embed-english-v3, or open-source BGE and E5 take any text and produce a single vector (usually 768-3072 dimensions) that captures its overall meaning.

This is what makes RAG possible. You embed all your documents, store the vectors in a vector database, and when a user asks a question, you embed the question and find the documents whose vectors are closest to it. We'll build this exact system in Lesson 34.

embeddings.pypython
# Generate embeddings with OpenAI
from openai import OpenAI
client = OpenAI()

texts = [
    "The cat sat on the mat.",
    "A feline rested on a rug.",
    "Interest rates rose 0.5% this quarter.",
]

response = client.embeddings.create(
    input=texts,
    model="text-embedding-3-small"  # 1536 dimensions
)
vectors = [d.embedding for d in response.data]

# Sentence 1 and 2 should have high similarity (both about cats on mats)
# Sentence 1 and 3 should have low similarity (different topics)
import numpy as np
sim_1_2 = np.dot(vectors[0], vectors[1])  # ~0.85 (high)
sim_1_3 = np.dot(vectors[0], vectors[2])  # ~0.25 (low)
print(f"Cat sentence vs Feline sentence: {sim_1_2:.3f}")
print(f"Cat sentence vs Interest rate sentence: {sim_1_3:.3f}")

Cosine Similarity — The Math Behind Semantic Search

To find "similar" vectors, you need a similarity function. The most common is cosine similarity, which measures the angle between two vectors:

  • 0° angle (vectors point same direction) → cosine = 1.0 → identical meaning
  • 90° angle (perpendicular) → cosine = 0.0 → unrelated
  • 180° angle (opposite) → cosine = -1.0 → opposite meaning

In practice, embedding vectors are normalized to length 1, so cosine similarity is just the dot product — a single multiplication and sum. This makes search extremely fast: millions of vectors can be searched in milliseconds.

cosine.pypython
# Cosine similarity in 3 lines
import numpy as np

def cosine_sim(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# If vectors are already normalized, just use np.dot(a, b)

Choosing an Embedding Model

Not all embedding models are equal. Here's the landscape as of late 2026, ranked by quality on the MTEB benchmark:

ModelProviderDimensionsCost / 1M tokensBest For
text-embedding-3-largeOpenAI3072$0.13Best proprietary quality
text-embedding-3-smallOpenAI1536$0.02Best cost/quality
embed-english-v3Cohere1024$0.10English-only, fast
voyage-3Voyage AI1024$0.12Code & mixed content
bge-large-en-v1.5Open-source1024Free (self-host)Best free option
E5-Mistral-7BOpen-source4096Free (self-host)Best quality self-host

For most projects, start with text-embedding-3-small — it's 7× cheaper than the large version and 95% as good. Move to large only if you have specific quality needs. For self-hosting, BGE or E5 are excellent.

🌍 REAL WORLD: Embeddings Power Modern Search
Google's BERT (2019) was the first major production embedding-based search system. Today, every major search engine uses embeddings: Google Search, Bing, Elasticsearch (with kNN), Postgres (with pgvector), and every RAG system you'll build. Embeddings replaced the keyword-only era — now you can search for meaning, not just for exact word matches.

Practical Exercise

Get an OpenAI API key from platform.openai.com (or use Ollama for free local embeddings).
Run the embeddings.py script above. See how the cat-related sentences cluster together.
Now embed these three: 'Python is a great programming language', 'Java is a great programming language', 'The snake slithered through the grass'. Notice how the first two (programming languages) cluster closer than to the third (a real snake).

🎯 Key Takeaways

  • Embeddings convert words/sentences/documents into vectors of numbers.
  • Words with similar meanings end up close in vector space — this is the foundation of semantic search.
  • Vector arithmetic captures meaning: king - man + woman ≈ queen.
  • Cosine similarity measures vector angle — fast and effective.
  • Start with text-embedding-3-small ($0.02/1M tokens); use BGE/E5 for self-hosting.
  • Embeddings power RAG, semantic search, recommendation systems, and clustering.
Previously: Previously: Lesson 06 covered hallucination causes and fixes.
Today: Today: you learned embeddings — the math that lets LLMs 'understand' meaning.
Next: Next: Lesson 08 compares the top 8 frontier LLMs in 2026 so you know which to pick for which job.
Test Your Knowledge
How did you find this?

Comments

Join the discussion! Sign in with your Google or Blogger account, or comment as Anonymous - no account needed. For quick questions, also reach me on Telegram @cytestch.

Comments