What You Will Learn
INTERMEDIATE
- The big idea: words as numbers (and why this unlocks everything)
- Why
king - man + woman ≈ queenworks — vector arithmetic on meaning - Cosine similarity in plain English — the math behind semantic search
The Big Idea: Words as Numbers
Computers can't do math on words. They can do math on numbers. So if we want a model to "understand" language, the first step is to convert words into numbers. But not just any numbers — we need numbers that capture meaning. The word "dog" should be numerically close to "puppy" and "pup", and far from "democracy" or "calculus".
This is what embeddings are: a way to convert each word (or sentence, or paragraph, or document) into a vector of numbers — usually 256 to 4,096 dimensions — where words with similar meanings end up close together in vector space. The model learns these vectors during training, by noticing which words appear in similar contexts. "Dog" and "puppy" both appear near "bark", "leash", "pet", "cute" — so they end up with similar vectors.
The Famous King - Man + Woman = Queen
Once words are vectors, something magical happens: vector arithmetic = meaning arithmetic. The classic example is king - man + woman ≈ queen. Here's why it works:
- The vector from
mantowomanencodes "gender direction" — maleness to femaleness. - If you start at
king(a male royal) and walk in the same "gender direction", you should arrive at the female equivalent. - That lands you near
queenin vector space.
This works for many relationships: Paris - France + Italy ≈ Rome (capital-of relation), walk - walked + swim ≈ swam (verb tense), big - bigger + small ≈ smaller (comparative). The model learns these relationships implicitly from training data. The math doesn't "know" about gender or capitals — it just learns that the offset between king/queen is the same as the offset between man/woman.
Sentence & Document Embeddings
Embeddings aren't just for words. You can embed entire sentences, paragraphs, or documents into a single vector. Models like OpenAI's text-embedding-3-large, Cohere's embed-english-v3, or open-source BGE and E5 take any text and produce a single vector (usually 768-3072 dimensions) that captures its overall meaning.
This is what makes RAG possible. You embed all your documents, store the vectors in a vector database, and when a user asks a question, you embed the question and find the documents whose vectors are closest to it. We'll build this exact system in Lesson 34.
# Generate embeddings with OpenAI
from openai import OpenAI
client = OpenAI()
texts = [
"The cat sat on the mat.",
"A feline rested on a rug.",
"Interest rates rose 0.5% this quarter.",
]
response = client.embeddings.create(
input=texts,
model="text-embedding-3-small" # 1536 dimensions
)
vectors = [d.embedding for d in response.data]
# Sentence 1 and 2 should have high similarity (both about cats on mats)
# Sentence 1 and 3 should have low similarity (different topics)
import numpy as np
sim_1_2 = np.dot(vectors[0], vectors[1]) # ~0.85 (high)
sim_1_3 = np.dot(vectors[0], vectors[2]) # ~0.25 (low)
print(f"Cat sentence vs Feline sentence: {sim_1_2:.3f}")
print(f"Cat sentence vs Interest rate sentence: {sim_1_3:.3f}")
Cosine Similarity — The Math Behind Semantic Search
To find "similar" vectors, you need a similarity function. The most common is cosine similarity, which measures the angle between two vectors:
- 0° angle (vectors point same direction) → cosine = 1.0 → identical meaning
- 90° angle (perpendicular) → cosine = 0.0 → unrelated
- 180° angle (opposite) → cosine = -1.0 → opposite meaning
In practice, embedding vectors are normalized to length 1, so cosine similarity is just the dot product — a single multiplication and sum. This makes search extremely fast: millions of vectors can be searched in milliseconds.
# Cosine similarity in 3 lines
import numpy as np
def cosine_sim(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# If vectors are already normalized, just use np.dot(a, b)
Choosing an Embedding Model
Not all embedding models are equal. Here's the landscape as of late 2026, ranked by quality on the MTEB benchmark:
| Model | Provider | Dimensions | Cost / 1M tokens | Best For |
|---|---|---|---|---|
| text-embedding-3-large | OpenAI | 3072 | $0.13 | Best proprietary quality |
| text-embedding-3-small | OpenAI | 1536 | $0.02 | Best cost/quality |
| embed-english-v3 | Cohere | 1024 | $0.10 | English-only, fast |
| voyage-3 | Voyage AI | 1024 | $0.12 | Code & mixed content |
| bge-large-en-v1.5 | Open-source | 1024 | Free (self-host) | Best free option |
| E5-Mistral-7B | Open-source | 4096 | Free (self-host) | Best quality self-host |
For most projects, start with text-embedding-3-small — it's 7× cheaper than the large version and 95% as good. Move to large only if you have specific quality needs. For self-hosting, BGE or E5 are excellent.
Practical Exercise
🎯 Key Takeaways
- Embeddings convert words/sentences/documents into vectors of numbers.
- Words with similar meanings end up close in vector space — this is the foundation of semantic search.
- Vector arithmetic captures meaning: king - man + woman ≈ queen.
- Cosine similarity measures vector angle — fast and effective.
- Start with text-embedding-3-small ($0.02/1M tokens); use BGE/E5 for self-hosting.
- Embeddings power RAG, semantic search, recommendation systems, and clustering.
Comments
Comments
Post a Comment