What You Will Learn
INTERMEDIATE
- The 3-stage training pipeline that every modern LLM goes through
- Why pre-training is the most expensive step (and why it can't be done at home)
- How RLHF turned GPT-3 into ChatGPT — the alignment trick that changed everything
The 3-Stage Training Pipeline
Every modern LLM you use — GPT-5, Claude 4, Llama 4, Gemini 2.5 — was created using essentially the same three-stage pipeline. Understanding this pipeline is what separates "LLM users" from "LLM engineers". The three stages are:
- Pre-training: Train on trillions of tokens of raw text. The model learns to predict the next token. This is where the model acquires "world knowledge" and basic language fluency. Costs $1M-$100M+. Can't be done at home.
- Supervised Fine-Tuning (SFT): Train on a smaller dataset (10K-1M examples) of high-quality instruction-response pairs. The model learns to follow instructions and converse. Costs $100-$10K. Can be done at home with LoRA.
- Alignment (RLHF or DPO): Train the model to prefer helpful, safe, honest responses. Uses human preference data. This is what made ChatGPT feel "smart" instead of just "fluent." Costs $1K-$100K.
Stage 1: Pre-Training — Predicting the Next Token
Pre-training is brutally simple: download a huge corpus of text (Common Crawl, Wikipedia, books, code, papers), feed it to the model in chunks, and ask "given these 4,096 tokens, what's the next one?" The model's prediction is compared to the actual next token, the error (loss) is backpropagated through the network, and the weights update. Repeat a trillion times.
This sounds dumb, but it works because to predict the next token accurately, the model is forced to learn grammar, facts, reasoning patterns, code conventions, and a staggering amount of world knowledge. By the end of pre-training, GPT-3 could write coherent essays, translate languages, and do basic arithmetic — all from "predict the next token" as the only training signal.
The catch: pre-training is enormously expensive. GPT-3 cost an estimated $4.6M to pre-train. GPT-4 likely cost $50-100M. Llama 3 405B cost ~$650M. This is why only a handful of organizations on Earth can pre-train frontier models. The compute required is staggering: thousands of GPUs running for months.
Stage 2: Supervised Fine-Tuning (SFT)
After pre-training, the model can predict the next token brilliantly — but it doesn't know how to have a conversation. If you prompt a base (pre-trained-only) model with "What's the capital of France?", it might respond with "What's the capital of Germany?" — because in its training data, a question like that is often followed by another question (e.g., in a quiz). It doesn't know you want an answer.
SFT fixes this. You collect a dataset of 10K-1M high-quality examples where each example is an instruction-response pair: {"instruction": "What's the capital of France?", "response": "Paris."}. You fine-tune the model on this dataset using the same next-token-prediction loss, but now the model learns that instructions should be followed with helpful responses.
# SFT data format (JSONL — one example per line)
{"messages": [
{"role": "user", "content": "Write a Python function to reverse a string."},
{"role": "assistant", "content": "def reverse(s):\n return s[::-1]"}
]}
{"messages": [
{"role": "user", "content": "What is 25 * 17?"},
{"role": "assistant", "content": "425."}
]}
The quality of your SFT data matters more than the quantity. OpenAI's GPT-3.5 was fine-tuned on roughly 12K-15K examples — but each example was hand-written and reviewed. A common rule of thumb: 1,000 excellent examples beats 100,000 mediocre ones. We'll cover SFT in depth in Lesson 61.
Stage 3: RLHF — The Trick That Made ChatGPT
SFT alone produces a model that follows instructions — but it's not aligned with what humans actually prefer. It might give correct but unhelpful answers, long-winded responses, or unsafe advice. RLHF (Reinforcement Learning from Human Feedback) fixes this by training the model to prefer responses that humans actually rate highly.
The RLHF process has three steps:
- Collect preferences: Generate 2+ responses to the same prompt. Have human raters choose which is better. Repeat for 10K-100K comparisons.
- Train a reward model: A smaller model learns to predict which response humans would prefer. This becomes your "goodness score" function.
- Optimize the LLM: Use reinforcement learning (PPO algorithm) to update the LLM's weights so it produces responses that score high on the reward model.
DPO: The Modern Alternative
RLHF works but is fiddly — PPO is unstable, the reward model can be hacked, and the whole pipeline is complex. In 2023, researchers at Stanford published DPO (Direct Preference Optimization), which simplifies the whole thing. Instead of training a separate reward model and running RL, DPO directly optimizes the LLM on preference pairs (chosen vs rejected). The math is cleaner, the training is more stable, and you don't need RL expertise.
By 2025, most open-source fine-tunes (Llama 3.1 Instruct, Mistral Instruct, etc.) used DPO instead of RLHF. The proprietary labs (OpenAI, Anthropic) still use RLHF variants because at their scale the extra engineering pays off. We'll cover DPO in depth in Lesson 62.
Why This Matters for LLM Engineers
You might be thinking: "I'm never going to pre-train a model. Why does this matter?" Because understanding the training pipeline tells you why models behave the way they do:
- LLMs hallucinate because pre-training taught them to produce plausible text, not retrieve true facts.
- LLMs are sycophantic (agree with the user) because RLHF rewarded "helpful" responses, and agreeing feels helpful.
- LLMs have a knowledge cutoff because pre-training ended on a specific date. New facts require fine-tuning or RAG.
- LLMs are bad at math because pre-training data contained mostly natural language, not enough structured math.
- LLMs can do few-shot learning because pre-training on diverse tasks taught them to generalize from examples.
Once you understand the training pipeline, every weird LLM behavior becomes predictable — and fixable.
🎯 Key Takeaways
- Every modern LLM is trained in 3 stages: pre-training, SFT, alignment (RLHF/DPO).
- Pre-training is the expensive leap ($10M+) — predicts next token on trillions of words.
- SFT teaches the model to follow instructions using ~10K-1M high-quality examples.
- RLHF/DPO aligns the model with human preferences — what made ChatGPT feel 'smart'.
- DPO (2023) is the modern, simpler alternative to RLHF — most open models use it now.
- Understanding training tells you WHY models hallucinate, are sycophantic, and have knowledge cutoffs.
Comments
Comments
Post a Comment