Keyboard Shortcuts N Next post
P Previous post
S Save / unsave
R Read aloud
T Toggle theme
/ Focus search
Esc Close panels
🔥
Ready to read...
LLM Engineering LLM Engineering: From Zero to Production Module 1 - LLM Foundations

How LLMs Are Trained: Pre-training, Fine-tuning, RLHF Explained

Reviewed & accurate
AI Summary

What You Will Learn

INTERMEDIATE

  • The 3-stage training pipeline that every modern LLM goes through
  • Why pre-training is the most expensive step (and why it can't be done at home)
  • How RLHF turned GPT-3 into ChatGPT — the alignment trick that changed everything

The 3-Stage Training Pipeline

Every modern LLM you use — GPT-5, Claude 4, Llama 4, Gemini 2.5 — was created using essentially the same three-stage pipeline. Understanding this pipeline is what separates "LLM users" from "LLM engineers". The three stages are:

  1. Pre-training: Train on trillions of tokens of raw text. The model learns to predict the next token. This is where the model acquires "world knowledge" and basic language fluency. Costs $1M-$100M+. Can't be done at home.
  2. Supervised Fine-Tuning (SFT): Train on a smaller dataset (10K-1M examples) of high-quality instruction-response pairs. The model learns to follow instructions and converse. Costs $100-$10K. Can be done at home with LoRA.
  3. Alignment (RLHF or DPO): Train the model to prefer helpful, safe, honest responses. Uses human preference data. This is what made ChatGPT feel "smart" instead of just "fluent." Costs $1K-$100K.
The 3-Stage LLM Training Pipeline Stage 1: Pre-trainingPredict next token on\ntrillions of words Stage 2: SFTLearn to follow\ninstructions Stage 3: RLHF/DPOAlign with human\npreferences Cost: $50M → $5K → $50K | Can be done at home: No → Yes → Yes
Pre-training is the giant leap. SFT + RLHF are what make the model usable as a chatbot.

Stage 1: Pre-Training — Predicting the Next Token

Pre-training is brutally simple: download a huge corpus of text (Common Crawl, Wikipedia, books, code, papers), feed it to the model in chunks, and ask "given these 4,096 tokens, what's the next one?" The model's prediction is compared to the actual next token, the error (loss) is backpropagated through the network, and the weights update. Repeat a trillion times.

This sounds dumb, but it works because to predict the next token accurately, the model is forced to learn grammar, facts, reasoning patterns, code conventions, and a staggering amount of world knowledge. By the end of pre-training, GPT-3 could write coherent essays, translate languages, and do basic arithmetic — all from "predict the next token" as the only training signal.

The catch: pre-training is enormously expensive. GPT-3 cost an estimated $4.6M to pre-train. GPT-4 likely cost $50-100M. Llama 3 405B cost ~$650M. This is why only a handful of organizations on Earth can pre-train frontier models. The compute required is staggering: thousands of GPUs running for months.

Why You Can't Pre-Train at Home
Pre-training Llama 3 70B required 6,000 H100 GPUs running for ~50 days. That's roughly 7.2 million GPU-hours. At $2/hour for H100s on Lambda Labs, that's $14.4M. And that's just for one model size. Plus the data pipeline, the eval harness, the distributed training infrastructure — pre-training is a sport for trillion-dollar companies.

Stage 2: Supervised Fine-Tuning (SFT)

After pre-training, the model can predict the next token brilliantly — but it doesn't know how to have a conversation. If you prompt a base (pre-trained-only) model with "What's the capital of France?", it might respond with "What's the capital of Germany?" — because in its training data, a question like that is often followed by another question (e.g., in a quiz). It doesn't know you want an answer.

SFT fixes this. You collect a dataset of 10K-1M high-quality examples where each example is an instruction-response pair: {"instruction": "What's the capital of France?", "response": "Paris."}. You fine-tune the model on this dataset using the same next-token-prediction loss, but now the model learns that instructions should be followed with helpful responses.

sft_dataset.jsonljson
# SFT data format (JSONL — one example per line)
{"messages": [
  {"role": "user", "content": "Write a Python function to reverse a string."},
  {"role": "assistant", "content": "def reverse(s):\n    return s[::-1]"}
]}
{"messages": [
  {"role": "user", "content": "What is 25 * 17?"},
  {"role": "assistant", "content": "425."}
]}

The quality of your SFT data matters more than the quantity. OpenAI's GPT-3.5 was fine-tuned on roughly 12K-15K examples — but each example was hand-written and reviewed. A common rule of thumb: 1,000 excellent examples beats 100,000 mediocre ones. We'll cover SFT in depth in Lesson 61.

Stage 3: RLHF — The Trick That Made ChatGPT

SFT alone produces a model that follows instructions — but it's not aligned with what humans actually prefer. It might give correct but unhelpful answers, long-winded responses, or unsafe advice. RLHF (Reinforcement Learning from Human Feedback) fixes this by training the model to prefer responses that humans actually rate highly.

The RLHF process has three steps:

  1. Collect preferences: Generate 2+ responses to the same prompt. Have human raters choose which is better. Repeat for 10K-100K comparisons.
  2. Train a reward model: A smaller model learns to predict which response humans would prefer. This becomes your "goodness score" function.
  3. Optimize the LLM: Use reinforcement learning (PPO algorithm) to update the LLM's weights so it produces responses that score high on the reward model.
RLHF in 3 Steps 1. Collect\nPreferencesHuman rates 2\nresponses (A > B) 2. Train\nReward ModelLearns to predict\nhuman preference 3. PPO\nOptimizationLLM tuned to\nmaximize reward Result: model that produces helpful, safe, honest responses
RLHF is what makes ChatGPT feel 'smart' instead of just 'fluent'.

DPO: The Modern Alternative

RLHF works but is fiddly — PPO is unstable, the reward model can be hacked, and the whole pipeline is complex. In 2023, researchers at Stanford published DPO (Direct Preference Optimization), which simplifies the whole thing. Instead of training a separate reward model and running RL, DPO directly optimizes the LLM on preference pairs (chosen vs rejected). The math is cleaner, the training is more stable, and you don't need RL expertise.

By 2025, most open-source fine-tunes (Llama 3.1 Instruct, Mistral Instruct, etc.) used DPO instead of RLHF. The proprietary labs (OpenAI, Anthropic) still use RLHF variants because at their scale the extra engineering pays off. We'll cover DPO in depth in Lesson 62.

Why This Matters for LLM Engineers

You might be thinking: "I'm never going to pre-train a model. Why does this matter?" Because understanding the training pipeline tells you why models behave the way they do:

  • LLMs hallucinate because pre-training taught them to produce plausible text, not retrieve true facts.
  • LLMs are sycophantic (agree with the user) because RLHF rewarded "helpful" responses, and agreeing feels helpful.
  • LLMs have a knowledge cutoff because pre-training ended on a specific date. New facts require fine-tuning or RAG.
  • LLMs are bad at math because pre-training data contained mostly natural language, not enough structured math.
  • LLMs can do few-shot learning because pre-training on diverse tasks taught them to generalize from examples.

Once you understand the training pipeline, every weird LLM behavior becomes predictable — and fixable.

🎯 Key Takeaways

  • Every modern LLM is trained in 3 stages: pre-training, SFT, alignment (RLHF/DPO).
  • Pre-training is the expensive leap ($10M+) — predicts next token on trillions of words.
  • SFT teaches the model to follow instructions using ~10K-1M high-quality examples.
  • RLHF/DPO aligns the model with human preferences — what made ChatGPT feel 'smart'.
  • DPO (2023) is the modern, simpler alternative to RLHF — most open models use it now.
  • Understanding training tells you WHY models hallucinate, are sycophantic, and have knowledge cutoffs.
Previously: Previously: Lesson 04 covered self-attention.
Today: Today: you learned the 3-stage training pipeline — pre-training, SFT, RLHF/DPO.
Next: Next: Lesson 06 explains why LLMs make things up — and why no amount of training will ever fully fix it.
Test Your Knowledge
How did you find this?

Comments

Join the discussion! Sign in with your Google or Blogger account, or comment as Anonymous - no account needed. For quick questions, also reach me on Telegram @cytestch.

Comments