What Made DeepSeek-R1 Different
Most large language models before 2024 got better at reasoning the same way: collect huge amounts of human-written chain-of-thought data, supervise-fine-tune on it, then apply reinforcement learning from human feedback (RLHF) to polish style and safety. DeepSeek-R1 took a different route. Its research team started with DeepSeek-V3-Base, a strong but not reasoning-specialized foundation model, and asked whether reasoning behavior could emerge from reinforcement learning alone — with no supervised chain-of-thought data at all in the first pass. That model, called DeepSeek-R1-Zero, is the interesting part: it learned to produce long, self-checking reasoning traces purely from a reward signal, without ever being shown an example of what "good reasoning" looks like.
The production model, DeepSeek-R1, adds back a small amount of curated supervised data to fix the readability problems that pure RL introduced, but the core training philosophy — reward verifiable correctness, let the model discover its own reasoning strategy — is what the rest of the industry took notice of.
Group Relative Policy Optimization (GRPO)
Standard RLHF pipelines use Proximal Policy Optimization (PPO), which requires training a separate critic network (a value model roughly the same size as the policy) to estimate how good a given state is, so the advantage of an action can be computed. That critic model doubles your memory and compute footprint and is notoriously unstable to train well.
GRPO removes the critic entirely. For each prompt, the model samples a group of outputs (for example, 16 or 64 completions) instead of just one. Each output gets a reward from a verifier — for math problems, whether the final answer matches the known solution; for code, whether it passes unit tests. The advantage for each output is then computed relative to the other outputs in the same group, by normalizing against the group's mean and standard deviation:
# Simplified GRPO advantage computation
rewards = [reward_fn(prompt, o) for o in group_outputs] # e.g. len 16
mean_r = mean(rewards)
std_r = std(rewards)
advantages = [(r - mean_r) / (std_r + eps) for r in rewards]
# Policy update pushes probability mass toward
# above-group-average outputs and away from below-average onesThis is cheaper to run at scale (no critic forward/backward pass), and it's a natural fit for tasks with verifiable rewards — you don't need a learned reward model at all if you can just check whether a math answer is correct or a unit test passes.
Reward Design: Accuracy and Format
R1-Zero's reward function was deliberately simple and rule-based rather than learned, to avoid reward hacking against a neural reward model:
- Accuracy rewards: for math, checking the final boxed answer against ground truth; for code, running the generated solution against test cases in a sandbox.
- Format rewards: requiring the model to wrap its reasoning in
<think>...</think>tags before producing a final answer, which made the reasoning trace separable from the response and easier to evaluate and later distill.
No neural reward model scoring "helpfulness" or "quality" was used at this stage — the reward comes directly from checking the answer, which sidesteps a huge class of reward-hacking failure modes where a model learns to exploit quirks of a learned reward function rather than actually getting better at the task.
The Emergent Behavior: Self-Verification and the "Aha Moment"
The notable result from R1-Zero's training runs is that reasoning length grew substantially over the course of RL training without anyone telling the model to write longer, and the model spontaneously began exhibiting behaviors like re-checking its own intermediate steps, backtracking from a wrong approach, and explicitly stating things equivalent to "wait, let me reconsider this" mid-generation. None of that was hand-coded — it emerged because those behaviors happened to correlate with getting the final accuracy reward, and gradient updates reinforced them. It's a clean demonstration that reasoning strategies can be discovered rather than imitated, given the right reward signal and enough rollouts.
Why R1-Zero Alone Wasn't Shippable
Pure RL from a base model has a real downside: R1-Zero's outputs suffered from poor readability and language mixing, where the model would blend languages mid-reasoning-trace or produce chains of thought that were hard for a human to follow even when the final answer was correct. Nothing in the reward function penalized readability, so nothing selected for it.
DeepSeek-R1's actual training pipeline fixes this with a multi-stage process:
- Cold-start SFT: a small set of curated, human-readable long chain-of-thought examples fine-tunes the base model first, giving it a reasonable starting policy before RL begins.
- Reasoning-oriented RL: GRPO training similar to R1-Zero, now with an added language-consistency reward term to keep outputs in a single coherent language.
- Rejection sampling and SFT round two: the RL checkpoint generates many samples, the best ones (by correctness and readability) are filtered and combined with general-purpose data (writing, factual QA, self-cognition), then used for a second supervised fine-tuning pass.
- Final RL for helpfulness and harmlessness: a last RL stage tunes for general alignment across a broader distribution of prompts, closer to a conventional RLHF pass.
Distillation to Smaller Dense Models
Perhaps the most practically useful outcome of the R1 project is that the reasoning ability of the large model was distilled into much smaller dense models — Qwen and Llama-architecture checkpoints ranging from 1.5B to 70B parameters — simply by fine-tuning them on R1's generated reasoning traces. The published results showed these distilled models substantially outperforming what you get from applying RL directly to a small model from scratch, which suggests that the reasoning patterns discovered by RL at large scale transfer more efficiently through distillation than by re-discovering them in a smaller model's own RL run.
Practically, this is why you can run a "DeepSeek-R1-distill" model on a single consumer GPU through tools like Ollama or vLLM and still see meaningfully better step-by-step reasoning than the base model it was distilled from — you're getting a compressed copy of reasoning behavior that took a much larger training run to originally discover.
The Bigger Picture
The architectural takeaway isn't "GRPO is better than PPO" in some abstract sense — it's that verifiable-reward RL, applied at scale with a critic-free algorithm, can bootstrap reasoning capability without depending on the ceiling of human-annotated demonstrations. That shifted a real portion of the field's post-training research effort toward reward design and verifier construction for domains beyond math and code, and toward distillation as a first-class way to get reasoning capability into smaller, cheaper-to-serve models rather than treating it as something only frontier-scale models can do.
Discussion & Insights