100 Days After DeepSeek‑R1: Surveying Replication Studies & Future Paths for Reasoning Language Models
1. Introduction
The release of DeepSeek‑R1 marked a pivotal moment in LLM evolution—ushering in purposeful chain-of-thought (CoT) and advanced reasoning behaviors. However, DeepSeek’s lack of transparency around training data and full reproduction of its multi-stage pipeline spurred a wave of replication studies. This survey synthesizes recent open-source initiatives that emulate DeepSeek‑R1’s two pillars—supervised fine-tuning (SFT) and reinforcement learning from verifiable rewards (RLVR)—to distill actionable insights for researchers and practitioners .
2. Supervised Fine-Tuning for Reasoning
2.1. Datasets & Curation
Replication teams have curated reasoning-oriented SFT datasets that mimic DeepSeek‑R1’s "cold-start" schema:
OpenThoughts-114k and OpenR1-Math-220k: Synthesized from reasoning traces or math-specific prompts.
Light-R1-SFT and SYNTHETIC-1: Multi-domain conversation and reasoning transcripts.
Stratos-17k, s1K-1.1, and LIMO: Smaller, focused reasoning sets tailored for distilled models .
Key principles in dataset design:
Length distribution and domain diversity mirror targeted benchmarks.
Data decontamination and cross-referencing sources are crucial to avoid data leakage .
2.2 Training Strategies & Results
Comparison of approaches reveals:
Smaller base models (7–32B) fine-tuned on curated SFT datasets reach 70–90% of DeepSeek‑R1’s zero-shot reasoning.
Multi-domain SFT gives superior generalization compared to math-only tuning.
Tiered SFT recipes—starting with broad general examples followed by domain-specific reasoning—enhance performance .
These studies collectively found that a robust SFT stage is essential to bootstrap credible CoT behaviors prior to any RL refinement .
3. Reinforcement Learning from Verifiable Rewards (RLVR)
3.1 RL Dataset Construction
Open-source RLVR datasets include:
Open-Reasoner-Zero, DeepScaleR-Preview, Skywork-OR1, OWM, LIMR, DAPO: Propose structured mix of math, code and logic challenges .
Their design aligns with DeepSeek’s original RL training: prioritizing verifiable rewards—correct boxed answers or passing code tests.
3.2 RL Algorithm Variants
Investigators applied and compared:
PPO, GRPO (Group Relative Policy Optimization), and DrPO.
Rewards combining correctness, format (e.g., self-verification), and language consistency.
Sampling via rejection or ranking to enhance policy robustness.
Insights:
GRPO simplifies reward modeling by ranking outputs rather than absolute scoring.
Verifiable, process-based reward signals (correctness + CoT tags) outperform purely preference-based tuning .
RL is more stable on LLMs, converging in fewer steps—all due to strong pretrained representations .
4. Comparative Analysis of SFT and RLVR Pipelines
| Stage | Replication Findings |
|---|---|
| SFT (Cold-start) | Mixed-domain, structured reasoning data is vital. Two-phase curriculum SFT stabilizes RL downstream . |
| RL (GRPO/PPO) | Group-relative optimization plus format rewards replicate DeepSeek-R1 dynamics; PPO is viable baseline . |
| Reward Design | Crowdsourced/verifiable rewards on solution correctness and CoT formatting outperform human preference tuning alone . |
| Model Sizes | Distilled 7–32B models trained via SFT+RL can approach DeepSeek‑R1‑level performance . |
| Context Length | Longer contexts yield better CoT outputs; replication includes matching 128K context setups . |
| Generalization | Curriculum SFT and RLVR confer out-of-distribution benefits; SFT excels in in-domain, RL aids broader logic tasks . |
5. Alternative Reasoning-Boosting Techniques
Beyond SFT+RL, replications are exploring:
Direct Preference Optimization (DPO): Enhances CoT responses via ranked data fine-tuning .
Curriculum Learning: Gradually increasing CoT complexity to stabilize learning.
Distillation: Using large models to generate expert reasoning traces for smaller students .
Tool-Augmented RL: Integrating external calculators, code execution checks, or knowledge-grounded retrieval during RL .
6. Challenges & Open Questions
6.1 Data Sharing & Transparency
Most replication efforts emphasize community access to reasoning datasets and training logs—filling gaps in DeepSeek’s black-box pipeline .
6.2 Stability & Reward Sensitivity
GRPO appears stable across reward designs, but edge cases (e.g., hallucinations) still occur; sampling diversity matters .
6.3 Model Size vs. Performance
Smaller models show promise, yet performance drops in complex reasoning and code execution; scaling up remains valuable .
6.4 Safety Implications
RL-only pipelines can miss malicious content; safety-oriented SFT or reward signals remain essential .
7. Future Directions
Open-Pipeline Initiatives: Projects like Open‑R1 (HuggingFace) aim to fully replicate DeepSeek’s pipeline, sharing datasets, code, and models .
Multimodal & Multilingual Expansion: Extending techniques to reasoning in images, code, and non-English languages .
Tool-Augmented Reasoning: Integrating interpreters or solvers directly into RLVR.
Fine-Grained Reward Schemas: Rewarding meta-skills such as interpretability, verification, and bias control.
Framework Standardization: Introducing benchmarks, checklists, and open-source pipelines to normalize RLM reproduction workflows.
8. Conclusion
The proliferation of replication studies within 100 days of DeepSeek‑R1 underscores a turning point in AI research: accountability via open access and reproducibility. Supervised fine-tuning and reinforcement learning—particularly within verifiable, curriculum-aligned pipelines—emerge as foundational elements in crafting robust reasoning models. As the community coalesces around Open‑R1 initiatives and innovations like DPO, curriculum SFT, and tool-assisted RL, the roadmap for future RLM research is both rich and promising.
If you'd like visual pipeline diagrams, code examples from HuggingFace, or a comparison of RL algorithms (PPO vs. GRPO), I’d be happy to provide them!
