Post-Training Evolution in V4: OPD Replaces Mixed RL, Distilling Multiple Expert Models into One

According to monitoring by Beating, the post-training methodology of DeepSeek V4 has undergone significant changes: the mixed RL phase of V3.2 has been completely replaced by On-Policy Distillation (OPD). The new process consists of two steps. In the first step, domain expert models are trained in areas such as mathematics, coding, agent behavior, and instruction following, based on the V3.2 pipeline. Each expert undergoes fine-tuning followed by reinforcement learning using GRPO. In the second step, a multi-teacher OPD distills the capabilities of over ten experts into a unified model: the student performs reverse KL divergence logit distillation on the full vocabulary for each teacher based on its own generated trajectories, aligning logits to merge multiple expert weights into a unified parameter space, thus avoiding the capability conflicts commonly seen in traditional weight merging and mixed RL. The report also introduces the Generative Reward Model (GRM): for tasks that are difficult to validate with rules, instead of training a traditional scalar reward model, RL data guided by rubrics is used to train the GRM, allowing the actor network to simultaneously generate and evaluate, enabling generalization to complex tasks with a small amount of diverse human annotations.