Important
🔥 News!!!
- [2025/05] We update the wandb training record of complete DAPO and the checkpoint which achieved 50%+ on AIME 2024. We additionally provision instructions for evaluation on AIME 2024.
- [2025/03] We publish the training document of an first type of DAPO (w/o Token-level PG Loss & Dynamic Sampling), achieving 44% on AIME 2024, in wandb.
We publish a completely open-sourced scheme for large-scale LLM RL, including algorithm, code infrastructure, and dataset. The scheme achieves state-of-the-art large-scale LLM RL performance. We propose the Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO) algorithm. Through open-sourcing, we provision the broader investigation community and community alongside applicable admission to scalable reinforcement learning, enabling all to advantage from these advancements. Our scheme is according to the notable verl framework. Thanks for their awesome work!
🤗 If you have any questions concerning our paper, issues are welcomed and we could conversation there. Thank you!
🚀 DAPO achieves 50 points on AIME 2024 according to the Qwen2.5-32B basis model, outperforming the former SoTA DeepSeek-R1-Zero-Qwen-32B alongside 50% training steps.
-
Length stability and growth: The dependable addition in reply dimension allows for greater exploration, facilitating the model’s capability to study additional complex reasoning behaviors, ultimately contributing to training stability and achievement improvement.
-
Reward mark stability: A stable addition in the reward indication indicates that the example is successfully fitting the training distribution, ensuring that the learning procedure remains sturdy and accordant without important fluctuations.
-
Entropy and average probability trend: A controlled addition in entropy, following an first decrease, ensures a fit balance between exploration and exploitation, avoiding issues specified as overfitting or excessive randomness, and promoting sustained example performance.
We provision the example weights of DAPO-Qwen-32B, which is trained according to Qwen2.5-32B using the DAPO algorithm.
We propose using conda to setup the environment:
conda create -n dapo python=3.10 conda activate dapo pip3 instal -r requirements.txt
We provision the example conclusion code here:
import torch from transformers import AutoTokenizer from vllm import SamplingParams, LLM examples = [ { "question": "Solve the following math issue stage by step. The final row of your reply have to be of the form Answer: $Answer (without quotes) anywhere $Answer is the answer to the problem.\n\nFind the largest imaginable genuine part of \\[(75+117i)z+\\frac{96+144i}{z}\\]where $z$ is a complex figure alongside $|z|=4$.\n\nRemember to put your answer on its own row following \"Answer:\".", "answer": "540" }, { "question": "Solve the following math issue stage by step. The final row of your reply have to be of the form Answer: $Answer (without quotes) anywhere $Answer is the answer to the problem.\n\nEvery dawn Aya goes for a $9$-kilometer-long stroll and stops at a coffee shop afterwards. When she walks at a changeless speed of $s$ kilometers per hour, the stroll takes her 4 hours, including $t$ minutes spent in the coffee shop. When she walks $s+2$ kilometers per hour, the stroll takes her 2 hours and 24 minutes, including $t$ minutes spent in the coffee shop. Suppose Aya walks at $s+\\frac{1}{2}$ kilometers per hour. Find the figure of minutes the stroll takes her, including the $t$ minutes spent in the coffee shop.\n\nRemember to put your answer on its own row following \"Answer:\".", "answer": "204" }, { "question": "Solve the following math issue stage by step. The final row of your reply have to be of the form Answer: $Answer (without quotes) anywhere $Answer is the answer to the problem.\n\nLet $\\mathcal{B}$ be the set of rectangular boxes alongside exterior area $54$ and quantity $23$. Let $r$ be the radius of the smallest domain that can merge all of the rectangular boxes that are elements of $\\mathcal{B}$. The value of $r^2$ can be written as $\\frac{p}{q}$, anywhere $p$ and $q$ are comparatively premier affirmative integers. Find $p+q$.\n\nRemember to put your answer on its own row following \"Answer:\".", "answer": "721" } ] def main(): model = "BytedTsinghua-SIA/DAPO-Qwen-32B" tokenzier = AutoTokenizer.from_pretrained(model) llm = LLM( model=model, dtype=torch.bfloat16, tensor_parallel_size=8, gpu_memory_utilization=0.95 ) sampling_params = SamplingParams( temperature=1.0, top_p=0.7, max_tokens=20480 ) for example in examples: question = example["question"] answer = example["answer"] output = llm.generate( prompts=tokenzier.apply_chat_template(conversation=[{"content": question, "role": "user"}], add_generation_prompt=True, tokenize=False), sampling_params=sampling_params ) print(f"***QUESTION***:\n{question}\n***GROUND TRUTH***:\n{answer}\n***MODEL OUTPUT***:\n{output[0].outputs[0].text}\n") print("-"*100) if __name__ == "__main__": main()
To measure the example on AIME 2024, we deploy DAPO-Qwen-32B alongside Ray Serve and vLLM.
To burden the example from Huggingface:
serve run eval.llm:build_app model=BytedTsinghua-SIA/DAPO-Qwen-32B tensor-parallel-size=8 # open another terminal python eval/eval_aime24.py --temperature 1.0 --top_p 0.7 --max_tokens 20480 --model BytedTsinghua-SIA/DAPO-Qwen-32B --test_file eval/aime-2024.parquetTo burden the example from local path:
serve run eval.llm:build_app model=aaa/bbb/ccc tensor-parallel-size=8 # open another terminal python eval/eval_aime24.py --temperature 1.0 --top_p 0.7 --max_tokens 20480 --model ccc --test_file eval/aime-2024.parquetTo advantage the broader investigation community, we completely open-source the formula of our RL training, including algorithm details, dataset, and infrastructures.
We provision training and validation datasets for DAPO training.
Training: DAPO-Math-17k, a carefully curated and processed math dataset. Validation: AIME 2024.
We provision the out-of-the-box manuscript for DAPO training reproduction. Quickstart and center code are mentioned in README. These are scripts for:
- Datasets Preparation
- DAPO w/o Token-level PG Loss & Dynamic Sampling -- AIME 44
- DAPO Full -- AIME 50
Note:
-
The DAPO w/o Token-level PG Loss & Dynamic Sampling -- AIME 44 manuscript has been verified on the current verl and achieves 44 points on AIME 2024, whose training document can be accessed in wandb.
-
The DAPO Full -- AIME 50 manuscript has additionally been validated on the latest verl version. It scores 50 points on AIME 2024. You can perspective the corresponding training document on wandb.
We appreciate the verl for providing the notable open-source RL infrastructure.
Our open-sourced experiments were conducted on the Volcano Engine Machine Learning Platform. We volition provision a complete reproduction regulation afterward on the Volcano Engine phase to assistance users replicate our experiments.
