[English Tutorial] / [中文教程]

• 2025.07: 🎉🎉🎉 We have updated the number of reviewed papers to over 1000. Additionally, we have added bilingual supports and updated our repository more friendly for Long-CoT beginner.
• 2025.04: 🎉🎉🎉 We have updated the number of reviewed papers to over 900. Additionally, we have enhanced the presentation with more engaging teaser figure.
• 2025.03: 🎉🎉🎉 We have published a survey paper titled "Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models". Please feel free to cite or open pull requests for your awesome studies.

Welcome to the repository associated with our survey paper, "Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models". This repository contains resources and updates related to our ongoing Long CoT research. For a detailed introduction, please refer to our survey paper.

Recent advancements in reasoning with large language models (RLLMs), such as OpenAI-O1 and DeepSeek-R1, have demonstrated their impressive capabilities in complex domains like mathematics and coding. A central factor in their success lies in the application of long chain-of-thought (Long CoT) characteristics, which enhance reasoning abilities and enable the solution of intricate problems.

However, despite these developments, a comprehensive survey on Long CoT is still lacking, limiting our understanding of its distinctions from traditional short chain-of-thought (Short CoT) and complicating ongoing debates on issues like "overthinking" and "test-time scaling." This survey seeks to fill this gap by offering a unified perspective on Long CoT.

1. We first distinguish Long CoT from Short CoT and introduce a novel taxonomy to categorize current reasoning paradigms.
2. Next, we explore the key characteristics of Long CoT: deep reasoning, extensive exploration, and feasible reflection, which enable models to handle more complex tasks and produce more efficient, coherent outcomes compared to the shallower Short CoT.
3. We then investigate key phenomena such as the emergence of Long CoT with these characteristics, including overthinking, and test-time scaling, offering insights into how these processes manifest in practice.
4. Finally, we identify significant research gaps and highlight promising future directions, including the integration of multi-modal reasoning, efficiency improvements, and enhanced knowledge frameworks.

By providing a structured overview, this survey aims to inspire future research and further the development of logical reasoning in artificial intelligence.

We aim to help newcomers quickly establish domain knowledge, so our design concept is as follows: briefly introduce the main technologies involved in reasoning large models and Long CoT, allowing everyone to understand which problems different technologies can address, so that when they wish to delve deeper into the field in the future, they will have a clear starting point.

We are a team of beginners in reasoning large models, and we hope that through our own learning experiences, we can offer some assistance to future learners, accelerating the popularization and application of reasoning large models. We welcome more friends to join our project, and we are also open to friendship and academic collaboration. For any inquiries, please feel free to contact us via email at charleschen2333@gmail.com.

Daily Knowledge Resources

• Social Media:
  • Recommended WeChat Public Accounts: JIQIZHIXIN, Paper Weekly, MLNLP...
  • Recommended Twitter Accounts: AK, elvis, Philipp Schmid, ...
• Cutting-edge Courses: CS336
• Community Sharing: MLNLP, JIQIZHIXIN, BAAI, NICE Academic

• OpenAI-o1 / o3 / o4: The earliest reasoning large language models exploring Long CoT, developed by OpenAI’s first-tier models.
• Gemini: First-tier reasoning large language models developed by Google.
• Deepseek-r1: The first open-source reasoning large language model with Long CoT.
• QwQ: The first open-source large-scale reasoning large language model with Long CoT.
• Qwen3: The most commonly used open-source reasoning large language models for Long CoT developed by Alibaba.
• Seed-Thinking-v1.5: ByteDance’s open-source reasoning model for Long CoT.
• Kimi-k1.5: The earliest multimodal reasoning model for Long CoT developed by Moonshot.
• MiniMax-m1: The open-source reasoning model for Long CoT developed by MiniMax.

In this chapter, we will provide the most representative technologies for each capability, along with the latest developments. A detailed list of papers can be found in the complete list.

The core of deep reasoning ability lies in the need for sufficient logical depth to manage a large number of reasoning nodes. Without this capability, the performance of reasoning large language models (RLLMs) significantly degrades. Current methods for enhancing deep reasoning can be categorized into two main approaches: Deep Reasoning Format and Deep Reasoning Learning.

Since reasoning models heavily depend on the format of reasoning, they tend to achieve the deepest reasoning paths in the forms they excel at. As a result, some works have begun exploring better reasoning formats for deeper reasoning.

Natural Language Deep Reasoning

• Core Idea: Aims to express deep reasoning through natural language formats.
• Representative Works:
  • Natural Program: ensures more structured and rigorous logical analysis.
  • Code I/O: restructures code-based reasoning patterns into natural language forms, further unleashing the reasoning potential of RLLMs.

Structured Language Deep Reasoning

• Core Idea: Aims to enhance deep reasoning through programmatic or symbolic language formats. Current research primarily focuses on using code to improve mathematical reasoning capabilities.
• Representative Works:
  • Program-of-Thought: enables models to think using code language, thereby enhancing their reasoning capabilities.
  • DeepSeek-Prover: converts natural language questions into formal statements, filters out low-quality statements, and generates proofs to create synthetic data, enhancing LLM’s theorem proving ability.
  • RBF: demonstrates why structured language is more effective than natural language in scenarios that require strong planning.

Latent Space Deep Reasoning

• Core Idea: Enhances LLM reasoning ability through continuous latent space operations.
• Representative Works:
  1. Token-driven: Early studies introduced implicit "planning tokens" or "thinking tokens" to guide the reasoning process in latent space.
     • Coconut (Chain of Continuous Thought): further expands this method by maintaining multiple parallel reasoning paths, enhancing complexity while ensuring efficiency.
     • Heima: performs efficient reasoning through latent hidden spaces, innovatively compressing the entire Long CoT process into a single token, resulting in significant computational resource savings.
  2. Vector-driven: Inserts an additional vector to guide the reasoning process in latent space.
     • LTMs: innovatively abstracts each layer of the LLM into "thinking blocks" and introduces the concept of a "thinking vector" for each layer. Through iterative deep computation in latent space, the model dynamically scales the computational load during testing.
  3. Manager-driven: Proposes a continuous manager mechanism to manage the latent space states.
     • Recurrent Block: uses iterative control over trained "recurrent blocks" as recursive "thinking blocks" to integrate deeper model layers during reasoning, enhancing performance without the need for specialized training data.
     • Implicit Thought Transformer (ITT): leverages raw Transformer layers as recursive "thinking blocks," using adaptive token routing to select key tokens and residual thinking connections to control reasoning depth, thereby achieving efficient processing of key tokens.
• Relevant Repositories:
  • Awesome-Latent-CoT: provides an overview of various thought chain representations in latent space, capturing complex non-linguistic thoughts that cannot be expressed by language alone.

The deficiency of deep reasoning abilities in RLLMs can significantly reduce model performance. As a result, the academic focus has shifted towards enhancing reasoning capabilities through training. Supervised fine-tuning (SFT), as a memory process, can stabilize model output, while reinforcement learning (RL) facilitates generalization and self-learning.

Deep Reasoning Imitation

• Core Idea: By imitating advanced reasoning systems, deep reasoning in RLLMs can be effectively achieved, enabling models to learn complex reasoning patterns and generalize across tasks.
• Representative Works:
  1. Imitation from Human
     • GSM8K/GPT-Verifier: Introduces early imitative learning based on human-annotated deep reasoning samples.
     • ALT: Enhances deep reasoning in RLLMs by generating a large-scale dataset of human-annotated logical templates.
  2. Imitation from Advanced RLLMs
     • AceMath: Uses few-shot prompting to distill Long CoT samples from advanced LLMs, improving performance through multi-stage quality-guided SFT.
     • DART-Math: Effectively distills difficulty-dependent deep reasoning samples through rejection sampling in the synthesis stage.
     • OpenThoughts / OpenCodeReasoning / NaturalThoughts: Extends this paradigm to mathematics, code, and general scenarios.
  3. Imitation from Scaling-augmented RLLMs
     • Bansal et al.: Find that expanding the sampling scale and length improves data quality.
     • Qwen-Math / PromptCoT: Further combine large-scale sampling with reward model sample selection to generate Olympic-level difficulty deep reasoning samples.
     • FastMCTS: Utilizes Monte Carlo Tree Search (MCTS) to identify optimal deep reasoning paths.
• Latest Developments:
  • Journey P2: Knowledge distilled from advanced RLLM APIs such as o1, R1 significantly boosts small LLMs' performance, with supervised fine-tuning methods surpassing teacher models in complex mathematical reasoning tasks.
  • s1 / LIMO: A small number of high-quality samples is sufficient to activate deep reasoning abilities in base LLMs.

Deep Reasoning Self-Learning

• Core Idea: Although simple imitation yields excellent performance, current models still heavily rely on human annotations or outputs from advanced models during imitation and distillation. To break through this limitation, research focuses on self-learning techniques to achieve more advanced reasoning capabilities.
• Representative Works:
  1. Self-Learning from Direct Sampling
     • STaR: Uses in-context learning (ICL) to sample deep reasoning results and treats the correctness of the final answer as an implicit reward for self-learning.
     • Reinforced Self-Training (ReST): Proposes the "Grow-Improve" paradigm, where the self-generated reasoning process is rewarded, enhancing with offline reinforcement learning.
     • ReST$^{EM}$: Generates rewards and iteratively optimizes LLMs to achieve peak performance on validation sets, significantly improving robustness.
     • TOPS: Finds that self-learning with deep reasoning samples at an appropriate reasoning depth is the most efficient.
  2. Self-Learning from Tree Search
     • PGTS: Uses policy-guided tree search, combining reinforcement learning with structured tree exploration.
     • ReST-MCTS*: Optimizes MCTS behavior through progressive trajectory extraction and curriculum preference learning, significantly improving LLMs' reasoning ability.
• Latest Developments: Introduce an error-correction adaptive mechanism by training a verifier or using entropy to filter and optimize the reward process, thus enhancing the quality of self-learning.
  • UnCert-CoT: Dynamically schedules thought chains based on entropy-aware uncertainty, activating multi-path reasoning only under high-entropy situations, significantly improving code generation accuracy and efficiency.
  • Wang et al.: Analyzes the impact of verifiable reward-based reinforcement learning on large language models' reasoning capabilities from the token entropy perspective, where high-entropy "branching" tokens dominate adjustments to multi-path reasoning strategies. Policy gradient optimization is applied exclusively to these high-entropy tokens.
  • CoT-Valve: Dynamically adjusts to reduce reasoning path length based on task difficulty, thus reducing computational overhead.

Feedback mechanisms provide multi-granularity evaluation signals for Long CoT, ranging from Overall Feedback, which evaluates the final outcome, to Process Feedback, which supervises individual steps of the reasoning process, and Hybrid Feedback, which combines both types. These mechanisms not only support reward modeling and path optimization but also lay the foundation for subsequent self-correction, serving as a crucial bridge to move RLLMs from static generation to dynamic evaluation.

Overall Feedback

• Core Idea: Overall feedback evaluates the complete reasoning process and the final result from a global perspective, commonly used to guide large language models in improving reasoning quality during reinforcement learning or self-optimization. Feedback forms include numerical rewards, rule checks, and natural language evaluations.
• Representative Works:
  1. Outcome Reward Models(ORM): Provide numeric reward signals to optimize output quality, suitable for tasks where accuracy is hard to assess directly.
     • Gen-Verifier: introduces the first generation verification framework based on reasoning accuracy;
     • Critic-RM: combines natural language criticism and reward prediction, significantly optimizing feedback quality;
     • Self-Rewarding LMs (SRLMs): introduce consistency mechanisms, achieving self-supervised rewards without human annotation.
  2. Rule Extraction: Uses in-task rules to verify and correct answers, enhancing the stability of feedback.
     • STaR / ReST: show that rule-based feedback based on final answers outperforms ORM in mathematical tasks;
     • OpenCodeInterpreter / AceCoder: generate program-level feedback using automated test cases in coding tasks.
  3. RLLMs Feedback (LLM-as-a-Judge): The model self-critiques and evaluates in natural language, enhancing reflection and error-correction capabilities.
     • EvalPlanner: distinguishes feedback between planning and reasoning;
     • RoT: combines reverse reasoning and reflection to assist models in discovering knowledge gaps;
     • AutoRace: provides task-specific evaluation criteria to improve feedback relevance.
• Relevant Repositories:
  • RewardBenchFor system evaluation of ORM methods.

Process Feedback

• Core Idea: Process feedback evaluates each step in the reasoning chain progressively, often in conjunction with reinforcement learning or tree search, guiding the model to fine-tune without relying on human annotations. The feedback sources primarily include process reward models and language models driven by natural language.
• Representative Works:
  1. Process Feedback from Process Reward Models (PRMs): Use automatically constructed or minimally annotated data to train stepwise reward functions, the mainstream approach in Long CoT.
     • PRM800K: Pioneers using human-annotated stepwise supervision to enhance reward stability;
     • Math-Shepherd: Automatically generates stepwise feedback using tree search to enhance PRM generalization;
     • Full-Step-DPO: Rewards the entire reasoning chain, encouraging holistic optimization;
     • AdaptiveStep: Dynamically segments reasoning steps based on confidence, enabling token-level fine-grained feedback.
  2. Process Feedback from RLLMs: Leverages the model’s own generation of natural language feedback to simulate reward signals, improving the flexibility and scalability of process supervision.
     • React / Reflexion: Generate language feedback after each action, enhancing decision-making rationality;
     • Step-DPO: Introduces a self-validation mechanism to construct positive-negative contrastive samples, optimizing the training process;
     • CACE: Proposes causal impact metrics between reasoning steps to make the entire chain more interpretable;
     • ORPS: Automatically optimizes reasoning strategies with program execution feedback, reducing human reliance.
• Relevant Repositories:
  • ProcessBench: Evaluates stepwise reasoning and reward model performance;
  • PRMBench: Focuses on comparative analysis of PRM methods in mathematical tasks.

Hybrid Feedback

• Core Idea: Hybrid feedback mechanisms combine the strengths of both overall and process feedback, assessing final outputs while focusing on intermediate reasoning steps. This unified multi-granularity evaluation system enhances the overall reasoning quality and error-correction capabilities of language models.
• Representative Works:
  • Consensus Filtering: Combines Monte Carlo estimation with LLM-as-Judge to integrate overall and stepwise feedback, enhancing reasoning consistency and accuracy;
  • Step-KTO: Merges PRM and ORM binary feedback mechanisms, emphasizing reflection-driven error correction, guiding the model to form more coherent Long CoT structures.

The Refinement mechanism focuses on self-correction capabilities based on feedback information, serving as a key step in achieving closed-loop optimization in Long CoT. Through Prompt-based Refinement, spontaneous reflection is achieved; SFT-based Refinement facilitates imitation learning; and RL-based Refinement strengthens self-correction strategies. As a result, the model gradually develops the ability of "self-diagnosis—self-updating," making the reasoning chain more robust and controllable.

Prompt-based Refinement

• Core Idea: By guiding the model to generate initial responses via prompts, and allowing for self-feedback and multi-round corrections in subsequent rounds, this method improves reasoning accuracy, reduces hallucinations, and supports stronger automated reflection capabilities.
• Representative Works:
  • ReAct / Reflexion: A typical implementation of the multi-round reflection and self-correction mechanism;
  • Self-Backtracking / Refiner / BackMath: Supports the model in autonomously backtracking and modifying during the reasoning process, streamlining decision paths;
  • MCTSr / ReST-MCTS: Combines tree search with confidence updates to enable multi-round dynamic reflection;
  • LLM2/ ReARTeR: Promotes the automatic evolution and stable convergence of refinement strategies in Long CoT tasks.

SFT-based Refinement

• Core Idea: By utilizing high-quality reflection data for supervised fine-tuning, the model imitates the self-correction behaviors of more advanced models, enhancing its step-by-step error correction and reflective capabilities. This is suitable for small model capability transfer and fine-grained training.
• Representative Works:
  • rStar: Enhances small model self-improvement capabilities through self-play methods;
  • Math-Minos: Trains the model using step-by-step rationale labels for fine-grained reasoning;
  • Journey Learning: Combines MCTS backtracking to generate supervision signals;
  • MM-Verify: Expands the refinement mechanism to multimodal image-text reasoning.

RL-based Refinement

• Core Idea: Through reinforcement learning mechanisms, the model is guided to self-reflect and correct during testing or reasoning processes, emphasizing self-refinement capabilities under reward guidance, thus reducing dependence on manual supervision.
• Representative Works:
  • SCoRe: Enhances the model's self-refinement ability during testing through self-generated correction trajectories and regularization;
  • DeepSeek-R1: Uses result-level reinforcement learning to activate the model's natural feedback and "aha" moments for corrections;
  • S$^2$R: Combines process-level reinforcement learning to achieve dynamic refinement in reasoning;
  • ReVISE: Introduces an internal verifier to decide when to trigger RL-guided reflective behaviors.

Extensive Exploration enables reasoning large language models (RLLMs) to explore multiple reasoning paths more deeply and comprehensively when dealing with complex problems, thereby improving problem-solving accuracy and robustness. From the perspective of exploration types, extensive exploration techniques can be divided into three categories: Exploration Scaling, Internal Exploration, and External Exploration.

Exploration Scaling aims to enhance the model's ability to solve more complex problems by increasing the number or length of reasoning paths. This approach is typically suitable when the reasoning task is more complex, and a single reasoning path may not effectively lead to the correct answer.

Sequential Scaling

• Core Idea: By extending the reasoning chain of a single path, the model gradually deepens its thinking, thereby improving its understanding and handling of complex problems. This is especially applicable to tasks with Long CoT that require multi-step reasoning to draw conclusions, such as mathematical proofs, logical deductions, and multi-step planning.
• Representative Works:
  • OpenAI-o1 / Deepseek-R1: Extends the reasoning chain to provide detailed multi-step reasoning processes, effectively improving the ability to solve complex problems in mathematics, coding, and other areas.
  • ITT(Inner Thinking Transformer:): redefines the layer computation in Transformer as "thinking steps," dynamically allocating computation resources to deeper reasoning on key tokens without increasing the total number of model parameters.

Parallel Scaling

• Core Idea: By generating multiple reasoning paths in parallel and combining, voting, or verifying their results, the model can effectively avoid the issue of a single path getting trapped in local optima or errors, thus improving robustness and accuracy in situations with high ambiguity, multiple possible solutions, or unclear outcomes.
• Representative Works:
  • Self-Consistency: Proposes generating multiple reasoning paths and selecting the most frequent answer from the results, effectively improving the stability and accuracy of the final answer.
  • ECM(Electronic Circuit Model): Borrowing the concepts of parallel and series circuits in electronics, combines reasoning paths in parallel or series, considering various possibilities and improving decision quality.

Internal Exploration primarily refers to large reasoning models (RLLMs) actively exploring and optimizing reasoning paths through their internal mechanisms (usually reinforcement learning strategies and reward mechanisms), allowing for more efficient and deeper solutions to complex reasoning problems. This method enables the model to autonomously adjust its reasoning strategy, reducing reliance on external guiding data.

RL Strategies

• Core Idea: Leverage reinforcement learning (RL) algorithms to guide models in actively learning and exploring diverse reasoning paths. This approach overcomes the limitations of overly uniform patterns in reasoning processes, enhancing model performance in tasks that involve high uncertainty or heavily depend on autonomous decision-making.
• Representative Works:
  • PPO(Proximal Policy Optimization): A classic RL algorithm that efficiently optimizes a model's internal decision-making mechanism through a policy-gradient-based approach, suitable for path exploration and optimization in complex environments.
  • DivPO(Diverse Preference Optimization): Encourages models to explore a greater variety of reasoning paths to maintain decision diversity, preventing convergence to local optima.
  • GRPO(Guided Reward Policy Optimization): Designs a guided reward mechanism that enables models to more effectively explore within complex logical reasoning spaces.

Reward Strategies

• Core Idea: Directly guide models to explore and optimize effective reasoning paths through carefully designed reward functions, which are particularly useful in scenarios where there are explicit optimization goals or specific reasoning bottlenecks to address.
• Representative Works:
  • Deepseek-R1: Proposes a specially designed reward function to incentivize models to optimize intermediate reasoning steps, aiding the model in building a high-quality internal reasoning process.
  • ReST-MCTS*: Combines Monte Carlo Tree Search (MCTS) with reward strategies, guiding the tree search algorithm through process rewards for more accurate exploration of effective reasoning paths, improving overall reasoning quality.

External exploration refers to the assistance of external tools, human knowledge, or other models in guiding the model to more effectively explore diverse reasoning paths and improve its ability to solve complex problems. This approach is often used in scenarios where fine-grained guidance or external knowledge is essential for effective problem-solving. External exploration can be subdivided into two types: Human-driven Exploration and Model-driven Exploration.

Human-driven Exploration

• Core Idea: Utilizes human intuition, experience, or feedback to guide the model in selecting and adjusting reasoning paths, especially in situations where the model's autonomous exploration ability is limited or the reasoning task is complex and requires decomposition into multiple sub-tasks.
• Representative Works:
  • Least-to-Most: Breaks down complex problems into simpler subproblems, solving each and using prior answers as inputs for subsequent steps, ultimately synthesizing a solution for the overall problem. This method was proposed to address the "difficulty generalization" bottleneck in traditional Chain-of-Thought approaches.
  • ToT (Tree-of-Thought): Expands the traditional "left-to-right" token generation reasoning process into a "tree structure exploration," where each node represents a thought unit. This supports multi-path attempts, backtracking, forward reasoning, and self-evaluation within the reasoning process.

Model-driven Exploration

• Core Idea: Uses auxiliary models or algorithms to automatically guide the current model's reasoning process, reducing the need for human intervention and allowing for the efficient search and optimization of numerous complex reasoning paths, thus improving automation and overall efficiency.
• Representative Works:
  • PPO-MCTS: Integrates MCTS (Monte Carlo Tree Search) with PPO-based training to enhance reasoning. The key is to retain the value network obtained during PPO training and use it during the reasoning phase to guide the MCTS in selecting more desirable output sequences, thereby improving the quality and consistency of the generated text.
  • MindStar: Reformulates complex reasoning problems (particularly mathematical ones) as search problems, where structured searches are performed over different reasoning paths to select the optimal one.
  • rStar-Math: Develops a strong mathematical reasoning system through MCTS + small model reward mechanisms + self-evolution processes, enabling small models to outperform o1-preview in mathematical capabilities.

Long CoT abilities naturally emerge after training, demonstrated by the model's ability to generate multi-step, coherent reasoning processes by internalizing logical structures and contextual examples from pretraining data, even in the absence of direct supervision. Related studies have described this phenomenon as follows:

• Wang et al. found that a small number of high-quality contextual examples can effectively guide RL-based language models (RLLMs) to generate clear, logically consistent reasoning chains, indicating that the model has internalized basic reasoning patterns during pretraining.
• Madaan et al. demonstrated that even without specific problem entities, the model can still generate reasonable reasoning chains by retaining only the logical structure information, showcasing its inductive and transfer abilities regarding structural information.
• Stechly et al. pointed out that by adjusting decoding strategies or constructing specialized prompts, latent CoT abilities within the model can be explicitly activated, resulting in multi-step reasoning in complex tasks.
• Guo et al. showed that rule-based RL strategies can directly induce models to form coherent reasoning chains during pretraining, significantly improving performance in multi-step tasks.

Large language models exhibit clear performance boundaries in Long CoT: when the depth or complexity of reasoning exceeds a certain threshold, model performance significantly degrades, sometimes even resulting in logical collapse. This phenomenon suggests that current models have a "reasoning boundary," which is the upper limit of reasoning complexity that can be supported by their parameter space and computational resources. Existing research has systematically explored this phenomenon from both theoretical modeling and empirical analysis:

• Chen et al. formally introduced the "reasoning boundary" concept, experimentally quantifying the performance critical points of models under different task complexities, and indicating that accuracy sharply declines when the reasoning task exceeds the model's capacity.
• Bi et al. observed that performance deteriorates drastically when the model attempts to mimic overly complex CoT examples in code generation tasks, indicating that beyond a certain complexity, Long CoT examples become counterproductive.
• Feng et al. proposed a mathematical model showing that models with fixed parameter sizes cannot perform numerical calculations exceeding a certain complexity, revealing a hard limit in accuracy.
• Zhou et al. constructed the GSM-Infinite dataset and demonstrated through experiments that the upper limits of reasoning abilities vary significantly across different tasks, further emphasizing that reasoning boundaries are related to task structure.

In Long CoT, extending the reasoning chain does not always lead to performance improvement. Studies have shown that once the reasoning length exceeds the model’s capacity, accuracy decreases, a phenomenon known as "overthinking," which reflects the non-linear marginal benefits of reasoning and error accumulation in the process.

• Chen et al. found that when the number of reasoning steps exceeds the model’s boundary, reasoning accuracy significantly drops, indicating that there is an optimal depth range for reasoning.
• Wolf et al. emphasized that the fundamental reason for performance degradation is the amplification of errors in intermediate reasoning steps, which affects the final judgment.
• Xie et al. experimentally showed that reasoning length does not have a monotonic relationship with accuracy, challenging the intuition that "longer CoT leads to better reasoning."
• Wu et al. established a mathematical model defining the "optimal reasoning length" interval under different models and task conditions, suggesting that performance reverses once the length exceeds the optimal range.
• Chen et al. introduced the "reasoning chain Ohm’s law," analogizing the non-linear relationship between reasoning length and performance to information flow resistance in the model.

The inference test-time scaling phenomenon refers to the increase in reasoning performance by extending the computational process (e.g., reasoning chain length or sample number) during inference. This phenomenon reveals the "dynamic amplification" potential of the model, but also comes with a trade-off between exploration depth and computational cost.

• Brown et al. observed that by repeating multiple rounds of inference attempts, even if the initial attempt fails, the correct answer can be found within a certain number of trials, introducing the "language monkey" phenomenon.
• o1 demonstrated that simply increasing the reasoning chain length improves accuracy, particularly in complex mathematical tasks.
• Jin et al. pointed out that while increasing the reasoning chain length initially leads to performance improvement, beyond a certain threshold, performance deteriorates, resulting in a typical nonlinear growth curve.
• Wu et al. found that there is a logarithmic relationship between the number of inference samples and the lower bound of error, suggesting an asymptotic relationship between computational complexity (FLOPs) and inference performance.
• Chen et al. established the theoretical upper-bound of parallel inference, indicating that no matter how the sample size is increased, the model’s verification performance cannot exceed its internal reasoning ceiling.

In reinforcement learning optimization, Long CoT tasks involve supervision of the model generation process. Researchers distinguish between two main strategies: Process Reward Model (PRM), which focuses on the reasoning process itself, and Outcome Reward Model (ORM), which only concerns whether the final output is correct. The two strategies differ significantly in terms of generalization ability, learning stability, and supervision cost.

• Lampinen et al. validated the causal relationship between intermediate steps and final answers in qualitative experiments, providing theoretical support for the rationale behind process supervision.
• Jia et al. theoretically proved that under sufficiently diverse data, ORM is not harder to optimize than PRM, with the two only differing by polynomial factors in terms of sample complexity.
• Guo et al. showed that rule-based PRM reinforcement learning significantly improves the model's Long CoT capabilities in complex tasks but also faces the risk of reward hacking.
• Tan emphasized the importance of reward distribution in intermediate reasoning steps for complex reasoning paths, which ORM cannot provide.
• Jiang et al. pointed out that PRM is more costly in terms of data collection, as it requires labeling each reasoning step, limiting its large-scale application.

The Aha Moment refers to the sudden integration of information during the reasoning process, leading to a key turning point in judgment, resembling human reflection and self-correction. This phenomenon highlights the model's dynamic cognitive adjustment abilities, but its occurrence depends on the collaboration between external stimuli and internal mechanisms.

• Guo et al. first triggered the Aha Moment behavior under unsupervised conditions through rule-based rewards, with models reflecting on intermediate reasoning and self-correcting.
• Xie et al. further demonstrated through experiments that this behavior can be replicated across multiple models, verifying that it is not an偶然 event but rather an inducible strategy.
• Zhou et al. extended the Aha Moment phenomenon to multimodal tasks, showing that it is not specific to text-based tasks but reflects the model's broader cognitive abilities.
• Liu et al. pointed out that in certain reinforcement learning frameworks (e.g., R1-Zero), Aha behavior may not genuinely exist, with lengthening the generation more likely a result of reward optimization rather than actual reflection.
• Yang et al. found that the Aha behavior often involves human-like language enhancement and dynamic uncertainty regulation, with the model more inclined to use expressions like "I think" under high-pressure tasks, reflecting its coping mechanisms for task stress.

In advancing large models to possess powerful Long CoT reasoning abilities, Supervised Fine-Tuning (SFT) plays a crucial role, bridging pre-training with more advanced alignment methods such as Reinforcement Learning from Human Feedback (RLHF). The core goal of SFT is to teach models how to follow instructions and initially master the ability to generate structured, step-by-step reasoning chains, thus laying the foundation for more complex reasoning tasks.

• In the context of deep reasoning, SFT is especially critical. Although the lack of sufficient reasoning depth in RLLMs significantly reduces performance, SFT stabilizes the model’s output format through a memorization process, allowing it to learn reasoning from human-labeled or distilled data. In contrast to reinforcement learning (RL), which focuses more on generalization and self-learning, SFT plays a vital role in deep reasoning imitation. It allows RLLMs to learn complex reasoning patterns by mimicking high-quality reasoning examples generated by humans, advanced RLLMs, or enhanced RLLMs, and generalizing them to new tasks. SFT not only significantly improves the model’s reasoning performance but, in some cases, enables even a small number of high-quality samples to activate the underlying LLM's deep reasoning capabilities, allowing it to predict events outside the model's knowledge base. This makes SFT one of the key technologies for enhancing reasoning levels and generalization abilities in RLLMs.

• For feasible reflection, SFT primarily focuses on optimization-based imitation (Refinement Imitation). In reflection-based LLM reasoning, SFT is a key mechanism for enabling self-optimization and error correction in the model. Through SFT, the model can directly learn the error-correction processes of advanced LLMs, significantly enhancing its reflective abilities, such as performing self-play reasoning, iterative feedback error correction, and even justifying and reflecting on the reasoning process through incremental natural language feedback. Additionally, SFT can integrate visual and textual reasoning in multimodal scenarios, improving the model’s critical thinking and self-correction abilities. SFT enhances the reasoning accuracy of LLMs through iterative feedback and self-correction strategies, which is especially beneficial for smaller models.

SFT consists of two core concepts: Instruction Tuning and Parameter-Efficient Fine-Tuning, PEFT。

Instruction Tuning

• Core Idea: By fine-tuning the model on a large number of instructions covering various tasks, the model’s zero-shot generalization ability on unseen tasks can be significantly enhanced. This enables the model to learn the skill of "following instructions."
• Representative Works:
  • Finetuned Language Models Are Zero-Shot Learners (FLAN): Google’s pioneering work demonstrating that multi-task instruction fine-tuning unlocks zero-shot capabilities in LLMs for unseen tasks.
  • Instruction Tuning for Large Language Models: A Survey: A comprehensive survey systematically introducing methods, datasets, challenges, and future directions in instruction tuning.

参数高效微调(PEFT)

• Core Idea: Given the high cost of full fine-tuning (Full Fine-tuning) for LLMs, PEFT methods have emerged. These methods achieve near-full fine-tuning effects by updating only a small subset of the model’s parameters, greatly reducing hardware requirements.
• Representative Works:
  • LoRA: Low-Rank Adaptation of Large Language Models: A revolutionary LoRA technique proposed to efficiently fine-tune models by injecting low-rank adaptation matrices, currently one of the most widely used PEFT methods.
  • QLoRA: Efficient Finetuning of Quantized LLMs: A further optimization of LoRA, combining 4-bit quantization, double-weight quantization, and page optimizers, making it possible to fine-tune massive models on a single consumer-grade GPU.
  • Adapter Tuning: Inserts small neural network modules (adapters) between layers of the Transformer, updating only the parameters of these adapters during training.
  • Prompt Tuning / P-Tuning: Instead of modifying the model’s weights, it learns one or more trainable virtual tokens (soft prompts) at the input end, guiding the model to perform downstream tasks more effectively.

Technical Comparison

1. Data Quality is Far More Important than Quantity:
   • Core Principle: It is better to use 1,000 high-quality, diverse data points than 10,000 low-quality, homogeneous ones. Low-quality data can teach the model incorrect patterns.
   • Format Consistency: Ensure that all training data follows a unified dialogue template (e.g.,ChatML), which is crucial for training the model to recognize roles and dialogue boundaries.
2. Choose the Right Fine-Tuning Strategy:
   • For most applications with limited resources, QLoRA should be prioritized as it strikes the best balance between efficiency and effectiveness.
   • If seeking optimal performance and with sufficient resources, full fine-tuning may be considered, though care should be taken to avoid the risk of overfitting.
3. Tuning Key Hyperparameters:
   • Learning Rate: SFT typically uses smaller learning rates than pretraining, usually ranging from 1e-5 to 5e-5.
   • Epochs: Usually, 1 to 3 epochs are sufficient. Too many epochs can lead to overfitting on small datasets, causing the model to "forget" the general knowledge learned during pretraining.
   • Batch Size: Within the memory limits, increasing the batch size appropriately helps stabilize training.
4. Evaluation and Iteration:
   • Comprehensive Evaluation: Do not rely solely on the loss function. Combine it with objective evaluation benchmarks (such as MMLU) and subjective human evaluation for a more thorough assessment of model performance.
   • Iterative Optimization: SFT is an ongoing iterative process. Based on evaluation results, continuously clean the data, adjust hyperparameters, and optimize the model.

• LLM4NLP

• West Lake University’s "Mathematical Principles of Reinforcement Learning"
  • Features: Starts with MDP and the Bellman Equation, derived using the policy gradient theorem.
  • Prerequisite Knowledge: Linear algebra, probability theory.
  • Focus: The mathematical essence of value iteration and policy optimization.
• Book-Mathematical-Foundation-of-Reinforcement-Learning (Beginner-friendly)

Authoritative Courses

Essential Basic Algorithms

• Basic Reinforcement Learning Algorithms

  • DQN: The beginning of Deep Reinforcement Learning
  • PPO: A key method in policy optimization, widely used in industrial applications
  • SAC: Incorporates exploration entropy, robust for continuous action spaces
  • TD3: Improves off-policy reinforcement learning using double delayed networks

• Model-Based Reinforcement Learning Algorithms

  • dreamer: A model-based reinforcement learning algorithm
  • tdmpc2: Significant advancement in model-based reinforcement learning algorithms

• Offline Reinforcement Learning Algorithms

  • CQL: Introduces conservative constraints, foundational work in offline reinforcement learning
  • decision-transformer: Introduces autoregressive models into offline reinforcement learning

• Large-Scale Model Reinforcement Learning Algorithms

  • PPO: Applying classic PPO to large language models
  • DPO: Preference optimization without rewards, an offline reinforcement learning algorithm for large models
  • GRPO: Group Relative Policy Optimization, core algorithm in DeepSeek-R1

Cutting-Edge Algorithms for Large-Scale Model Reinforcement Learning

• DAPO: Four improvements on GRPO
• LUFFY: Off-policy version of GRPO, introduces high-quality external trajectories
• Absolute-Zero-Reasoner: A large-scale reinforcement learning algorithm requiring no annotations
• One-Shot-RLVR: One-shot optimization for large model inference
• SPIRAL: Reinforcement learning in self-play game environments, successfully enhancing mathematical reasoning abilities
• High-Entropy Minority Tokens Drive Effective RLVR: Reinforcement learning driven by high-entropy tokens (20%)
• Spurious_Rewards: Random rewards can also enhance LLM reasoning abilities
• SwS: Reasoning reinforcement learning driven by self-perceived weaknesses

Basic Reinforcement Learning Frameworks

• stable-baselines3 (Quick experimentation, with well-established and stable baselines)
• legged_gym (Quadruped robot control)

Large-Scale Model Reinforcement Learning Frameworks

• verl: A high-performance and user-friendly open-source reinforcement learning training library based on Ray, vLLM, ZeRO-3, and HuggingFace Transformers, with features like efficient resource utilization, scalability, and production readiness. (Complex structure, highly reusable, excellent performance)
• OpenRLHF: An open-source RLHF framework released by teams such as NVIDIA, based on Ray, vLLM, ZeRO-3, and HuggingFace Transformers. It supports algorithms like PPO, GRPO, and REINFORCE++, and provides dynamic sampling and asynchronous agent mechanisms to accelerate training.
• AReaL: Asynchronous reinforcement learning framework
• ROLL: Supports training large models with 600+ billion parameters
• Hugging Face TRL: An RLHF full-stack library maintained by Hugging Face, integrating SFT, GRPO, DPO, Reward Modeling, and other modules. It supports multiple model architectures and distributed scaling, making it one of the most active RLHF tools in the community. (User-friendly, quick start, active community)
• RL4LMs: An open-source RLHF library for language models, providing end-to-end tools for reward model construction and policy network training, helping researchers quickly build custom RLHF pipelines.

Additionally, there are some interesting extension repositories:

• Sachin19/trlp: An end-to-end RLHF library based on the TRL stack, supporting not only language models but also extending to Stable Diffusion models. It includes steps like SFT, reward modeling, and PPO, with example code for experimentation.
• OpenRLHF-M: An extension of OpenRLHF, optimized for multimodal models. It leverages DeepSpeed and HuggingFace Transformers to achieve higher throughput and richer training scenarios.
• HumanSignal-RLHF: An archived resource repository, gathering links and tutorials on RLHF data collection, system construction, and best practices, suitable for beginners to quickly understand the full RLHF pipeline.
• MichaelEinhorn/trl-textworld: A derivative version of TRL focused on performing RLHF experiments in the TextWorld environment, demonstrating how to train models like GPT2 using PPO to generate text that meets specific feedback requirements.

Classical RL Tests

• OpenAI Gym: Classic Control

• Atari 2600: Games

• Box2D: Physics Simulation

• MuJoCo: Robotic Control

• Other Special Environments

Extended Resources:

• Safe RL: Safety-Gymnasium (task with constraints)
• Autonomous Driving: CARLA/AirSim (high fidelity simulation)
• Multi-agent: PettingZoo (compatible with Gymnasium API)

💡 Full environment list can be found at: Gymnasium Documentation | OpenAI Gym Wiki

Large Model RL Tests

The ability of LLM Agents to solve complex problems fundamentally relies on their reasoning and planning capabilities. The core mechanism of this ability is Long CoT, which breaks down complex tasks into smaller, logical steps. The characteristics of Long CoT, particularly its depth of inference, extensive exploration, and feasibility reflection, are not just additional features but the foundation for realizing these abilities. If an agent cannot "think longer" and engage in a "thinking-critique-improvement" cycle, its ability to make independent decisions and adapt in unfamiliar scenarios will be severely limited, causing it to revert to "predefined pipelines" or "iterative interactions with humans." Models such as o1 and DeepSeek-R1 have made breakthroughs in using Long CoT to solve complex tasks, directly proving this causal relationship: enhanced reasoning depth directly leads to an improvement in agent capabilities (autonomy in complex tasks). Therefore, the future development of AI agents will be closely linked to breakthroughs in Long CoT.

AI Agent Online Courses and Resources

• Andrew Ng's "How to Build, Evaluate, and Iterate LLM Agents": A seminar by LlamaIndex and TruEra team experts (March 2024), explaining how to build LLM agents using tool frameworks like LlamaIndex and evaluate agent performance, detect hallucinations and biases using observability tools like TruLens. The video provides both Chinese and English subtitles, making it suitable for learning about agent development and evaluation methods in production environments.
• Coursera AI Agent Developer Specialization (Vanderbilt University): A series of 6 courses for beginners with Python experience, focusing on building and deploying intelligent AI agents using Python, tools, memory, and reasoning. Topics include creating custom GPTs, applying prompt engineering, designing reliable AI systems, and implementing multi-agent collaboration systems.
• Hugging Face Agent Course: A free online course introducing agents.

Open Source Frameworks for Building LLM AI Agents

• LangChain: The most widely used framework for LLM agent development, offering a modular and extensible architecture, unified LLM interfaces, pre-built agent toolkits (for CSV, JSON, SQL), Python and Pandas integration, and vector storage capabilities. It supports React-style agents and provides a memory module to maintain context.
• CrewAI: An open-source framework for orchestrating role-playing AI agents, emphasizing multi-agent collaboration through defined roles and shared goals. It is independent, streamlined, and offers deep customization, supporting "Crew" (team) and "Flow" (event-driven workflows).
• Dify: An open-source framework for LLM applications with a visual prompt orchestration interface, long context integration, API-based development, multi-model support, and RAG pipelines.
• OpenAI Agent Demo: OpenAI's official platform for setting up Agent client services (visual platform, no additional code required).

For more frameworks, refer to Awesome LLM Agent Frameworks.

End-to-End RL Learning for Complex Agent Trajectories

• Agent-R1: An open-source framework aimed at accelerating research and development at the intersection of RL and agents. It uses end-to-end reinforcement learning to train agents in specific environments, allowing developers to define domain-specific tools and reward functions without complex process engineering. It supports multi-round tool calls and multi-tool coordination.
• RAGEN: A framework for training LLM reasoning agents with RL in interactive, stochastic, and multi-round environments. It introduces the StarPO (State-Think-Act-Reward Policy Optimization) framework, which features staggered rollout and update phases for trajectory-level optimization.

RL-enhanced Tool Use and Search Capabilities

• ReCall: A novel framework that trains LLMs for tool invocation reasoning with RL, without the need for supervised data on tool usage trajectories or reasoning steps. It is designed to enable LLMs to use and combine any user-defined tools in an agent-like manner.
• OpenManus-RL: An extension of the OpenManus framework, specifically focused on enhancing AI agents via RL techniques like GRPO, to enable training across multiple environments and performance tuning for specific tasks.
• R1-Searcher, Search-R1: Research exploring the use of RL to enhance the search capabilities of LLMs.

Awesome Blog

• Neptune.ai blog: Provides a detailed step-by-step guide, such as "How to Build LLM Agents with AutoGen," covering components, RAG pipelines, planning, tools, and memory integration.
• n8n.io blog: Offers insights into the capabilities of LLM agents (such as strategic planning, memory, and tool integration) and includes a practical tutorial on building agents.
• NVIDIA Developer Blog: Provides an introductory article on LLM reasoning and AI agents.
• Botpress blog: Explains chain-of-thought prompting and discusses various AI agent frameworks.
• SuperAnnotate blog: Offers a comprehensive overview of LLM agents, their capabilities, and their future.
• Smythos blog: Discusses how LLM agents are revolutionizing task automation and AI integration.
• Unite.ai: Provides a detailed discussion on how reinforcement learning, combined with chain-of-thought, transforms LLMs into autonomous reasoning agents.
• Holistic AI blog: Delves into the architecture of LLM agents, including multimodal enhancement, tool usage, and memory.
• ProjectPro and Lightrains blog: Discuss various AI agent design patterns, including reflection, tool usage, and planning patterns.

Awesome GitHub Repositories

• Awesome-LLM-Agents: A curated list of various LLM agent frameworks, serving as a valuable starting point for exploring the ecosystem.
• Awesome-LLM-Agents-Scientific-Discovery: A curated list of papers focused on LLM-driven AI agents' applications in biomedical research and broader scientific discovery.
• Awesome-Agent-RL: A specialized collection of papers and resources focused on unleashing the potential of AI agents through reinforcement learning.
• Awesome-LLM-APPs: A curated collection of excellent LLM applications built using RAG, AI agents, multi-agent teams, MCP, speech agents, and more.

• LLM Evaluation Frameworks:

  • OpenCompass is a comprehensive evaluation platform for large language models (LLMs) that supports assessments of a wide range of open and closed models across over 100 datasets. It covers multiple dimensions such as language understanding, reasoning, and code generation, and supports various evaluation modes, including zero-shot, few-shot, and Chain-of-Thought (CoT), as well as distributed evaluation capabilities.
  • DeepEval is an easy-to-use, open-source LLM evaluation framework designed for evaluating and testing large language model systems. It aims to help developers efficiently assess the quality of model-generated content based on key metrics such as relevance, factual consistency, bias, and toxicity. Its usage is similar to the Python unit testing framework, Pytest.

• MLLM Evaluation Frameworks:

  • VLMEvalKit is an open-source toolkit launched by OpenCompass specifically designed for the evaluation of large vision-language models. It supports one-click evaluations for over 220 vision-language models across more than 80 benchmark tests, covering tasks such as image question answering, image-text matching, and visual reasoning. It provides evaluation results based on both exact matching and LLM-based answer extraction.
  • EvalScope is a model evaluation framework introduced by the MoTower community, supporting performance benchmarking for various types of models, including large language models, multimodal language models, embedding models, and AIGC models.

• CoT Evaluation Frameworks:

  • ROSCOE aims to provide a set of automated metrics to evaluate the reasoning quality of models without requiring reference answers.
  • ReCEval is a reasoning chain evaluation framework proposed by Archiki Prasad and colleagues. It aims to provide a detailed analysis of the multi-step reasoning process generated by large language models through two dimensions: "correctness" and "informativeness."

This section focuses on evaluating the final performance of Long CoT reasoning from a holistic perspective, emphasizing whether the reasoning chain is ultimately sound and accurate.

• Complex Mathematics

• Complex Coding

• Commonsense Puzzles

Here is the translation of the table into academic English:

• Scientific Reasoning

• Medical Reasoning

The focus is on the local perspective or the individual abilities of the model during the Long CoT reasoning process, examining finer granularity by investigating whether each step of the model's reasoning is correct and logical. For instance, whether it can correctly identify errors and correct them, or whether it can complete complex tasks step by step.

• Deep Reasoning

• Exploration Benchmarks

• Reflection Benchmarks

Benchmarks designed specifically to evaluate large language models' capabilities in complex reasoning, cross-domain knowledge integration, and multimodal understanding. As basic evaluations are gradually saturated by top-tier models, researchers have started developing more challenging benchmarks to more accurately measure models' performance on real-world complex tasks.

• Agentic & Embodied Reasoning

• Multimodal Reasoning

  • Complex Mathematics:

  Name	Number of Problems	Release Date	Authors	Description	Relevant Links
  MathVista	6,141	2023	Pan Lu et al. (UCLA)	MathVista is a multimodal mathematical reasoning evaluation benchmark jointly released by UCLA, the University of Washington, and Microsoft Research. It is designed to systematically assess the mathematical reasoning capabilities of large language models and multimodal models within a visual context.	🤗dataset 🌐repository 🌐website
  MathVision	3,040	2024	Ke Wang et al. (Chinese University of Hong Kong)	MathVision (MATH-V) is a multimodal mathematical reasoning evaluation benchmark released by the Chinese University of Hong Kong, among others. It aims to systematically evaluate the mathematical reasoning abilities of large vision-language models within visual contexts. The benchmark includes 3,040 problems across 16 mathematical disciplines, divided into five difficulty levels, with problems sourced from real mathematics competitions.	🤗dataset 🌐repository 🌐website
  MathVerse	~15,000	2024	Zimu Lu et al. (Chinese University of Hong Kong)	MathVerse is a multimodal mathematical reasoning evaluation benchmark jointly released by MMLab at the Chinese University of Hong Kong and the Shanghai AI Lab. It is designed to comprehensively assess multimodal large language models' ability to understand mathematical diagrams. The benchmark includes 2,612 problems spanning areas such as plane geometry, solid geometry, and functions, annotated by experts. It generates six versions of multimodal information, totaling approximately 15,000 test samples. MathVerse introduces a Chain-of-Thought (CoT) evaluation strategy, leveraging GPT-4V for fine-grained analysis of model reasoning processes.	🤗dataset 🌐repository

  • Complex Code:

  Name	Number of Problems	Release Date	Authors	Description	Relevant Links
  HumanEval-V	253	2024	Fengji Zhang et al. (City University of Hong Kong)	HumanEval-V is a multimodal code generation evaluation benchmark proposed by the University of Hong Kong, aiming to test the capabilities of large multimodal models in complex diagram understanding and code generation tasks. This benchmark includes 253 Python programming tasks, each accompanied by key diagrams and function signatures, requiring the model to generate executable code based on visual information.	🤗dataset 🌐repository 🌐website
  Code-Vision	1,000+	2025	Hanbin Wang et al. (Peking University)	Code-Vision is a multimodal code generation evaluation benchmark jointly released by Peking University, Northeastern University, and the University of Hong Kong. It aims to test the ability of multimodal large language models to understand flowcharts and generate corresponding code. This benchmark fills the gap in existing benchmarks, which mainly focus on textual reasoning and lack a systematic evaluation of code generation in visual contexts, providing a unified evaluation framework.	🌐repository 🌐website
  ChartMimic	4,800	2024	Cheng Yang et al. (Tsinghua University)	ChartMimic is a multimodal code generation evaluation benchmark jointly released by Tsinghua University, Tencent AI Lab, and other institutions. It aims to evaluate the cross-modal reasoning abilities of large multimodal models in chart understanding and code generation, addressing the gap in existing benchmarks that focus mainly on textual reasoning and lack systematic evaluation of chart understanding and code generation. It includes two task types: Direct Mimic and Customized Mimic, with data sourced from scientific papers across multiple fields.	🤗dataset 🌐repository 🌐website

  • Complex Science:

  Name	Number of Problems	Release Date	Authors	Description	Relevant Links
  ScienceQA	21,208	2022	Pan Lu et al. (UCLA)	ScienceQA is a multimodal multiple-choice dataset consisting of 21,208 problems across natural sciences, language sciences, and social sciences, designed for K-12 grade levels. The dataset provides context with images and text, explanations, and detailed answers, supporting Chain-of-Thought (CoT) reasoning, aiming to assess and enhance the multi-step reasoning abilities and interpretability of AI models.	🤗dataset 🌐repository 🌐website
  M3CoT	11,459	2024	Qiguang Chen et al. (HIT-SCIR Lab)	M3CoT is a multimodal, multi-domain, multi-step reasoning dataset built upon ScienceQA, designed to assess the capabilities of AI models in complex reasoning tasks. Compared to ScienceQA, M3CoT-Science has an average reasoning step increase from 2.5 to 10.9, and the average text length grows from 48 to 294, significantly increasing task complexity. The dataset spans science, common sense, and mathematics, emphasizing cross-reasoning between image and text information, challenging the reasoning capabilities of existing multimodal large models.	🤗dataset 🌐repository 🌐website
  MolPuzzle	234	2024	Kehan Guo et al.	MolPuzzle is a multimodal, multi-step reasoning dataset designed to evaluate large language models in molecular structure analysis tasks. The dataset involves various spectrometric data types, including infrared spectroscopy (IR), mass spectrometry (MS), and nuclear magnetic resonance (1H-NMR and 13C-NMR), as well as molecular formula information. Tasks are divided into three stages: molecular understanding, spectral analysis, and molecular construction, simulating real chemical reasoning processes.	🤗dataset 🌐repository 🌐website

  • Commonsense Puzzle:

  Name	Number of Problems	Release Date	Authors	Description	Relevant Links
  PuzzleVQA	2,000	2024	Yew Ken Chia et al.	PuzzleVQA is a multimodal reasoning dataset consisting of 2,000 abstract graphic puzzles, designed to evaluate the visual perception, induction, and deduction abilities of large multimodal models in basic concepts such as color, numbers, shapes, and sizes. Experiments show that even advanced models like GPT-4V achieve an average accuracy of only 46.4% on single-concept puzzles, significantly lower than human performance, exposing limitations in abstract pattern recognition and multi-step reasoning.	🤗dataset 🌐repository 🌐website
  LEGO-Puzzles	1,100	2025	Kexian Tang et al. (Shanghai AI Lab)	LEGO-Puzzles aims to evaluate the capability of large multimodal language models in multi-step spatial reasoning tasks. The dataset contains 1,100 visual question answering (VQA) tasks based on LEGO bricks, covering 11 task types, including spatial understanding, single-step and multi-step sequence reasoning.	🤗dataset 🌐repository 🌐website
  CVQA	10,374	2024	David Romero et al. (MBZUAI)	CVQA is a multimodal visual question answering dataset designed to assess models' abilities to integrate multiple visual cues for combined reasoning. The dataset includes three task types requiring models to extract and synthesize key information from multiple images to answer complex questions.	🤗dataset 🌐website

• AI4Research:

To build and enhance models with strong Long CoT capabilities, numerous open-source training datasets have emerged. These datasets provide foundational supervision signals for various domains such as mathematics, science, medicine, programming, and general reasoning. Based on their construction methods, we classify the datasets into four major categories: Manual Annotation, Direct Distillation, Search-based Distillation, and Validated Distillation.

In this section, we systematically list representative datasets under each category, covering key information such as their sources, modalities, applicable domains, and data scale, providing researchers and developers seeking suitable training resources with a comprehensive guide and convenient reference.

These datasets are created through manual annotation or rule-based construction, typically offering high-quality samples with interpretable reasoning paths. While smaller in scale, they are critical for guiding the alignment and evaluation of initial models.

• R1-OneVision combines high-quality data from LLaVA-OneVision with datasets from specific domains. It bridges the gap between visual and textual understanding, providing rich, context-aware reasoning tasks across natural scenes, science, mathematics, OCR-based content, and complex graphs.
• M3CoT lays the foundational work for multi-domain, multi-step, multi-modal chain-of-thought research.
• Big-Math-RL-Verified is designed for RL training with large language models (LLMs) such as PPO, GRPO, etc.
• GSM8K is a high-quality, linguistically diverse dataset of elementary school math word problems.

The method utilizes large language models to generate training data through prompt-based or chain-of-thought reasoning. These datasets can be scaled up to millions of examples, covering a wide range of domains.

• NaturalReasoning validates through knowledge distillation experiments that NaturalReasoning can effectively extract and transfer reasoning capabilities from powerful teacher models. It is equally effective for unsupervised self-training using external reward models or self-rewarding.
• NuminaMath-CoT employs a chain-of-thought (CoT) format for solving each problem. The dataset covers Chinese high school math exercises, US and international Mathematical Olympiad problems, etc., collected from online exam PDFs and math forums. Processing steps include: (a) OCR recognition of the original PDFs; (b) segmentation into problem-solution pairs; (c) translation into English; (d) rearranging to generate chain-of-thought reasoning formats; and (e) final answer formatting.
• NuminaMath-TIR focuses on problems that produce numerical outputs selected from the NuminaMath-CoT dataset. A pipeline was constructed using GPT-4 to generate reasoning paths similar to TORA, execute code, and produce results until the final solution is completed, filtering out solutions where the final answer differs from the reference answer.
• DART-Math-uniform constructs datasets through the application of DARS-Uniform.
• DART-Math-hard is a mathematical question-answer pair sample constructed using query sets from the DARS-Prop2DiffMATH and GSK8K training datasets. It achieves SOTA results on many challenging mathematical reasoning benchmarks, introducing a deliberate preference for hard queries, in contrast to traditional rejection sampling.
• DART-Math-pool-math is a data pool synthesized from query sets of the MATH training dataset, including all samples with correct answers and additional metadata generated during the process. DART-Math-- datasets are extracted from the DART-Math-pool-- data pools.
• DART-Math-pool-gsm8k is a data pool synthesized from query sets of the GSM8K training dataset, including all samples with correct answers and additional metadata. DART-Math-- datasets are extracted from the DART-Math-pool-- data pools.
• OpenO1-SFT is a dataset for fine-tuning language models on chain-of-thought activations using SFT.
• OpenO1-SFT-Pro is a dataset for fine-tuning language models on chain-of-thought activations using SFT.
• OpenO1-SFT-Ultra is synthesized based on existing open-source datasets using the openo1-qwen-sft model.
• Medical-o1 is an SFT medical reasoning dataset constructed based on medically verifiable questions and an LLM verifier.
• AoPS-Instruc is a large-scale high-quality question-answer pair dataset for advanced mathematical reasoning created and maintained using a scalable approach.
• Orca-Math is a high-quality synthetic dataset of 200,000 math problems created in a multi-agent setup where agents collaborate to generate the data.
• MATH-plus collects 10 million naturally occurring instruction data from pre-trained network corpora.
• UltraInteract-SFT is designed for complex reasoning tasks and helps explore preference learning in reasoning tasks. It is applicable for supervised fine-tuning and preference learning. Each instruction includes a preference tree consisting of: (1) reasoning chains with multiple planning strategies and consistent formats; (2) multi-round interaction trajectories with the environment and comments; and (3) paired data for facilitating preference learning.
• MathCodeInstruct is a novel high-quality dataset containing mathematical problems and their code-based solutions.
• MathCodeInstruct-Plus $Paper¹, ²$ is a novel high-quality dataset containing mathematical problems and their code-based solutions.
• OpenMathInstruct-1[HuggingFace] is a math instruction adjustment dataset, generating 1.8 million problem-solution pairs using the Mixtral-8x7B model. The problems are sourced from the GSM8K and MATH training subsets; solutions are synthesized by the Mixtral model using text reasoning and Python interpreter-executed code blocks.
• OpenMathInstruct-2 is a large-scale math reasoning dataset for training large language models (LLM).
• AceMath-Instruct is a training dataset for cutting-edge mathematical reasoning models in AceMath.
• QwQ-LongCoT integrates prompts from multiple high-quality sources to create diverse and comprehensive training data.
• SCP-116K is a high-quality set of scientific question-answer pairs automatically extracted from web-scraped documents. Each question is accompanied by a matching solution extracted from the source material, along with responses and reasoning processes generated by advanced language models.
• R1-Distill-SFT is distilled using DeepSeek-R1-32b; generated using Numina-math and Tulu; each prompt samples a response.
• Sky-T1-Data contains 5k encoded data from APPs and TACO, as well as 10k math data from the AIME, MATH, and Olympiads subsets of the NuminaMATH dataset. It also maintains 1k science and puzzle data from STILL-2.
• Bespoke-Stratos-17k is a reasoning dataset that includes questions, reasoning traces, and answers. It replicates and improves upon the Berkeley Sky-T1 data pipeline using SFT distilled data from DeepSeek-R1.
• s1K contains 1,000 diverse, high-quality, and difficult problem examples (from Gemini Thinking), refining reasoning paths and solutions.
• MedThoughts-8K
• SYNTHETIC-1 is the largest open reasoning dataset generated by Deepseek-R1, covering reasoning trajectories for tasks in mathematics, programming, science, etc., with correctness verified by task-specific validators.
• Medical-R1-Distill-Data is an SFT dataset distilled from Deepseek-R1 (Full Power Version), based on HuatuoGPT-o1’s medically verifiable questions.
• Medical-R1-Distill-Data-Chinese is an SFT Chinese version dataset distilled from Deepseek-R1 (Full Power Version), based on HuatuoGPT-o1’s medically verifiable questions.
• RLVR-GSM-MATH is used to train the Tulu3 model.
• LIMO Less is More for Reasoning.
• OpenThoughts-114k is an open synthetic reasoning dataset containing 114k high-quality examples covering math, science, code, and puzzles.
• Magpie-Reasoning-V2 generates high-quality alignment data by aligning LLMs using pre-query template prompts.
• Dolphin-R1 is an 800k sample dataset similar in composition to the datasets used for training the DeepSeek-R1 Distill model.

The dataset based on search is constructed through an automated search algorithm, which explores the reasoning tree to generate the optimal reasoning trajectory. Although the scale is limited, these datasets typically generate high-quality and deep reasoning samples.

• STILL-1 enhances the reasoning capabilities of large language models (LLMs) through a reward-guided tree search algorithm.

The validated datasets contain rule-based filtering, test case verification, or LLM validation to ensure quality. These datasets strike a balance between scalability and reliability.

• KodCode-V1 provides verifiable solutions and tests for coding tasks; specifically designed for supervised fine-tuning (SFT) and reinforcement learning (RL) optimization; covering various domains (from algorithms to domain-specific software knowledge) and difficulty levels (from basic coding exercises to interview and competitive programming challenges).
• KodCode-V1-SFT-R1 provides verifiable solutions and tests for coding tasks; specifically designed for supervised fine-tuning (SFT) and reinforcement learning (RL) optimization; covering various domains (from algorithms to domain-specific software knowledge) and difficulty levels (from basic coding exercises to interview and competitive programming challenges).
• OpenR1-Math is a large-scale mathematics reasoning dataset, generated by DeepSeek R1 for NuminaMath version 1.5 problems, with two to four reasoning trajectories per question.
• Chinese-DeepSeek-R1-Distill-Data is a Chinese open-source distilled dataset from DeepSeek-R1, containing not only math data but also a significant amount of general-type data.
• AM-DeepSeek-R1-Distilled includes problems from numerous open-source datasets, which have been semantically deduplicated and cleaned to eliminate test set contamination. The answers are extracted from reasoning models (primarily DeepSeek-R1) and undergo rigorous validation: mathematical problems are verified through answer checking, coding problems through test case validation, and other tasks through reward model evaluation.

• Awesome-Long-Chain-of-Thought-Reasoning (Our Official Paper List, 1000+ papers)

• Awesome-System2-Reasoning-LLM

If you find this work useful, welcome to cite us.

@misc{chen2025reasoning,
      title={Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models}, 
      author={Qiguang Chen and Libo Qin and Jinhao Liu and Dengyun Peng and Jiannan Guan and Peng Wang and Mengkang Hu and Yuhang Zhou and Te Gao and Wanxiang Che},
      year={2025},
      eprint={2503.09567},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2503.09567}, 
}

For any interesting news about Long CoT, you can also @Qiguang_Chen on Twitter or email me at charleschen2333@gmail.com to follow and update it at our GitHub repo.

Hope everyone enjoy the Long CoT era :)