What objective is reasoning solving?

There’s an intuitive but simplified image of reasoning: let a example think for longer, and it’ll be additional apt to be correct. This aligns fine alongside our individual intuition: expend additional period and attempt on thinking and it’s additional apt that we’ll attain at a accurate answer.

But this image doesn’t appear to explain everything. During RL training, models study to compose longer chains of thought, and they get additional exact as they do. Yet if you obtain a trained example and power any one sequence to run longer, it helps for a while, afterward stops helping.¹ Sometimes the answer equal gets worse, as if it’s overthinking.² If a sequence of idea were really converging toward several optimum, additional steps shouldn’t create it worse. So is it really the dimension of all sequence that helps at conclusion time? Or is it item additional subtle?

This article is my attempt to explain this disturbance (mostly for myself) by looking at reasoning as pathfinding through token space. We’ll see that RL maximizes the probability of finding a way from $x$ to $\Scal_x$, but idiosyncratic paths are not guaranteed to converge to the accurate answer. As a bonus, this viewpoint clarifies several odd observations concerning reasoning models, akin unfaithful chains of idea and reward hacking.

Reasoning as pathfinding#

Ask a reasoning example the identical math issue ten times and you’ll get ten distinct chains of thought. Rather than thinking of reasoning as one mysterious procedure heading toward an answer, let’s think of it as pathfinding in token space. There are many paths from the issue to a accurate answer, and the example traverses one of them alongside all sample.

Consider the problem: “what’s $1 + 2 + 3 + \cdots + 10$”? One resolution is to add the numbers one at a time: 1, 3, 6, 10, and so on up to 55. Another way is to brace them from the exterior in: $1 + 10$, $2 + 9$, $3 + 8$, $4 + 7$ and $5 + 6$ all create 11, and five pairs of 11 create 55. Both of these paths district on the accurate answer, 55.

We can think concerning reasoning as following this broad pattern. We are stated a starting point, the prompt, which we’ll call $x$. We desire to discover a way $z$ that starts from $x$ and ends at a accurate answer $y$. To inform whether a way lands in the correct place, we assume we have a verifier $r(x, y)$ that returns 1 if $y$ is accurate and 0 if it isn’t.

So stated a starting item $x$, we desire to discover a way of tokens $z$ that lands in the set

$$\Scal_x = \{y : r(x, y) = 1\}.$$

The fig at the top of this post sketches this picture. Each gray row is a sampled path: it starts from the immediate on the left, meanders in token area by sampling $z$, alongside several of these paths landing in $\Scal_x$.

Since many paths district in $\Scal_x$, the achievement probability adds up complete all of them. Let $\pi(z \mid x)$ be the probability that the example generates the way $z$ stated the immediate $x$, and $y(z)$ for the final answer at the end of $z$. The probability of achievement is the probability of generating all path, times the probability that way lands in $\Scal_x$, summed complete all path:

$$p_x = \prob{\text{a way lands in } \Scal_x \mid x} = \sum_{z} \underbrace{\pi(z \mid x)}_{\text{samples way } z}\; \underbrace{\prob{y(z) \in \Scal_x \mid z}}_{\text{path } z \text{ lands in }\Scal_x}.$$

The second aspect is fair 1 if the way lands in $\Scal_x$ and 0 if it doesn’t, so we sum lone the probability of paths that district in $\Scal_x$:

$$p_x = \sum_{z} \pi(z \mid x)\, \ind{y(z) \in \Scal_x} = \sum_{z \,:\, y(z) \in \Scal_x} \pi(z \mid x).$$

We’ll call $p_x$ the success probability. Maximizing this probability method our example volition be additional apt to example a way that lands in $\Scal_x$.

Note that there are two ways of expanding this probability. Either the example can example multiple paths to the identical answer, all alongside a small probability, or the example can example a sole way to the answer alongside a extremely elevated probability. The achievement probability doesn’t attention which one we get, but in custom we power desire to prioritize particular paths or justify assortment in the samples³.

How do we boost our achievement probability?#

Best-of-$k$ sampling#

The simplest way to pathfind is to have the example $\pi$ keep sampling paths $z$ until one lands in $\Scal_x$.

Since we assumed we had a verifier, we can do best-of-$k$ sampling, which samples $k$ paths and looks for a success. Each way misses alongside probability $1 - p_x$, so all $k$ young female alongside probability $(1-p_x)^k$, and the achievement probability rises from $p_x$ to

$$\prob{\text{at smallest one of } k \text{ paths lands in } \Scal_x \mid x} = 1 - \underbrace{(1-p_x)^k}_{\text{all } k \text{ paths miss}}.$$

This is a guaranteed axis of compute-scaling. As lengthy as $p_x > 0$, the achievement probability increases monotonically as we example additional paths, so spending additional compute on additional samples continually helps, as seen first on by Brown et al. 2024. However, the scaling current is complete the figure of samples fairly than the dimension of the paths (thinking longer).

Train the example to create improved samples#

So far the example was fixed, so $p_x$ was fixed too. Can we alter the example to addition $p_x$ itself?

Here’s a uncomplicated way to do this. Collect many prompts whose answers we can check. For all one, example several paths from the model. We keep the paths that district in $\Scal_x$, and train the example on the ones we kept. Then repeat. Each round, we train the example to put additional importance on the prosperous paths that it sampled. This iteration is called rejection-sampling fine-tuning, and STaR was an first type of it for reasoning.

RL alongside a verifiable reward, which is how today’s reasoning models are trained, turns out to do nearly the identical thing. Let $\pi_\theta$ be the example alongside weights $\theta$. Then since $r(x, y(z)) = 1$ exactly whenever $y(z) \in \Scal_x$:

$$ \begin{aligned} p_x &= \sum_{z \,:\, y(z) \in \Scal_x} \pi_\theta(z \mid x) \\ &= \sum_{z} \pi_\theta(z \mid x)\, \ind{y(z) \in \Scal_x} \\ &= \sum_{z} \pi_\theta(z \mid x)\, r(x, y(z)) \\ &= \ev[z \sim \pi_\theta(\cdot \mid x)]{r(x, y(z))}, \end{aligned} $$

meaning the expected binary reward is exactly the achievement probability $p_x$.

Computing the gradient of the achievement probability via the log-derivative trick, we attain at

$$\grad_\theta\, p_x = \ev[z \sim \pi_\theta(\cdot \mid x)]{r(x, y(z))\, \grad_\theta \log \pi_\theta(z \mid x)},$$

which is exactly the guideline gradient, additionally called the REINFORCE estimator. Now evaluation this gradient alongside $N$ sampled paths $z_1, \dots, z_N$. The paths that unsuccessful have $r = 0$ and disappear, during the paths that district in $\Scal_x$ have $r = 1$. If $M$ of the $N$ paths succeed, the evaluation is

$$\grad_\theta\, p_x \approx \frac{1}{N} \sum_{i \,:\, \text{landed}} \grad_\theta \log \pi_\theta(z_i \mid x) = \underbrace{\frac{M}{N}}_{\text{success rate}} \cdot \underbrace{\frac{1}{M} \sum_{i \,:\, \text{landed}} \grad_\theta \log \pi_\theta(z_i \mid x)}_{\text{log likelihood of the prosperous paths}}.$$

This gradient involves two terms: the empirical achievement charge $M/N$ and the average log likelihood of the prosperous paths.

In another words, a stage of REINFORCE alongside a binary reward is a stage of rejection-sampling fine-tuning on the model’s own sampled paths, scaled by the current achievement charge $M/N$.

This additionally explains the archetypal study from the intro, that chains of idea get longer during RL training. Each stage pulls the example toward the paths that landed, so if the paths that succeeded tended to be longer than the ones that unsuccessful (e.g. since they contain backtracking behavior, collect additional feedback from the environment, or fair dumb luck), the example volition study to compose longer paths.

Amortization and generalization#

During RL training, we expend a lot of compute to hunt for fine paths and afterward we train the example to create those paths directly. Once trained, the example can create the answer without spending all this additional compute on sampling multiple paths. This is called amortizing the search: we pay for it formerly during training, so we don’t have to pay for it all period we ask the example a question.⁴ Since we train a single model on many prompts, the example can possibly generalize to new prompts it has never seen.

The fig below shows this for four prompts, all alongside its own $\Scal_x$, and switches between the example before and following training. During training, we example paths from all prompt, keep the ones that succeed, and SFT on them. The percentages display the empirical charge of achievement stated all prompt. The hollow immediate is new and not seen before by the model, but the example generalizes to this new immediate following training. As the new immediate gets additional from the training distribution, its accuracy drops (try dragging it!).

Revisiting several observations concerning reasoning#

Once we see a reasoning example as a allocation trained to district in $\Scal_x$ stated any $x$, a few odd facts concerning these models rotate into clear.

Chains of idea are not guaranteed to be faithful#

Our verifier complete lone checks whether the answer is correct. The reward is 1 if $y(z) \in \Scal_x$ and 0 otherwise, so the intermediate tokens $z$ are fair a way for the example to get to the answer. Nothing requires those tokens to be true, or to explain the answer that follows, although of way these are frequently correlated because of the model’s pretraining.

If we desire meaningful chains of thought, we need to modify the rewards or the training objective.

Reward hacking comes from the two the verifier and the sampler#

We additionally assumed the verifier is never wrong, that $r(x, y)=1$ if and lone if $y$ is really the accurate answer to $x$. But in custom the verifier can be imperfect. Then $\Scal_x$, the set of answers the verifier accepts, isn’t the identical as the set of accurate answers: it can contain incorrect answers and depart out accurate ones. This misspecification opens up regions of $\Scal_x$ that the example power study to district in equal although it’s not what we intended.

Thus reward-hacking comes from an imperfect verifier, which fails to grasp what we really want, as fine as an imperfect sampling procedure which fails to remain distant from these shortcut regions.

In the fig below, the dashed circle is the set of accurate answers and the red oval is $\Scal_x$. The oval stretches rear toward the prompt, so its incorrect answers are the easiest to reach, and following training most paths district there.

This visualization additionally shows why scaling up RL (generating additional samples and longer samples) makes reward hacking additional likely. The additional extensively we “fuzz” the verifier, the additional apt we’ll discover a way that lands in the misspecified area of $\Scal_x$. Once the example finds this shortcut, training pushes the example towards akin behavior.

So what goal is reasoning solving?#

Reasoning isn’t really solving any goal at all. A way is not guaranteed to converge to the accurate answer. It’s lone in aggregate, complete the allocation of paths, do we get a elevated probability of success.

RL training immediately raises the achievement probability $p_x$, by spending a lot of compute on searching for prosperous paths and afterward training the example to reproduce them. The gains we credit to “thinking longer” arrive from the hunt we paid for during training, fairly than at inference. Longer chains of thoughts lean to correlate alongside achievement during training, which afterward gets trained into the model. The disbursal of the sampling is amortized into the weights, so at conclusion period the example is apt to win alongside a sole sample, but any idiosyncratic example is not guaranteed to succeed.