I have written a lot concerning reasoning models current complete the final two years on Ahead of AI, including a latest part on mechanisms to control reasoning attempt levels in contemporary LLMs. Reasoning models are motionless as applicable as ever. In fact, nearly all current-generation LLMs today are reasoning models or recommendation a reasoning mode.
For those who are curious in how reasoning models work, including conclusion scaling, reinforcement learning, and distillation, I wrote my latest Build Reasoning Model (From Scratch), which implements these from the dirt up in PyTorch, which, in my experience, is motionless the finest way to comprehend and study how things really work.
Now, in the final few months, I additionally sat downward and recorded accompanying lectures I wanted to portion alongside you. So, below, you volition discover eight reasoning model-from-scratch coding videos (11 hours of content).
We commencement alongside a pre-trained LLM and execute methods to measure and enhance it in Python and PyTorch. So, if you enjoyed my Coding LLMs from the Ground Up way final year, this is basically the sequel (but you can additionally observe it without the previous way as a prerequisite).
I’m currently taking a abbreviated interrupt and volition be rear shortly alongside additional breathtaking LLM investigation roundups next month. In the meantime, I anticipation you bask coding along!
As a small flank note, and since I frequently get asked how reasoning models connect to today’s AI agents, here’s a abbreviated division on this before getting started alongside the videos.
A reasoning example is basically an upgrade from a accepted LLM, that is, adding additional post-training via reinforcement learning alongside verifiable rewards (and optionally conclusion scaling techniques) to create it additional capable of solving additional challenging problems. Classically, reasoning models are developed for math and code, but the capabilities can generalize to another issue domains as well.
A reasoning example afterward functions as the motor inner an delegate harness, anywhere the delegate harness is a application or tooling example about the LLM/reasoning example to tackle equal additional challenging problems (bigger coding tasks, journey booking, and all another sorts of individual aide tasks). The project of the harness is advanced environment administration and giving the LLM admission to tools.
Figure 1: An motor difference for accepted LLMs, reasoning models, and delegate harnesses. Figure from my Components of a Coding Agent article.
In this difference depicted in the fig above, the example is the engine, and the delegate harness adds the machinery that lets us use it for assorted things (as an upgrade from a uncomplicated chatbot interface). An delegate can use either a accepted LLM or a reasoning model, but reasoning models are the additional representative choice as they are additional capable. (As mentioned above, most current-gen LLMs are reasoning models alongside changeable reasoning attempt settings.)
So, the base row is that reasoning models, as the “engine,” remain an crucial part of understanding contemporary LLM agents.
The resources accompanying these topics shown complete are:
The Build a Large Language Model (From Scratch) book and videos
The Build A Reasoning Model (From Scratch) book (and videos below)
The Components of a Coding Agent article on Substack
The videos below concentration on evolving the reasoning example that can power specified a system.
The reasoning-from-scratch videos concentration on a small pre-trained Qwen3 model, anywhere we execute the reasoning methods ourselves in Python and PyTorch. Starting alongside a pre-trained example lets us concentration on the engaging part: evaluation, inference-time scaling, reinforcement learning, and distillation.
Regarding the prerequisites, I propose being comfortable alongside Python and PyTorch. I designed the reasoning materials as standalone content, so the former Build A Large Language Model (From Scratch) satisfied is recommended but not strictly required.
Compute-resource-wise, you can activity through the first text-generation and evaluation examples on a CPU. For the afterward experiments, particularly training, a GPU makes the activity much additional practical. The archetypal video covers the setup and several additional recommendations; the repository additionally includes hardware and surroundings recommendations.
Figure 2: An overview of the topics being covered.
These videos escort my book, Build a Reasoning Model (From Scratch). If you akin to peruse through the clarification in additional item and at your own pace, activity through exercises, and so on, I propose the publish as a companion here.
For the most in-depth learning experience, my recommendation is to peruse a chapter, activity through its code alongside the corresponding video, and afterward try the exercises.
The publish is accessible from Manning and Amazon, and the imprint type is in color!
Figure 3: A random excerpt from the Build a Reasoning Model (From Scratch) imprint book.
Thank you to everyone who has picked up a copy or recommended it to a friend. Your assistance helps me keep creating these materials.
As a last, quick flank note before getting to the chief videos below: For a shorter, additional conceptual overview, I additionally propose my AI Reasoning Models way on LinkedIn Learning (it have to be liberated for 24h whenever accessing it via the nexus there, which shouldn’t afterward necessitate a LinkedIn Premium subscription).
The way is concerning 90 minutes and explains how reasoning models work, covering the chief ideas rearward inference-time scaling, reinforcement learning, and distillation. It’s a fine optional preview if you’d akin the chief ideas without operating through the complete implementations below.
To get to the chief part of this article, current are the eight chief address videos.
The archetypal video starts alongside an clarification of the association between accepted LLMs, reasoning models, and agents. Next, the chief topic of the video is environment up the coding environment. This includes installing the Python and PyTorch requirements (with uv), among others. I additionally conversation hardware choices and average setup issues.
Topics covered
The association between accepted LLMs, reasoning models, and delegate harnesses
On the benefits of learning through code implementations
The range of this video series
Installing Python dependencies
Choosing hardware and checking CUDA or Apple Silicon support
Using a distant GPU alongside VS Code and troubleshooting the environment
Watch video 1 · Setup instructions
Before adding reasoning methods in the upcoming videos, we need a operating basis model. Instead of training it from scratch, since that would be a entire another publish (see Build A Large Language Model (From Scratch)), we commencement alongside a pre-trained model.
Here, we burden the pre-trained weights of a Qwen3 0.6B model. Then, we execute the text-generation procedure via a streaming content generator and add KV caching and example compilation to create content generation additional efficient.
These implementations provision us the baseline and basis components we volition measure and enhance in the following videos.
Topics covered
Encoding content as tokens and decoding it rear into text
Downloading Qwen3 weights and loading the example on the selected compute device
Going complete tensor shapes, logits, and next-token predictions
Implementing greedy decoding and streaming content generation
Handling end-of-sequence tokens and measuring generation speed
Adding a KV cache to reuse notice keys and values
Combining KV caching alongside torch.compile and comparing hardware performance
Watch video 2 · Chapter 2 code
In this lecture, we build a math-answer verifier stage by step, from answer extraction to normalization and checking mathematical equivalence. We afterward use it to measure the example on MATH-500.
We use this verifier to measure the basis model’s achievement on the MATH-500 evaluation dataset.
However, the verifier additionally has a second job afterward in the series: providing the reward indication for reinforcement learning alongside verifiable rewards (RLVR).
Topics covered
Intro to verifiers and how they fit into LLM evaluation methods (other methods are covered in the book’s appendix)
Reusing answer verification as a reward indication for RLVR (more on that in division 6)
The verifier flow, including extracting final answers, normalizing answers and checking mathematical equivalence
Building an answer grader and an evaluation iteration for MATH-500
Comparing basis and reasoning models on MATH-500
Watch video 3 · Chapter 3 code
Now we examine ways to enhance answers during keeping the example weights fixed. But before implementing particular inference-scaling techniques, we archetypal execute methods to create varied answers from the identical immediate (this is norm custom in proprietary LLM chatbots, specified as ChatGPT).
Concretely, we execute heat scaling, multinomial sampling, and top-p filtering to create varied responses. Then we merge multiple responses through self-consistency, using bulk voting complete their final answers.
We additionally analyze the tradeoff between answer accuracy and the additional computation needed to create additional candidates.
Topics covered
Comparing how additional compute benefits example achievement during training and during inference
A gentle intro to inference-scaling through chain-of-thought prompts
Implementing heat scaling and multinomial sampling to create varied answers
Adding top-p filtering to authority the norm of the varied answers
Combining applicant answers through self-consistency, among the most famous conclusion scaling techniques
Evaluating MATH-500 accuracy against the disbursal of additional conclusion compute
Watch video 4 · Chapter 4 code
Building on the former video, we execute scoring functions to difference multiple answers generated from the identical prompt. We can additionally use these scoring methods to interrupt ties in self-consistency (covered in the former lecture).
The scoring methods covered current contain computing token and sequence log probabilities (this volition additionally be applicable in the afterward reinforcement learning chapters). We afterward execute a self-refinement iteration in which the example critiques an answer and attempts a revision.
The evaluation lets us inspect how helpful those additional steps are for our model.
Topics covered
Using scoring functions for comparing multiple answers and breaking ties
Implementing a uncomplicated heuristic scorer
Computing token probabilities and sequence likelihoods (also applicable for the following lectures)
Using log probabilities for improved numerical stability
Building a self-refinement iteration for generating critiques and revised answers
Evaluating self-refinement on MATH-500
Watch video 5 · Chapter 5 code
In this lecture, we move on to implementing a training method called Reinforcement Learning alongside Verifiable Rewards (RLVR). We archetypal execute a simplified type of Group Relative Policy Optimization (GRPO), create groups of responses, mark them alongside our verifier, and use their related rewards for the example importance updates during training.
Topics covered
Generating reasoning traces and understanding the association between answer lengths and problem-solving abilities
Discussion and difference of two crucial LLM post-training methods: Reinforcement Learning alongside Human Feedback (RLHF) and Reinforcement Learning alongside Verifiable Rewards (RLVR)
Turning verifier scores into rewards and related advantages
Computing sequence log probabilities and a simplified GRPO loss
Assembling the training iteration alongside logging and checkpoint saves
Evaluating checkpoints and checking training stability and recollection needs
Watch video 6 · Chapter 6 code
Now that we implemented the training iteration in the former lecture, we can revisit it and analyze the training runs in additional detail.
Concretely, this video adds additional training metrics for inspection and analysis, including entropy and advantage statistics, clipped guideline ratios, KL defeat term values, and formatted rewards.
We afterward assessment the training logs and evaluation results to comprehend what these additions do and whether additions specified as clipped guideline ratios and KL defeat conditions are helpful.
Topics covered
Running longer training experiments and interpreting their logs
Tracking advantages and entropy to comprehend training behavior
Implementing clipping in the guideline defeat and analyzing its effects
Adding a KL defeat term and discussing reward hacking
Understanding restrictions of the simplified KL formulation
Implementing format rewards and conditioning them on answer correctness
Comparing checkpoints on MATH-500 and exploring additional GRPO research
Watch video 7 · Chapter 7 code
Finally, we examine another way to train a small reasoning model, that is, learning from answers generated by a stronger (and bigger) instructor model. We conversation distinct forms of distillation, prepared reasoning data, and execute the supervised training loop.
We complete this address and way by evaluating the distilled example on MATH-500 and comparing the results. This rounds out the sequence alongside an substitute to evolving reasoning capabilities through reinforcement learning.
Topics covered
Comparing the distinct blend and approaches to distillation, including gentle and difficult distillation, and on-policy and off-policy distillation
Creating and collecting teacher-generated answers for the MATH dataset (using DeepSeek R1)
Preparing the distillation dataset, including tokenization and training/validation/test splits
Computing answer-token log probabilities and understanding the cross-entropy loss
Implementing supervised training and tracking training and validation losses
Evaluating checkpoints on MATH-500 and comparing distillation results
Watch video 8 · Chapter 8 code
As you observe these lectures, I advance you to intermission occasionally and try a small test of your own if item engaging comes to mind.
For example, awareness liberated to alter the sampling settings, inspect a few unsuccessful answers, or try variations of distinct ideas akin to the suggestions in the exercises.
And if you would akin the complete written walkthrough, figures, and exercises alongside the videos, you can discover Build a Reasoning Model (From Scratch) at Manning and Amazon.
Thanks for study and supporting Ahead of AI. I am taking a abbreviated interrupt to regenerate, and I appearance onward to being rear alongside the next circular of LLM investigation articles next month.
In the meantime, blessed viewing and tinkering!
