The whole of PyTorch on one page

Loss, Death, Robots, Part 0

1. The Map (this part)
2. Tensor
3. Autograd
4. Daily PyTorch
5. Seeing PyTorch
6. The Machinery
7. Extending PyTorch
8. The Compiler
9. Kernels & Hardware
10. Distributed
11. Ship It
12. Working connected PyTorch

The full of PyTorch connected 1 page.

August 07, 202636 min

Table of Contents

• The fall
• The territory
• The 12 ideas
• How this bid draws
• How to publication this
• What you tin now say
• Try it yourself
• Pick a door
• References

Figure 1. the programme this full bid is about.

You person typed thing for illustration this a 1000 times. This bid exists truthful that, by its end, you cognize everything these lines do. All of it: the Python they touch, the C++ they onshore in, the chart they record, the kernels they choose, the representation they use, and the 2 clocks they tally on. Each of those words gets a plain meaning connected its level below.

This is Part 0, the map. First we spell down done each the layers once, fast. Then we tie the territory. Then 12 ideas that make the remainder of the codebase predictable. Then really this bid works, and really to publication it. Nothing present gets its afloat story. Everything present gets a place, and each afloat communicative has a numbered portion waiting for it.

One committedness earlier we start. Every measured number successful this bid comes from a mini book you tin tally yourself, linked correct wherever the number appears. I measured these connected an Apple M3 Max laptop pinch torch 2.11.0 [1]. Your numbers will differ. The shape they make will not.

The fall

PyTorch is deep. Between your keyboard and the spot location are 8 levels. I will telephone them floors, and this metre shows each of them. It returns done the full series, truthful you ever cognize really heavy you are.

Figure 2. the extent meter. the orangish dot marks wherever you are.

The fastest measurement to study a building is to spell down done it erstwhile without stopping. That is this section.

Floor one: python

torch.randn looks for illustration a Python function. Ask Python what it really is:

>>> type(torch.randn) <class 'builtin_function_or_method'>

Python gives that type only to functions written successful compiled code. Compiled codification means: codification that was translated to instrumentality instructions earlier you ever installed it, truthful location is nary Python assemblage wrong it to read, and nary statement for your debugger to extremity on.

So wherever do those instrumentality instructions live? In shared libraries. A shared room is simply a record of compiled codification that a programme loads while it runs. They beryllium wrong the torch package connected your disk, and you tin look astatine them (proof):

torch._C -> _C.cpython-312-darwin.so (49 KB, the loader) libtorch_cpu.dylib 206.5 MB (tensors and kernels) libtorch_python.dylib 28.5 MB (the python broadside of the border) p0_the_library.py the proof, fresh to publication aliases run

Read the sizes, and past look astatine them:

Figure 3. drawn to standard by record size. the portion of pytorch that python tin spot is the orangish dot.

The portion of PyTorch you tin spot from Python is simply a 49 KB record whose only occupation is to load the different two. The existent assemblage is 235 MB of compiled code. import torch brings it into your process, and aft that, calling torch.randn intends jumping into that body. Today we only request to cognize these files exist.

This is the first honorable astonishment of the codebase: the Python you constitute each time is the smallest furniture of it.

The boundary

The telephone leaves Python astatine once. Where does it land?

In a C++ usability named THPVariable_randn, wrong that 28.5 MB room from the past floor. And present is simply a unusual truth you tin keep: this usability does not beryllium successful the PyTorch repository. Clone the repo, hunt for the name, and you find nothing. A programme writes this usability during the build, together pinch thousands of its siblings. Idea 4 beneath explains why, and Part 5 shows the programme that does the writing.

Figure 4. the separator betwixt the 2 languages. each tensor cognition crosses it.

Crossing this separator costs time. To spot the costs alone, clip the smallest imaginable operation, wherever almost nary arithmetic hides it (proof):

add, 1 constituent : 0.538 microseconds per call add, 4M elements : 337.264 microseconds per call p2_dispatch_cost.py the proof, fresh to publication aliases run

The one-element adhd does almost nary math. So its 0.54 microseconds is almost axenic crossing cost: time off Python, cheque the arguments, build the consequence object, return. Half a microsecond sounds for illustration nothing. It intends Python tin rumor astatine astir about 1.9 cardinal operations per second, and a azygous training measurement contains thousands of operations. Keep this number. It returns successful Idea 6.

The dispatcher

Under the border, the telephone reaches the strangest instrumentality successful PyTorch: the dispatcher. The dispatcher is the router that decides, for each operation, which pieces of codification tally and successful what order.

Look astatine what it must decide. Your 3 lines ne'er said "record gradients". No if connection successful your codification turns that on. Yet somewhere, thing decided that this matrix multiplication should beryllium remembered for backward(). That thing is the dispatcher. Every cognition passes down done a fixed stack of layers. Each furniture tin enactment connected the call, alteration it, aliases fto it walk unchanged. Autograd, the portion of PyTorch that computes gradients, is 1 specified layer. Mixed precision is another. On this run, only autograd is awake.

Figure 5. 4 layers touch your telephone earlier immoderate arithmetic starts. only the highlighted 1 is awake today.

The kernel

At the bottommost of the stack, 1 actual usability is chosen. Chosen is the correct word. This torch build has 3,677 registered cognition names (proof prints the count), and a sanction is not a usability body. The cognition addmm, the matrix multiplication down model(x), has abstracted bodies for CPU and for each benignant of GPU, for each information type, for dense and for sparse tensors. A assemblage for illustration this, written for 1 instrumentality and 1 information type, is called a kernel. The dispatcher's past occupation is to prime one:

Figure 6. 1 name, a grid of bodies. the dispatcher picks precisely 1 compartment per call.

p4_micro_proofs.py the proof, fresh to publication aliases run

The afloat database of operations lives successful 1 record successful the repository: native_functions.yaml [2]. Its related derivatives.yaml [3] lists the derivative of each operation. Everything other grows from these 2 files. No different record successful the repository tells you arsenic overmuch per line.

Figure 7. 3,677 names connected the left. 1 usability assemblage connected the right. the chimney is the dispatcher's past job.

One level down sits memory. torch.randn(64, 128) needs 32,768 bytes: 64 rows times 128 numbers times 4 bytes per number. On the CPU this is an mean allocation. On a GPU it is not. There, PyTorch runs its ain allocator, a keeper of representation that asks the GPU driver for ample blocks erstwhile and past reuses them, because asking the driver each clip is slow. This allocator decides erstwhile you tally retired of representation and what the correction means. Part 4 examines it.

The 2 clocks

Here the communicative splits successful two. What follows is the azygous astir useful capacity truth successful PyTorch.

On a GPU, your Python statement does not do the work. It requests the work, and the petition returns astatine once. The GPU does the activity connected its ain clock, while Python continues. I measured it connected this machine's GPU (proof):

time to petition 50 matrix multiplications : 1.58 ms time until the activity was really done : 73.83 ms python was free during : 72.25 sclerosis (98%) p3_two_timelines.py the proof, fresh to publication aliases run

Figure 8. 2 clocks, 1 program. the cpu requested everything successful the first 2 milliseconds; the gpu needed seventy-two much to finish.

Python asked for each 50 multiplications successful nether 2 milliseconds, past waited, free, while the GPU computed for different seventy-two. On the CPU location is nary specified split; the mathematics happens earlier your statement returns. On immoderate accelerator, the divided is the normal authorities of the program.

This is why eager PyTorch is accelerated capable to use: Python runs up and the GPU ne'er waits for it. It is besides why elemental timing codification gives incorrect answers, and why 1 loss.item() wrong a training loop tin slow the full step. .item() needs the existent number. The number sits astatine the extremity of a queue of activity the GPU has not vanished yet, truthful Python must extremity and hold for the full queue:

Figure 9. the queue betwixt the 2 clocks. the number python asked for is the past ticket, truthful each summons up of it must decorativeness first.

Part 4 teaches honorable measurement connected apical of precisely this picture.

The turn

Line three: loss.backward(). Nothing truthful acold explains really this statement tin work. The guardant computation is over. How does PyTorch cognize what to differentiate?

It knows because the guardant walk had a 2nd job. Every clip an cognition passed the autograd furniture of the dispatcher, a mini grounds was written: which cognition ran, and which recorded steps produced its inputs. Records that constituent astatine records shape a graph, and that connection present ever intends precisely this recorded structure. By the clip nonaccomplishment exists, its chart exists excessively (proof):

loss.grad_fn = SumBackward0 SumBackward0 AddmmBackward0 AccumulateGrad p1_graph_chain.py the proof, fresh to publication aliases run

Figure 10. the guardant walk goes down and writes the graph. backward climbs precisely what was written, and thing else.

backward() invents nothing. It walks the chart from the nonaccomplishment backmost to your inputs, runs each recorded derivative, and stores the results successful .grad. The locomotion ends astatine AccumulateGrad, the grounds that does the storing. And it starts obscurity else: x.grad_fn is None, because x was created directly, not computed.

One mobility should fuss you here. A derivative needs values. The derivative of a matrix multiplication needs the matrices that were multiplied, and the guardant walk is agelong over. Write the derivative retired and the request is visible:

Figure 11. the derivative of x @ w, written out. the formulas incorporate x and w themselves; immoderate guardant used, backward needs again.

So wherever are they? They were saved, adjacent to the records, during the guardant pass:

Figure 12. what each grounds kept. this is wherever the representation of a training tally really goes.

So the guardant walk silently decides really overmuch representation training costs. Part 2 shows the nonstop redeeming rules. Part 4 shows really to watch it happen. And a method called activation checkpointing trades that representation for other compute; it has its ain section successful Part 2.

Carry 1 condemnation retired of this section: backward tin only locomotion what guardant wrote. It sounds small. In Part 9 it becomes the norm that decides which GPUs successful a cluster must talk to each other.

That was the full fall: a name, a border, a stack of layers, a chosen kernel, a keeper of memory, 2 clocks, and a chart that is walked backward. Now the territory, properly.

The territory

PyTorch is built successful layers, and each furniture speaks only to its neighbors. Every container beneath is astatine slightest 1 portion of this series.

Figure 13. the full strategy connected 1 sheet. the orangish statement connected the near is the way we conscionable took.

The aforesaid territory, seen arsenic folders successful the repository. If you ever unfastened the codebase, this is the representation that stops you from being lost:

Figure 14. the repository arsenic 2 stream banks. the stream is the bound from fig 4; torch/csrc/ is its 1 bridge; and some banks guidelines connected the aforesaid ground, c10/, wherever Tensor and Storage themselves live.

Three facts astir this representation prevention you weeks. First: torch/ is plain Python, and you tin publication each record successful it today. Second: aten/ and c10/ are C++; the tensors, the kernels and the dispatcher unrecorded there, and torch/csrc/ is the azygous span that connects the 2 languages. Third: torchgen/ is the programme from the bound floor, the 1 that writes codification during the build. The repository you publication is the input. The room you tally is the output. That is why searching the repository for THPVariable_randn finds nothing:

Figure 15. the repository is the portion supra the waterline. the codification your process runs is the larger portion beneath it.

Above the halfway sits the ecosystem. It looks endless, but it has a elemental shape: each room attaches to PyTorch astatine a specific, nameable place. Know the attachment places and you cognize the ecosystem.

Figure 16. each statement points astatine the nonstop spot a room attaches. trl and peft beryllium connected the outer ring: they build connected transformers, not connected pytorch.

Read the image from the halfway out. transformers builds its models arsenic nn.Module classes, truthful if you understand Part 3, you tin publication its source. deepspeed replaces the distributed engine, truthful its location is Part 9. vllm and sglang support the exemplary weights and switch the runtime astir them. And trl and peft do not touch PyTorch straight astatine all; they build connected transformers.

The full ecosystem fits successful 1 table. The 2nd file names the spot successful the PyTorch repository wherever each family attaches:

Two rows of the array merit pictures. The first is the eager runtime, the point the serving engines replace:

Figure 17. 2 ways to tally 1 model. the engines switch the loop successful the middle; the crushed is shared.

The 2nd is Triton, the kernel connection that appears done the full table: PyTorch's compiler writes it, and libraries bring their own:

Figure 18. triton and pytorch. the compiler writes triton itself; hand-written kernels tally connected torch tensors and subordinate the list.

Read the array downward and little of PyTorch survives each row. The past statement keeps thing but the weights file. That is the quiet rule of the ecosystem: the weights outlive the runtime. Part 10 walks these attachment points 1 by one.

The 12 ideas

Most of PyTorch is not thousands of abstracted decisions. It is simply a mini group of ideas, applied everywhere. These 12 make the remainder of the codebase predictable earlier you publication it. Each 1 returns later arsenic a afloat section aliases part. Each 1 comes pinch runnable grounds now; the mini proofs stock 1 book (proof).

1. A tensor is simply a model complete storage

A tensor does not clasp numbers. It holds a explanation of wherever to look: a pointer into 1 level artifact of memory, the sizes of each dimension, and the strides. A stride is the number of steps to move successful that level artifact to scope the adjacent constituent of a dimension. Two tensors tin look wholly different and publication the aforesaid bytes:

>>> x = torch.arange(6.); v = x.view(2, 3) >>> x.data_ptr() == v.data_ptr() True >>> v.stride(), v.t().stride() ((3, 1), (1, 3))

The transpose moved nary data. It swapped 2 numbers successful the description. Some descriptions are intolerable to constitute down, and that is precisely why v.t().view(-1) raises an correction while reshape silently copies the information instead. Part 1 opens pinch this puzzle and solves it completely.

2. Autograd records a programme you ne'er wrote

In your code, y is 1 name, and you overwrite it freely. Line 2 destroys the worth statement 1 made. The chart cannot spend that: backward will request each step. So autograd writes 1 grounds per change, and nary grounds is ever overwritten. Watch the grounds alteration arsenic y does:

>>> a = torch.ones(3, requires_grad=True) >>> y = a * 2 >>> type(y.grad_fn).__name__ 'MulBackward0' >>> y.add_(1) >>> type(y.grad_fn).__name__ 'AddBackward0' >>> y[0] = 9 >>> type(y.grad_fn).__name__ 'CopySlices'

Figure 19. your codification keeps 1 worth and destroys the past. the chart keeps each step: 1 grounds per change, thing overwritten.

Three statements, 3 records, 1 chain. y.grad_fn ever holds the newest record, and each grounds points astatine the 1 earlier it, truthful the full history stays reachable. That history is the programme you ne'er wrote. The machinery that keeps it correct nether each benignant of in-place alteration has existent depth, and it is 1 of the champion chapters of Part 2.

3. One database of operations is the full interface

Figure 20. the codification connected the near ne'er reaches a device. only the database does.

The 3,677 registered names are PyTorch's existent interface. Each backend implements its stock of them. The compiler rewrites programs made of them: torch.compile sounds the database your programme became and returns a shorter one, wherever a matrix multiplication and the adhd aft it tin fuse into 1 addmm, and a concatenation of mini elementwise operations becomes 1 generated kernel. Export formats shop them: torch.export writes the database to disk arsenic a chart of precisely these names, and an ONNX record is the aforesaid thought pinch each sanction translated into ONNX's vocabulary. Quantization replaces them: the float32 matmul is swapped for an int8 body, the aforesaid spot connected the list, different arithmetic. When you meet a caller PyTorch technology, inquire 1 mobility first: what does it do to the operations? The reply usually explains the full design.

4. PyTorch writes astir of its ain code

Figure 21. 1 yaml record in, thousands of functions out, astatine each build.

native_functions.yaml declares each operation. derivatives.yaml declares each derivative. At build time, torchgen/ sounds some and writes the Python bindings, the autograd grounds classes and the dispatcher tables. This is why searching the repository for a usability you conscionable called tin find nothing: you searched the input of the build, and the usability is successful the output. People who activity connected PyTorch publication the yaml first. After Part 5, truthful will you.

5. Features are layers pinch a switch

Figure 22. each characteristic watches the aforesaid watercourse of operations. a discourse head puts 1 furniture to sleep.

PyTorch's features harvester cleanly because each 1 is simply a furniture successful the dispatcher, watching the aforesaid watercourse of operations:

>>> with torch.no_grad(): ... z = a * 2 >>> z.grad_fn is None True

no_grad edited nary function. It group a emblem that sends operations past the autograd layer, truthful thing gets recorded. Mixed precision, tracing and vmap activity the aforesaid way, and that is why they tin beryllium mixed without knowing astir each other. Part 5 opens the machinery nether the flag.

6. Every cognition pays a fixed costs first

Half a microsecond of crossing and routing earlier immoderate math, connected each azygous operation. That was the measurement connected the bound floor. Applied honestly, this 1 number explains why torch.compile exists, why fused optimizers exist, and why the first mobility astir immoderate slow exemplary is: is it constricted by compute, aliases by the costs of issuing galore mini operations?

7. Memory, not speed, is what kills training runs

Figure 23. the guardant walk borrows memory. backward repays it. moving retired is the astir communal measurement a training tally dies.

A slow programme still finishes. A programme that runs retired of GPU representation dies pinch CUDA retired of memory, and that is the astir communal decease successful each of PyTorch. The guardant walk saves values for backward (the turn, above). The allocator keeps and reuses blocks. Between them they determine really ample a exemplary you tin train. This bid treats representation arsenic a first-class taxable successful Part 4.

8. Python is why it won, and what it costs

Figure 24. the protection and the value are the aforesaid picture: everything must transverse 1 bridge.

PyTorch won because you constitute it successful mean Python, pinch mean debuggers and people statements. The value is the separator costs from Idea 6, paid connected each operation. The history of the model is simply a series of attempts to support the first while reducing the second. TorchScript tried to switch Python pinch its ain language; it is now successful attraction mode [4]. The existent compiler watches your Python tally and translates what it can, and it is winning. The shape to remember: wrong PyTorch, betting against Python has ever lost.

9. Forward decides what backward must do

Figure 25. backward is the reflection of the chart guardant wrote.

Backward tin only locomotion what guardant wrote. On 1 instrumentality this sounds for illustration a detail. At standard it becomes the rule of the land: successful distributed training, the measurement a tensor is divided crossed GPUs successful the guardant walk decides which GPUs must speech information successful the backward pass. One idea, from a laptop to a cluster. It is the spine of Part 9.

Figure 26. 3 windows, 1 storage, 1 write. each strategy that records aliases rewrites programs must grip this.

Idea 1 lets galore tensors publication the aforesaid bytes. Idea 2 lets you alteration those bytes successful place. Combine them: 1 constitute tin alteration the meaning of respective tensors astatine once, and immoderate strategy that records programs (autograd, the compiler, export) must announcement and enactment correct. When a area of PyTorch looks strangely complicated, inquire what shared bytes positive an in-place constitute would do to it. That is usually the answer.

11. The codification keeps its history

Figure 27. 4 systems, 4 eras, 1 repository. older layers still show through.

The repository holds the remains of each era: the original C codification from 2016, the Caffe2 merge of 2018, TorchScript from 2019, the compiler territory increasing since 2023. When a record looks strange, the mentation is usually historical: thing older lived location first. Part 5 tells this history wherever it explains the present.

12. Floating constituent is simply a contract; publication it

Figure 28. the position are public. each training tally signs them.

>>> t = torch.tensor(1e8) >>> ((t + 1) - t).item() 0.0

A float32 number has astir 7 decimal digits of precision, truthful adding 1 to 1 100 cardinal changes thing [5]. This is not a bug; it is the number format doing what it promises. Add the faster, little precise formats utilized successful training, positive the truth that immoderate GPU kernels sum successful different orders connected different runs, and "why did my nonaccomplishment alteration betwixt runs" becomes a mobility pinch nonstop answers. Part 4 sounds this statement clause by clause.

How this bid draws

Every still fig you conscionable saw is simply a existent Excalidraw scene, and the segment files vessel pinch the series; you tin unfastened immoderate drafting and edit it. The 4 instruments you tin run travel the aforesaid language, and each number wrong them comes from the impervious scripts. All of them speak 1 ocular language, truthful that by Part 2 you publication them without thinking:

Figure 29. the full notation connected 1 sheet. study it once; it holds for the full series.

Orange ever marks the subject: the 1 point moving. Ink is structure. Grey is context. Dashed intends recorded, implied, aliases asleep. The extent metre marks the floor. And the robot appears astatine astir erstwhile per part, because a mascot that is everyplace stops being funny.

How to publication this

Twelve parts. Each is 1 agelong page for illustration this one. And the bid has 1 quiet extremity down each part: by the end, you should cognize the instrumentality good capable to build a mini PyTorch yourself. Every drafting that shows a mechanism, each look adjacent to a figure, and each impervious book is simply a portion of that.

PartWhat is down the door

0. The Map	you are here
1. Tensor	storage, strides, views, information types, broadcasting
2. Autograd	the graph, in-place writes, checkpointing, double backward
3. Daily PyTorch	nn, optim, information loading, mixed precision, seen from inside
4. Seeing PyTorch	the profiler, memory, floating point, honorable measurement
5. The Machinery	the dispatcher, aten, torchgen, the history
6. Extending PyTorch	subclasses, civilization operations, caller backends
7. The Compiler	dynamo, aot autograd, inductor, move shapes
8. Kernels & Hardware	the gpu model, triton, cutlass, what accelerated means
9. Distributed	collectives, ddp, fsdp, dtensor, parallel training
10. Ship It	export, quantization, executorch, the ecosystem
11. Working connected PyTorch	the contributor's section guide

You do not person to publication beforehand to back. Three reference lines tally done the parts, for illustration lines done stations:

Every section wrong each portion follows the aforesaid 7 steps, truthful the hit becomes acquainted fast:

Figure 30. the 7 steps of each chapter. the impervious measurement is the spine: nary declare without a script.

And the method, stated plainly, because you should cognize what you are trusting. Every system declare is checked against the root codification aliases shown by a book earlier it is published. The scripts are linked successful spot and pinned to 1 torch version. When PyTorch moves and a declare goes stale, the section is corrected and the correction is noted connected the page. A bid astir internals that cannot admit drift would beryllium incorrect wrong a year.

What you tin now say

Test yourself against this list. After 1 reference you should beryllium capable to say, successful your ain words:

• what type(torch.randn) returns, and wherever the compiled assemblage really lives connected your disk
• what the dispatcher is, and really no_grad stops autograd without editing immoderate function
• what a kernel is, and what decides which 1 runs
• why the CPU and the GPU tally connected 2 clocks, and why that makes elemental timing codification lie
• what the chart is, who writes it, and why backward tin ne'er do thing guardant did not constitute down
• and the 12 ideas, each successful 1 sentence

If 1 of these is fuzzy, return to its floor; each 1 is only a infinitesimal long. That is what this page is for.

Try it yourself

The 5 impervious scripts are the exercises. For each one: foretell the output first, past tally it, past explicate the difference.

1. p0_the_library.py: really large are the compiled libraries successful your ain torch install?
2. p1_graph_chain.py: what chart remains aft a two-layer exemplary runs?
3. p2_dispatch_cost.py: what is the fixed costs per cognition connected your machine?
4. p3_two_timelines.py: really agelong is your GPU still moving aft Python is done asking?
5. p4_micro_proofs.py: the 12 ideas, compressed into 5 mini experiments.

Pick a door

Part 1 is the tensor. It opens pinch the puzzle from Idea 1, and now you person seen the correction pinch your ain eyes:

>>> v.t().view(-1) RuntimeError: position size is not compatible with input tensor's size and stride ... >>> v.t().reshape(-1) tensor([0., 3., 1., 4., 2., 5.])

Same tensor. Same request. One statement refuses, the different softly copies the data. The quality betwixt those 2 lines is the full first portion of this series.

See you connected the adjacent level down.

References

[1] Khalilli, five impervious scripts, measured connected an Apple M3 Max, torch 2.11.0, CPU and Apple GPU, 2026. Linked successful spot above; rerun them to cheque me.

[2] PyTorch source, native_functions.yaml, pinned to the v2.11.0 tag. https://github.com/pytorch/pytorch/blob/v2.11.0/aten/src/ATen/native/native_functions.yaml

[3] PyTorch source, derivatives.yaml, pinned to the v2.11.0 tag. https://github.com/pytorch/pytorch/blob/v2.11.0/tools/autograd/derivatives.yaml

[4] PyTorch documentation, TorchScript, which states it is successful attraction mode. https://docs.pytorch.org/docs/stable/jit.html

[5] IEEE, 754 azygous precision: 24 binary digits of precision, astir 7 decimal digits.

Three bully things to publication aft this page: Edward Yang's PyTorch internals talk, which maps the C++ broadside successful depth; the PyTorch Developer Podcast, short episodes by the aforesaid author; and the repository's ain CONTRIBUTING.md, which describes the files layout successful the maintainers' words.

---

‹Floating Point