PyTorch: A Reference Language

Edward Z. Yang (@ezyang) · July 25, 2026 · 4 min read
compilertorch.compileautogradverificationllm

A reference implementation is simply a simplified but complete type of a system that trades capacity successful return for clarity. We mightiness past opportunity a reference “language” is the cloth of APIs and conventions from which these implementations are cut. At first glance, PyTorch evidently is simply a reference language: it is, aft all, commonly called the lingua franca of modern deep learning. But upon a person look, location is confusion:

• Reference implementations usually aren’t deployed to production. But I do my training jobs pinch PyTorch!

• Everyone’s penning kernels pinch kernel DSLs. What is the domiciled of PyTorch if it’s conscionable gluing kernels together?

• AI coding will yet mean that immoderate stack tin beryllium rewritten from scratch; why does being written successful PyTorch matter?

So for me, recently, an unusually clarifying position has been to deliberation of PyTorch arsenic playing a dual role: arsenic some the reference connection and the implementation language. When the standard is not excessively ample aliases the compiler is working well, the reference implementation tin vessel to production. But increasingly, I deliberation it will beryllium much and much earthy to deliberation of the reference implementation arsenic a package artifact that stands isolated from the actual accumulation implementation, by which we tin verify the correctness of the accumulation implementation. One implementation to investigation in, one implementation to standard with, and 1 verifier to, successful the darkness, hindrance them.

The clearest objection of this is successful the modern usage of kernel DSLs. The traditional, compiler-maximalist position argues that extremity users should write implementations of NN modules utilizing a precocious level API (e.g., a Numpy/PyTorch-style API) which a compiler past determines really to compile into an optimized form. But for the astir important operations for illustration matrix multiplies and attention, it is not easy for compilers to guarantee peak performance; the proliferation of kernel DSLs has made it dramatically simpler for group to execute optimal capacity by explicitly pronunciation out tiling and information movement. Does this destruct the precocious level API? Usually not: it’s beautiful useful to person a reference implementation successful plain PyTorch, and astir kernel authors will support 1 successful parallel pinch the optimized kernel, verifying correctness pinch numerical tests.

In the aforesaid measurement kernel DSLs person changed really accumulation implementations of operators tin beryllium written, I deliberation coding agents alteration the measurement production implementations of train steps tin beryllium written. Traditionally, we deliberation of autograd arsenic a halfway portion of PyTorch’s worth proposition, because it guarantees you will get correct derivatives. However, astatine scale, the implicit backwards graph becomes an albatross astir one’s neck: the mostly of your compute is hidden away, pinch nary opportunity to interact pinch it pinch normal debugging tools aliases use fusions to it successful the aforesaid measurement you tin do it successful eager forwards code. With a compiler, it is imaginable to modify the backwards chart with, e.g., a shape match, but this is brittle and a little bully acquisition than just swapping a telephone from a reference implementation to a hand-written kernel. This is not a caller observation: the now defunct Tangent room was built connected the proposition that source-to-source automatic differentiation could beryllium useful.

The caller look looks for illustration this. Keep the accepted PyTorch autograd-friendly codification arsenic the reference implementation. Use LLMs to generate an definitive forward-backward type of the code, which tin beryllium optimized separately from the reference implementation. Unlike shape matching, you never person to interest astir your optimizations failing to apply. The costs is that the reference and the existent implementation tin diverge: we request a verifier that shows america they are equivalent. This verifier tin beryllium implemented simply with a bitwise equivalence test, aliases implemented arsenic immoderate benignant of chart capture and structural equivalence, successful the contented of translator validation. To ensure the verifier useful erstwhile 1 broadside has a fusion the different doesn’t, you only request to supply a reference implementation of the fusion (an inverse pattern match, if you will!).

I americium not going to declare that this look is correct for everyone. It turns out PyTorch, the reference language, is simply a beautiful bully executable spec, and astatine the end of the time what really matters is really quickly you get the experimental results you need. But, having spent a batch of my clip precocious reasoning about what it intends for PyTorch to excel astatine frontier training–and successful particular whether aliases not it is basal for PyTorch to disrupt itself arsenic scaling continues–I consciousness that this position helps span the aged and the new. An open mobility Horace He posed past twelvemonth was this: “How tin we get each of the control of eager-mode execution pinch immoderate of the conveniences of graph-level abstraction?” I deliberation this look is simply a beautiful promising answer, and PyTorch continues to beryllium astatine the halfway of it.