Continuous Diffusion Language Models (CDLM's)

A flurry of caller activity successful the abstraction of continuous diffusion models for language, aft a fewer years of comparative dormancy, suggests that this attack is making thing of a comeback. Fully discrete diffusion methods had mostly supplanted earlier attempts to make continuous diffusion activity for language, but the tide is starting to turn. In this post, I want to return a person look astatine what’s going on, and why it is happening now.

The caller influx of caller investigation successful this abstraction inspired maine to constitute up immoderate of my thoughts. I person written astir diffusion connection models before, truthful this chiefly serves arsenic an update to screen everything that’s happened since then. This will beryllium a reasonably subjective relationship – different perspectives and dissenting opinions are very invited successful the comments and elsewhere! I’ll talk immoderate method aspects of continuous diffusion for connection later on, but first, immoderate humanities context.

Challenging the autoregressive hegemony

Modern connection models are, by and large, autoregressive: they make sequences 1 token astatine a time. This is simply a earthy decomposition of a difficult procreation task into smaller, easier sequential steps. All steps are instances of the aforesaid underlying task (predict a token fixed preceding tokens), which enables parameter sharing crossed the series dimension. In spite of this inherently sequential generative process, the Transformer architecture1 admits businesslike parallel training crossed each series positions utilizing coach forcing2. This has turned retired to beryllium an highly scalable recipe3, which has brought america ample connection models (LLMs).

However, autoregression is not the only measurement to conception an iterative generative process for sequences. Inspired by early successes successful the audiovisual domain, researchers sought to use diffusion to connection procreation instead. Rather than generating a series 1 constituent astatine a time, the generative process of diffusion models is defined by reversing a corruption process, which gradually destroys information. The canonical measurement to do this is to adhd Gaussian sound small by little, until it wholly overpowers the signal.

2021: early discrete diffusion models

After early successes successful image procreation successful 20194 and 20205 6, the first attempts to use this thought to connection arrived successful 2021, and progressive replacing a continuous corruption process pinch a discrete one to alteration modelling of categorical data: multinomial diffusion7, D3PM8 and SUNDAE9.

Back then, the power of autoregression was not arsenic well-established arsenic it is today: GPT-310 had turned immoderate heads, but the ‘ChatGPT moment’ wouldn’t travel until precocious 2022. At the time, discrete diffusion seemed to reside immoderate existent theoretical flaws successful the autoregressive paradigm, for illustration vulnerability bias owed to teacher forcing and the comparative trouble of applying it to infilling and constrained procreation tasks. Note that location had been immoderate exploration of non-autoregressive and any-order autoregressive approaches successful the preceding years11 12 (especially for instrumentality translation13 14), but not yet from a diffusion perspective.

2022: continuous diffusion for discrete data

In 2022, respective attempts to apply continuous diffusion to connection modelling appeared, starting pinch Diffusion-LM15. This attack addresses the incompatibility betwixt categorical information and corruption pinch Gaussian sound successful a different way: simply correspond the discrete categories pinch continuous embedding vectors, which are perfectly amenable to Gaussian sound corruption. That way, the Gaussian diffusion mechanism, which useful truthful good for images, tin beryllium applied without immoderate changes.

Diffusion-LM touted the advantages of this replacement generative paradigm for controllable matter procreation successful particular. In the past fewer months of 2022, rather a fewer different papers utilizing variations of this attack were published, including DiffuSeq16, SSD-LM17, Difformer18, SeqDiffuSeq19, GENIE20, LD4LG21 and besides 2 papers that I worked on: self-conditioned embedding diffusion (SED)22 and continuous diffusion for categorical information (CDCD)23.

At the time, the allure of these continuous methods was that they could use from each the insights, devices and machinery that were being discovered and developed for continuous diffusion, arsenic it wholly took complete audiovisual generation. For example, applying immoderate of the sampling and distillation techniques developed for continuous diffusion models to discrete diffusion was often overmuch little straightforward, aliases moreover downright impossible.

Late 2023: the continuous extinction

Then, thing absorbing happened: after 2023, virtually each caller investigation successful this abstraction utilized discrete diffusion, and continuous diffusion for connection went extinct. A sketch from a 2025 study paper24 astir diffusion connection models intelligibly shows this:

— Sander Dieleman (@sedielem) August 19, 2025

Continuous methods are marked successful yellow, discrete methods successful green. The modulation from 2023 to 2024 is rather stark! It is difficult to opportunity for definite why this happened, but I tin deliberation of a fewer imaginable factors: 1 is the ChatGPT moment, which gradually shifted the attraction of connection diffusion investigation from theoretical advantages and elegance to earthy performance. Now, the extremity was to effort and match powerful autoregressive models astatine scale, aliases moreover outcompete them successful circumstantial settings. It seems that group felt closing the capacity spread would beryllium easier to execute pinch afloat discrete methods, possibly because they are conceptually much intimately related to autoregression.

Another facet could beryllium that the science of scaling connection diffusion models had started to beryllium explored, and first observations for continuous methods weren’t looking promising. In May 2023, Gulrajani & Hashimoto25 quantified the training ratio spread for a likelihood-based continuous diffusion connection exemplary (Plaid-1B): 64x little efficient. (Note that the sketch supra marks Plaid arsenic discrete, but it is simply a continuous method.)

At a clip erstwhile the LLM organization was still very overmuch focused connected the pareto frontier of training compute versus perplexity (Chinchilla-optimality26), immoderate modelling attack whose training ratio was almost 2 orders of magnitude worse than an autoregressive baseline was difficult to return seriously. The first LLaMA27 model, which challenged this training ratio attraction and based on for taking the conclusion fund into account, had only conscionable been released a fewer months earlier (February 2023), truthful I judge it is plausible that the organization had not yet internalised this shift.

Needless to say, this is each highly speculative. Perhaps it was conscionable a coincidence, and discrete methods ended up having much momentum astir that clip purely by chance. If you person immoderate thoughts astir what could person caused the late-2023 continuous extinction event, I’d beryllium keen to perceive them successful the comments!

Personally, I had stopped moving connected diffusion connection models by that constituent (I sewage excessively engaged building image and video procreation models: Imagen and Veo, and later on, Nano Banana and Omni), truthful I was conscionable watching this improvement from the sidelines. I recovered it somewhat suprising, because I believed continuous diffusion has a fewer cardinal advantages, for illustration an expertise to correspond uncertainty astatine the individual token level, and a rich | toolbox of sampling algorithms and tricks to tie on. Giving those up seemed for illustration it could beryllium a mistake, but the investigation organization arsenic a full intelligibly figured that this was the measurement to go.

Adapting continuous diffusion to discrete data

We will talk astir what’s been happening successful the diffusion connection modelling abstraction much precocious successful the adjacent section, but first, I deliberation it is useful to talk really continuous diffusion tin really beryllium applied to discrete information successful a spot much detail. This discourse will beryllium adjuvant to understand what mightiness beryllium driving caller events.

The first point to see is the quality of the discrete information we are trying to model. Usually erstwhile group opportunity ‘discrete’, they really mean categorical, i.e. the output abstraction (at the token level) is an unstructured set, and location is nary narration (ordinal aliases otherwise) betwixt the different values that each discrete adaptable tin assume. Digital images represented arsenic pixel grids are also discrete, technically speaking, but because the discrete values assumed by the pixel colour channels correspond an underlying continuous beingness awesome (light intensity), we thin to simply disregard that, and dainty them arsenic continuous anyway.

Assuming we are moving pinch categorical data, location are a fewer basal ingredients to make continuous diffusion activity well: an embedding strategy, a loss function, and a sensible noise schedule. In addition, location is simply a instrumentality that pops up successful almost each insubstantial connected this topic, which turns retired to person a immense effect connected performance: self-conditioning. We’ll return a person look astatine each of these successful turn. For brevity, I will usage the acronym CDLM to mention to continuous diffusion connection models going forward, and DDLM to mention to their discrete counterparts.

Embedding strategies 📍

Modern neural networks typically person real-valued parameters and activations. Therefore, the first point that usually happens successful immoderate neural web that processes discrete data, is embedding the discrete inputs successful a continuous practice space. From that constituent on, the web exclusively manipulates real-valued representations. These embeddings are usually conscionable treated arsenic further parameters, which tin beryllium optimised jointly pinch the remainder of the model. It’s worthy pointing retired that this is besides the lawsuit for DDLMs and autoregressive models – the internals of the neural networks powering these models are still continuous.

To use continuous diffusion to discrete data, we tin simply ‘lift’ the corruption process from the discrete input abstraction into a continuous embedding space. In different words, compared to discrete diffusion, it’s conscionable a mobility of changing the bid of operations. Rather than applying discrete corruption followed by continuous embedding during training, we first embed the inputs and past use continuous corruption instead.

Schematic sketch of (a) an autoregressive model, (b) a discrete diffusion exemplary (masked diffusion) and (c) a continuous diffusion model. Green blocks bespeak continuous embeddings of the discrete tokens. Blue blocks correspond the layers of the model. For a and b, the embedding shape is typically considered besides portion of the model. Corruption for b happens astatine the token level, earlier embedding. For c, corruption is applied by adding sound to the continuous embeddings.

The style and building of the embedding abstraction profoundly impacts the quality of the continuous corruption process that happens wrong it. Various embedding strategies person been explored:

• Explicit (e.g. SSD-LM17): arguably the simplest attack is to usage thing for illustration a one-hot representation, wherever each constituent successful a vocabulary of size \(V\) is associated pinch a \(V\)-dimensional vector that has \(V-1\) zeros and a azygous one. Since a vocabulary is simply a set, assigning representations to the elements requires arbitrarily picking a circumstantial ordering. This benignant of embedding abstraction tin beryllium cumbersome to activity pinch for modern connection models, because \(V\) tends to beryllium beautiful ample nowadays. A imaginable workaround is to usage compact binary patterns instead, arsenic successful Analog Bits28.

• Pre-trained (e.g. SED22): we tin usage a practice learning strategy to study embeddings, and past repurpose them for usage successful a diffusion connection model. For example, they could beryllium borrowed from an autoregressive connection model, aliases taken from a bidirectional connection exemplary for illustration BERT29. They tin besides beryllium made contextual, i.e. the embedding for a fixed token tin dangle not only connected the token itself, but besides connected adjacent tokens, resulting successful a richer embedding space21.

• Jointly learned (e.g. CDCD23): we tin effort to fresh the embeddings together pinch the denoiser exemplary successful a azygous learning procedure, perchance letting them co-adapt.

The second mightiness look for illustration the astir earthy point to do, because associated learning of the embeddings is what useful for DDLMs and autoregressive LLMs. An end-to-end single-stage learning attack is besides wide considered the astir charismatic nowadays, some from a conceptual and from a applicable standpoint. But the elevated domiciled of the embedding abstraction successful CDLMs (relative to DDLMs) intends that this comes pinch immoderate challenges: naive formulations are prone to embeddings collapsing aliases increasing uncontrollably. For example, denoising correction tin beryllium minimised pathologically by making each embeddings the same, truthful this suggests that immoderate benignant of trade-off betwixt aggregate constraints aliases position successful the nonaccomplishment usability mightiness beryllium necessary.

There has been a adjacent spot of chat successful the lit astir the value of the geometry of the embedding space. It is often assumed aliases suggested that embedding spaces pinch meaningful semantic building lend themselves amended to continuous diffusion connection modelling. Concretely, this intends that the embeddings should beryllium organised successful specified a measurement that a fixed magnitude of corruption creates commensurate disorder betwixt tokens from a semantic perspective; e.g. if a mini magnitude of sound is added to the embedding for ‘cat’, it mightiness go indistinguishable from the embedding for ‘dog’ pinch the aforesaid magnitude of sound added, but it will still look very different from the embedding for, say, ‘umbrella’, astatine the aforesaid sound level.

It is still unclear to maine really important this really is, if what we attraction astir is earthy connection modelling performance. This is not a facet that is usually considered successful the discourse of DDLMs aliases autoregressive LLMs. It has been suggested successful the lit that immoderate strategies and learning objectives for CDLMs person a dispersive effect30 (i.e. pushing embeddings for semantically related tokens apart, alternatively than together), and that this mightiness beryllium a bad thing, which surely seems plausible.

Related to this is the mobility of whether continuous embeddings should correspond individual tokens, tokens successful discourse (i.e. contextual embeddings), aliases thing much hierarchical, for illustration sequences of aggregate tokens, aliases moreover full sentences aliases paragraphs. While the attraction successful this blog station is connected token-based approaches, I’ll concisely talk these higher-level alternatives (often framed arsenic ‘latent diffusion for language’) later on.

As the extremity of CDLMs is yet to make a discrete token sequence, an unembedding strategy is besides needed. Neural networks for classification tasks usage a softmax nonlinearity to make predictions successful a categorical space, and construe the web outputs arsenic probabilities (which are themselves continuous). This attack tin besides beryllium utilized for denoisers: moreover if we are denoising continuous vectors, we tin usage the knowledge that they each correspond 1 of a finite number of discrete vocabulary elements to constrain the predictions (in CDCD23, we took advantage of this study and called it ‘score interpolation’, framing it arsenic an replacement to ‘score matching’). In astir useful however, predictions are made straight successful the continuous embedding space, without specified constraints, and a last discretisation measurement is performed astatine the extremity of sampling. This is often done simply by clamping the embeddings to the nearest vocabulary element, but the process tin besides beryllium much involved.

Loss functions 📉

There is immoderate absorbing assortment successful the nonaccomplishment functions utilized for training CDLMs. I won’t enumerate each the options, but I do want to constituent retired immoderate trends. Usually, the prime of nonaccomplishment usability is intimately tied to the unembedding strategy. If the denoiser makes predictions straight successful the continuous embedding space, the accustomed mean squared error (MSE) nonaccomplishment tends to beryllium used, conscionable for illustration successful continuous diffusion models for audiovisual data.

If the denoiser outputs probabilities crossed vocabulary elements, it tin beryllium trained utilizing the categorical cross-entropy nonaccomplishment instead, which makes things look much akin to the autoregressive setting. Note that this attack only useful pinch per-token embeddings and is not compatible pinch contextual aliases higher-level embeddings: predicting probabilities for each imaginable output is feasible astatine the per-token level if the vocabulary size is not excessively large, but not beyond that.

Another attack is to commencement from the maximum likelihood principle25, and travel up pinch an nonsubjective that bounds the likelihood from beneath (in the aforesaid measurement that variational autoencoders31 32 are trained). Some CDLM variants that jointly study the embeddings and the denoiser see further nonaccomplishment position to regularise aliases constrain them15, but sometimes these constraints are handled done parameterisation alternatively (e.g. forcing the embeddings to beryllium normalised vectors23).

Several useful person explored various ways to constrain the continuous diffusion process to the \(V\)-simplex: the abstraction of valid categorical probability distributions crossed \(V\) categories33 34 35 36. In this setup, intermediate noisy vectors are themselves constrained to beryllium valid probability distributions crossed each vocabulary elements, which besides requires replacement nonaccomplishment functions. While it seems for illustration a bully thought successful theory, successful practice, this usually adds important complexity and it doesn’t look to beryllium very scalable to ample vocabulary sizes. Most successful applications of this thought person really been successful biology, wherever absorbing discrete series modelling problems pinch overmuch smaller vocabularies beryllium (e.g. \(V=4\) for DNA, \(V\approx22\) for amino acids).

Noise schedules 📻

The corruption process of a diffusion exemplary is governed by the sound schedule, which determines the complaint astatine which the sound level increases complete the people of the process. I wrote a batch much astir sound schedules for continuous diffusion models in an earlier blog post.

Ideally, the schedule is chosen truthful that accusation is destroyed gradually, allowing the generative process to beryllium surgery up into smaller subtasks that each resoluteness mini amounts of uncertainty. A poorly chosen sound schedule results successful ample segments of the corruption process wherever thing happens (i.e. almost nary accusation is lost, and truthful the denoiser has thing to learn), and immoderate segments wherever a batch of accusation is destroyed each astatine once, making for a very difficult denoising task.

For CDLMs, getting the sound schedule correct is particularly important, arsenic a naive strategy will almost surely consequence successful a very uneven corruption process. This is simply a nonstop consequence of the truth that embeddings are usually high-dimensional vectors, which correspond discrete underlying categories. In that setting, meaningful corruption happens crossed a comparatively mini scope of sound levels. Most sound levels either destruct almost nary accusation astir token personality (too low), aliases destruct almost each accusation (too high). It is important to debar spending denoiser modelling capacity connected those sound levels, arsenic it will not beryllium capable to study thing useful there.

A communal strategy has been to explicitly accommodate the sound schedule to the geometry of the embedding space, either offline aliases done online adjustment of the sound schedule during training. This creates a feedback loop, wherever the exemplary predictions are utilized to find which sound levels are of interest, and subsequently the distribution of sound levels sampled to corrupt training examples is adapted to attraction training connected precisely those sound levels.

The original inspiration for specified online adjustment mechanisms was the variational diffusion models (VDM) paper37, which utilized this thought to minimise the variance of the training objective, successful bid to accelerate convergence. In the discourse of CDLMs, this attack was adapted to get a balanced corruption process pinch a attraction connected sound levels wherever the level of corruption is conscionable correct to alteration learning of meaningful structure. This tin beryllium achieved by learning a schedule \(\sigma(t)\) that linearises the entropy of the denoiser predictions23 38 (in position of \(t\)), aliases the decoding correction rate39. With \(t\) sampled uniformly, entropy linearisation ensures that each diffusion sampling measurement resolves the aforesaid number of bits of information.

Figure from the CDCD insubstantial showing the effect of adapting the sound schedule during training. The comparative weight of the different sound levels (middle plot) becomes highly non-uniform, and the attraction is connected those sound levels wherever the entropy changes the fastest. In position of the learnt schedule (referred to present arsenic 'uniform time'), the entropy increases astir linearly. If sampling steps are spaced evenly successful azygous time, the magnitude of entropy they resoluteness is astir constant.

Self-conditioning 🔄

Diffusion sampling is stateless, successful the consciousness that the adjacent update measurement successful the sampling process only depends connected the existent canvas. One could see the canvas itself to represent a shape of state, but crucially, it is ever afloat observed. There is nary further hidden discourse that the exemplary tin manipulate during sampling, which is why the sampling process tin beryllium (and often is) framed successful position of differential equations40.

This led immoderate group to wonderment if possibly denoisers utilized for diffusion sampling are doing redundant work: astatine each sampling step, they compute the optimal denoising guidance from watching the existent noisy canvas, without entree to their ain erstwhile predictions from earlier steps. But if the steps are mini enough, the optimal denoising guidance mightiness really beryllium rather akin to the erstwhile prediction, truthful this seems for illustration it could beryllium wasteful.

Self-conditioning28 was introduced to reside this: simply pass the denoiser’s erstwhile prediction to the adjacent step arsenic an other input. This allows the denoiser to study really to modify a unsmooth estimate, alternatively than having to make predictions from scratch. To train specified a denoiser, the further ‘previous prediction’ input is sometimes near blank, and sometimes provided during training by utilizing the denoiser itself to make a prediction from scratch (hence ‘self’-conditioning). This clever system ensures that the denoiser still useful erstwhile nary erstwhile prediction is available, but besides knows what to do pinch it erstwhile it is provided.

Illustration of self-conditioning from the Analog Bits paper, which introduced it.

For CDLMs, it was discovered beautiful quickly that self-conditioning tends to supply a immense boost successful performance, and truthful almost each useful successful this abstraction make usage of it. This is successful spite of the truth that it breaks the statelessness presumption built into various diffusion machinery, astir notably sampling algorithms based connected differential equations (ODEs and SDEs). It is adjacent to presume that it astir apt biases the modelled distribution successful hard-to-understand ways, but everyone uses it anyway, because it makes specified a immense quality to sample value that it would beryllium an enactment of self-sabotage not to.

Exactly why this useful truthful good for connection diffusion successful peculiar is still unclear – particularly because attempts to use the thought for audiovisual generative modelling person been acold little fruitful (Recurrent Interface Networks41 are a notable exception). The underlying discrete building of the output abstraction seems to play a domiciled successful this. A caller insubstantial by Yoo et al.42 reanalyses diffusion pinch self-conditioning arsenic an businesslike approximation of a fixed-point exemplary embedded wrong a diffusion model, almost for illustration a nested for-loop. This position provides an mentation arsenic to why the statefulness of denoisers pinch self-conditioning does not look to beryllium a problem successful practice: it is simply a broadside effect of approximating the nested for-loop pinch a azygous level loop.

Some homemade recipes 🧑‍🍳

To wrap up this section, I want to exemplify really these ingredients tin travel together successful a fewer different ways, utilizing immoderate early useful successful the CDLM abstraction that I contributed to. All of these day backmost to precocious 2022, because I stopped moving connected connection aft that.

Simplex diffusion33 uses a non-Gaussian corruption process: the alleged Cox-Ingersoll-Ross (CIR) process43. This operates connected strictly affirmative existent values, and it was primitively utilized to exemplary liking rates. It comes pinch a built-in presumption that these liking rates cannot beryllium negative, truthful arsenic you tin imagine, it mislaid a spot of traction for that intent aft 2008! That spot does make it very well-suited to exemplary (unnormalised) probabilities, though. We utilized the score-based SDE formalism (📉) pinch this replacement process, which (somewhat surprisingly) yields tractable, if somewhat exotic formulas for each the quantities of interest. For example, the modulation density is simply a non-central chi-squared distribution, alternatively of the accustomed Gaussian. This was a theoretical exploration during the task that later became CDCD (see below). We ended up not pursuing it further, because it seemed to standard poorly to larger vocabulary sizes.

Self-conditioned embedding diffusion22 (SED) is built astir pre-trained embeddings (📍) obtained utilizing a BERT model, which is somewhat modified to person a low-rank bottleneck, arsenic diffusion connected lower-dimensional embeddings was recovered to execute better. Note that moreover though they travel from a BERT model, the embeddings themselves are per-token, not contextual. The nonaccomplishment usability is simply a operation of the accustomed denoising MSE and a cross-entropy-based unembedding nonaccomplishment (📉), the sound schedule is simply a cosine schedule (📻 reasonably modular for the time), and self-conditioning is an important constituent (🔄 it’s successful the name!).

Continuous diffusion for categorical data23 (CDCD) is built connected the rule that connection diffusion would beryllium much apt to spot wider take if it looks arsenic acquainted arsenic imaginable to existing LLM practitioners. The insubstantial frames it arsenic a type of BERT, but pinch Gaussian sound alternatively of masking noise. It uses modular Gaussian diffusion, but pinch a cross-entropy nonaccomplishment usability (📉 score interpolation), and pinch embeddings learnt connected the fly, jointly pinch the denoiser (📍). As this makes the embeddings prone to uncontrollable growth, a normalisation furniture is utilized to unit them to person portion norm. It besides relies heavy connected self-conditioning to execute bully capacity (🔄). Another cardinal capacity facet is the adaptive sound schedule based connected an entropy linearisation heuristic (📻 time warping), which ensures that some training and sampling walk much clip and capacity connected the sound levels that matter most.

To my ain delight, galore of the CDCD ingredients person go reasonably mainstream successful modern CDLM useful (several of which I’ll talk successful the adjacent section). Adaptive schedules characteristic frequently, and they often usage immoderate benignant of linearisation heuristic. The people interpolation strategy that primitively enabled cross-entropy-based training of continuous denoisers has been rederived successful a much modern mounting (i.e. travel matching44 and travel maps45), and fixed a stronger theoretical underpinning. Self-conditioning is now ubiquitous.

The caller comeback

With each of that successful mind, let’s prime up wherever we near disconnected astatine the extremity of the first section, and return a look astatine what’s been going connected pinch CDLMs much recently.

After 2023, this abstraction was very quiet for a agelong clip arsenic group focused connected discrete diffusion. Two strategies for discrete corruption are commonly used: masked discrete diffusion corrupts tokens by gradually replacing each of them pinch disguise tokens, until the series is afloat masked. uniform-state discrete diffusion corrupts tokens by replacing them pinch random tokens instead, until the series is afloat randomised. The erstwhile attack has a azygous deterministic absorbing extremity authorities (fully masked), whereas the extremity authorities of the second is that each imaginable token sequences are arsenic apt (uniform categorical distribution).

In the 2nd half of 2025, group started trying to bring backmost immoderate continuous flavour successful the shape of hybrid methods, combining discrete and continuous approaches successful various ways to effort and get the champion of some worlds. This was followed successful 2026 by a full-on resurgence of continuous methods.

2025: discrete and continuous, why not both?

Sahoo et al.46 started disconnected this inclination by watching a adjacent relationship betwixt Gaussian continuous diffusion and uniform-state discrete diffusion. They recovered that mapping continuous noisy intermediate states to discrete states utilizing the \(\arg \max\) usability besides implicitly turns the Gaussian corruption process into a uniform-state corruption process. They called this narration the diffusion duality, and utilized it to use consistency distillation47 to discrete diffusion models, arsenic good arsenic for training nonaccomplishment variance reduction.

CADD48, CCDD49 and CANDI50 each propose different ways to combine discrete and continuous corruption into a azygous process. CADD uses continuous intermediate representations to augment masked diffusion, successful bid to guarantee that accusation is mislaid successful a much gradual fashion. CCDD uses associated diffusion complete discrete and continuous representations simultaneously, to pat into the accrued expressivity of continuous diffusion, while avoiding the situation of decoding continuous embeddings backmost into discrete tokens.

CANDI alternatively tries to reside a scaling rumor pinch continuous diffusion for discrete data, which they telephone temporal dissonance: for high-dimensional vocabularies, the discrete personality of individual tokens decays quickly arsenic the corruption process progresses, but their comparative rank among each posibilities decreases overmuch much slowly. By the clip location is thing absorbing to study astir the semantic building of the continuous embedding space, each token identities person already been corrupted, and the exemplary will person a really difficult clip learning astir the conditional relationships betwixt tokens arsenic a result. They place this arsenic a cardinal problem holding backmost continuous methods, and propose to reside it by utilizing discrete masking and applying Gaussian corruption only to the masked positions, thereby decoupling these 2 kinds of corruption.

These useful each appeared wrong a fewer months of each other, making it almost look for illustration a coordinated effort to rehabilitate continuous diffusion for discrete data, while still sticking intimately to the ascendant discrete paradigm. Not agelong aft this, the purely continuous attack besides saw a revival, arsenic we will talk next.

2026: continuous diffusion strikes back

2025 had besides seen the improvement and accelerated take of flow maps45 51, the taxable matter of my erstwhile blog post. A travel representation is fundamentally the integral of a diffusion model. In a diffusion model, a denoiser is learnt, which tin beryllium utilized to move done input abstraction from sound to information by many times taking mini steps successful the predicted denoising direction. Flow maps effort to do this successful 1 spell instead, aliases astatine least, successful arsenic fewer steps arsenic possible. To execute this, a web is trained to straight approximate the output of the diffusion sampling procedure. (I’m deliberately cutting immoderate corners present for brevity’s sake, but that is the gist of it.)

Although step distillation of diffusion models had already been a fruitful investigation taxable agelong earlier that, the improvement of specified a powerful model seemed to animate respective groups of researchers to revisit continuous methods for connection modelling, successful hopes of bringing the benefits of the model to this people of models arsenic well. Early 2026 saw the quality of 3 intimately related useful successful accelerated succession: Categorical Flow Maps52, Flow Map Language Models39 and Discrete Flow Maps53. All 3 extend travel maps to the discrete categorical setting utilizing definitive one-hot embeddings (📍) and cross-entropy-based nonaccomplishment functions (📉), bringing their exertion to connection modelling wrong reach.

In the outpouring of 2026, this was followed by thing of a Cambrian detonation successful the abstraction of CDLMs. A bid of papers appeared that revisited and extended various recipes:

• LangFlow54, Spherical flows55 and Hyperspherical flows56 build connected CDCD23, making usage of jointly learned normalised embeddings (📍), the cross-entropy nonaccomplishment (📉) and adaptive sound schedules (📻). The second 2 useful constrain the corruption process to the sphere (we tried a naive version of this successful CDCD and called it ‘renormalisation’, but that didn’t activity very well).

• Latent diffusion connection models (LDLM)57 and Embedded connection flows (ELF)58 travel the creation of Diffusion-LM15 and SED22, applying the modular continuous diffusion look successful an appropriately chosen embedding space. A awesome quality is that they make usage of contextual embeddings, alternatively than per-token ones (📍 pursuing LD4LG21). They are jointly learnt pinch the denoiser for LDLM, whereas ELF (primarily) uses pre-trained and stiff embeddings.

• Continuous bitstream diffusion (CoBit)59 uses definitive embeddings successful the shape of spot sequences (📍), applying and extending the Analog Bits28 attack to language.

• RePlaid30 revisits Plaid25 and modernises its likelihood-based attack (📉) by drafting from caller DDLMs.

Compared to their predecessors, these useful characteristic a modernised framing, improved implementations, caller theoretical insights, updated information methodologies and accrued scale. Several of them reason that the previous statement astir discrete diffusion having the separator is incorrect. RePlaid and LangFlow moreover make the other declare successful their respective insubstantial titles: ‘Continuous Diffusion Scales Competitively pinch Discrete Diffusion for Language’, ‘Continuous Diffusion Rivals Discrete successful Language Modeling’.

Why continuous? And why now?

The revival of CDLMs is very overmuch ongoing, truthful it is astir apt a spot excessively early for a humanities position that tries to afloat explicate the back-and-forth betwixt discrete and continuous methods complete the past 5 years. Nevertheless, I want to constituent retired a fewer trends that person apt influenced it.

First of all, continuous methods person gotten simpler and easier to use complete the years: earlier perspectives required knowing people matching4, heavy latent adaptable models6 aliases differential equations40; modern explanations trust mostly connected basal concepts for illustration linear interpolation betwixt information and noise44. This has lowered the obstruction of introduction for practitioners successful adjacent fields to besides research these methods.

Interest successful replacement modelling paradigms for connection beyond autoregression has besides accrued overall, acknowledgment to the occurrence of DDLMs and the quest to research caller substrates for reasoning. The imaginable pay-off of uncovering the adjacent large point has only accrued arsenic ample connection models person go large business: making connection models faster and much elastic has go highly economically valuable.

Attempts to alteration the number of steps required to sample from DDLMs thin to deed a wall: simultaneously sampled tokens are assumed to beryllium conditionally independent fixed antecedently sampled tokens. In the limit of single-step sampling, this intends each tokens are needfully sampled independently, and the models are past fundamentally incapable to seizure immoderate correlations betwixt them. This makes measurement distillation challenging, and overcoming that problem mightiness require introducing important further complexity.

Continuous methods side-step this rumor completely: trajectory-based measurement distillation methods (like travel representation methods) alteration moreover single-step models to seizure each correlations, astatine slightest successful mentation – successful practice, the constricted capacity of the models still makes this challenging to do successful conscionable 1 step. Nevertheless, I deliberation it is adjacent to opportunity that few-step sampling comes overmuch much people to the continuous setting. This distillability advantage is astir apt the main logic why CDLMs are backmost pinch a vengeance today.

Step distillation doesn’t conscionable alteration faster sampling: it besides unlocks possibilities for reward-based steering and fine-tuning that were challenging to execute pinch diffusion-based connection models earlier (see this section successful my erstwhile blog post). Given the awesome domiciled that post-training plays successful the occurrence of modern LLMs, this is besides an important consideration.

Whether alleged flow representation connection models will displace the existent position quo wholesale remains to beryllium seen, but I afloat expect them to proceed to summation traction, and efforts to standard them up are underway60. I powerfully urge reference Floor Eijkelboom’s deep dive connected flow-based connection generation for a much method curen of this topic. Jiaming Song besides has a bully write-up connected the domiciled of diffusion and travel maps successful connection modelling.

It is absorbing to ponder really the ascendant position of continuous diffusion models for audiovisual procreation is motivated by almost wholly different reasons. I person antecedently written extensively astir the spectral position connected diffusion, and the link betwixt sound levels and characteristic scales successful the ocular domain. I judge the truth that we tin manipulate the diffusion nonaccomplishment to emphasise perceptually applicable contented is simply a cardinal logic for their success, but that intelligibly does not use to connection modelling astatine all.

One commonly heard statement successful favour of CDLMs is that they make multimodal integration easier: we tin usage continuous diffusion crossed each modalities and harvester them into a azygous model. While that is true, I deliberation it misses the constituent a small bit. The situation of building multimodal generative models is not truthful overmuch astir bridging aggregate modelling paradigms, which is really not that difficult (see e.g. Diffusion Forcing61, Transfusion62). A much pertinent situation is the semantic gap that exists betwixt connection representations and representations of perceptual signals utilized successful diffusion models (i.e. latents aliases patches of pixels): connection tokens are semantically abstract, audiovisual tokens are not. I will astir apt person much to opportunity astir that successful a early blog post. For now, each I person is simply a spicy tweet 🌶️:

— Sander Dieleman (@sedielem) June 1, 2026

Discrete = obsolete?

Is this the extremity of the statement for discrete diffusion, then? That seems alternatively unlikely. Dimitri von Rütte made the lawsuit that diffusion connection models are the future a while backmost (uniform-state discrete models successful particular). More recently, Volodymyr Kuleshov and colleagues shared a station describing the building blocks of DDLMs, and Junbo Zhao wrote a blog post astir their move into the mainstream. A batch of group look to beryllium bullish capable astir the attack to blog astir it!

The caller merchandise of DiffusionGemma63, an open-weights uniform-state DDLM developed by my colleagues astatine Google DeepMind, arsenic good arsenic NVIDIA’s Nemotron Diffusion64, besides importantly accrued consciousness that autoregression is not the only crippled successful town.

Step distillation of DDLMs is not a complete impossibility either: methods specified arsenic discrete infinitesimal matching distillation (D-MMD)65 and inverse-distilled diffusion connection models (IDLM)66 show that immoderate approaches tin beryllium ported complete from the continuous to the discrete setting.

It is worthy noting that displacing autoregression wholly is not the only measurement for diffusion connection models to beryllium successful. They tin coexist, sometimes moreover wrong the aforesaid system. A communal strategy to accelerate sampling from autoregressive connection models is speculative decoding, wherever a faster draught exemplary is utilized to foretell aggregate tokens astatine a time, which tin past beryllium verified successful parallel by the autoregressive model. Discrete diffusion is progressively being utilized for drafting successful this discourse (e.g. DFlash67).

A different position connected the aforesaid thought is that autoregressive verification tin beryllium utilized to mitigate the effect of independency assumptions successful few-step discrete diffusion sampling68. DMax69 uses a operation of continuous relaxations and post-training to trim the effect of these independency assumptions instead.

In the meantime, immoderate caller useful person continued to research hybrid continuous-discrete approaches. Sticky Jump Diffusions70 are an effort to create a unified position of hybrid methods, building connected the SDE formalism, pinch some discrete masked diffusion and continuous diffusion arsenic typical cases. Posterior Refinement71 wraps continuous diffusion wrong masked diffusion. This creates a nested sampling loop, wherever the soul loop tin beryllium distilled down to very fewer steps utilizing flow representation methods. This results successful a shape of masked diffusion wherever each measurement tin besides seizure correlations betwixt simultaneously unmasked tokens (unlike modular masked diffusion, which assumes their conditional independence).

What’s next?

The existent inclination of diffusion connection models moving into the mainstream shows nary signs of slowing down. Whether discrete aliases continuous methods will travel to predominate is difficult to foretell – possibly they will coexist, alongside autoregression. As the LLM organization moved connected from the Chinchilla position (focusing exclusively connected training efficiency), replacement modelling paradigms person gradually received much attention, which diffusion connection models are undoubtedly benefiting from. In the longer term, different facet of diffusion models whitethorn summation importance: their accrued information ratio comparative to autoregression72.

In the meantime, I judge it is important for the investigation organization to activity to reside a communal weakness of diffusion connection modelling papers: the evaluation methodology. Because of their elastic sampling procedure, diffusion models are peculiarly amenable to tuning the trade-off betwixt value and diverseness astatine sampling time. If this trade-off is not cautiously quantified and accounted for, this tin lead to definite models appearing to beryllium importantly amended than others, moreover if they simply correspond different points on this trade-off. Because the information methodology is not presently standardised, different papers usage different approaches and results tin beryllium unintentionally misleading.

Moreover, because galore formulations do not readily admit the estimation of likelihoods aliases perplexities nether the models themselves, surrogate autoregressive models are often utilized to measurement these alternatively (referred to arsenic generative perplexity, GenPPL), which biases the information towards the capabilities and weaknesses of the surrogates used.

Patrick Pynadath and colleagues propose quantifying the trade-off by looking astatine generative frontiers (i.e. plotting perplexity vs. entropy) successful a caller blog post. Sam Acquaviva besides has a blog post identifying respective diffusion connection exemplary information issues and imaginable fixes. Franca and Tong73 show conscionable really easy it is to crippled GenPPL arsenic a metric, and reason that it should not beryllium utilized moreover erstwhile entropies are matched. Metrics specified arsenic MAUVE74 were projected to effort and reside the situation of evaluating open-ended matter generation, but yet still trust connected pre-trained autoregressive models.

A investigation guidance that continues to seizure people’s imagination is latent diffusion for language: learning a higher-level and perchance much coarse-grained continuous practice for connection that is easy to exemplary pinch vanilla continuous diffusion. The main situation present continues to beryllium learning the latent abstraction itself, not truthful overmuch the diffusion part. As antecedently discussed successful my blog station connected latent diffusion, connection is simply a very different beast compared to perceptual signals, and practice learning techniques that activity good for the second mightiness wholly neglect for the former.

Aside from LD4LG21 and LDLM57 (previously mentioned), location has been a dependable watercourse of activity connected learning higher-level connection representations astatine the token, phrase, condemnation aliases paragraph level, including Time Control75, PLANNER76, Large Concept Models77, Segment-level Diffusion78, LaDiR79, Latent Thought Flows80 and AURORA-LM81. Mapping retired this abstraction would lead america excessively far, but this statement of activity cannot spell unmentioned erstwhile talking astir CDLMs, moreover if it is not the attraction of this post.

Closing thoughts

I wanted to constitute a speedy statement astir the caller resurgence of CDLMs, fixed my earlier activity successful the space, my fondness for the idea, and my disappointment erstwhile DDLMs seemed to return complete wholly aft 2023. Inevitably, it turned into an effort and a humanities relationship of connection diffusion investigation complete the past 5 years – I conscionable can’t look to thief myself 🤷. Some cardinal takeaways to wrap up:

• As of this year, CDLMs are backmost connected the menu, I judge chiefly because of their distillability advantage. Step distillation enables faster sampling, but besides creates caller possibilities for post-training and steering during sampling.
• The level of liking successful CDLMs comparative to DDLMs has ebbed and flowed complete the years, but presently it’s high tide! Recent incarnations echo their predecessors, but widen and amended them successful various ways (including but not constricted to scaling up).
• There are a assortment of approaches, and truthful far, no clear convergence connected a peculiar recipe. This makes the abstraction each the much breathtaking from a investigation perspective.
• For a while, location was an implicit consensus successful the organization that DDLMs activity better than CDLMs astatine scale, but that seems to beryllium waning. As pinch galore things successful instrumentality learning, really powerfully we collectively judge successful an thought and really difficult we effort to make it activity could importantly effect the outcome!
• To guarantee that this guidance of investigation continues to beryllium taken seriously, sorting retired the evaluation methodology will beryllium crucial, arsenic this is presently a anemic constituent of galore contributions successful the space.

I besides want to item a fewer upcoming diffusion connection modelling workshops:

• Non-Autoregressive Language Models for Fast & Flexible Text Generation astatine COLM 2026 successful San Francisco, USA
• Diffusion Language Models: Foundations, Efficiency, and Reasoning astatine NeurIPS 2026 successful Sydney, Australia
• Beyond Next‑Token Prediction — Diffusion & Flow Models for Next‑Generation Decoding astatine NeurIPS 2026 successful Sydney, Australia

The submission deadline for the NeurIPS workshops is successful a fewer days! Thanks for reading, and arsenic usual, please consciousness free to stock your thoughts successful the comments, connected Twitter, aliases via email.

Disclosure regarding the usage of AI successful producing this blog post: I want to constitute successful my ain voice, and I want to respect everyone who takes the clip to publication what I write. Therefore, you will not find immoderate passages aliases sentences successful this station that are afloat AI-generated. (Even the em dashes are each mine!) That said, I do occasionally consult AI erstwhile considering a peculiar move of phrase, aliases to thief maine find the champion wording (like a souped-up type of thesaurus.com). I chiefly usage it to thief maine understand papers and the narration betwixt them, and sometimes to create images and diagrams. AI was extensively utilized successful the making of this blog post, but the prose is wholly ‘artisanal intelligence’. That is the level of AI engagement I americium presently comfortable with.

If you would for illustration to mention this station successful an world context, you tin usage this BibTeX snippet:

Acknowledgements

Many acknowledgment to James Thornton, Oliver Wang, Sheel Shah, Jinwoo Kim, Justin Deschenaux, Patrick Pynadath, Oscar Davis, Luca Ambrogioni and Zhengyang Geng for sharing their thoughts and insights connected this and galore different topics. I’d besides for illustration to convey the organisers, different speakers and participants of the awesome EEML 2026 summertime school successful Cetinje, Montenegro, and the ICML 2026 diffusion circle crew. As usual, acknowledgment to my colleagues astatine Google DeepMind and the wider investigation community.

References