Read next

What changes whenever a tool doesn't have to pay for itself

October 5, 2026 3 infinitesimal read

Years ago, if an app annoyed me, I shrugged and dealt alongside it. I could sometimes have fixed it or built my own, but the annoyance rarely justified the effort. With LLMs that calculation has changed, and this former weekend it changed for a domain lookup tool.

Occasionally I need to inspect DNS records or registration info for a domain. At my device I use dig or whois. Otherwise I Google "domain lookup tool," click on one, and paste in the domain. Most of what comes up is slow, buried in upsells, and unpleasant to use for item that have to be simple.

They aren't seriously made -- they're fine made for a distinct job. A domain lookup answers a inquiry nobody pays to ask, so many of the liberated tools that display up in hunt are the ones a revenue chimney justifies: the lookup brings in person who power buy a domain or hosting, and the leaf is built to position for "domain lookup tool" and to sell. The tools you discover are the ones built to be found. Being fine at the lookup was never what paid for them.

I wanted item I'd really bask using, so I opened Claude Code and built heade.rs. In three prompts and concerning an hr I had a archetypal operating type deployed to Cloudflare: DNS and registration records, live latency, email configuration, HTTPS details. The remainder of the day went into making it awareness good: animations, graphics, an interface I akin looking at. It expenses small than a cup of coffee a duration to run. It doesn't have to market anything, so it can fair be good.

Before, a project akin this needed justifying. I'd have asked whether I'd use it adequate to create the period worthwhile. If I expected to expend weeks on it, I'd have started thinking concerning another users, pricing, how it could pay for itself. A small annoyance would have rotate into a much bigger commitment; a weekend is much easier to justify. I wanted it to exist, and that was enough.

A few hours used to get a programmer a script; now it gets a polished app, and it gets a non-programmer item alternatively of nothing. More group can apparent the bar, and additional ideas apparent it: small tools that resolve a individual annoyance, weird ideas you're inquisitive about, things you build merely since you'd bask having them or the earth would be additional beautiful if they existed. The spectators can be one person.

The identical economics cut the another way: heade.rs volition likely not position for "domain lookup tool," and neither volition most things built akin it. Cheaper application method additional group shipping, and another demo in a crowded nourish is uncomplicated to scroll past. If you desire to attain additional people, you motionless have to provision them a logic to care. I'm additional curious whenever person explains what bothered them, what they made, and the choices they cared concerning alongside the way. The narrative gives me a logic to try the thing.

Sometimes sharing a individual project leads location unexpected. Four years ago Paragraph started as item I built for myself, afterward another group wanted to use it too. I'm glad I shared it.

But I additionally akin that heade.rs doesn't need to prosecute that path. If it stays a small tool I use occasionally to appearance up a domain, it was motionless value building.

example.com is a reserved domain name set apart by the Internet Assigned Numbers Authority (IANA) for records purposes. It has been accessible as an example URL for developers and writers since the delayed nineties.

While the general messaging of the leaf has been accordant complete time, the ocular appearance has received multiple important overhauls.

In this article, we obtain a appearance at the latest changes as fine as the development of example.com complete the final 20+ years.

What changed on 28 September 2026?​

Until 28 September 2026, example.com was a fixed English-language page. The redesign introduces multi-language assistance using JavaScript, showing the satisfied in a new tongue all 5 seconds.

Example.com shows an clarification of the site's intent in English, Arabic, Chinese, French, Russian and Spanish. This is the English-language message:

This domain is for use in records examples without needing permission. This is not a service, evade relying on it for evaluation and monitoring purposes.

Along alongside introducing multi-language support, the JavaScript code additionally inserts an SVG publish icon.

How does the tongue passage animation work?​

The new location doesn't fair substitute the satisfied whenever showing the next language. Instead, there's a gradual opacity transition.

The opacity animation itself fair uses transition: opacity .4s;. But how is the character-by-character passage implemented?

Each character is wrapped in its own extend element, and all extend gets a slightly larger CSS transition-delay than the one before it. That method all character's opacity animation happens at a slightly distinct time.

Statement from IANA​

IANA provided the following declaration explaining the reasoning rearward the latest changes:

The changes we create to the location from time-to-time are normally informed by a desire to either decrease the general bandwidth demands of serving the site, or enhance its utility. At its bosom it is a placeholder for a location no-one have to be accessing, but we have historically showed a communication there to communicate visitors concerning why the domain is registered and how they may use it.

As it tends to be heavily trafficked, the general bandwidth is a key consideration, and to that end among the changes we made this week was to divided the leaf satisfied into a additional essential page, augmented by a distinct Javascript document alongside additional satisfied beyond that. Since most of the traffic to the location is automated it tends not to fetch the Javascript file, reducing general data requirements.

The essential intent of the domain is for use in documentation, specified as explanatory examples you may discover in instructions. We cognize that group may, for example, cut-and-paste a example configuration document and ignore to customize entries that merge an example domain, and we anticipate we get several secondary traffic from that. However, the domain isn't intended to be a broad intent endpoint for things akin preparedness evaluation so we do not advance that benevolent of usage. There is no necessity there is a HTTP assistance current on the presenter in command to fulfill its purpose, we fair run it as a courtesy.

January 20, 2002: What did the earliest type appearance like?​

According to Wikipedia, the example.com website launched on January 1, 1999. The earliest type accessible to perspective in the Wayback Machine is from January 20, 2002.

The earliest type of the website lone featured a catalog of domain names reserved by the Internet Corporation for Assigned Names and Numbers (ICANN) and IANA.

In the .com, .net, and .org top-level domains (TLDs), assorted domain names are reserved. These names are reserved from first registration, but to the degree they are already registered the existing registrant may renew them.

Structurally, the HTML used a array layout, alongside the ICANN logo positioned in the top remaining border of the page.

The server headers display that the location used Apache 1.3.22 at this point.

March 28, 2002: simplified messaging​

The archetypal type of the example.com wording we acknowledge today came 67 days following the archetypal accessible screenshot:

You have reached this web leaf by typing "example.com", "example.net", or "example.org" into your web browser.

These domain names are reserved for use in records and are not accessible for registration.

February 7, 2003: RFC 2606 citation added​

In February 2003, the satisfied was revised to contain a nexus to RFC 2606, the authoritative document for reserved top-level and second-level domains, so visitors could discover additional data on the website's purpose.

There was additionally a small specialized alter about this time: the server was upgraded to Apache 1.3.27 on Red Hat Linux.

July 30, 2010: .edu added to domain list​

The website afterward went unchanged for seven years. On July 30, 2010, a minor alter was made: example.edu was now pointed to the landing page, alongside the three former domains.

July 29, 2013: immigration and landing leaf redesign​

From January 2011, example.com didn't assist its own page. Instead, it redirected visitors to a leaf on IANA's website.

That changed on July 29, 2013, whenever the redirect was removed and example.com began serving its own leaf from the EdgeCast Content Delivery Network (CDN) alternatively of Apache, alongside a important redesign of the landing page. The new scheme used a rounded cardstock alongside an <h1> study 'Example Domain' and a new clarification of the website's purpose.

The viewable domain catalog was now gone, alongside a nexus to IANA's website for additional information.

October 17, 2019: font and copy changes​

A new update came six years following the biggest alter in the website's history. The layout remained the same, but the font stack was modernized to contain additional fallback fonts for devices specified as Macs, iPhones, and Windows PCs.

The copy was tweaked slightly, alongside 'is for use in explanatory examples' replacing 'established to be used' and 'in literature' replacing 'in examples'.

January 15, 2025: infrastructure switch​

Almost dozen years following the toggle from Apache to the EdgeCast CDN, another infrastructure alter took place: the Server header disappeared entirely. The response headers solitary don't disclose which phase was serving the leaf at this point.

A alter of provider about this period isn't surprising. EdgeCast had been acquired from Yahoo by Limelight Networks in 2022 and rebranded as Edgio. Edgio afterward submitted for Chapter 11 bankruptcy in delayed 2024, alongside Akamai acquiring choose Edgio client contracts.

October 9, 2025: redesign​

Later in 2025, the leaf was redesigned. The cardstock scheme was dropped, and the HTML was minified. The copy was changed to a additional straightforward message, again linking to IANA's website, alongside the nexus content changed from 'More information...' to 'Learn more'.

December 17, 2025: Cloudflare migration​

Less than a twelvemonth following the former switch, the infrastructure changed again: the location now serves from Cloudflare, as indicated by the cloudflare server header and CF-RAY header in the response. Whatever phase served the leaf following the January 2025 change, it has now been replaced alongside a Cloudflare-fronted setup.

June 9, 2026: favicon and HTML tweaks​

The next update, on June 9, 2026, brought no ocular changes. The lone HTML changes on the leaf itself were a favicon tag being added and the <p> tags being closed. The leaf now includes an bare favicon:

<link rel="icon" href="data:," />

This avoids a lot of unnecessary requests, as alternatively browsers appearance for a favicon at /favicon.ico.

Avoid relying on it for evaluation and monitoring purposes​

Example.com archetypal introduced an "Avoid use in operations" communication in 2025.

For developers, it's tempting to use example.com to inspect if the network is operating or test HTTP requests. The example server apt receives a lot of traffic through this.

However, as the leaf now clarifies, it's not a assistance you can depend on. When we appearance at our uptime monitoring data, we can see that all now and afterward the example.com server doesn't react correctly.

Using infrastructure that your institution controls can be additional reliable.

Conclusion​

Looking at the ocular timeline, we can see that changes to example.com happen sporadically. Sometimes a brace of small tweaks happen in back-to-back months. Other times the website remains untouched for periods of up to seven years.

We have been monitoring the page for multiple years in our community demo project. The Chrome UX Report (CrUX) data shows how leaf speed has changed complete time.

To study additional concerning how your website is performing, sign up for a liberated 14-day DebugBear trial.

Read next

Routes are computed in your browser complete 160,000 street segments, using USGS 1 m lidar height and the Overture / OpenStreetMap street graph. The slider runs alongside the entire trade-off between extend and climbing: all path that nothing alternatively strikes on the two counts, from the shortest way to the flattest one value walking (where a ft of ascend expenses 200 ft of walking; beyond that the routes halt being routes). Sliding correct never shortens the path and never adds climbing. Climbing is cumulative gain, not the net difference between the endpoints. Stairways are allowed on ft and excluded on a bike. Faint lines display the another routes alongside the way. Place hunt is offline: street intersections, places and addresses for San Francisco are built into this page. Source, data and the complete analysis.

In 2022, whenever YuChiang Cheng invested in Hero Bread on behalf of 444 Capital and stepped in as the company’s interim chief promotion officer, the low-carb, zero-sugar brand was at a crucial crossroads. 

Cole Glass had founded the business in 2019 to location his serious food allergies connected to pollen. By October 2021, a alliance alongside Subway seemed akin an breathtaking way forward. The idea was to get the macro-friendly bread products into as many hands as imaginable so group could sensation it for themselves. 

However, about the period Cheng joined the company, it was apparent the component economics didn’t create sense. The Subway agreement wasn’t profitable; Hero Bread was operating out of money. 

“ We were fair too small at the period and couldn’t assistance it, didn’t have the resources and all the infrastructure put in location to efficiently provision that,” Cheng says. “So we backed out of it.”

Hero Bread brought in $54 myriad in income final twelvemonth

Cheng pivoted to a direct-to-consumer (DTC) strategy, allowing the brand to obtain feedback from consumers additional quickly and run word-of-mouth sales, he says. 

Cheng assumed the function of CEO in November 2023 and has continued to develop the company.

In a backing circular announced in June 2024, Hero Bread raised $21 myriad in under 90 days. The round, co-led by Cleveland Avenue, DNS Capital and Composite Ventures, brought the company’s total backing to date to additional than $68.5 million. 

Hero Bread brought in $54 myriad in annual income final year. Additionally, the business additional than doubled its retailing existence in two years, expanding from concerning 4,000 to 10,000 doors, which includes a latest Target launch. 

But Hero Bread’s prosperous shove into retailing — a feat many CPG brands that initiate DTC anticipation to accomplish — doesn’t average the business plans to put the brakes on its DTC scheme anytime soon.

Entrepreneur sat downward alongside Cheng to study additional concerning what he calls the “third chapter” of Hero Bread’s DTC playbook, and how it’s continuing to unlock growth for the company. 

Hero Bread experimented alongside distinct retailing channels

After strolling distant from the Subway alliance and prioritizing DTC sales, Hero Bread focused on five SKUs: pale bread, seeded bread, burger buns, hot dog buns and flour tortillas. 

The line-up additionally made awareness as the business entered retailers: The goal was to recommendation mundane staples that would run reiterate purchases on a weekly basis. 

Initially, though, Cheng and the squad weren’t fairly certain which retailers would be the correct fit. 

“It’s a extremely mainstream sensation profile,” Cheng explains. “So there was a lot of debate, if this is going to be additional of a Publix, Target category product, or is this going to be additional of a natural Whole Foods, Sprouts, category of product.”

So Hero Bread opted to test alongside distinct types of retailers. The business launched in Market District, Publix and Sprouts and planned to lean into whichever conduit performed best. But six months later, the company’s products had an notable track document at all three. 

“We were very, extremely fortunate in that,” Cheng says. “So alternatively of picking [retail] channels, we decided to choice retailers that were additional enthusiastic despite of what conduit they were in.” 

It’s a distinct maneuver than many CPG brands use; they lean to zero in on one retailing conduit and go regional, the CEO notes. 

Striking the balance between affordability and convenience

Of course, CPG brands that develop into retailing have the advantage of economies of scale. 

“You can move lots of pallets on a truck much additional affordably than you can dispatch idiosyncratic packages through ecommerce,” Cheng says. 

As a result, retailing typically gives consumers a additional affordable way to buy. DTC, on the another hand, serves as a additional concierge way to acquisition not lone Hero Bread’s staple products, but additionally another offerings that power not be capable to hit retailing shelves as seamlessly. 

Hero Bread’s Crafted Collection, which includes items akin croissants, citrus poppy kernel scones, shortbread biscuits (hand-baked in Sonoma alongside brown margarine made by a Michelin chef), lends itself fine to the model. The company’s latest collaboration on pretzel bites alongside Auntie Anne’s does too. 

“ [These are] small-batch, mostly handmade products that are exclusive to hero.co, anywhere it fair wouldn’t create financial awareness for us to hand-roll 300,000 to 400,000 croissants,” Cheng explains. 

What’s more, DTC can act as an priceless evaluation dirt for products alongside the possible to market efficiently in a brick-and-mortar setting. 

That’s exactly what the business did alongside its bagels, unveiling a limited-edition initiate online in December 2024. 

“ We weren’t certain if group would desire to buy bagels from us,” Cheng says, “so we initially made them in smaller batches, sold them at hero.co, and they quickly became one of our best-selling items.” 

Hero Bread’s bagels landed on retailing shelves inside six months. 

Meeting Hero Bread’s consumers anywhere they may be

Cheng doesn’t accept a successful entrepreneur or endeavor chief is defined by how many times they win. 

“It’s how many times you can try again, despite of how many times you’ve fallen downward or how much lifeblood you get on your face,” the CEO says. “There is a flat of bravery that is really necessary.” 

It’s a inclination toward resilience that’s helped Cheng guide Hero Bread through several crucial years that required an enthusiastic willingness to modify to all difficulty — and the CEO looks onward to what comes next. 

“We’re continually trying to encounter the person anywhere they are,” Cheng says. “There’s a particular Hero fan, and afterward there’s the overarching bread fan. We accept that the additional group who try us, the additional group who volition accept [in us].”

You don't anticipate to see Apple's latest and top products on sale, particularly not weeks following their first release. While there are plenty of discounts on Apple products this October Prime Day, including on several Apple Watches, I certainly wouldn't have predicted that Apple's latest flagship would get a cost cut. And yet, current we are. The Apple Watch Series 12 starts at $399. But during October Prime Day, Apple has the observe $50 off. With the discount, it now starts at $349, a awesome cost for a observe alongside all of Apple's latest wearable perks.

Apple Watch Series 12

Buy One Apple Watch, and Get a Second Apple Watch $300 off at T-Mobile

Get Deal

Get Deal

Buy One Apple Watch, and Get a Second Apple Watch $300 off at T-Mobile

The Apple Watch Series 12 comes alongside Apple's newest wearable processor, the S11 chip. This processor is what powers things akin Siri AI and Apple Intelligence, as fine as the new "Audio Intelligence," which can provision you a content recap of the final 15 seconds of an in-person conversation—a characteristic that has generated a bit of controversy. It additionally comes alongside a new bosom charge sensor, which can notably inspect your beat all five seconds, or 20 times a minute. This example supports up to 24 hours of power division existence on a sole charge, and its Ceramic Shield 2 display should forestall additional scratches and cracks than former Apple Watches in this category. Lifehacker elder health publishing company Beth Skwarecki thinks the Series 12 is value the upgrade at its first MSRP, so this is a bit of a no-brainer if you were already interested.

The lone caveat current is that the Series 12 isn't the lone Apple Watch discount value considering this October Prime Day. While the Apple Watch Ultra 4 isn't on sale, the Apple Watch SE 3 is. This isn't a "new" watch, per se, but it is Apple's finest observe for most people. You don't get any of the complete perks of the Series 12, but you do get the capable S10 chip, Apple's norm bosom charge monitor, sleep and exercise tracking, and the capability to dispatch and obtain texts. There's a presenter of fitness and health crossover here, and, if you opt for the cellular model, you'll nexus to the identical 5G network as the cellular Series 12. While $349 is a awesome starting cost for the Series 12, the SE 3 currently starts at $199 for October Prime Day. That's a durable cost to beat.

Samip Dahal, Bishwas Mandal, Serdar Gülbahar, Akshay Vegesna

October 2026Correspondence to s@qlabs.sh·Code·

TL;DR

• We current the archetypal zeroth-order method that is rivalrous alongside backprop at pretraining transformer tongue models. Dust perturbs activations (node perturbation) independently at all token, so all token is a virtual community associate and one onward continue evaluates them all in parallel.
• Dust approximates backprop closely at ample community (i.e. substantially additional compute) and in multiple settings equal exceeds it. This hints that in a compute-rich administration we power be capable to surpass backprop.
• Dust is orders of dimension additional productive than weight-space ES. From 1M tokens up, Dust is on the command of $10^3$ to $10^4$ times additional productive than a transformer implementation of EGGROLL, a state-of-the-art ES method, according to our extrapolations.
• Zeroth-order methods are extensively believed not to measure to ample networks. Strikingly, we discover larger models are additional population-efficient, not less: a 243M-parameter example outperforms a $120\times$ smaller example at most community sizes.
• Dust’s gradient estimates align improved alongside backprop’s as community grows, and remain fine aligned at all measure we test, up to 1B tokens, which is encouraging for scaling.

Contents

1 Introduction

Deep learning has been built about backprop, the lone credit project algorithm capable of training contemporary neural nets, including transformer-based tongue models. Backprop requires differentiability and produces first-order gradients, and profound learning’s architectures, optimizers, and hardware have co-evolved about this constraint.

However, as the amount of compute accessible in the earth increases, we power favor additional generic and brute-force learning algorithms according to search complete inductive biases akin differentiability, backprop, and approximations of higher-order gradients. The bitter lesson (Sutton, 2019Richard S. Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019. Blog post.) is that broad methods that measure alongside compute eventually win, and AlphaGo Zero (Silver et al., 2017David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the equivalent of Go without individual knowledge. Nature, 550 (7676): 354–359, 2017. doi: 10.1038/nature24270.) is the apparent example. Bootstrapping AlphaGo on individual data helped the network study faster initially, but alongside a lot of computation the purely self-play network overtook it. Similarly, differentiability and backprop power be fine inductive biases in the low-compute regime, anywhere they create learning efficient, but in the high-compute administration they bounds the area of architectures that work. Even inside an architecture, gradient-based methods neglect to examine the defeat landscape optimally (Liu et al., 2020Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad earth minima be and SGD can attain them. In Advances in Neural Information Processing Systems, quantity 33, 2020.). This power additionally explain why current neural nets necessitate enormous amounts of data to generalize. A additional elastic credit project algorithm according to hunt is apt an crucial stage towards much improved generalization.

In this paper, we aim to substitute backprop alongside a learning algorithm based much additional on brute-force computation and much small on analytic structure. We call it Dust. Dust is a zeroth-order optimization algorithm that perturbs activations, rewards all perturbation by how much it lowers the loss, and averages the reward-weighted perturbations complete a community to evaluation the gradient. Traditional ES methods that perturb weights (Salimans et al., 2017Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable substitute to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.), akin EGGROLL (Sarkar et al., 2025Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652, 2025.), measure alongside population, but scaling the community is costly since all associate must be materialized and evaluated. We eliminate the two expenses alongside the idea of virtual population, anywhere we evade materializing all associate by bypassing importance area entirely and alternatively perturb activations, as in node perturbation (Werfel et al., 2003Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient decline in linear feedforward networks. In Advances in Neural Information Processing Systems, quantity 16, 2003.; Widrow and Lehr, 1990Bernard Widrow and Michael A. Lehr. 30 years of adaptive neural networks: Perceptron, Madaline, and backpropagation. Proceedings of the IEEE, 78 (9): 1415–1442, 1990. doi: 10.1109/5.58323.). We do so independently at all token, so all token is a associate and one onward continue evaluates them all in parallel.

Activations are a additional engaging area to hunt complete than weights. Mechanistic interpretability has shown that reasoning, whether verbalizable or not, lives in the activations (Gurnee et al., 2026Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Verbalizable representations form a earth workspace in tongue models. arXiv preprint arXiv:2607.15495, 2026.; Lindsey et al., 2025Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, et al. On the existence discipline of a ample tongue model. Transformer Circuits Thread, 2025.), which method this method could rotate training into a hunt complete latent reasoning (Vegesna and Dahal, 2025Akshay Vegesna and Samip Dahal. Decoupling hunt and learning in neural net training. arXiv preprint arXiv:2509.10973, 2025.). We afterward brace the activation-space perturbation alongside a extremely generic credit project regulation that assigns distinct token-level rewards to distinct tier types in a transformer block. Those two biases, alongside alongside a few implementation particulars and effectiveness measures, akin avoiding interference between perturbed modules, are the entire algorithm.

We create the following contributions.

• We current the archetypal zeroth-order method that is rivalrous alongside backprop at pretraining transformer tongue models. At ample populations Dust exceeds backprop in multiple settings, which suggests that in a compute-rich administration we power be capable to surpass backprop.
• Dust is orders of dimension additional productive than weight-space ES. From 1M tokens up, Dust is on the command of $10^3$ to $10^4$ times additional productive than a transformer implementation of EGGROLL, according to our extrapolations.
• Contrary to accepted wisdom, larger models are frequently additional population-efficient, not less, and can create use of larger populations. This gives a new perspective of overparameterization as a larger hunt area alongside possibly improved geometry.
• Dust’s gradient estimates align improved alongside backprop’s as the community grows, and the alignment holds up at all measure we test, up to 1B tokens, which is encouraging for scaling.

The goal of this document is to lay the foundations of a search-based credit project algorithm that is rivalrous alongside backprop on the hardest task we could think of: pretraining transformers. We do not attempt to create it compute-efficient adequate to substitute backprop today. We additionally do not train the new kinds of neural nets it makes accessible, akin nets alongside an external program in the iteration or transformers looped complete many steps that backpropagation through period struggles to train. Both are remaining to forthcoming work.

2 Method

Dust plant as follows. We add Gaussian noise to the output of all linear layer, independently at all token, run a onward pass, and reward all token’s noise by the alter in defeat at that token. The reward-weighted noise, averaged complete draws, is the estimated error at the layer’s output, and its external merchandise alongside the layer’s input is the importance gradient. Attention internals get a type of it: they are credited through the estimated error at the notice output complete current and forthcoming tokens, alternatively of the tokens’ defeat directly. The center intuition is that during weight-space ES evaluates one community associate per onward pass, we measure one per token, in parallel, and a associate is materialized by adding noise to a hidden state, which is cheap. On a contemporary transformer a sole onward continue hence evaluates a community at smallest three orders of magnitude larger than weight-space ES. We depict all component in item below.

2.1 Activation-Space Perturbation

The bottleneck of development strategies is community size. Every associate needs its own perturbed copy of the weights and its own onward pass. EGGROLL (Sarkar et al., 2025Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652, 2025.) makes the copies cheap alongside low-rank perturbations, but all associate is motionless one sequence component of the batch, so the community is bounded by the onward passes one can afford. We perturb activations instead, independently at all token. At that token the network behaves as if a low-rank perturbation had been applied to the weights of the tier that produced the activations, without the perturbation always being materialized in the weights. We call this a virtual population. A sequence in a transformer has a few thousand tokens, so one onward continue evaluates a few thousand members per sequence alternatively of one. Every importance in the example is trained this way apart from the $2L$ residual mixing scalars, which are trained by average weight-space ES.

Adding noise to activations fairly than weights is node perturbation (Widrow and Lehr, 1990Bernard Widrow and Michael A. Lehr. 30 years of adaptive neural networks: Perceptron, Madaline, and backpropagation. Proceedings of the IEEE, 78 (9): 1415–1442, 1990. doi: 10.1109/5.58323.), and the customary disagreement for it is dimensionality (Ren et al., 2023Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling onward gradient alongside local losses. In International Conference on Learning Representations, 2023.; Werfel et al., 2003Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient decline in linear feedforward networks. In Advances in Neural Information Processing Systems, quantity 16, 2003.). A layer’s output has $d_{\mathrm{out}}$ entries and its weights have $d_{\mathrm{out}} \times d_{\mathrm{in}}$, so activation noise lives in a much smaller space. Naively, the dimensionality disagreement doesn’t clasp for transformers alongside ample activations throughout many tokens. The noise on one sequence is a $T \times d_{\mathrm{out}}$ tensor, which has at smallest as many entries as the importance matrix formerly $T \ge d_{\mathrm{in}}$. However, alongside per-token autonomous perturbations and rewards, what perturbing activations gives is a new, productive community alongside the token axis that is orthogonal to the lot axis EGGROLL already relies on.

2.2 Credit Assignment

For a linear tier $y_t = W x_t$ we jitter its output at all tokens, $y_t \to y_t + \sigma a_t$ alongside $a_t \sim \mathcal{N}(0, I)$ and $\sigma$ the noise scale, and run the onward pass. At all token $s$ we compute the centered defeat decrease $c_s = \tilde{\ell}_s - \ell_s$, anywhere $\ell_s$ is the perturbed defeat and $\tilde{\ell}_s$ is the average perturbed defeat at that token throughout draws evaluated together in the identical batched forward. The reward of the jitter at token $t$ is the defeat decrease at $t$ and, alongside a decay $\gamma$, the defeat reductions at the tokens following it, which the jitter additionally reaches through attention,

With $\gamma = 0$ a jitter is rewarded by its own token alone. We depart it to the tuning of Section 2.3 to decide which layers see the forthcoming tokens. One autonomous jitter of all tokens is a draw, and a community is $K$ draws. Averaged complete the draws, the reward-weighted noise

is the estimated error at the layer’s output, and its external merchandise alongside the layer’s input, which the onward continue already computed, summed complete tokens, is the importance gradient,

Backprop forms the identical external merchandise alongside the identical input. The lone difference is that it gets the output error from the sequence regulation and we get it from the population. For an embedding tier $x_t$ is one-hot, so the external merchandise is a scatter-add of $\hat g_t$ into the token’s row.

With unlimited community the estimator complete is the entire method, and all perturbation could go into one onward pass. Each draw’s reward-weighted noise is the gradient affirmative an error alongside no preferred direction. Over draws the gradient adds up linearly during the errors add up as the quadrate root, so their proportion falls as the community grows and the interference vanishes in the limit.

2.3 Interference and Tuning

At a community we can afford, the chief disbursal is interference, ie we jitter many layers at many tokens in the identical onward pass, so the defeat alter that rewards one token’s noise additionally picks up the consequence of all another perturbation in that pass. We decrease specified interference in three ways. First, distinct tier types are jittered in distinct onward passes, all alongside its own noise scale, and all obstacle gets its own passes. These passes are cheaper than complete forwards since the spotless onward is cached and a diagram for obstacle $l$ lone reruns blocks $l$ onward. Second, the notice internals (query, key, value, gate, value embedding) are jittered separately. Token losses barely enroll their jitters, so they are rewarded through the notice output instead, as described below. Third, the tongue modeling caput is jittered immediately on the cached logits, re-evaluating lone the cross-entropy and lone a slab of the vocabulary per draw, which expenses a small fraction of a onward continue and lets the caput run a much larger population.

For the notice internals we recompute only their block’s notice output alongside the jitter, from the cached spotless activations. We mark the jitters using the alignment alongside the estimated notice output gradients,

where $\Delta o_s$ is the alter the jitter makes in the notice output at token $s$ and $\hat g_s$ is that output’s estimated error from Equation 2. The reward is Equation 1 alongside these scores, computed per head, in location of the defeat reductions.

The hyperparameters, the noise measure of all tier type, the credit decay of the notice internals and the portion of the community all tier gets, can be tuned in two ways. One is a grid hunt that trains alongside all environment on a small token prosperity and keeps the environment that lowers the defeat most. It is dependable but expensive. The another is a grid hunt that maximizes the cosine between our evaluation and the backprop gradient on a sole batch, which needs no training at all. A larger cosine on one lot doesn’t continually lesser the defeat following training, so the cosine picks candidates and training decides. Either way the tuning is mostly a one-time cost, since the settings it finds mostly generalize throughout token budgets and community sizes, alongside one exception. At the largest community at 10M and 20M tokens, a slower credit decay and a change of draws toward the notice flank motionless pay (Appendix F). Hence, this hunt recovers broad principles of the method fairly than settings for one run. As expected, all layers necessitate no credits from forthcoming tokens apart from the keys, values, gates and value embeddings, which are peruse by the afterward tokens that act to them and get a $\gamma$ near to one.

3 Pretraining Without a Backward Pass

3.1 Setup

We train GPT-style transformers on FineWeb alongside a 4096-token BPE tokenizer, a lot of 16k tokens (8 sequences of 2048 tokens), one epoch, and SGD alongside momentum at a changeless learning rate. The basis example has 8 layers and width 512. Every method gets the identical protocol and three seeds per cell, and is tuned separately at all token prosperity and population. Dust and backprop portion one momentum and learning-rate grid; we execute EGGROLL alongside the identical transformer architecture (named EGGROLL-Transformer) and it is tuned complete its own grid of stage size, momentum, noise measure and fitness shaping. Validation and test are held-out sets of 544 sequences each. We study the test defeat at finest validation checkpoint.

We figure community in draws for Dust and in onward passes of the lot for EGGROLL (Sarkar et al., 2025Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652, 2025.), our weight-space baseline. A diagram is one jitter of activations throughout all tokens of selected layers, rewarded alongside the token losses, and the community $K$ is the figure of draws per update. A diagram is slightly cheaper than a onward pass, since the spotless onward is cached and a diagram that jitters obstacle $l$ reruns lone the blocks from $l$ on. $K$ leaves out the draws of the caput and the notice internals, which are a small fraction of the update’s FLOPs (Appendix E). Taken together, from a community of 256 up Dust uses small compute than EGGROLL at the identical population, so the difference is lenient to EGGROLL.

3.2 Main Results

We clear the token prosperity from 100k to 20M against populations from 64 to 16k, alongside backprop tuned on the identical grid at all prosperity (Table 1, Figure 2). At 100k and 1M tokens Dust ends below backprop, from a few hundred draws at 100k and from a thousand at 1M. At 10M and 20M tokens, the gap alongside backprop shrinks alongside population. At 10M the ladder has flattened and its fitted bounds lands fair complete backprop. At 20M the ladder is motionless falling at 16k draws. Its power law fit puts the bounds at 4.431 (95% duration 3.89 to 4.58), below backprop’s 4.633, but alongside the ladder motionless falling the fit is loosely constrained, so we peruse it as evidence that the gap keeps decision alongside community fairly than as a measured limit.

Weight-space ES is far small efficient. With 256 times the population, EGGROLL at 16k motionless does not attain Dust at 64 draws. It comes inside 0.02 at 100k tokens and stays 0.4 to 0.6 complete at 1M, 10M and 20M (Figure 2). EGGROLL’s ladder is motionless falling steeply at 16k, so it would keep improving alongside additional population, but continuing the ladder puts what it needs to equivalent Dust’s smallest community at multiple thousand to concerning $10^4$ times that community (Appendix D). Figure 1 additionally shows the gap during training; Dust at 1k and 16k draws stays near to backprop’s curve throughout, during all EGGROLL curve falls rearward first and flattens fine complete Dust at 64 draws.

3.3 Dust Under Adam

While we concentration chiefly on SGD for the remainder of the paper, we reiterate the 1M token ladder alongside Adam on all three methods (Figure 3), re-tuning backprop’s learning rates, Dust’s own hyperparameters and EGGROLL’s stage size, momentum and fitness shaping at all population. Interestingly, EGGROLL gains nearly nothing from Adam and its tuned Adam ladder lands inside 0.01 of its SGD ladder in Table 1 at all population. However, Adam improves the two Dust and backprop and leaves the ladder’s form akin to before. Dust closes on backprop alongside a ample community and the duration on its bounds sits below backprop (Figure 3). So Dust’s evaluation already plant alongside contemporary optimizers, equal although contemporary optimizers were optimized for backprop gradients. We doubtful that coevolution of optimizers alongside Dust could guide to additional gains and depart this to forthcoming work.

4 Search in High-Dimensional Space

4.1 Overparameterization

The accepted perspective is that zeroth-order methods cannot train ample networks (Lillicrap et al., 2020Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21 (6): 335–346, 2020. doi: 10.1038/s41583-020-0277-3.). A onward continue returns a sole scalar, so the variance of the gradient evaluation grows alongside the figure of perturbed dimensions, and alongside it the community needed for a helpful update (Nesterov and Spokoiny, 2017Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17 (2): 527–566, 2017. doi: 10.1007/s10208-015-9296-2.; Werfel et al., 2003Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient decline in linear feedforward networks. In Advances in Neural Information Processing Systems, quantity 16, 2003.). We test this immediately by training four sizes, 2M, 7M, 38M and 243M parameters (a $120\times$ range in indicator count), at a fixed 10M tokens throughout the community clear (Figure 4; see Appendix F for tuning details). We create two striking observations that difficulty accepted wisdom.

• Bigger models are frequently additional population-efficient, not less. At all community from 256 up the defeat falls from 2M to 7M to 38M parameters, and the 243M example is lone slightly worse. Even at the smallest community we test, a example $120\times$ larger does concerning the same, and is improved at all another population. Up to a point, a larger example gains additional from its size than it loses to variance. What we observe, however, is that the gap to backprop at the identical community and example size increases alongside example size, but barely noticeable, esp at ample community sizes.
• Bigger models keep improving alongside community anywhere small ones flatten. Small models saturate sooner, during big models keep improving alongside ample community sizes. Past 1k draws the 38M and 243M models acquire concerning 30% additional than the 2M and 7M ones, and no amount of community makes the small example competitive.

The correct way to think concerning example size is hence as the size and the geometry of the hunt space. A larger example has a larger area to hunt over, which is what lets it put a ample community to use, and possibly a better-conditioned defeat landscape geometry, which could be why the hunt is additional productive equal at small populations.

4.2 Emergence of Backprop-Like Gradients

We measure the cosine between Dust’s evaluation and the backprop gradient on the identical batch, per tier category and per layer, on backprop-trained checkpoints spanning two orders of dimension in tokens, ie from 10M to 1B tokens, alongside the community expanding from 64 to 128k onward passes per stage (Figure 5). The cosine rises alongside community for all tier category at all phase of training, and a two-parameter law

fits all tier category alongside RMSE below 0.06, alongside $c_{\max}$ the ceiling and $c$ the community at which the tier reaches $c_{\max}/\sqrt{2}$. Useful gradients emerge from the ample community alone, alongside nothing concerning the sequence regulation built in, fair ray tuning of the hyperparameters to maximize the cosine similarity as described in Section 2.3. For the 100M token checkpoint, the fits to tier category average cosine similarities have RMSEs of $0.0032$–$0.0367$ (see Appendix G for details). Dust’s gradients method backprop’s in cosine, but how near they get varies by tier category and by layer. EGGROLL’s estimate, measured the identical way at the identical figure of onward passes, is expanding alongside community but is motionless below 0.05 in cosine at 128k for all category but the head, which power explain why training on it went nowhere in Section 3.2.

Importantly, the cosines clasp up throughout most layers as the token figure grows, which is encouraging for scaling. Section 3 shows the community required to equivalent and exceed backprop grows alongside tokens. However, the cosine stays flat throughout two orders of dimension in tokens at ample populations, which method that at a ample adequate community it’s imaginable that this necessity no longer grows alongside additional tokens. Furthermore, the fact that Dust’s gradients method backprop’s but don’t converge to them exactly is in fact a fine property. The evaluation points in a akin direction without being backprop’s gradient, which leads to a distinct optimization trajectory, and in our experiments that trajectory can equal be improved than backprop’s (Section 3.2).

5 Conclusion

Since Rumelhart et al. (1986)David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323: 533–536, 1986., backprop has been the algorithm that trains neural nets, and the architectures, optimizers and hardware we have were all built about it. As compute becomes additional abundant, we think much improved alternatives are possible. We introduced Dust, an algorithm that drastically improves upon existing ES algorithms and approximates backprop closely at pretraining transformers, equal exceeding it alongside ample amounts of computation.

There are many engaging open questions. The archetypal is whether, and how, Dust can discover improved directions than backprop’s first-order gradient by implicitly exploring the defeat landscape, picking up higher-order curvature that pulls the hunt toward flat regions. We have hints that it can, since at ample populations it sometimes ends up below backprop, but the scheme is not clear. The second is that Dust opens up the hunt area complete architectures, since it does not need the network to be end-to-end differentiable, and it may do improved anywhere backprop is known to struggle, akin recurrent or looped computation trained by backpropagation through time. The third is compute efficiency, which was not the concentration of this paper. We must have orders of dimension additional compute effectiveness before Dust becomes a applicable substitute to backprop at current levels of compute.

Early activity on zeroth-order pretraining of tongue models (Allaire et al., 2025Nathan Allaire, Mahsa Ghazvini Nejad, Sébastien Le Digabel, and Vahid Partovi Nia. Zeroth command optimization for pretraining tongue models. In Proceedings of ICPRAM, pages 113–121, 2025. doi: 10.5220/0013261100003905. URL https://doi.org/10.5220/0013261100003905.) examined the difficulty of training transformers from scratch alongside importance perturbations. Their afterward method, KronZO (Allaire et al., 2026Nathan Allaire, Sébastien Le Digabel, Dominique Orban, and Vahid Partovi Nia. Zeroth-order Kronecker optimization for pretraining tongue models. SN Computer Science, 7, 2026. URL https://www.gerad.ca/en/papers/G-2025-44. Article 162.), uses compact perturbations alongside Kronecker construction and selective directional updates to enhance pretraining during reducing recollection use. EGGROLL (Sarkar et al., 2025Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652, 2025.) makes ample populations of importance perturbations productive on GPUs through low-rank structure. Dust alternatively searches complete activations and uses rewards for all token to extract additional credit from all onward pass.

For fine tuning, MeZO (Malladi et al., 2023Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning tongue models alongside fair onward passes. In Advances in Neural Information Processing Systems, quantity 36, 2023.) showed that tongue models can be adapted alongside onward passes solitary and recollection use near to inference. Evolution Strategies at Scale (Qiu et al., 2026Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Meyerson, Babak Hodjat, and Risto Miikkulainen. Evolution strategies at scale: LLM fine-tuning beyond reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2509.24372. arXiv:2509.24372.) demonstrates fine tuning of all parameters alongside ES in tongue models alongside billions of parameters. Neural Thickets (Gan and Isola, 2026Yulu Gan and Phillip Isola. Neural thickets: Diverse project experts are compact about pretrained weights. arXiv preprint arXiv:2603.12228, 2026. URL https://arxiv.org/abs/2603.12228.) finds helpful project experts by randomly perturbing pretrained weights, selecting the finest candidates and ensembling their predictions. These results display how much hunt can accomplish about a pretrained model. Our experiments location learning the representations themselves through pretraining from scratch.

Our activation perturbations build on node perturbation (Werfel et al., 2003Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient decline in linear feedforward networks. In Advances in Neural Information Processing Systems, quantity 16, 2003.). GEMINI (Le Cun et al., 1988Yann Le Cun, Conrad C. Galland, and Geoffrey E. Hinton. GEMINI: Gradient evaluation through matrix inversion following noise injection. In Advances in Neural Information Processing Systems, quantity 1, pages 141–148, 1988. URL https://papers.neurips.cc/paper_files/paper/1988/file/a0a080f42e6f13b3a2df133f073095dd-Paper.pdf.) injected noise into the archetypal hidden tier and restored layerwise gradient estimates through iterative matrix inversion. Zoop (Hu et al., 2025Xixi Hu, Bo Liu, Qiang Liu, Xiaocong Du, Bhargav Bhushanam, Louis Feng, Chengyue Gong, and Kaizhao Liang. Zoop it! Efficient zero-order optimization alongside output perturbation. In ICML Workshop on Tiny Titans: The next motion of On-Device Learning for Foundation Models, 2025. URL https://openreview.net/forum?id=Tc8vFyRhPO.) uses output perturbations for tongue example fine tuning, converting estimated output gradients into indicator updates alongside local derivatives. Scaling Forward Gradient alongside Local Losses (Ren et al., 2023Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling onward gradient alongside local losses. In International Conference on Learning Representations, 2023.) combines activation perturbations and local losses alongside onward manner automatic differentiation to decrease estimator variance. Forward gradients alongside multiple tangents (Flügel et al., 2025Katharina Flügel, Daniel Coquelin, Marie Weiel, Charlotte Debus, Achim Streit, and Markus Götz. Beyond backpropagation: Optimization alongside multi-tangent onward gradients. In International Joint Conference on Neural Networks, 2025. URL https://arxiv.org/pdf/2410.17764v2.) additionally use onward manner differentiation, combining multiple directional derivatives through orthogonal projection to enhance gradient estimates.

A distinct row of activity replaces the earth backward continue alongside local learning dynamics: Sakana AI’s PC-ALM (Seely and Gould, 2026Jeffrey Seely and Julian Gould. Augmented Lagrangian predictive coding. arXiv preprint arXiv:2605.31022, 2026. URL https://arxiv.org/abs/2605.31022.) propagates credit through local predictive coding dynamics and Lagrange multipliers. It uses local derivatives and is evaluated on depiction classification tasks. Dust estimates credit from onward perturbations during transformer pretraining, combining rewards for all token alongside local targets for notice outputs.

References

Nathan Allaire, Mahsa Ghazvini Nejad, Sébastien Le Digabel, and Vahid Partovi Nia. Zeroth command optimization for pretraining tongue models. In Proceedings of ICPRAM, pages 113–121, 2025. doi: 10.5220/0013261100003905. URL https://doi.org/10.5220/0013261100003905.

Nathan Allaire, Sébastien Le Digabel, Dominique Orban, and Vahid Partovi Nia. Zeroth-order Kronecker optimization for pretraining tongue models. SN Computer Science, 7, 2026. URL https://www.gerad.ca/en/papers/G-2025-44. Article 162.

Katharina Flügel, Daniel Coquelin, Marie Weiel, Charlotte Debus, Achim Streit, and Markus Götz. Beyond backpropagation: Optimization alongside multi-tangent onward gradients. In International Joint Conference on Neural Networks, 2025. URL https://arxiv.org/pdf/2410.17764v2.

Yulu Gan and Phillip Isola. Neural thickets: Diverse project experts are compact about pretrained weights. arXiv preprint arXiv:2603.12228, 2026. URL https://arxiv.org/abs/2603.12228.

Wes Gurnee, Nicholas Sofroniew, Adam Pearce, Mateusz Piotrowski, Isaac Kauvar, Runjin Chen, Anna Soligo, Paul Bogdan, Euan Ong, Rowan Wang, Ben Thompson, David Abrahams, Subhash Kantamneni, Emmanuel Ameisen, Joshua Batson, and Jack Lindsey. Verbalizable representations form a earth workspace in tongue models. arXiv preprint arXiv:2607.15495, 2026.

Xixi Hu, Bo Liu, Qiang Liu, Xiaocong Du, Bhargav Bhushanam, Louis Feng, Chengyue Gong, and Kaizhao Liang. Zoop it! Efficient zero-order optimization alongside output perturbation. In ICML Workshop on Tiny Titans: The next motion of On-Device Learning for Foundation Models, 2025. URL https://openreview.net/forum?id=Tc8vFyRhPO.

Yann Le Cun, Conrad C. Galland, and Geoffrey E. Hinton. GEMINI: Gradient evaluation through matrix inversion following noise injection. In Advances in Neural Information Processing Systems, quantity 1, pages 141–148, 1988. URL https://papers.neurips.cc/paper_files/paper/1988/file/a0a080f42e6f13b3a2df133f073095dd-Paper.pdf.

Timothy P. Lillicrap, Adam Santoro, Luke Marris, Colin J. Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21 (6): 335–346, 2020. doi: 10.1038/s41583-020-0277-3.

Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, et al. On the existence discipline of a ample tongue model. Transformer Circuits Thread, 2025.

Shengchao Liu, Dimitris Papailiopoulos, and Dimitris Achlioptas. Bad earth minima be and SGD can attain them. In Advances in Neural Information Processing Systems, quantity 33, 2020.

Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning tongue models alongside fair onward passes. In Advances in Neural Information Processing Systems, quantity 36, 2023.

Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17 (2): 527–566, 2017. doi: 10.1007/s10208-015-9296-2.

Xin Qiu, Yulu Gan, Conor F. Hayes, Qiyao Liang, Yinggan Xu, Roberto Dailey, Elliot Meyerson, Babak Hodjat, and Risto Miikkulainen. Evolution strategies at scale: LLM fine-tuning beyond reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2026. URL https://arxiv.org/abs/2509.24372. arXiv:2509.24372.

Mengye Ren, Simon Kornblith, Renjie Liao, and Geoffrey Hinton. Scaling onward gradient alongside local losses. In International Conference on Learning Representations, 2023.

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323: 533–536, 1986.

Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable substitute to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.

Bidipta Sarkar, Mattie Fellows, Juan Agustin Duque, Alistair Letcher, Antonio León Villares, Anya Sims, Clarisse Wibault, Dmitry Samsonov, Dylan Cope, Jarek Liesen, Kang Li, Lukas Seier, Theo Wolf, Uljad Berdica, Valentin Mohl, Alexander David Goldie, Aaron Courville, Karin Sevegnani, Shimon Whiteson, and Jakob Nicolaus Foerster. Evolution strategies at the hyperscale. arXiv preprint arXiv:2511.16652, 2025.

Jeffrey Seely and Julian Gould. Augmented Lagrangian predictive coding. arXiv preprint arXiv:2605.31022, 2026. URL https://arxiv.org/abs/2605.31022.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the equivalent of Go without individual knowledge. Nature, 550 (7676): 354–359, 2017. doi: 10.1038/nature24270.

Richard S. Sutton. The bitter lesson. http://www.incompleteideas.net/IncIdeas/BitterLesson.html, 2019. Blog post.

Akshay Vegesna and Samip Dahal. Decoupling hunt and learning in neural net training. arXiv preprint arXiv:2509.10973, 2025.

Justin Werfel, Xiaohui Xie, and H. Sebastian Seung. Learning curves for stochastic gradient decline in linear feedforward networks. In Advances in Neural Information Processing Systems, quantity 16, 2003.

Bernard Widrow and Michael A. Lehr. 30 years of adaptive neural networks: Perceptron, Madaline, and backpropagation. Proceedings of the IEEE, 78 (9): 1415–1442, 1990. doi: 10.1109/5.58323.

Appendix

← Back to Q

Well, sort of — since the Matic wasn’t banned to commencement with.

Here’s what’s really happening: On October 2nd, the FCC added the first Matic, example 4MA0001, and its charging dock example 4DK0008 to its ban waiver list. That allows the FCC to endorse these broadcast products for import, marketing, and sale.

But the Matic doesn’t need a waiver, since its radios were already approved years ago. The FCC’s abroad drone, robot, and router bans lone keep future products from getting imported into the United States, not existing ones whose radios have already been approved.

Besides, Matic’s products don’t get FCC endorsement to commencement alongside — instead, Matic ships them alongside an off-the-shelf Intel AX210 Wi-Fi and Bluetooth part that gets its own endorsement from the FCC. (It’s the identical benevolent of Wi-Fi cardstock you’ll discover inner a desktop or laptop computer.) Matic additionally doesn’t need to concern concerning an import ban specifically, since Matic assembles its robots correct current in Mountain View, California.

A waiver could theoretically assistance if Matic were to build new products exterior the United States and try to import them here, apart from that the FCC hasn’t granted Matic a covering endorsement on a brand of robot, akin it did with Asus and Netgear’s routers. Instead, it specifically granted a waiver for the Matic models 4MA0001 and 4DK0008, which, again, have already made it through the FCC.

The waiver power not be entirely symbolic. The FCC is currently trying to near many of the loopholes that currently create its ban impotent, is weighing whether to create manufacturers disclose all the components in their gadgets, and is retroactively cracking down on several companies it sees as flaunting the rules. While Matic assembles its robots in the US, it admits it doesn’t yet origin 65 percent of its components from US suppliers, the threshold for it to be seen as Made in the US, and perchance one of those abroad components could trip it up if not for the waiver.

Even so, Matic already gets its “core compute platform” from Nvidia, co-founder and CEO Mehul Nariyawala tells The Verge, and sources another “critical components” from US companies “whenever possible.”

And dissimilar several of the another companies anywhere the FCC has mysteriously granted a waiver, Matic is committing to additional US manufacturing and onshoring.

“Up till 3 months ago, the two our R&D offices and manufacturing accommodation were in the identical agency in Mountain View, concerning 24,000 quadrate feet,” Nariyawala tells The Verge. “We’ve turned that complete facility into manufacturing facility and moved our agency to Menlo Park.”

He says Matic went from shipping 1,500 robots a twelvemonth ago to 17,000 now, and is trying to discover additional and additional US suppliers so it can use equal small abroad parts.

“At the instant we power be the lone flat cleaning robot built in America,” he says. “That is becoming among the key value propositions for our product.”

For a lengthy time, there’s been an serious run in device recollection investigation to create materials in between these two extremes. For this purpose, it helps to have a apparent image of what these extremes mean.

Today, I’ll portion what we found. A squad of AI agents and I designed one applicant magnet and established another, archetypal made in 1999, that our calculations foretell has the properties we were after.

But before diving into the details, let me archetypal lay out a magnet primer that takes all of 90 seconds, assuming you are not an undergraduate in discipline or chemistry.

A 90 second primer in magnets

Spin

Each speck has a quantum mechanical asset called ‘spin’, which is liable for its magnetic moment. We can example the rotation direction for all speck as either pointing up or down.

Spintronics

We use rotation for storage. A magnetised matter stores data according to the rotation up/spin downward inclination of its electrons, in the identical way traditional magnets shop data according to pointing up/down. The most notable example of spintronics is the difficult run peruse head, the equipment that says the magnetic bits on the disk. MRAM is another category of spintronics that uses the identical regulation to shop binary data in a non-volatile way.

In the earth of spintronics, we desire to kind electrons by their rotation inclination so we can read/store their information. Ferromagnets do this naturally: the electrons that transport current are mostly of one spin, up or down. Ordinary antiferromagnets, however, cannot differentiate up/down electrons.

This leads us into the next section.

Three kinds of magnets

Ferro

Ferromagnetic materials are characterised by having a macroscopic magnetic field, or a site that leaks out from the surface. This is why a fridge magnet sticks to your refrigerator door.

The problem, however, is that the magnetic site interferes alongside nearby materials and is difficult to authority for retention purposes. In addition, switching magnets rear and onward is comparatively dilatory and consumes a lot of power.

Another characteristic of ferromagnetic materials is that the spins are sorted (by up/down orientation) according to energy level. We can see this whenever looking at the energy spectrum of the electrons: near the edges of the spectrum, the electrons all have the identical spin.

Antiferro

If we obtain the complete and flip the logic, meaning, the spins are unsorted, we end up alongside an average antiferromagnetic material.

Here the electrons of the identical energy flat volition alternatively have blended spins. This leads us to two properties of interest:

• Because spins are not sorted according to energy level, it is difficult to read/store data using spintronics techniques.
• However, the deficiency of macroscopic magnetic site allows us to group these materials nearer together, enabling higher achievement for retention devices. In conditions of speed, AF materials are additionally concerning a thousand times faster to switch.

The lone issue is, if the spins are blended at all energy level, how would we always kind them by energy? How do we get a way to read/store data using spintronics alongside this deficiency of sorting?

Luttinger compensated

This brings us to a third type, Luttinger compensated (LC).

LC materials are antiferromagnets anywhere the spin-up atoms and spin-down atoms have the identical dimension of magnetism, making the net rotation instant zero (i.e. they cancel out). However, dissimilar in average antiferromagnets, the up and downward atoms sit in inequivalent environments: they can be distinct elements (e.g. one component points up and another points down), or the identical component in two distinct kinds of site. The name comes from Luttinger’s theorem: in an insulator, the net rotation instant of all repeating component of the gemstone must be a entire number, so formerly it is zero it stays locked at zero. Strictly, that holds for a ideal gemstone near complete zero; smaller effects specified as spin–orbit coupling, and heat, can depart a minor imbalance.

Since the up and downward atoms are not equivalent, up/down spins can now be divided (sorted) by energy, fair akin in ferro materials.

Let’s obtain the former sections and appearance at how rotation is distributed throughout the energy landscape for ferro, antiferro and LC magnets.

What matters for retention is the “spin window”: the piece of energy at the border of the collection gap anywhere all accessible speck province has the identical spin. The larger this opening compared alongside the thermal jiggling at area heat (about 26 meV), the improved the electrons remain sorted.

This method we’d affection to have a semiconductor alongside a collection gap, without losing the capability to distinct spins by their energy levels, and alongside zero net magnetism.

This is anywhere our AI agents (and me) act the scene. Let’s dive into how these agents established two promising materials for next generation device memory.

The agents ran quantum-mechanical simulations of all gemstone alongside the norm method for this, density functional theory, at two levels of approximation: a faster one (PBE+U) and a slower, normally additional exact one (HSE06). The collection gaps and rotation windows below arrive from the additional exact one.

Candidate 1: Designed a Luttinger Compensated Magnet, YBaMnFeO₅

First, let’s see what our AI agents designed:

• A new compound made of lone 5 elements (yttrium, barium, Mn, Fe, O). As far as we could find, it has never been made, nor projected as this benevolent of magnet
• The compound was predicted to be a semiconductor
• This Luttinger-compensated magnet is predicted to have a 2.35 eV collection gap, anywhere rotation sorting occurs at the two sides of the collection gap: a opening of 1.0 eV for holes and 1.4 eV for electrons. Note that thermal agitation at area heat lone causes about 26 meV of fluctuation
• Finally, our delegate predicted the matter to keep magnetism up to an elevated temperature: concerning 420 K in the raw simulation, or concerning 490 K following calibrating the simulation against a known magnet

While this seems akin a beautiful promising candidate, the scheme needs the Mn and Fe atoms to sit in a ideal checkerboard. When our agents simulated how the atoms arrange themselves at distinct temperatures, that checkerboard cut distinct into a random mix at about 950 K. Making this benevolent of oxide takes concerning 900–1300 °C, and at lesser temperatures the atoms barely move, so norm synthesis would apt provision a scrambled crystal.

A scrambled gemstone loses the rotation sorting, so this scheme may be difficult to create in its helpful form.

Candidate 2: Identified a Luttinger Compensated Magnet in KV[Cr(CN)₆] from 1999

Shortly after, my delegate established this engaging material, earlier reported lone in 1999, that matches many of the requirements above. Its zero net magnetism is not new: the chemists who made it designed the two metals’ magnetism to cancel. Even its rotation sorting was already on paper. A 2008 study using hybrid functionals akin ours plotted its speck states rotation by spin, and the two collection edges in that storyline transport the identical spin. But that study was concerning magnetic coupling under force and never remarked on it. As far as we found, nobody had pointed out that this makes KV[Cr(CN)₆] a Luttinger-compensated semiconductor, put numbers on its spin-sorted windows, or tested how sturdy they are. It had been hiding in plain sight.

KV[Cr(CN)₆] belongs to the identical family as Prussian blue, the 300-year-old pigment.

Our delegate identified it as a Luttinger-compensated material:

• It predicts the matter volition have a collection gap of concerning 2.1 eV, alongside the two collection edges sorted into the identical spin: windows of 2.6 eV for holes and 1.6 eV for electrons.
• The 1999 example stayed magnetically ordered up to 376 K (103 °C), complete area temperature, as measured by the chemists who made it (365 K following the example had been heated).
• And most importantly, its construction locks all metal element into its own site: chromium bonds to the carbon end of all cyanide and vanadium to the nitrogen end, which is exactly what YBaMnFeO₅ lacked.

These predictions are for a perfect, dry crystal. The lone example so far, from 1999, is a powder alongside h2o in its pores, and it showed a small leftover magnetic instant (0.125 Bohr magnetons per equation unit, anywhere the ideal gemstone would have zero). Our two simulation methods differ on how much the h2o weakens the effect: the additional exact one (HSE06) says the rotation sorting survives, during the faster one (PBE+U) says the gap opening shrinks by additional than half. Neither the collection gap nor the rotation sorting has been measured yet.

The base line

We identified a matter that combines properties of the two acquainted kinds of magnet and could be helpful in spintronic devices. We accept the two materials discussed in this part are promising examples of a category of materials that have the two antiferromagnetic and ferromagnetic properties.

The key features for the two materials: in our calculations for ideal crystals they have zero net rotation moment, yet they are semiconductors whose electrons are sorted by rotation complete windows far larger than the thermal jiggling at area temperature. With small stray magnetic field, they shouldn’t disturb their neighbours much.

Here is the mark of our two materials:

• YBaMnFeO₅: designed candidate; may be difficult to make.
• KV[Cr(CN)₆]: archetypal made in 1999. Near-zero net magnetic instant in the 1999 example (zero predicted for a ideal crystal), magnetic up to 376 K.

This suggests that materials for the next stage in spintronics may already exist, waiting to be recognized. A 2025 study that predicted two another Luttinger-compensated semiconductors established that the two endure their magnetic command below area temperature, and named a room-temperature one as the open goal. KV[Cr(CN)₆] has already been made, and its 1999 example stayed ordered up to 376 K. Identifying room-temperature Luttinger-compensated semiconductors is a stage toward applicable spin-based technologies, particularly if their magnetic properties can be controlled through chemistry. The next stage is to create KV[Cr(CN)₆] again and measure its rotation sorting directly.

In the soul of transparency, I invitation the audience to go through all the computations that produced the complete predictions, including the calculations rearward the applicant designs. The input files, raw outputs and inspection code rearward the numbers in this post, alongside alongside a one-command checker, autonomous re-runs and a catalog of known caveats, have been shared on GitHub in a community repository I created to document my investigation journey: github.com/spicylemonade/compensated-magnet-ledger.

If you bought one of Apple's M5 MacBook Airs at launch, it would have disbursal you $1,099 for the basis model. Then, the recollection emergency took its toll, and all of a sudden, that identical device attempt up in cost by $200 without any new features or changes. The MacBook Air, which for years has started at $999 and jumped to $1,099 alongside the M5 (while additionally giving you additional storage, at least), now starts at $1,299. If you desire additional storage, or to upgrade to the 15-inch model, it'll lone disbursal you more.

These prices lone shove customers towards Apple's small costly MacBook Neo. That's not necessarily a bad thing: the Neo comes alongside most of the features that group are looking for in an Apple laptop, at concerning fractional the disbursal of the MacBook Air. But these are not identical machines. Users choosing the Neo complete the Air endure out on a bigger, higher-quality display; a processor made for macOS; a backlit keyboard; and MagSafe charging, fair to name a few perks. These are all outstanding features, but it's durable to validate the additional $600. But what if you could preserve $200? That makes those perks a bit additional value the upgrade.

M5 MacBook Air (13-inch)

$1,099.00 at Amazon

$1,299.00 Save $200.00

Get Deal

Get Deal

$1,099.00 at Amazon

$1,299.00 Save $200.00

For October Prime Day, Amazon has an outstanding agreement on the 13-inch M5 MacBook Air, taking $200 off Apple's new and beefier MSRP, bringing the disbursal downward to...well, the M5's initiate price. If you've been kicking yourself for not purchasing one of these machines between March and June, now's your chance to rotate rear the clock.

This MacBook Air comes alongside a 13.6-inch display (Apple calls it the "13-inch" for promotion purposes), alongside a resolution of 2560 by 1664 and a maximum brightness of 500 nits. You get the M5 chip, which, according to Geekbench, outperforms the MacBook Neo's A18 Pro by nearly 22% in single-core tasks and 134% in multi-core tasks. That additional $400 isn't fair going towards an "Apple tax," as you'll notice real-world achievement increases for your money. You'll additionally get a 12MP Center Stage camera for video conferencing and FaceTime calls, Touch ID built into the keyboard, and assistance for MagSafe charging—though you can additionally accusation using either of the two USB 4 ports.

Importantly, this example comes alongside 16GB of RAM and 512GB of storage. That have to be plentifulness for most users, but Amazon is additionally hosting discounts on the two the 1TB model ($1,399 alternatively of $1,599), as fine as the 1TB example alongside 24GB of RAM ($1,599 alternatively of $1,799). But again, most group volition create do fair fine alongside the basis configuration. And for $1,099, it's a awesome deal.

The Amazon October Prime Day sale officially starts tomorrow, but plentifulness of deals are already live, including several important cuts on iPads, Kindles, and another tablets. Right now, the 2024 Amazon Kindle is downward to $103.99 (originally $149.99), and it’s a agreement value considering for anyone in the market for an e-reader—especially after Amazon's cost hike before this year.

2024 Amazon Kindle 16GB

$103.99 at Amazon

$149.99 Save $46.00

Get Deal

Get Deal

$103.99 at Amazon

$149.99 Save $46.00

The essential 16GB example checks all the boxes of a straightforward e-reader: It’ll provision weeks of power division life, has a sharp, compact screen, and lacks the annoying notifications or apps competing alongside the publish you’re trying to read. It has a six-inch, 300-ppi glare-free display, an adjustable forefront light, Dark Mode to improved defend your eyes, USB-C charging, and plentifulness of retention room. 

At 5.56 ounces, it's Amazon’s lightest and most compact Kindle, making it a firm choice for commuters or travelers, alongside power division existence enduring up to six weeks per charge. The attractiveness is its simplicity; it’s designed to keep you immersed in your book, and is particularly convenient for those who buy ebooks through Amazon or use a archive app that supports Kindle delivery.

If you desire a heated study ray or waterproofing, you’ll have to flat up to the Kindle Paperwhite (also 32% off for Amazon Prime Day 2026). Still, at about a $100 cost point, the 2024 Amazon Kindle is motionless among the finest options for shoppers who are seeking e-reader simplicity without paying additional for Paperwhite features—or opting for a additional full-featured tablet.

Read next