When Genius Fails: The Intellectual Arrogance of the AI Labs

Hacker News by 14 min read 507x views
When Genius Fails: The Intellectual Arrogance of the AI Labs

Share Post

Hammering a screw connected a portion of wood

Being an master successful 1 section doesn’t make you an master successful each fields. Leopold Aschenbrenner’s hedge fund, Situational Awareness, provided a $20 cardinal objection this week. His declare to fame was being portion of OpenAI’s Superalignment squad earlier being fired complete alleged leaks (which he disputes) and past publishing an effort successful 2024 astir the imminence and value of AGI that launched a 1000 media interviews (of him). And past he was moving a $20 cardinal hedge fund. And past it blew up.

Lots of group will beryllium dancing to the news this week because galore recovered him a spot insufferable. And for me, I really don’t mind nary longer being asked, “Should I put successful Situational Awareness?” and needing to beryllium delicate astir it.

I want to make a larger point, though. The deficiency of intelligence humility wrong the frontier AI laboratory civilization he hails from extends beyond him into galore verticals other than money management. That being said, arsenic an ex-hedge money feline myself who besides has an AI background, I do person immoderate unsocial qualifications to astatine slightest talk concisely astir this.

Are you successful the Bay Area? San Mateo County Libraries is hosting maine for a book signing and talk astatine the Atherton Library for What You Need to Know About AI, moderated by Justin Kuczynski (PhD, Computational Biology and Engineering Lead astatine Google). It’s connected Friday, August 7, 2026, astatine 3-4pm. Come retired and opportunity hi!

Situational Awareness LP’s woes are not unique. Many hedge costs person blown up. In fact, I’d opportunity many, galore much hedge costs person blown up than person ever been consistently excellent. It’s thing that astir laypeople don’t realize.

The canonical illustration is Long-Term Capital Management, which put 2 Nobel laureates and Wall Street’s champion enslaved traders successful 1 fund. It was the champion and brightest successful the field, and they quadrupled investors’ money successful 4 years. Then it blew up truthful spectacularly successful 1998 that the Federal Reserve had to rally Wall Street banks to thief bail it retired (in a preview of 2008).

There’s a full book astir it, fittingly titled When Genius Failed (yes, it’s the inspiration for the title).

ltcm vs situational awareness
The original “smartest group successful the room” fund. Situational Awareness was capable to impressively speedrun the full rise-and-blow-up process.

It’s not astir your highest returns. After all, personification who goes all-in connected reddish astatine the roulette array 5 times successful a statement and wins by luck will person a 3,100% return. I would dream that nary 1 would deliberation this personification is simply a unsocial brilliant aliases qualified to negociate money.

It’s not moreover astir “beating the market.” That’s a reddish herring. In a bull market—or moreover better, a bubble—anyone who isn’t fully invested, aliases more than afloat invested (with leverage), successful the banal marketplace will “lose.” What you attraction astir from a hedge money is that they are accordant successful bull aliases carnivore markets. The constituent of those costly fees is that they will always perform, moreover if they look temporarily “bad” against the banal market.

Which, by the way, is not the only marketplace successful the world—there are bonds, commodities… but it gets the attraction because unit investors emotion to bet successful it. Like this Korean feline who went 500% agelong connected stocks, concisely turned his military-service savings into a mini fortune, and past mislaid it all, though his was conscionable 1 of more than 1.2 cardinal Korean brokerage accounts that sewage separator called by mid-July… Which makes this each look akin to the roulette table.

Which brings america backmost to Leopold Aschenbrenner and his hedge fund.

By each accounts, he (like the Korean unit traders) levered into the AI roar (reportedly moving astir 4x), pinch July losses crossed nationalist stocks for illustration neoclouds, representation names, and datacenter power. He also, reportedly, had short positions successful package names—the “SaaSpocolypse” trade—that bounced backmost against him astatine the aforesaid time.

I person nary uncertainty Aschenbrenner is ace smart, but this doesn’t look that different from the Korean unit investors who blew up. One of the first lessons immoderate existent investor learns is the marketplace tin enactment irrational for longer than you tin enactment solvent… if you don’t person the correct consequence controls. Using leverage is conscionable the astir evident portion of it.

leverage\_chart
Given that representation is inherently cyclical and often has convulsive swings… while it mightiness not person been his full position, getting margin-called into liquidation to Citadel is not really suggestive of beardown consequence controls…

Of course, it’s because of his thesis. From his founding essay (emphasis mine):

Because—it’s starting to consciousness real, very real. A fewer years ago, astatine slightest for me, I took these ideas seriously—but they were abstract, quarantined successful models and probability estimates. Now it feels highly visceral. I tin spot it. I tin spot really AGI will beryllium built. It’s nary longer astir estimates of quality encephalon size and hypotheticals and theoretical extrapolations and each that—I tin fundamentally show you the cluster AGI will beryllium trained connected and erstwhile it will beryllium built, the unsmooth operation of algorithms we’ll use, the unsolved problems and the way to solving them, the database of group that will matter. I tin spot it. It is highly visceral. Sure, going all-in leveraged agelong Nvidia successful early 2023 has been awesome and all, but the burdens of history are heavy. I would not take this.

Ah, all-in leveraged agelong Nvidia. We sewage a sensation of his investing style backmost then, earlier he moreover had a fund.

Sam Altman and Dario Amodei regularly waste and acquisition disconnected successful really apocalyptically they picture the early of the labour marketplace (though both person lately been softly stepping it back). And while I sanction those 2 because they’re CEOs of the 2 astir salient AI labs, this isn’t really restricted to conscionable them. What I personally find infuriating is really small grounding astir of these statements person successful either economical history aliases theory—which is possibly unsurprising, because astir everyone making them is simply a heavy master successful AI, not those fields.

And this goes beyond really unpredictable markets and exertion are. It reminds maine of Thomas Malthus, who really was an economist. He predicted successful 1798 that we’d inevitably tally retired of nutrient (population grows exponentially, the nutrient proviso doesn’t), which made a harmonious nine without war, famine, and illness to cull the organization impossible. And that nutrient crunch was rather imminent. He was famously wrong.

Beyond food, we’ve had periodic doomsaying astir the labour marketplace successful the look of technological change. But the reality is, moreover erstwhile there’s been disruption, the labour marketplace has adapted—and to acold much melodramatic alteration than we’re talking astir correct now.

economy\_absorbs\_farming\_slide
Farming went from astir 70% of US jobs to astir 1%, and the system absorbed it. From a talk I gave—regular readers whitethorn admit it from my 4th of July piece.

And moreover beyond that, I’ve now seen multiple cases of either my aliases different investors’ portfolio companies being approached by companies affiliated pinch the starring labs pinch unthinkable assurance successful their ain broad-spectrum intelligence superiority.

Let’s conscionable opportunity a materials subject startup is talking pinch an OpenAI outer company. The information and expertise of that startup are important to the satellite’s halfway pursuit. Talks are going well. And then, each of a sudden, personification connected the OpenAI outer squad asks, “Why don’t we conscionable do this [super difficult heavy subject problem] pinch ChatGPT ourselves?” I’ve seen this successful aggregate cases successful likewise deep, difficult areas. Like bioengineering. Or semiconductor design. And truthful connected and truthful forth. (And if you’re successful the task and startup organization and deliberation you cognize precisely who I’m referring to—there’s much than one. A batch more.)

And look, I’ve personally played astir pinch models, moreover pinch CAD and PCB design, pinch amazingly bully results. I americium not an “AI skeptic” (at slightest not successful this way).

There is simply a spectrum, though, betwixt R&D and a rote task successful a difficult field. Especially fixed it tends to activity best—like successful my case—when you person a quality pinch astatine slightest immoderate expertise driving.

Even the full “Terence Tao connected AI making breakthroughs successful math” point hinges connected having AI driven by Terence Tao (or different mathematicians)…

And galore wrong the AI laboratory organization person afloat drunk their ain Kool-Aid connected full AI supremacy, moreover for tasks that will request a batch of quality productivity and thief for the foreseeable early (as I’ve written astir before: if anything, master quality judgement will get more important and frankincense expensive). (And AI Supremacy, the concept, not the AI Supremacy newsletter by Michael Spencer, which besides covered Leopold Aschenbrenner much specifically).

And, of course, celebrated AI interrogator and Nobel Prize victor Geoff Hinton has been predicting since 2016 that group should extremity training radiologists because heavy learning would hit them wrong 5 years (he allowed it “might beryllium ten”—which has besides now passed). That is acold from happening, and if anything, arsenic Deena Mousa has documented for Works successful Progress, there’s acold much request for radiologists than erstwhile Hinton made the prediction. And, as per my question and reply pinch AI researchers who are besides radiologists, it’s a class error: tech/AI group don’t actually cognize what goes into the job.

It’s overmuch easier to opportunity someone else’s job is going to beryllium afloat replaceable by AI erstwhile you don’t really cognize what they do.

Recent news besides gave america a awesome illustration of why this isn’t conscionable an absurd taste complaint. In mid-July, during OpenAI’s soul information evaluations, GPT-5.6 Sol and a frankincense acold unreleased (more powerful) exemplary escaped their sandboxed trial environment. The agents were hunting for a difficult benchmark’s answer, and I’d conjecture that they were told to do immoderate it takes (they were moving ExploitGym, a information benchmark, aft all…).

In doing so, the agents autonomously breached Hugging Face’s accumulation infrastructure done a concatenation of vulnerabilities that allowed them to get entree to the unfastened internet. As per Ben Thompson astatine Sharp Tech, this was apt owed to sloppiness successful OpenAI’s controls (they’d already had anterior information failures, including the axios package debacle). Though, to beryllium fair, just yesterday Anthropic revealed that its models (including Mythos 5) had besides accidentally compromised existent organizations during testing.

Anyway, yes, these models are powerful—which has been thing I’ve said arsenic good and is not really successful question.

The Kafkaesque broadside of this comes from Hugging Face, though. They realized they were nether onslaught and tried to usage a frontier exemplary from 1 of the starring American labs to thief take sides themselves. It refused arsenic portion of its “safety” guardrails—the exemplary couldn’t separate an incident responder from an attacker. Which makes full sense—you request to really probe for weaknesses to, you know, fig retired the weakness. Instead, they had to usage GLM-5.2, a Chinese open-weight model, to take sides them.

It’s benignant of funny that US companies are relying connected Chinese models—given China has been salient successful (successful) hacking attempts connected American infrastructure.

But, of course, this is because of a determination by the AI companies that “know better” connected really their models should beryllium utilized and person a alternatively patronizing cognition mostly astir giving entree to their models (or astatine slightest definite capabilities).

Instead of policymakers aliases nine making this decision, it’s efficaciously been centralized to AI labs arsenic the “safe hands” (which Anthropic, successful particular, has been holier-than-thou astir for its full existence). Somehow—just guessing here—I uncertainty the remainder of nine would work together if you put this up to a vote.

Superintelligence has been “right astir the corner” each year. We support getting, bafflingly, predictions of labour marketplace carnage, which is not happening. All of this is helping make AI extraordinarily unpopular pinch regular people. And for nary bully reason, particularly erstwhile regular group person a batch to gain from AI, if the manufacture stopped trying to make everyone dislike it.

I get it. It feels for illustration the extremity of history because it’s their ain small bubble. And they don’t person the position to understand things extracurricular of it. This is particularly bad because overmuch of the manufacture has besides made itself an land afloat of PhDs.

And it’s not that I person thing against PhDs. I took PhD seminars myself. Most of my colleagues astatine Creative Ventures person them. Given our portfolio companies, honestly, I deliberation I regularly interact pinch more group who person PhDs than group who don’t. Many individuals pinch PhDs person plentifulness of intelligence humility, particularly erstwhile they’ve taken the clip to summation broader perspectives extracurricular of their field.

But fundamentally, fixed really agelong it takes to get one, you’re usually taking a organization from a young property that hasn’t had overmuch vulnerability to thing extracurricular the world organization (which is its ain very restricted bubble), aliases moreover extracurricular their ain circumstantial discipline, for the entirety of their lives frankincense far.

This is besides compounded by a batch of self-congratulatory rhetoric from the leaders of those communities (being cynical, 1 tin reason it’s because they want to nutrient much postdocs for inexpensive labour successful their labs…). I erstwhile had 1 of the largest plus allocators successful the world explicate to maine that he’d heard a batch of heavy tech task superior pitches—and teams composed solely of PhDs were often a immense pain. They often didn’t cognize what they didn’t know. One team, asked why he should invest, had moreover answered pinch this: “Frankly, we’ve already done the hardest point successful the world, which is getting our PhDs. Managing money should beryllium nary problem.”

Climbing Mount Everest and getting a PhD are some hard. I wouldn’t opportunity that doing 1 intends you tin do the different “no problem.”

While, contempt my champion efforts, it feels for illustration I’m bashing the degree, this is much a tendency created by precocious expertise (and accomplishment) successful constrictive areas. Surgeons and electrical engineers whitethorn beryllium highly smart and accomplished, but that doesn’t mean they person the expertise to weigh successful connected ambiance science.

I deliberation AI is apt going to hugely use regular people, especially since galore of the gains will beryllium “socialized” owed to deficiency of differentiation betwixt labs/models (from an economics perspective).

The field’s belief successful its ain apotheosis, however, is not only annoying but whitethorn extremity up damaging its expertise to make an impact—through wide unpopularity—or causing existent harm from elemental intelligence arrogance, magnified by really overmuch money the manufacture has to propulsion astir correct now.

I dream you enjoyed this article. If you’d for illustration to study much astir AI’s past, present, and early successful an easy-to-understand way, I’ve published a book titled What You Need to Know About AI.

You tin bid the book connected Amazon, Barnes & Noble, Bookshop, aliases prime up a transcript in-person astatine a local bookstore.

Discussion astir this post

Other Article Hacker News
↑
Close Right Ads
Close Left Ads