The Education of a Doomer

If you travel maine connected Twitter, aliases publication this blog, you person noticed that I went from being mostly optimistic and excited astir AI to being extremely concerned. And I thought, I should explicate why I changed my mind. Each section of this station is astir immoderate facet of AI wherever my views shifted. I statesman each section by explaining my erstwhile beliefs, and why I held them, and past explain why those beliefs changed.

1. Economics
2. Alignment
3. Control
   1. Writing
   2. Software Engineering
   3. Deferring to AI
4. Stagnation

Automation has been good. We’ve automated 99% of the jobs group did successful 1790, and the consequence is not wide unemployment, rather, we are wealthier, healthier, more educated, we person much leisure, etc. I had this vague, inductive thought that, while I can’t foretell what jobs will beryllium aft AGI, location will beryllium request for me to do something. If thing else, the overmuch higher economical maturation of the post-AGI world intends that the quality niche, while mini successful absolute terms, might be overmuch larger than today’s economy.

And this whitethorn yet beryllium true. Or it whitethorn show a deficiency of imagination connected my part. If AI develops specified that we person enduring complementarity betwixt humans and AIs, then we mightiness still person jobs successful the post-AGI future. But if AI becomes truly general, the G successful AGI, and connected apical of that it is vastly smarter, faster, and cheaper than humans, past location mightiness beryllium thing for america to do, isolated from unrecorded off UBI.

When group talk astir UBI, they typically interest astir the problem of meaning in a world without work. I’ve ne'er had this worry. When I was funemployed last year, I spent my clip reference books and penning codification and hanging retired with friends. If the early is an infinite UBI-funded vacation, I cognize what I’ll do. “Before the Singularity, publication books and propulsion location parties; aft the singularity, etc.”

But past I started reasoning astir the governmental consequences of AGI, and started writing astir it:

• No-One Escapes the Permanent Underclass: if humans are economically useless, the authorities does not request them. Why salary retired UBI to group who have neither economical nor governmental power?
• When The Future Doesn’t Need Us: mill workers tin sabotage the machines, motortruck drivers tin unopen down logistics. But if humans are economically useless, location is nary measurement for group to veto the governmental order by withdrawing their publication to it. And if the AIs conflict wars, past the state tin beryllium arbitrarily repressive.
• Mathematics Without Mathematicians: astir arguments astir humans moving “one occupation up” neglect because the AI tin do those jobs too.
• Our Servants Will Do That For Us: moreover for jobs we deliberation of as uniquely human, we mightiness for illustration to person machines do those jobs too.

I had dream that alignment would move retired to beryllium a normal engineering problem, that we lick done empirical experimentation and investigation, like everything else. It helps that the early LLMs were not incomprehensible alien minds, for illustration thing from a Stanisław Lem novel, but alternatively immensely human. It’s difficult not to anthropomorphize them. The immense halfway of unsupervised learning successful an LLM understands quality morality conscionable fine: you tin talk to them about it, they will explain, eloquently, successful detail, why thing is “good” or “bad” according to immoderate civilized system. And, because capabilities were weaker, the failures were very small. What’s the worst ChatGPT successful 2022 could do?

Since for illustration 2024, reinforcement learning has been the main method to push the frontier forward. And reinforcement learning agents activity precisely for illustration Yudkowsky says. Consequently, capabilities person accrued markedly but the models are harder to understand (literally: their prose is progressively incomprehensible) and are progressively misaligned arsenic RL scrambles their brains successful pursuit of reward. Incidents of superior misalignment are much communal and more consequential. It’s clear that AI capabilities are increasing far, acold faster than our expertise to power aliases moreover understand them.

I thought—or, rather, I implicitly believed—that group would want to remain successful power of the AIs. And if we want to clasp control, and solve alignment, past we will enactment successful control. Simple enough. But precocious I started to think: no, we will astir apt manus complete power to the AIs.

The anemic type of the disempowerment thesis is thing for illustration the prisoner’s dilemma: people/companies/polities that manus much powerfulness to AI outcompete those that don’t, truthful there’s competitory unit towards disempowerment. This is easy to believe.

The beardown type of disempowerment is: the AIs will beryllium truthful smart, knowledgeable, personable, civilized etc. that we will willingly, voluntarily hand power to them. We’ll think, “they tin do a amended occupation than us”, and we’ll be right. Believing this requires you to beryllium somewhat cynical astir humanity’s desire for autonomy vs. worldly considerations. But, complete the past fewer months, I person go much cynical astir it.

This was not a abrupt “oh shit” penetration and much a slow, gradual accumulation of tiny small grains of grounds that each constituent somewhat successful the guidance of disempowerment.

Writing

The proliferation of AI writing, I think, is grounds for disempowerment.

Early on, I’d commencement reference a blog post, and it’s AI slop. I’d unfastened a nexus to a GitHub task that looks interesting, and the README is slop. I’d get a pull request connected 1 of my projects, and the codification is slop. You publication articles connected major newspapers, and they’re written by AI. And that was simply frustrating.

Then it sewage worse. I publication a insubstantial arguing against the usage of LLMs in mathematics—and the insubstantial was AI slop. I reviewed an full book about the post-AGI world, and it was AI slop. I publication that immoderate conference started utilizing Pangram to select retired a deluge of AI-written submissions, and I read a tweet from a assemblage professor—a assemblage professor, penning nether their own name!—defending the usage of AI to constitute papers.

I don’t deliberation utilizing AI to constitute is analogous to calculators, aliases hunt engines, or other specified things. There are only finitely galore things you tin automate. And once you’ve automated writing, there’s nary higher-order activity to move to. People think “the ideas are mine, the penning is the AI”, i.e., they provender the AI a list of rambling slug points that the AI massages into a blog post, aliases a paper, or whatever. And they deliberation that measurement is “mere writing”, while their creation of the slug points was “thinking”. But they are wrong, because writing is thinking. And so, by letting the AI write, you are giving up astir of what makes up thinking.

Software Engineering

Claude Code was released a small complete a twelvemonth ago. In that short time, software engineering has been wholly transformed. Materially, it mightiness beryllium positive: higher productivity, though astatine the costs of a messier codebase. Socially, it has been a disaster.

The sermon astir package engineering has gotten dumber. It’s for illustration everyone in the manufacture mislaid 30 IQ points. People utilized to talk astir compilers, type systems, logic. Now they talk astir “prompts”, “harnesses”, “loops”. The discourse is narrower, shallower, and much repetitive. There’s only truthful many times I tin perceive astir “agentic harnesses” earlier I suffer my mind.

Then there’s the nonaccomplishment of quality superior formation: location is thing to learn. Prompting is not a skill, astatine least, it’s a overmuch shallower accomplishment than software engineering. The instrumental magnitude of the activity has improved in that group tin get much output per portion of effort, but the magnitude of work that’s astir building up quality superior has collapsed. And possibly this is rational. Why study to codification astatine all? The computers tin do that for us. And truthful the rigorous, systematic reasoning you request to believe successful bid to beryllium a good programmer: each gone. The machines tin beryllium logical for us. We tin conscionable vibe.

Deferring to AI

What’s the past awesome life determination you made without astatine slightest consulting an AI? How galore group person you met who dainty ChatGPT for illustration an oracle? And this is today, erstwhile AI has galore limitations (no online learning, unreadable prose, hallucination). If group dainty today’s very flawed AI for illustration an oracle, really much worse will it beryllium successful 2030? In 2035? In 2040?

People utilized to reason online, occurrence each other, get angry. And that wasn’t good, but astatine slightest it was human. Now they conscionable reply pinch screenshots of ChatGPT “refuting” what the different personification said, pinch zero liking successful whether that refutation is correct. So we don’t moreover occurrence each different online anymore. If we’re moreover automating our vices, what’s left?

This isn’t really an area wherever my beliefs changed. Rather, it’s 1 of the reasons that I was primitively optimistic astir AI.

I was calved successful 1994, truthful I lived astir of my life successful the great stagnation. As I teenager, I publication Engines of Creation, The Diamond Age, Orion’s Arm; I dreamt astir each astonishing technologies we would someday have, the possibilities they would unfastened up to us. Yet, it all felt infinitely distant. There’s the adjacent future, which is thinner screens and cheaper star panels; and there’s the distant future, which is molecular manufacturing and Dyson spheres and mind uploading and interstellar travel, and you won’t spot immoderate of it, unless you motion up for cryonics and it works.

Before LLMs, what was location to beryllium excited astir successful the adjacent future? Solar power? Self-driving cars? The energy transition? It’s laughable. When AI came connected the scene, for the first clip successful my life, the early I dreamt astir felt for illustration it might beryllium wrong reach, it felt breathtaking and imminent, and not this dreadful monotony of phones and apps and firm memphis and managed decline.

I often think: what if GPT-3 hadn’t worked? What would we person to look forward to, successful this alternate 2026?