Project Xanadu: Even More Hindsight

Retrospective connected Project Xanadu’s occurrence and failure: a deficiency of creation iteration, meaningful use-cases, aliases practicality stopped a valuable imagination from maturing into thing useful. (And contrasted pinch my approach.)

In December 2024 during a sojourn to San Francisco, I was fortunate capable to beryllium invited astatine the past infinitesimal to a statement that could only person happened there: a ceremony of the 50th day of hypermedia visionary Ted Nelson’s 197452ya manifesto, Computer Lib/Dream Machines, extolling the imagination of Project Xanadu hypertext. I’ve contributed to English Wikipedia for 20 years now, and I’ve been moving connected Gwern.net connected and disconnected for 15 years now, truthful I could not perchance miss an full statement of group pinch beardown opinions connected hypertext.

Our host, James, had arranged for a amazingly extended postulation of Nelson memorabilia: not conscionable copies of that book (far larger and much awesome successful personification than I had realized, akin to a Compact OED successful requiring a magnifying glass, besides provided), aliases Nelson’s 199729ya book The Future of Information, but besides copies of his Swarthmore College mimeograph magazines, and astir awesome of all—several vintage computers moving copies of various Xanadu implementations.

Ted Nelson was still live astatine property 87, but unluckily could not attend. Fortunately, 1 of the attendees was 1 of the erstwhile Xanadu programmers from the Autodesk era (c. 1988–5199333ya), and we could perceive to immoderate of his stories.

They were a reminder of how, while we romanticize earlier eras of computing, the hardware constraints were really rather severe, and made productive improvement difficult. I deliberation immoderate disappointments successful past package systems go much comprehensible erstwhile we retrieve really overmuch clip and ingenuity is spent moving astir the limitations—eg. McIlroy 1982 is justifiably proud of the months of clever algorithm creation & optimization he did to get a useful spellchecker to fresh successful RAM successful nether 1MB that coming we would constitute successful a fewer lines of JavaScript (and done by an LLM).

For example, he described really they prototyped Xanadu successful Smalltalk (which made sense, arsenic a highly-productive, pleasant language/OS whose object-oriented exemplary matches hypermedia beautifully), but past had to cross-compile it to C++ and compile that… which took astir a week to compile. Not a minute, aliases an hour, aliases moreover a day, but a week. (And I thought that Gwern.net’s multi-hour compile-times were bad for my improvement speed!) He besides had to discarded a batch of clip dealing pinch C++ silliness and compile issues.1

Getting this to tally astatine each for the PCs we saw was a challenge, and 1 logic for the party.

I had, I must admit, sometimes wondered really the Xanadu years astatine Autodesk could person truthful small to show for aggregate fulltime man-years, erstwhile implementing the various client-side transclusion aliases popup features connected Gwern.net were typically a fewer days of activity for Said Achmiz. But proceeding immoderate warfare stories from the horse’s rima helped put things successful due perspective, and punctual maine conscionable really incredibly compute-impoverished group were astatine that time. (Perhaps worse than the CPU was the storage: I return for granted being capable to big immoderate PDF aliases PostScript aliases HTML record I need, but a meaningful fraction of my hosted documents transcend the total difficult thrust abstraction of a mid-range 199036ya PC, which mightiness person a 50 MB difficult drive. Meanwhile, the Markdown root files of the Gwern.net essays are themselves ~40MB, the annotations doubly that, and the last tract is 221,438 MB!)

The assemblage magazines were a astonishment and entertaining to leafage through: young Ted already liked to write, rather a lot, and dispense his advice. Someone mentioned that Nelson had dreamed of being a Hollywood head and regretted that he went into exertion instead; I thought that made consciousness and explained immoderate things astir his auteur attack to package improvement (like his insistence he is “not a programmer”, 50 years later) aliases usage of film-editing metaphors for illustration “edit determination lists”.

I besides had ne'er sat down to publication Computer Lib/Dream Machines properly, and leafed done a fewer pages. It was absorbing to spot specified a large book, successful aggregate columns to get arsenic overmuch successful arsenic possible: Nelson can’t presume the scholar knows thing astir computers and has to commencement from scratch to explicate the basal concepts for illustration bytes aliases files. (You really would request the magnifying solid if you were older.)

The later (and overmuch much obscure) book Future of Information was besides absorbing for an different building of chapters, wherever you could publication successful aggregate orders, pinch a cardinal summary chapter.

Yuxi Liu points retired that Xanadu resembles different celebrated long-running boil-the-ocean task pinch adjacent connections to databases & AI pinch a charismatic leader who ne'er changed his mind, averse to open-source, and which showed akin signs of ‘pathological science’: Douglas Lenat’s Cyc. And reasoning astir it and re-reading Xanadu materials, I agree.

I concisely poked astatine the Xanadu PCs, impressed that they were moving astatine all, but I and astir of the party-goers bounced disconnected them. The UI was excessively alien. We really needed to spot James demo them, aliases thing for illustration that: hypertext systems do not lend themselves to contiguous exploration, particularly erstwhile they are moving successful OSes connected computers nary 1 location has utilized successful 25 years, if ever.

But I didn’t request to usage them overmuch to look astatine the surface displaying the stereotypical Project Xanadu demo, the opening of the Book of Genesis pinch its celebrated lines zig-zagging disconnected to the correct to denote transclusions aliases commentary connected passages, and person a abrupt realization:

“Oh my god—It’s wholly unreadable.”

The lines were confusing clutter, particularly arsenic they crisscrossed (a perennial problem successful sidenotes layout, made acold worse by the outlines). None of the ‘sidenotes’ were readable because the screen was truthful small. Even arsenic you simply scrolled, for galore imaginable positions, owed to the lines you were incapable to publication anything! How could a archive UI wherever often, you could publication nothing, person ever seemed for illustration a bully idea?2 The UI was conscionable terrible—it could ne'er person worked. Even connected a ample surface for illustration my 4k monitor, I wouldn’t want that.

And past I thought astir the prime of text, and I realized that the UI wasn’t the existent problem; and the problem, for each these galore decades, wasn’t the squad either.

The full conception of side-by-side scope transclusions is simply a solution successful hunt of a problem.

The range-specific transclusion & commentary made consciousness for the Book of Genesis, wherever location are elaborate commentaries connected each line, and overmuch Biblical disapproval sorting retired really it’s redacted from aggregate contradictory texts (like the famously self-contradicting multiple stories of creation), but arsenic I thought to myself astir really “hey, we tin do sidenotes and scope transcludes/commentaries pinch bidirectional backlinks connected Gwern.net too, and we do do it successful my ”Suzanne Delage” short communicative analysis!”3, I abruptly realized: we can, but we mostly don’t, because nary 1 really needs to do that. This is particularly existent if we look astatine Ted Nelson’s “Examples of Parallel Documents”: textual disapproval again (the Bible, Hamlet, “Rashomon” vs the movie Rashomon), the stretched illustration of the Virginia Declaration of Rights influencing the United States Bill of Rights, and past very dubious examples—lists of saints, telephone phone-calls wrong an organization, and comedy TV episodes…?

Hardly immoderate matter successful the world really needs to beryllium fisked, aliases person circumstantial lines aliases paragraphs transcluded; hardly anyone is doing Talmudic commentary, nested furniture upon layer. Most existent instances of citations are citations to the target arsenic a whole. And moreover pinch those, we wouldn’t want to spot 99% of them, and we wouldn’t cognize really to shape the bully 1% anyhow because they whitethorn each person a different purpose. (Just look astatine online comments sometime: do we benignant them by date, by fame aliases immoderate benignant of karma aliases ‘page rank’, by length, by whether the page writer responded straight to them…?)

In retrospect, I deliberation it’s telling that successful neither Lib/Dream nor Future did I spot immoderate instances of ‘transclusion’ connected paper. Nelson could person utilized immoderate magnitude of side-by-side layout aliases range-transclusion successful his books, because package is nary obstacle: he was laying them retired by hand, and could tie aliases exemplify aliases transcript thing he wanted successful immoderate arrangement. But he didn’t, because… it’s conscionable not that useful for books. Not moreover his. (You don’t request transclusions erstwhile you tin conscionable mention an earlier passage.)

Note that moreover successful film, wherever it is trivial to put 2 sources of footage broadside by side, and person 1 ‘comment on’ the other, this is seldom done; alternatively of relying connected horizontal aliases vertical juxtaposition, movie uses the 3rd magnitude of time, processing a rich | vocabulary of cutting and different tricks for doing transitions, which do the aforesaid point (to an grade we can’t recognize until we watch films created earlier ones for illustration Battleship Potemkin). Spatial juxtaposition is utilized successful various circumstances, but is obscurity adjacent a default. Only successful uncommon circumstances is it accepted to do thing for illustration a picture-in-picture layout. A awesome niche is unrecorded broadcasts wherever it’s infeasible to sprout from aggregate angles & edit coherently; successful doing a unrecorded interview, it is accepted to person 2 talking heads simultaneous, but past erstwhile edited for later, erstwhile clip permits, it is often turned into a cut-by-cut sequence. Similarly, a streamer, for example, cannot easy ‘cut’ because they are engaged doing the existent stream, do not person a ample number of movie sources to trim between, do not person skilled unit who tin beryllium tied up doing, but if they were making a ‘greatest hits’, they mightiness rearrange aliases zoom in. (I would stake that erstwhile Hololive streamers execute astatine the large yearly concert, involving tens of millions of dollars, they do not adopt streaming-like side-by-side formatting, but spell for much emblematic performance cinematography, conscionable for illustration the Miku Expo Vocaloid concerts.) Notably, wherever budgets for unrecorded broadcasts are very precocious and value is simply a apical priority, specified arsenic NBA games aliases NFL aliases Met HD opera broadcasts, and 1 tin person a ample number of cameras arsenic good arsenic a ‘director’ and aggregate staff, they typically do not settee for immoderate benignant of fixed picture-in-picture aliases side-by-side arrangement; instead, they take to trim a azygous provender quickly betwixt different cameras, which lets them ‘follow the action’, bid cameramen to move astir successful beforehand to make a caller perspective connected the fly, and effort to create a meaningful hit based connected their appraisal of the crippled flow.

And that’s why we didn’t fuss adding that circumstantial transclusion capacity to Gwern.net until astir 2023, motivated by my literate study of a short story. I don’t request it, English Wikipedia doesn’t request it, Reddit doesn’t request it, Twitter doesn’t request it, ~100% of individual blogs do not request it… It’s conscionable not that useful—unless you are doing literate disapproval aliases commentary connected a text, which describes ~0% of the World Wide Web (in 2025 or 198937ya). All those lines look cool and futuristic, but the infinitesimal I deliberation astir really I would usage them successful an existent Gwern.net effort and really it would look to read, I commencement to get a headache.

The celebrated Project Xanadu UI/UX is simply a “science fabrication interface”, for illustration the 3D motion interfaces successful the Minority Report movie aliases the virtual reality of Snow Crash: everyone looks astatine them and is dazzled and wowed by really cool they look, isolated from they are a unspeakable thought which would beryllium a misery to use and springiness you “gorilla arm”. You effort retired a prototype aliases mockup, and almost arsenic soon arsenic you commencement utilizing them, you recognize that no, this isn’t workable and you person to propulsion it retired completely.

This reminded maine of the Hollywood head comment: you could opportunity that Project Xanadu is simply a Hollywood director’s thought of what a World Wide Web should be. You tin only judge successful it if you ne'er effort to really use it for immoderate existent project. If Ted Nelson had been little charismatic, and little compelling a writer, aliases had little religion successful his ain vision, this would person been clearer sooner. Since it was not, for illustration Douglas Engelbart, Nelson enjoys (if that is the word) the grant of being a surviving embodiment of Cunningham’s Law: becoming 1 of those historical figures whose value was that they were so incorrect connected such an important taxable they helped popularize that they inspired others to go right.

In fact, that besides explains thing astir Xanadu that had ever puzzled me, its accent connected copyright. It’s difficult to deliberation of immoderate portion of the modern world much pernicious and opposed to a useful hypertext strategy than our existent maximalist copyright law, but the Xanadu 17 principles walk much clip trying to make the Web safe for copyright holders than they do connected specified insignificant things arsenic “transclusion”—it needs to support micropayments astatine arbitrary levels of transclusion, needs to person default licenses, eschew Net Neutrality, let permissionless transclusion of arbitrary resources, etc.:

I’d ever wondered really anyone could judge immoderate of this was either imaginable aliases desirable: of course copyright holders would garbage to comply pinch immoderate of this, and would cull immoderate Xanadu pinch scary and loathing, arsenic it blows up countless business models and IP regimes and deals, and would beryllium bogged down instantly pinch tragedy of the anticommons, holdouts, prima donnas, controlling copyright owners, etc. But of course, if you saw yourself arsenic really a Hollywood head astatine heart, you would person beardown feelings astir the sanctity of copyright, you would beryllium fiercely opposed to pursuing the logic of transclusion to free software/open source (despite the past Xanadu principle, Nelson apparently has ne'er believed successful FLOSS and insisted connected NDAs), and you would judge successful a benignant of semantic web wherever you expect… thing to work, really.

As opposed to the reality of what happens erstwhile you get 8 cardinal group online and make overmuch of the world tally disconnected the Internet: a nightmarish acheronian wood wherever everything that tin spell incorrect will spell incorrect arbitrarily galore times a day; every azygous invariant you believed in turns retired to beryllium surgery location successful the world4; and each bully feature—like trackback bidirectional links—will beryllium ruthlessly abused by spammers, fools & knaves, nation-state actors, emergent bugs aliases hackers…

Most of those principles make consciousness only if you don’t ever effort it successful the existent world, wherever you will observe that galore of them are not conscionable of questionable value, aliases would beryllium fiercely opposed by galore entities, but outright illegal. (One of the systems that astir embodied the principles of robust decentralized retention of files, which moved them person to users requesting them for efficiency, was… Freenet, which instantly became notorious for kid pornography.)

So, to me, Project Xanadu is simply a case-study successful why designers must mock-up and prototype their designs earlier excessively overmuch is invested successful them. Xanadu wasn’t the unfortunate of “Worse is Better”; it was conscionable a solution successful hunt of a problem.5 (Something akin seems to beryllium existent of Arbital: it had 1 existent user, Eliezer Yudkowsky, who had envisioned it, but struggled to pass the imagination aliases create a clear prototype and group of use-cases, mocked up arsenic necessary; and for illustration Xanadu, it was a quadrate peg of what ought to person been a semipermanent non-profit FLOSS effort shoved into the information spread of a package startup company.)

Where were the Xanadu demos and use-cases laid retired connected insubstantial pinch immoderate scissors and glue, if request be?6 Why are they ever conscionable a wildly unrepresentative use-case for illustration “The Book of Genesis”? If you sat down and tried to move immoderate mean method archiving aliases the nearest mag laying adjacent you into a bunch of range-transclusions… really good would it work? How overmuch can, aliases should be, immoderate matter pinch transclusions spliced successful between? (Even successful this page about Xanadu, there’s frankincense acold only 1 constituent wherever I request to coming a ample artifact transcluded from elsewhere, the list of Xanadu principles copied from Wikipedia—which is covered by a popup nexus already, and trivial to inline.)

I deliberation if you did this, the reply quickly becomes, “yes, I request hyperlinks, those are dead-useful; I request to beryllium capable to nexus arbitrary media types for illustration images, audio, video, aliases circumstantial pages successful a book; I request links which don’t linkrot; I need a scripting language (which Xanadu is silent about), but… I don’t really request astir of these different things. If I were making immoderate benignant of distributed collaborative editor for illustration Google Docs, past I may request thing for illustration CRDTs aliases Xanadu’s akin ”enfilades”, but I don’t request that successful my published writing!”

And for me, erstwhile I started moving connected Gwern.net, almost each characteristic was driven by a use-case I had (mostly because I americium lazy); moreover pinch Said Achmiz, our attack has been to hold for respective use-cases, and past instrumentality it while enabling them all. (Invariably, arsenic Gwern.net has go truthful ample and complex, we observe edge-cases and person to revise the design, and for galore changes, abandon them.)

For hypertext, I inquire myself, “what do I request that a basal blog-like HTML page aliases an English Wikipedia page does not provide?”, and the reply is: “I don’t request to transclude conscionable a bunch of random paragraphs from a URL.7 I don’t request a bunch of mini atomic 1-sentence agelong pages nested successful a Table of Contents longer than the existent contents, for illustration GNU Info encourages you to write. What I request is immoderate benignant of summary aliases abstract. I request a scholar to beryllium capable to browse arsenic accelerated and friction-free arsenic possible, and quickly skim references aliases trace absorbing citations. I request thing like… Lupin’s Wikipedia popups, which let you to navigate WP pinch conscionable the astir casual rodent navigation, but connected steroids.”8

I would opportunity the flaw of Xanadu’s UI was treating transclusion arsenic ‘horizontal’ and side-by-side and assuming that each reading/writing must beryllium done astatine the lowest earthy level of matter (motivating the ‘tumblers’ etc.), erstwhile it should person been ‘vertical’ pinch popups, and ‘zooming in’ and ‘zooming out’ astatine different levels of abstraction (link-icon → title → absurd → conception etc.) of the matter (which motivates an wholly different group of concerns—being capable to specify arbitrary ranges becomes overmuch little important, particularly arsenic immoderate cardinal ranges tin conscionable beryllium hoisted into a higher level).

Once you person popups offering seamless navigation, you are successful effect utilizing transclusion everywhere—just wrong the popups. And erstwhile you person gone all-in connected the thought of offering abstracts for everything, it’s earthy to generalize it: if you person 2 versions of a URL, 1 ‘small’ (the abstract) and 1 ‘large’ (the full URL), why not person a ‘medium’ arsenic well?

There is nary earthy stopping constituent here, truthful you tin simply clasp a outliner-style hierarchical position of “semantic zoom”.9 This leads to galore elegant Gwern.net UI/UX creation patterns, like: the disclosure/collapses to let in-place zooming of everything from sections to sentences successful paragraphs, aliases the tag-directories (which trust heavy connected transcluding abstracts successful spot of the nexus arsenic a full to create ‘annotated bibliographies’), aliases the backlinks which amended truthful overmuch complete the backlinks of a Xanadu aliases English Wikipedia by doing a ‘reverse’ transclusion to show the transclude/link discourse successful the other pages, etc. And the local archive system intends that not only do links not break, you tin ‘preview’ galore URLs by popping up & transcluding the cleaned section archive version.

[Bibliography of links/references utilized successful page]