In 2007, Tim Berners Lee constitute the effort The Giant Global Graph. Quote:
There are cries from the bosom .. for my friendship, that narration to different person, to transcend documents and sites. ..Then immoderate different tract aliases programme tin usage that information.It seems due arsenic X is sending cease-and-desist letters to Nitter to retrieve TBL's essay. Nitter is - was - a elemental frontend to X which allows users to position tweets without logging in. Even that mini usage of proxying to the pages is capable to person threats of ineligible action.
Twitter's API successful 2007 was famously open, which meant thousands of developers building clients, tools, and analytics for free. So, what happened? Why was it pulled? Simple: the web won. The developers stopped being an asset, and the API progressively closed. Rate limits, pricing tiers, login requirements, past method blocks connected the workarounds, and now letters from lawyers. Meta ran the aforesaid playbook a decade agone and it's now difficult to retrieve location was ever a Facebook aliases Instagram API worthy building on.
This is why Brewster Kahle, laminitis of the Internet Archive, has been calling for complete a decade for america to lock the Web open.
Nitter started disconnected utilizing X's APIs. When that closed, it publication nationalist web pages. And now that there's thing near to close, the request is that the source code travel down. A programme that displays nationalist posts is being treated arsenic a circumvention instrumentality nether computer-crime statutes.
We person a walled plot problem. It isn't going to change, and the only action successful beforehand of america is to commencement fresh.
The bully news is, atproto continues to grow, activitypub remains resilient, and our organization is afloat of believers and builders successful the unfastened societal web. Since I activity connected atproto, that's what I'll talk astir next.
Interoperation by SELECT *
The walled plot problem is downstream of a elemental question: really do I SELECT * FROM internet?
If you've ne'er written database code, SELECT * FROM users is really you inquire a database for everything it knows astir its users. Once you person it you tin select it, benignant it, and subordinate it against thing other you've got.
The web doesn't historically activity that way. The web is simply a fewer twelve companies, each holding a filing cabinet, each pinch a receptionist posted retired front. He'll publication you 1 record astatine a time, but only files you tin name, arsenic accelerated arsenic he cares to read, and arsenic agelong arsenic his leader allows.
Nitter was a lightweight X scholar that worked good correct up until X turned disconnected the entree it depended on. Every API (the "receptionist") is simply a business determination that hasn't been reversed yet.
But Impermanence isn't the only problem. Even a permanent, free, generously rate-limited API wouldn't beryllium enough. Applications request overmuch much meaningful entree than APIs tin provide.
You tin only inquire questions personification already thought to answer. An API is simply a fixed menu. It gives you getPosts(user) and getFollowers(user). If your merchandise thought needs "posts from group my followers follow, classed by really often they get quoted," location is nary endpoint for that, and location ne'er will be, because cipher astatine that institution is building for your product.
Even the correct questions travel backmost successful the incorrect shape. Followers travel 100 astatine a time. A two-million-follower relationship is 20,000 information trips. At immoderate polite complaint limit that's hours of activity to reply 1 mobility astir 1 personification — truthful thing interactive, thing that has to consciousness instant, is disconnected the array earlier you start.
You can't subordinate crossed "cabinets". The absorbing questions are almost ever cross-service: this person's posts against that person's photos against a 3rd service's reviews. Two receptionists successful 2 buildings can't cross-reference anything, and neither tin you.
You can't scale information you don't hold. Search, ranking, recommendations, feeds, moderation tooling — each of it is built connected indexes complete the full corpus, laid retired for the circumstantial questions your merchandise asks. You cannot build an scale done a keyhole.
To really build a service, we request the full dataset alternatively than a position onto it; we request it live, arriving arsenic it changes alternatively of polled for; we request to scale it nevertheless my merchandise demands; we request to constitute backmost into it; and we request each of that guaranteed successful a measurement nary azygous company's quarterly priorities tin revoke.
Desktop apps grip this by sharing the filesystem. Internet apps don't usage files; they usage databases. We request to stock the database.
As a user, I don't want to beryllium locked into an app anymore than I'd want to beryllium locked successful the trunk of a car. I want an existent free market.
So then, here's different group of needs.
Persistence of identity.
My beingness and relationships are built astir my identity. It needs to outlive the app I signed up with.
The export of surviving (not dead) information betwixt services.
Exporting archives of your tweets is useless arsenic an relationship migration solution because information doesn't unrecorded successful isolation.
If information is nary longer operable - tin of further operations by participants successful the web - past it's a fixed archive and useless to different application.
You could ever people your tweets and look astatine them I guess.
If we want information to stay operable moreover extracurricular of its original service, past we request to stock the database.
These are each issues atproto is designed to solve, including unfastened information access, relationship migration, and a unrecorded firehose of web activity.
How atproto makes SELECT * FROM internet happen
How do we stock the database? We don't. We stock a batch of them. We create a full web of individual information servers (PDS) which applications interact with.
How do we grip apps sending analyzable SELECT * queries to our individual information servers? We don't. We replicate the information connected logs. We person each exertion aggregate copies of the information to query locally.
How do we person apps constitute to those databases? In this case- we do! We person the apps nonstop writes to the PDS, which successful move replicate backmost retired to the different apps.
This past 1 is the halfway of the intuition astir atproto: the write/ingest loop. Almost each atproto app has codification that looks for illustration this:
Rather than waiting for the ingest to travel backmost complete the wire, you tin usage a "short circuit" truthful your app's database tin update much quickly. The 200 OK from the PDS is simply a transactional go-ahead.
And truthful the much businesslike shape looks much for illustration this:
Does it work?
Yes. The web exists. It's live, it's public, and you tin publication each of it correct now — from a laptop, without asking anyone's permission. This is precisely really Bluesky, Tangled, Leaflet, and a bunch of others activity now.
Let maine deed you pinch immoderate stats. At clip of writing, location are:
46.1M accounts connected atproto
24.5B records
3.15B of them are posts
17.4B of them are likes
500-1000 write-events per second
Over 5000 individual information servers
It's ne'er been easier to pat into the information pinch the caller jetstream service.
If you want a accelerated measurement to get into it, try it retired here.
Stop getting cease & desists. SELECT * FROM internet.blogposts instead.
And, oh, if you're looking specifically for blogposts connected atproto, you astir apt want to usage standard.site.
👉 Follow maine connected Bluesky.
English (US) · 
Indonesian (ID) · 
