By Lambert Strether of Corrente.
We had a lively weekend on the Twitter, driven by the decisions and actions of the world’s richest human as measured in dollars, Elon Musk, squillionaire; necessarily so, since Musk is the platform’s sole owner, and committed to a “hands-on” management style. Since Musk’s decision-making process is completely opaque, I am left in the pleasant position of being entirely free to speculate on the causes of the error-strewn debacle episode, which I shall shortly do. First, however, I will present an apologia for the platform, explaining why it is important and interesting. Then I’ll give a chronology of the weekend’s events, followed by competing theories on why what happened happened. Then I’ll look at the business implications of the weekend’s excitement, and conclude.
Why Twitter Is an Important and Interesting Platform
I realize there are some Twitter h8ers round and about, so let me explain I use the platform in my workflow, since that directly benefits not only me personally, but you, dear reader, as well:
As a sidebar, let me confess at the outset that I am a dedicated Twitter user. I curate what I read very carefully, and reject Twitter’s frequent offers to let their algorithm take over my feed[1]. I inhabit various quiet neighborhoods that are important to me; photography, among other things. Twitter also makes finding Antidotes for Links a breeze, much easier than it was, pre-Twitter. Further, there is no better way to follow breaking news (especially with a properly curated feed). I could never have followed the twists and turns of the Covid epic without Twitter, and that very much includes the science. (It’s not easy to make a complex technical argument in a series of tweets, but some have mastered the form; not as well as a blog can, but still not badly.) In short, Twitter does a lot to make my real life, and especially my work life, more productive (and more pleasurable, because I discover things I never would have discovered otherwise). And all for free! Not a bad deal, the sort of deal a decent public utility should offer. End sidebar.
Here is the essential fact about Twitter: Twitter is a universal address space[1]. At least until this weekend, Twitter was a social graph where every user — not merely logged-in account, but user — is one degree of separation away from every other user. If I want to go find some idiotic eugenicist statement from Rochelle Walensky, I can do so with a search, logged in or not. (If I want to call Walensky a eugenicist, I must log in, but that seems only fair, and I can still do it.) Further, through embeds, Twitter’s graph extends outside the platform, which is why you see Tweets quoted everywhere. Twitter’s universal, open address space is in great contrast to the “walled gardens” of other platforms, especially The Zuckerberg™’s evil and decaying Facebook.
Many valuable use cases flow from Twitter’s universal address space. Twitter is ideal for public service announcements like emergency services and information about the weather. Twitter is ideal for breaking news, as users spontaneously coalesce around threads following the event. Twitter is ideal for public collaboration across geographic and institutional barriers; I am 100% convinced that the aerosol scientists would not have been able to take it the droplet dogmatists and win the day on the science without Twitter; the same goes for mask users and the Covid-conscious[2].
Of course, there’s are downsides to Twitter’s universal address space as well; in the same way that I can get at Rochelle Walensky, bots and dark triad personalities (10% of the population) can get at me (though none have, so far as I can tell). Worse, Twitter can be swept by moral panics and dogpiling. Even worse, committed partisans, ideologues, and activists may be confronted by people who disagree with them. Even worse, Twitter can be source of intelligence, training data for AIs, and, for the Censorship Industrial Complex, a venue for covert manipulation and deceit.
With that background, let’s turn to the events of the weekend.
How Twitter Blew Up Over the Weekend: A Chronology
By accident or design, our story begins on June 30, the Friday before Fourth of July weekend, and ends, or at least stabilizes, on the third day of the four-day long weekend. The sequence is best understood through a timeline of Musk’s tweets (since almost all the reporting simply repeats them). Here they are:
1) 7:12 AM, Jun 30, 2023:
JUST IN – Twitter has completely disabled public profiles. To see tweets, you now have to log in to the platform.
— Disclose.tv (@disclosetv) June 30, 2023
Notice that a login wall cripples the notion of Twitter as a universal address space; at the very least, as readers have noticed, it wrecks embed functionality.
2) 9:02 AM, Jun 30, 2023:
Temporary emergency measure. We were getting data pillaged so much that it was degrading service for normal users!
— Elon Musk (@elonmusk) June 30, 2023
Musk is vague here about who the “pillagers” actually are[3].
3) 8:45 PM, Jun 30, 2023:
This will be unlocked shortly. Per my earlier post, drastic & immediate action was necessary due to EXTREME levels of data scraping.
Almost every company doing AI, from startups to some of the biggest corporations on Earth, was scraping vast amounts of data.
It is rather…
— Elon Musk (@elonmusk) July 1, 2023
In case readers can’t click through past the ellipses, Musk’s tweet ends: “It is rather galling to have to bring large numbers of servers online on an emergency basis just to facilitate some AI startup’s outrageous valuation” (or very existence –lambert). Hence, the rate limits, to control the scraping, and save on server expense. Yoel Roth, Twitter’s former head of Trust and Safety, disagrees::
[Roth] says it’s not the first time the site’s been broken by someone ‘bumbling around in the rate limiter.’ It just doesn’t pass the sniff test that scraping all of a sudden created such dramatic performance problems that Twitter had no choice but to put everything behind a login.[4]
4) 1:01 PM, Jul 1, 2023:
To address extreme levels of data scraping & system manipulation, we’ve applied the following temporary limits:
– Verified accounts are limited to reading 6000 posts/day
– Unverified accounts to 600 posts/day
– New unverified accounts to 300/day— Elon Musk (@elonmusk) July 1, 2023
5) 2:29 PM, Jul 1, 2023:
Rate limited due to reading all the posts about rate limits
— Elon Musk (@elonmusk) July 1, 2023
As a sidebar, I got a boatload of “Something went wrong,” “Rate limit exceeded,” and plenty of other weird error messages and odd behaviors over the weekend, and did a lot of fiddling with my VPN, my system clock (as readers know, because I forgot to reset it), had to re-authenticate my (favorite of several) account(s), and just generally acted like an ant pushing a crumb around a hole until it falls in. Ultimately, things got back to normal. I am a tiny, tiny account, and all I can say about rate limits is that I read tweets for Links and Water Cooler like a baleen whale sucks down krill, and they haven’t affected my workflow at all[5].
Needless to say, the media — who still use Twitter, no matter how loudly they whinge — was not happy, and shared their unhappiness:
As did the users who could still log into the platform:
Why Twitter Blew Up Over the Weekend: Competing Theories
We know what Musk said, but is it true? We can’t be sure, since Twitter is completely opaque. Assume Musk isn’t lying. Who’s doing the scraping and manipulating?
One possibility is AI; Reddit has the same objection to AI’s “pillage.” From the Wall Street Journal:
Social-media platforms can be a rich trove for the large language models that power AI chatbots because they contain enormous volumes of the types of verbal exchanges those systems are trying to learn. Other social-media platforms also have instituted new policies to deal with the rise of data-hungry AI companies.
Reddit has been facing blowback over its decision to start charging some third-party developers to obtain its data.
“Reddit needs to be a self-sustaining business, and to do that, we can no longer subsidize commercial entities that require large-scale data use,” Steve Huffman, Reddit’s chief executive, said last month in a post on the platform.
A second possibility is the spooks, as this video vividly proposes:
This guy wins todays internet award for correctly analyzing all the factors in Twitters lockdown.
The intelligence community built a “Censorship Deathstar” by scrapping everyone’s data and Elon just threw a fat monkey wrench into the whole machine.
pic.twitter.com/JbDmUyP1Si
— Zach Vorhies / Google Whistleblower (@Perpetualmaniac) July 1, 2023
A third possibility is AI and the spooks:
Some people asked me to share what I just shared in a space about the rate limits. I don’t work for Twitter but, I do architect IT cloud solutions as my day job.
“Data scraping” is a big deal. This is where automated systems load the website or app and pull your tweets/data. It’s a huge security issue. Automated systems are pulling every tweet/word/user account information to store in an unknown database somewhere else.
This could be state actors like China, the US Government, Australia, or other bad political actors like PAC’s that are trying to gain access to everyone’s information to analyze and use for nefarious things. Manipulating what is said on the site can be done at scale with data scrapping.
It could also be used to figure out the identity of Anons or to punish people in their country for what they tweet. Looking at you #Australia and #Canada and #UnitedKingdom [three of the Five Eyes]
And:
The temporary measures of limiting tweets is to protect users just as much as it is to protect the entire Twitter network from going down. They are currently scrambling to get ahead of this and tune their network security to block it from happening again.
It’s also important to note that twitter has 500,000+ servers. That’s not free. In cloud data centers, the companies that use them have to pay for what is called “ingress and egress” of data going “in and out” of the servers. A data scrapping event that is large enough for them to start limiting means that it was a MASSIVE event that could be considered an attack on the site. It would also put massive load on their servers and cost them so much money it could threaten the site’s financial ability to keep running. It could be on purpose to put twitter out of business from cost alone.
And a fourth possibility is businesses. Charged for the API, they decided to scrape:
It seems very likely that killing free access to the Twitter API led to a big increase in scraping, since countless businesses, organizations, and individuals used it for their projects. It’s also plausible that these issues are entirely unrelated.
Now let’s assume Musk is lying. Why did Elon rate limit the site?
One possibility is that Google was unhappy Elon didn’t pay his bills:
Twitter has resumed paying its Google Cloud contract, according to Bloomberg. If you missed the initial news of the impending showdown, Platformer reported on June 10th that Twitter had been refusing to pay Google for its cloud services ahead of their contract’s June 30th renewal date. The possibility of losing access to Google’s infrastructure led to a frantic rush at Twitter to migrate as many of its services off of Google’s servers. However, that effort was reportedly “running behind schedule,” opening the door for some of the company’s in-house tools to go offline come the end of the month.
(This is plausible, since Musk has stiffed other vendors.)
A second possibility is to built Twitter’s base of paid subscribers:
Some users believe the view limitation move is an attempt to encourage Twitter Blue subscriptions – where users can view 10,000 tweets daily.
A third possibility is a technical own goal:
But on Mastodon this morning, web developer Sheldon Chang noticed another source of unusual traffic: a bug in Twitter’s web app that is constantly sending requests to Twitter in an infinite loop: “Twitter is firing off about 10 requests a second to itself to try and fetch content that never arrives because Elon’s latest genius innovation is to block people from being able to read Twitter without logging in.”
(I confess to some pleasure in the idea that Musk broke Twitter by breaking its universal address space.)
A fourth possibility is another technical own goal: a back-end debacle:
This is weird to say out loud, but I actually am kinda an expert in rate limiting, so I’m gonna explain some stuff.
I don’t know what happened at Twitter today, but I don’t think Elon woke up and decided to shut it all down – my bet is some ‘bottom up’ problem (but not necessarily the DDOSd yourself problem everyone is tweeting about – that could be an effect of getting limited, not the cause)…
At the simplest level, a rate limiter is a program that says “This computer can only do x requests per second” and stops all the others with “429 too many requests”.
Twitter has a really good [rate limiter] because they had a really exceptional infra team until a year ago.
BEGIN PURE SPECULATION, I DO NOT KNOW!
My hypothesis – Twitter lost a big part of a critical back end system – maybe they stopped paying their GCP bill, maybe they lost a critical cache and everything was reading other data, I truly do not know.
At this point, their probably very good adaptive rate limiter said ‘ohshit’ and brought the number of requests WAY WAY down throughout the system.
END SPECULATION
A final possibility is election interference:
Lauren Jauregui is sharing her opinion after Elon Musk set viewing limits on Twitter. She continued, “This place was one of our most effective organizing tools during 2020 and is where a lot of discourse and accountability happens. Please understand nothing these billionaires do is ever a coincidence.”
“Slowly but surely is the game, so we are apathetic instead of paying attention. When everything feels random and confusing, just know it is anything but. Just connect the dots,” she ended.
Implications of the Weekend’s Events for the Twitter Platform
Always remembering that Twitter is a squillionaire’s play-toy, there are some odd things going on from a business perspective.
First, does Musk believe that crashing traffic is the way to generate advertising revenue?
Search Engine Roundtable’s Barry Schwartz found that Google now has about 52 percent fewer Twitter URLs in its index than it did on Friday. It’s still showing recent tweets in the Search carousel, but normal indexing seems to be broken at the moment. ‘Not that a site command is the best measure, but… Twitter is down [around] 162 million indexed pages so far since this change,” Schwartz tweeted [ha].
Second, why is Twitter’s new CEO maintaining radio silence?
Will the advertisers be happy about paying for advertisements on a platform where its controlling shareholder publicly declares a desire for users to spend less time on the platform? Questionable.
It still remains unclear to what extent the new CEO was involved in this change. Yaccarino has neither publicly addressed it nor responded to the change on her Twitter account.
Third, will other platform owners be emboldened to take their own sites down?
[Musk’s] influence on other social media moguls has been undeniable.
These CEOs have been taking notes and mimicking parts of his strategy, from cost-cutting to paid verification.
Beginning in November, Musk laid off more than half of Twitter’s workforce, citing a need to cut costs and focus on rapid change. Reddit announced this month that it would lay off about 5% of its workforce.
Reddit isn’t alone in following Musk’s lead. Other apps that are more or less rivals of Twitter have experimented with actions that Musk has taken at the company, suggesting that the mercurial billionaire, despite being a new entrant in the social media industry, is already making a mark.
Fourth, what about Twitter’s competitors? In the last Twitter debacle, Mastodon was supposed to benefit. That fizzled. This time, it’s Jack Dorsey’s Blue Sky. From New York Magazine:
Whatever the real underlying reason for the throttling, the restrictions have prompted yet another wave of people tweeting that they’re leaving the site, as well as the trending of names of various Twitter alternatives. One, the vintage-Twitteresque platform Bluesky, said Saturday that it was temporarily suspending new sign-ups amid “record-high traffic” following Musk’s rate-limit announcement.
Ho hum. What New York Magazine forgets to mention is that “signing up” for Blue Sky doesn’t get you access to the platform; it gets you the promise of an invite code so you can set up an account and get access at some future date. So if Blue Sky has to suspend a system that simple, they’re not ready for prime time.
Conclusion
I leave you with this nugget of information. As readers know, I really like Twitter’s art bots (and deplore Musk’s decision not to carve out an exception for them in his API rate structure). Anyhow, I was poking around some tweets on Van Gogh, and got this screen:
Musk seems to have made Twitter into a platform that can accept payments, which could be, as we say, a gamechanger[6] (and could also break PayPal’s monopoly, which would make we NCers very happy). Tips is not currently available on the web, but I should have seen it on my iPad. I’ll have to look into it!
NOTES
[1] Twitter’s universal address space is in great contrast to Mastodon’s federated address space, which (IMNSHO) is siloed by definition. How do you “search Mastodon”? You don’t, because there is no single Mastodon entity.
[2] Not relevant to Twitter’s architecture, but nevertheless to be thrown into the “Good Things Twitter Has Enabled” bucket, we have The Twitter Files (see NC here and here), which showed the unholy merger of key Democrat factions, spooks, the press, and content moderators into a gelatinous mass of election thieves.
[3] Trying to formulate a joke about “it takes a pillage,” referring to the grandiose theft (“original accumulation”) of data for training sets by AI firms, but time presses…
[4] The rest of Roth’s comment is odd, but tangential, so I’m putting it in a note:
“Scraping was the open secret of Twitter data access. We knew about it. It was fine.”
It was? Why? The servers had to be paid for. Twitter has been unprofitable for years. Was it taking one for Silicon Valley’s AI team? Or the spooks? More:
There’s some legitimacy to Twitter and Reddit being upset with AI companies for slurping up social data gratis in order to train commercially lucrative models,” Roth added. “But they should never forget that it’s not *their* data — its ours. A solution to parasitic AI needs to be user-centric, not profit-centric.”
A pious statement, given that nobody’s doing it! Whose is “ours”?
[5] It’s not all that clear what the rate limit metric actually means:
Since Musk delivered this news in a tweet (as is tradition, at this point), it’s fairly unclear as to what actually constitutes a ‘view’ or a ‘post’ – interestingly, Musk didn’t use the word ‘tweet’ at all in his announcement.
Replies appear to count towards the limit, but ads might not; and as for what ‘viewing’ a post means, as far as we can tell you could scroll past it at lightspeed and Twitter will still consider it viewed. Hit your limit, and you’ll get either empty space or a message that reads ‘rate limit exceeded’.
From my experience, simply scrolling past doesn’t count toward the limit.
[6] This got lost in all the noise:
Mush just enabled blog posts on Twitter (although with some content limits). Interesting!