August 27 TCRF DDoS Attack Postmortem

…but first, a quick digression concerning what apt started it.

Our stance on AI agents isn’t much of a secret. I lately added a new characteristic to The Cutting Room Floor: If you sojourn the location alongside a “Claude-code” person agent… it adds you to a Claude person ban list. Then, if you try and sojourn the location later, without Claude — perchance since you wanted to examine the “prompt injection” leaf it received — you’re greeted alongside a particular error leaf telling you to get out, featuring a pixelated Claude logo:

A Twitter bluecheck ran into this, evaded the ban, afterward proceeded to get increasingly mad at the fact he got banned. Not lone did he dig up long-inactive versions of the “prompt injection”, but he afterward made up an complete narrative concerning how it zeroed a VM and wiped the OS, sent that made-up sob narrative to our presenter multiple times, and afterward got a Twitter mob going complete it. Kotaku equal covered several of this nonsense. Notably, whenever they asked the two of us for comment, I responded, during the another guy… asked Grok for lawful direction on suing for defamation.

LLMs rot your brain. Not equal once.

I citation this backstory, since the ban and Twitter meltdown happened just before the assault started. The timing seems additional than uncomplicated coincidence.

The DDoS assault itself

When discussing DDoSes affecting TCRF, they’re typically done via bots or scrapers. They petition hundreds of pages, trying to exhaust the server’s CPU period generating responses nobody volition read. Attacks akin this can be worked against alongside mitigation tools akin Anubis, which halt a bot from requesting pages until they expend a second doing math.

That wasn’t the case alongside this attack. The perpetrator current aimed to merely saturate the server’s network connection, overwhelming it alongside literal refuse traffic to a flat nothing lawful could get through. It was bad enough, and extended enough, that Linode had to null-route (disable) our server’s association — the DDoS traffic was starting to power Linode’s another customers. Their infrastructure couldn’t grip the sheer amount of refuse traffic being sent to us.

Unfortunately, anti-bot proxies akin Anubis aren’t productive for this category of problem; you need a big adequate pipe to grip all the traffic.

Below is a timeline of what happened, and what we did concerning it.

Pre-attack

※ (All times in this article are in Pacific Time unless alternatively specified.)

The archetypal item I noticed item was up can be dated to this screenshot, taken August 27, 12:11 PM. The chief indicator is the purple section of the CPU; this is from the scheme dealing alongside a huge influx of refuse traffic. It’s not really busy activity (the green and red sections), it’s fair trash.

We were getting a massive flood of incoming traffic. The server wasn’t doing item beyond dumping all of this in the garbage, but there was so much of that it was all it could do. This assault lasted for perchance fractional an hour, but was adequate to create accessing the website nearly unattainable during that time.

In retrospect, it’s imaginable that Linode’s automated DDoS mitigation kicked in temporarily, fairly than the attacker themselves assistance off.

Main assault starts (August 27)

At 7:39 PM, another motion of the assault began, knocking things offline. The assault was sending a huge amount of trash to harbor 80, and equal alongside most of it being filtered out before hitting any genuine resources (either by ufw or nginx rules), traffic at that flat was adequate to overwhelm the server.

But before long, the traffic stopped. All web traffic. Even known-safe traffic, explicitly allowed through our server’s firewall, was getting dropped.

Initial inquiry and response

We spent several period trying to troubleshoot. The Linode network chart showed no traffic; attempts to nexus from assorted sources didn’t work, including ones I had specifically whitelisted in the on-server firewall. After exhausting all options, at 8:34 PM, we reached out to Linode client support.

At 10:44 PM, we received a answer from Linode:

Hello,

Thank you for your patience during we investigated this.

We confirmed that an automated mitigation obstacle targeting incoming TCP harbor 443 traffic was triggered at the router flat in reply to the latest DDoS action against your IP (tcrf.net).

Because this obstacle is managed by our automated DDoS defence system, we do not have a particular ETA for its manual removal. However, formerly the DDoS event subsides and assault traffic is clear, the scheme volition automatically eliminate the obstacle and reconstruct normal HTTPS traffic.

We are actively monitoring the circumstance on our end to justify connectivity is restored as shortly as traffic conditions stabilize. Please let us cognize if you have any additional questions in the meantime.

In short: “There was so much traffic that we had to rotate it off since it was interfering alongside our hardware. When the traffic subsides, it volition rotate rear on automatically. There is nothing you can do.” Not ideal, but, well. There’s not much we could do: it was merely unreachable.

Day Two (August 28)

The DDoS assault had not stopped by this point, complete 12 hours. We reached out to Linode and received this response, at 8:56 AM:

Hey there,

Thank you for following up, and I sincerely apologize for the postpone and continued disruption to your service.

The automated router-level mitigation that W earlier mentioned is motionless energetic as our network systems continue to grip the ongoing assault traffic against your IP. We are actively tracking the circumstance on our end and volition answer current alongside an update as shortly as we can.

In the meantime, delight awareness liberated to attain out if you have any additional questions or if we can assistance alongside item else!

Best regards,
K

At this point, I began preparing a second server to presenter a “status page” during the assault was ongoing. Its lone intent was to assist a small HTML leaf (under 3 KB). We switched complete the DNS entry, and presto! Status page.

Job done, I got up and decided to do item alternatively productive, akin have breakfast, obtain a shower, perchance equal run an errand.

Overexcited presenter voice: You won’t believe what happened next!

Yep. Temporary location online? Better tap that one downward too. And so they did; downward goes the backup server.

If that wasn’t enough, our “friend” was additionally going following auxiliary services I run, including this extremely blog:

In this case, there’s nothing I can do. Most of these are uncomplicated shared hosting, not equal a VPS, and in those cases I couldn’t equal use item akin Anubis if I wanted to; it’s entirely out of my hands. Thankfully, the attacks on the another websites mostly stopped following a small while.

While all of this was going on, I had additionally checked to see if Linode had item that power help. They recommendation a “cloud firewall”, so I ask concerning it (along alongside the broad position of things). At 8:36 PM, I get a response, accent added:

2026-08-28

Hi there,

I entirely comprehend wanting a way rear online sooner fairly than later. Unfortunately, Cloud Firewall wouldn’t assistance in this particular case, since the obstacle is happening additional upstream, before traffic equal reaches your Linode’s own firewall rules. The most dependable way current is letting the automated mitigation run its course, it’s specifically designed to lift formerly the assault traffic settles, and that’s the resolution we’d propose position by for.

We’re actively monitoring the circumstance on our end and volition let you cognize as shortly as we see things stabilize.

Regards,
N

We’re keeping a near eye on this to get you rear to normal as quickly as possible. Please don’t hesitate to attain out alongside any questions in the meantime, we’re current to help!

So, there’s still beautiful much nothing I can do but wait, and now they’ve knocked downward the two the chief and the backup server.

Well, at smallest it can’t get any worse.

(gritting teeth) Well, at smallest it can’t get any worse. Ultimately, the another gathering unsuccessful to comment (again, asking Grok for lawful direction instead), and you can peruse that entire narrative above.

On top of all of this… we were additionally getting spurious abuse reports sent to Linode / Akamai, alongside no particulars and made up URLs, forcing us to document likewise pointless “this URL does not exist, and has never existed” responses. (Linode’s reply at one item included a note that they were obligated to open them, and thanked me for continually promptly responding.)

Day Three (August 29)

Today starts off alongside what seems to be fine news. Linode assistance messages me at 7:07 AM:

Hello,

I checked a few moments ago and I didn’t see any null routes for your IPv4 location in our system, so it looks akin the assault has stopped and the assistance is rear to regular operation.

Can you delight let us cognize how things appearance on your end at this time?

All the best,
A

This is concerning one and a fractional days into the attack, now, and it appears akin things have eventually let up. We commencement checking how things appearance on our end, and… nothing. The uptime tracker motionless reports it’s down; during my SSH connections are up, the traffic logs on the two servers are motionless empty. At 7:50 AM, we answer alongside our findings, and at 9:09 AM Linode assistance responds:

Hello,

I took a appearance at the two of your Linodes to see what is holding up the traffic. We can verify the automated mitigation blocks were removed for the two IP addresses.

The obstacle on your first IP (xx.xx.xx.xx) was cleared yesterday at 4:43 AM EDT, and the obstacle on your secondary IP (yy.yy.yy.yy) was removed today at 11:08 AM EDT.

I went onward and reset the networking on our end to create certain everything is apparent on the routing side.

[…]

They additionally provided several direction (they recommendation enterprise-level firewalls, and a third-party proxy resolution is apt best). But, we have to be fine to go, right?

Support and I go rear and forth, troubleshooting the networking — booting into Rescue Mode, doing additional network tests — but, following multiple distinct steps… 2026-08-29 3:08 PM:

Hello,

Thank you for confirming that for us. I wanted to verify that our engineers are actively examining this alongside us now. They have discovered that the IP xx.xx.xx.xx is currently being filtered hence of our DDoS protection, and I can verify that the current connectivity matter to that IP is a outcome of that block.

They are currently operating to eliminate that obstacle if they decide that traffic has returned to harmless levels. We volition keep you updated on this and volition portion any new data as it becomes available. We have noted the urgency of this circumstance to those involved, and so we are operating to determine this as shortly as possible.

Thank you for your continued patience throughout this. Please don’t hesitate to communication us if you have any questions in the meantime.

Regards,
C

From 7 AM to 3 PM, I was sitting about my desktop, stuck between operating assorted troubleshooting/diagnosis steps, and waiting for assistance replies… and, well, it was all for naught. The obstacle was never lifted. The assault hadn’t ended. Support confirmed this at 3:49 PM following I asked them to explain the discrepancy, accent added:

Hello,

I apologize for the confusion. The communication you’ve quoted from my teammate V was based off of the data and monitoring that our Support squad had accessible to us at that time, which indicated the obstacle had cleared.

Because connectivity had not resumed following our Support squad observed messages indicating a elimination of the block, we afterward requested additional inquiry from our inner teams. Further inquiry alongside those teams has now confirmed that the obstacle has remained in location for the duration, and has not really been lifted.

They have additionally now confirmed that the DDoS assault traffic targeting your Linode IP has not returned to harmless levels since its first onset, which occurred concerning an hr previous (August 27 about 10:35pm ET) to you beginning this ticket.

Since the quantity of the traffic is motionless elevated at this time, the DDoS mitigation obstacle remains in location and volition be removed automatically formerly the traffic to the IP returns to satisfactory levels.

At this point, the assault has been going on for 1 day, 20 hours: 8/27 7:35 PM to 8/29 3:49 PM.

I get up and obtain a interrupt for a while, to regard next steps. It’s apparent waiting isn’t going to work. Whoever is doing this has additional prosperity than sense. Since it’s clean trash traffic, item akin Anubis won’t help. We need a layer-3 service that can grip it.

It’s not much of a concealed that I’m not the biggest fan of Cloudflare. I resolve on trying out one of their competitors: Fastly.

Life in the Fastly lane

Fastly has (had?) an “Under Attack” nexus in the header, which suggests signing up for an account and routing your traffic through “Fastly’s DDoS mitigation”. So I do, environment it up in forefront of our backup server. While Linode won’t let me allocate a new IPv4 to the attacked device outright, it will let me swap IPs… so I create a new Nanode™, toggle the IPs around, and now the backup server has a caller IP address. Nothing to it.

My first belief of Fastly isn’t too bad. We get it online about Aug 30, 12:00 AM. I set a $10 monthly expend limit, get things configured, and before long, the backup server is rear online! I’ve spruced it up a bit by this item to characteristic several uncomplicated icons and position updates as things progressed.

Job done. Backup location online. I go to bed.

Day Four (August 30)

Linode reaches out this dawn at 10:37 AM (2 days, 15 hours into the attack):

Hello,

The assault appears to be ongoing. If you would akin to try and admission your linode using another IP, you can try adding a second one to see if you can use it to connect, as P stated before.

[…]

They are suggesting more or less what I’ve done alongside the backup server: a second IP location that we keep private. We’re going to do fair that, and the backup server is the preparedness for it. Making certain configurations and specified set are up before unleashing it on the “real” server.

So far, so good! Everything’s coming up Milhouse.

Move to Fastly and things break

With nearly comical timing, at 11:00 AM, I get an automated e-mail from Fastly:

Spend bounds exceeded
Your expend bounds for the duration is set to $10.00. Your current month-to-date expend is $21.00. You can detect use in real-time using Observability.

To perspective products alongside the most use sojourn the plan use page on the Fastly Control Panel.

Uhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh. Keep in intellect that the entire backup location is concerning 25 KB uncompressed. Let’s inspect out that Observability:

Our uncomplicated downtime leaf has accumulated nearly 700k requests in 14 hours. (The complete wiki typically serves concerning 5,000k/day.) Is that a lot for Fastly? I’m not sure. I’m confused as to how I spent $20 already. Let’s inspect out that scheme use page…

Excellent! I comprehend everything now.

At several point, some leaf reveals it is the “DDoS protection” I activated:

$20 on DDoS protection. I checked any leaf it was, and it stated it had blocked exactly 0 requests and allowed 100% of them. I’m not really certain what it did, another than disbursal money. Since it had a literal 0% obstacle rate, and it was item I activated on top of the essential account, I cancel it.

At several item about August 30, 2:30 PM, concerning 15 hours following we started using it… Fastly suspends our account.

I have no emails, no notifications, no anything to signify why. I am not certain what the “violation” is. We gag that it’s since I griped concerning their billing interface during Carnival Night Zone played. I dispatch a assistance email at 2:46 PM.

We’ll skip onward in the timeline momentarily to their reply (Aug 31, 11:17 AM, nearly a complete day later), as Fastly exits the narrative afterwards:

Hello X, 

Thank you for your patience during I worked alongside inner teams on finding the logic here.

We were capable to see the logic of the document being turned downward was marked as Excessive Bandwidth Usage, and was impaired for bandwidth abuse. This appears to be since the document was a Dev account, and those are mostly used for evaluation functionality or creating a POC, and not fielding that quantity of bandwidth.

I’ll be honest: in the procedure of penning this (currently 9/23), I had to go inspect the archives to double-check that the instructions said, quote,

Under energetic attack? Create a liberated account to path your traffic through Fastly’s network in minutes for contiguous DDoS mitigation.

What they don’t inform you is that the “immediate DDoS mitigation” is additionally measured in minutes (~900). At smallest for us.

The funniest item concerning it happened a few days later, whenever this communication hit my inbox:

I had elevated hopes for Fastly, but beautiful much all stage (aside from first setup!) was a blunder.

Meanwhile…

Back anywhere we were, item alternatively was happening to the backup server:

In my haste to get Fastly set up, I had made a crucial opsec error: Make certain the backend server only accepts requests from authorized frontends. (In practice, this fair method “drop all traffic apart from for the particular proxy origins they provision you”, it’s not that scary.)

I doubtful that our attacker scanned Linode’s IP area and established the backup server by merely asking all one for our site, until he established the one that gave it. Oops. Oh well, instruction learned. We made certain to add the firewall rules first next time.

Enter Cloudflare

At Aug 30, 3:52 PM, tcrf.net was added to Cloudflare. At 6:17 PM, “Under Attack” manner was activated, and has remained on since. We decided to depart the backup server in location for a during until we could be certain everything was ironed out, set up several essential rules, etc… and it was a fine item we did: about 6:12 PM, we were hit by a DDoS through Cloudflare:

The perpetrator of this particular attack directly reached out to me via Telegram and Discord. I immediately blocked both. (I don’t have any firm evidence that any another attacks were connected to this person.) Aside from that event, things were smooth: the backup location stayed online.

Day Five (August 31)

I dispatch a communication at 10:18 AM thanking Linode for the additional IP address, and mentioning that I’m inquisitive if there’s any clue as to fair how much traffic has been blasting our server.

Linode greets me alongside an update comparatively early, at 10:52 AM:

Hi,

That’s extremely understandable, and I concur that this assault seems unusually sustained and targeted. […]

Due to safety concerns, we mostly aren’t capable to portion the particular amount or benevolent of traffic that volition trigger our null routing, but I can verify that this does appear to be fairly a ample measure attack. I additionally confirmed that the assault is motionless ongoing, as a obstacle was fair temporarily removed and afterward re-added a small bit ago.

Please let us cognize if you have any another questions for our squad during this time, we appreciate you for your patience. We’ll provision any crucial updates current as we obtain them

Regards,
B (he/him)

We’re now 3 days, 15 hours into this attack. I really, really amazement how much item of this measure costs.

I spent most of the remainder of the day preparing the chief server to be online again, cleaning out older firewall rules for newer and simpler ones, preparing the Linode Cloud Firewall for Cloudflare traffic, and alternatively getting prepared for the relaunch.

Day Six (September 1)

Midnight, September 1 — 4 days, 4 hours following the assault started — we’re rear online. There’s a few tweaks to be made, safety settings to be tweaked, and so on, but the location is rear and browsable for everyone. The assault itself wouldn’t end for multiple days yet, but it was now completely mitigated.

The End (September 9)

September 9, 12 PM — 12 days, 16 hours following the assault began — it ends. Incoming refuse traffic eventually disappears from the two the chief and backup servers.

At 4:22 PM, Linode assistance confirms that it’s over, and their automated mitigations have ended:

Hello,

I checked the two of those IPs and it looks akin the final networking obstacle was removed about the identical period that you additionally noticed the traffic dip, possibly indicating an end to these attacks! We’re going to keep this ticket open for a few additional days fair to create certain that things have really subsided, awareness liberated to attain out alongside item alternatively in the meantime.

Regards,
B (he/him)

Finally.

Ongoing Attacks

While the chief DDoS assault ended, we motionless see sporadic HTTP-level attacks, enduring concerning 10 minutes each. They’re comparatively infrequent, perchance one or two per day, and not adequate to majorly disrupt service. We can fair inform group “yeah, delay a few minutes and it’ll work”; we’re motionless operating to mitigate equal those, but it’s a lot small crucial now that the location is rear to 99.5% uptime 🙂

For akin reasons, we lately had to execute Cloudflare for our Rusted Logic domain. Publishing this article (and example notice to the attack, again) method that it’s extremely imaginable I’ll have to put this blog and the remainder of xkeeper.net rearward it, too. At smallest the procedure has been comparatively painless.

To whoever is doing this: hi, delight discover a hobby that isn’t trying to obtain downward my sites.

Positive Outcomes

In the procedure of mitigating, migrating, and updating things, we’ve made a lot of upgrades and improvements to the wiki and infrastructure:

• Improved caching / distribution
• Full IPv6 support
• Automatic blocking of many bots and another nuisances, including Tor exit nodes
• Unblocking most VPNs
• Better embeds for Discord, Bluesky, etc.
• Cloudflare’s automated Wayback Machine archiving
• Better security, firewall, etc.
• Support for HTTP/2 and HTTP/3
• Basic analytics (more than uncomplicated nginx logs, at least)
• Potentially additional benefits?

My eventual goal is to have a non-Cloudflare proxy for logged-in users, so that group who can’t admission it through Cloudflare can motionless perspective the location (e.g. ancient hardware, etc). But that’ll arrive formerly we complete getting the application additional up to date and such; for now I’m fair enjoying several (relative) harmony and quiet.

Closing thoughts

I’d akin to appreciate everyone for their patience and assistance during we weathered this attack. The assistance helped keep us going during the lengthy nights spent trying workarounds and troubleshooting, and ultimately we came out of it stronger and improved than before.

If you’d akin to assistance us and the wiki, regard joining our Patreon or supporting us via Ko-fi. As always, appreciate you; it method a lot to us.

(Please keep any comments respectful.)