Modern Botnets & The Hidden Cost of Rate Limiters: Truths and Lessons
Why standard defenses fail against modern botnets
Standard Defenses Have Blind Spots
When the massive Layer 7 attack hit our infrastructure, my first instinct was to flip all the standard switches. I turned on Cloudflare's "Under Attack" mode, I ratcheted up our rate limits, and I prayed.
It didn't work. The attackers adapted, and the community forums were filled with people offering standard advice that simply failed against a modern, highly distributed botnet. Here is a breakdown of what failed, and why.
Lesson 1: Rate Limiting is Defeated by Distribution
We had a strict rate limit of 500 requests per minute. Under normal circumstances, this catches rogue scrapers perfectly. But the botnet we faced utilized tens of thousands of unique residential IPs.
Because the attack was "low and slow" on a per-IP basis, no single node ever triggered the 500 req/min rule. Yet, in aggregate, they pushed millions of requests past the WAF and directly onto our origin servers.
Lesson 2: Rate Limiting Blows Up Cache Databases
Even if you attempt to use a custom sliding-window rate limiter in your application backend (e.g., storing IP counts in Redis), it will fail under ground-level attacks.
When 30 million requests hit your application, your cache database has to perform 30 million read and write operations just to check the rate limits. This instantly blows up your infrastructure, spikes your database bills to astronomical levels, and causes the Redis cluster to crash entirely.
Lesson 3: The SEO Cost of Blunt Instruments
"I'm Under Attack" mode (JS challenges) is incredibly effective at stopping dumb bots. But it is a sledgehammer. When we turned it on, it stopped the attack, but it also stopped our business.
- Googlebot was blocked, meaning our pages were dropping out of the index.
- Social previews broke, so sharing links on WhatsApp or Twitter just showed a generic security challenge instead of our content.
You cannot leave these blunt instruments running indefinitely. They are emergency brakes, not steering wheels.
Lesson 4: Legacy Automation Chokes Under Load
The standard advice for IP blocking is to run `fail2ban` on your Nginx server, read the error logs, and call the Cloudflare API to block bad actors.
But under extreme load, servers freeze. When your origin is already buckling under 100% CPU usage just trying to drop connections, a heavy log-parsing script like fail2ban will lock up. You cannot rely on a struggling origin server to defend itself. The defense must happen at the edge.
Lesson 5: Origin IP Leaks Bypass Everything
If you only enable Cloudflare proxying (the "orange cloud") after an attack starts, your origin server's real IP is already exposed to the world.
Attackers routinely scrape historical DNS records and certificate transparency logs. If they have your origin IP, they will completely bypass Cloudflare's WAF, rate limits, and Under Attack mode by sending requests directly to your server. You must proxy from day one and strictly lock down your origin to only accept Cloudflare IPs.
The Path Forward
The attack proved that you can't block ASNs (because botnets use residential proxies), and you can't rely on rate limits or blunt challenges without hurting real users.
The only solution is surgical, edge-native IP blocking. We needed a system that acts like a human reading logs, but works autonomously at the edge without touching our origin servers. That realization is exactly what led to the creation of FlareStack.