Surviving the holiday peak: two CDNs, an origin shield, a WAF and bot management
An online retailer shipping to the United States and Canada, a leader in its segment.
- Scale
- ~38,000 requests/s at peak, ~45 Gb/s edge egress, 2 CDNs, 1 origin shield tier, 3 regions
- Engagement
- Design, load tests and cutover over 5 months, ahead of the holiday season
The challenge
Every Black Friday the site slowed down in the same order: product pages first, then search, then checkout. A single CDN absorbed the traffic, but cache misses still reached the origin in bursts. Scalper bots bought limited drops in seconds, credential-stuffing runs hit the login page at night, and one regional CDN problem the previous year had sent shoppers to an error page for forty minutes.
For an online retailer the year is decided in a few days. Every Black Friday this company's site slowed in the same order: product pages, then search, then checkout. Cache misses reached the origin in bursts, scalper bots bought limited drops in seconds, credential-stuffing runs hammered the login at night, and one regional CDN fault the year before had shown shoppers an error page for forty minutes.
We put two CDNs in front of the site, both active. DNS steers traffic with health checks and a 30‑second TTL, so each region follows the faster and healthier provider, and a degraded CDN is drained in roughly a minute. Active-active matters: the second provider is already warm and already trusted by the origin, not a plan on paper.
Between the edges and the origin we added a shield tier. Instead of every edge location asking the origin for the same product image, the shield asks once, with request coalescing and stale-while-revalidate to hold the line during a purge. The origins accept only the CDNs, through allow-lists, mutual TLS and a signed header, so the application cannot be reached by going around the edge.
On top sits the WAF, built on the OWASP Core Rule Set. We ran it in log-only mode for weeks, tuned out the false positives, then switched it to block, with rate limits on login, search, add-to-cart and checkout. Bot management combines TLS fingerprints (JA3/JA4), HTTP/2 behavior and device signals with a challenge, and applies different rules to scalpers, credential stuffers and price scrapers, while known good crawlers pass.
Before the season we ran a game day: traffic replayed at roughly twice last year's peak, a forced CDN failure and a simulated login flood. Every change had a rollback plan. At peak, about 95 percent of requests are answered at the edge, and checkout is fast because it is almost the only thing that has to reach the origin. HTTP/3, TLS 1.3 and modern image formats helped shoppers on mobile in particular.
What our team delivered
- Two CDNs in active-active mode, steered by DNS with health checks and a 30-second TTL, so traffic follows the faster or healthier provider by region, and a degraded CDN is drained in roughly a minute
- An origin shield tier that collapses cache misses from every edge location into a single request per object, with request coalescing, stale-while-revalidate and stale-if-error, so a purge or a viral product never turns into a stampede on the origin
- Origins locked to the CDNs: network allow-lists, mutual TLS and a signed header, so the application cannot be reached by going around the edge
- A WAF on the OWASP Core Rule Set tuned in log-only mode first, with per-path rate limits on login, search, add-to-cart and checkout, and virtual patches ready for a new CVE
- Bot management that combines TLS fingerprints (JA3/JA4), HTTP/2 behavior, device signals and a challenge, with separate policies for scalpers on limited drops, credential stuffing at login and scrapers on price pages, and a pass for known good crawlers
- HTTP/3 over QUIC, TLS 1.3 and Brotli at the edge, images resized and served in modern formats, and personalization pushed to edge-side includes so more of each page can be cached
- A pre-season game day: replayed traffic at roughly twice last year's peak, a forced CDN failure, and a simulated login flood, with a status room and a rollback plan for each change
Result
At peak, about 95 percent of requests are answered at the edge and the origin sees a small fraction of the load. Bots no longer decide who gets the limited drops, a provider fault moves traffic in about a minute without a page of errors, and checkout stays fast because it is the only part that has to reach the origin.
A flood stopped in four layers
An attack is cut down before it reaches your servers: upstream scrubbing takes the bulk, XDP drops what remains at the network card, and BGP FlowSpec and RTBH push new rules outward. Clean traffic keeps flowing the whole time.
Flood in, clean traffic flat
A volumetric UDP flood climbs to about 180 Gb/s across two 100G transit links. XDP drops it at line rate on the edge and a BGP FlowSpec rule (RFC 8955) pushes the filter upstream within about three seconds; delivered traffic stays at its normal 20 Gb/s after one short blip. Illustrative figures.
A request through the edge, WAF and shield
Each request lands on the nearest anycast edge, is checked against the OWASP Core Rule Set, scored by bot management and served from cache or from an origin shield. The origin accepts only mutual-TLS connections from the edge, so skipping the edge is not an option.
Failover in under a second
ISP A fails at 300 ms. BFD (RFC 5880) with a 50 ms interval and multiplier 3 declares the session down 150 ms later, BGP withdraws the routes, and the forwarding table repoints to ISP B. Traffic is back at full rate in well under a second; without BFD the router would wait for the 90 s BGP hold timer. Illustrative figures.
Facing something similar?
Tell us about your environment. We will tell you plainly what we would do and what it would take.