Part 2 of our multi-region failover series
Last week I wrote about running monthly chaos engineering exercises in production to validate our disaster recovery architecture.
The obvious follow-up:
"How do you get there safely?"
That's the right question.
Because chaos engineering in production is not where you start.
It's where you arrive — after building the operational maturity that makes failure survivable.
Here are the challenges you need to solve before you get there.
🔍 Challenge 1: Observability Gaps Will Expose You
Before triggering your first failover exercise, ask yourself:
If failover started right now, could you tell exactly what was happening?
Not after the fact.
In real time.
Can you see:
- traffic shifting between regions?
- application health in both regions?
- user impact during the transition?
Most dashboards are built for normal operations. Failover creates a completely different signal spanning infrastructure, DNS, networking, and applications simultaneously.
If you can't observe the recovery process, you can't safely test it.
Maturity bar: Build dashboards specifically for failover scenarios, not just general infrastructure health.
❤️ Challenge 2: Health Checks That Lie
Does your health check confirm the application is ready to serve traffic — or just that the process is running?
Applications can report healthy while:
- connection pools are still initializing
- caches are cold
- downstream services are unavailable
A failover mechanism that trusts shallow health checks can route traffic to a region that isn't actually ready.
Maturity bar: Validate readiness, not existence.
A lying health check is worse than no health check at all.
🌐 Challenge 3: DNS TTL Is Not Your Friend
Route 53 failover is not a switch.
Traffic does not instantly move from one region to another.
DNS caches.
Clients cache.
Resolvers cache.
Even with aggressive TTLs, some traffic continues flowing to the original region during transition.
Maturity bar: Understand your propagation behavior before running production exercises.
What looks like a failure may simply be DNS doing exactly what DNS does.
🚀 Challenge 4: Cold Start Reality vs. Cold Start Assumption
Recovery timelines often look great on architecture diagrams.
What they rarely account for:
- connection pool initialization
- cache warmup
- downstream dependency stabilization
- application readiness under real load
Container startup is only the beginning.
Maturity bar: Measure end-to-end recovery under load and let observed behavior define your recovery objectives.
🛑 Challenge 5: Blast Radius Without Boundaries
Chaos engineering gives automation authority over production infrastructure.
Without guardrails, a controlled exercise can become an actual incident.
Every exercise should have:
- hard limits
- abort conditions
- rollback procedures
- a designated kill switch
Maturity bar: Define the boundaries before the exercise starts.
💓 Challenge 6: The Traffic Shift Window Is Invisible Without a Heartbeat
During failover:
- the primary region is degrading
- the secondary region is coming online
- DNS is propagating
The question isn't:
"Did failover work?"
The real question is:
"Did users experience downtime?"
To answer that, we run an external synthetic heartbeat every 30 seconds through the same public endpoint users access.
The heartbeat records:
- success/failure
- latency
- timestamps
After every exercise we have evidence.
Not:
"We think there was no downtime."
But:
"Every heartbeat succeeded during the entire failover window."
The heartbeat doesn't prevent outages.
It proves the absence of them.
Maturity bar: Build an independent synthetic monitor before attempting production chaos.
✅ The Maturity Stack Before Production Chaos
Before running chaos engineering in production, you should have:
- Full observability
- Deep readiness-based health checks
- External heartbeat monitoring
- Tested recovery automation
- Practiced failback procedures
- Defined blast-radius controls
- Team readiness and communication plans
- Successful non-production validation
🎯 Final Thought
Chaos engineering in production is not about proving you're brave.
It's about proving your recovery process works.
The architecture matters.
The automation matters.
But confidence comes from continuous validation.
The heartbeat monitor proves users weren't impacted.
The recovery tests prove automation still works.
Together they turn disaster recovery from a theoretical capability into a continuously validated one.
That's not chaos.
That's engineering.
