Progressive collapse: civil engineering's lesson for distributed

When a single gas explosion in a London tower block in 1968 brought down an entire corner of the building, killing four people, structural engineers coined the term "progressive collapse" to describe how a localized failure can cascade into catastrophe. Sam Newman, author of O'Reilly's Building Microservices, argues that the same phenomenon plagues distributed software systems. His QCon presentation makes the case that the techniques civil engineers developed after Ronan Point apply directly to modern cloud architectures.

The core insight is uncomfortable: highly connected systems fail spectacularly. A faulty nut on a gas fitting triggered the Ronan Point explosion. The blast itself was survivable. Resident Ivy Hodge was blown across the room and knocked unconscious, but she lived. What killed four people was the building's response. The explosion knocked out a single load-bearing panel. That panel's absence transferred its load to adjacent panels, which couldn't handle the extra stress. They failed. The cascade continued until an entire corner of the 22-story building lay in rubble.

Why do localized failures become total outages?

Newman's argument rests on a structural parallel. Post-war British housing needed to go up fast. Ronan Point used a prefabricated "large panel system" design. Walls and floors arrived as finished concrete slabs, bolted together on site. Construction was quick and cheap. It was also brittle. Each panel depended on its neighbors for stability. Remove one, and the loads redistribute in ways the design never anticipated.

Distributed systems share this topology. Microservices call other microservices. Databases depend on network paths. Caches sit in front of origin servers. Each component assumes its dependencies will respond. When one doesn't, requests pile up. Timeouts trigger retries. Retries amplify load. The healthy components, now handling traffic meant for the failed one, start dropping requests themselves. The cascade spreads.

The 2017 AWS S3 outage demonstrated the pattern at scale. An engineer's typo during a routine debugging task took more servers offline than intended. S3's index subsystem couldn't recover quickly because it hadn't been fully restarted in years and needed to rebuild state. During the four-hour outage, services across the internet failed. Slack, Trello, IFTTT, and thousands of others lost functionality. Cost estimates exceeded $160 million.

Three strategies from structural engineering

After Ronan Point, building codes changed. Engineers developed three approaches to prevent progressive collapse, and Newman argues each has a software analog.

The first is strengthening individual components. Buildings now require key structural elements to withstand loads beyond their normal operating conditions. In software, this translates to hardening critical services. Can your authentication service handle 10x normal load if a frontend misconfiguration sends all traffic its way? Can your database survive a surge of reconnection attempts after a network blip? The question isn't whether components fail. It's whether they fail gracefully or explosively.

The second strategy is isolation. Modern buildings use structural compartments. If one section collapses, physical discontinuities prevent the failure from propagating. The software equivalent is bulkheads. Separate thread pools for separate dependencies. Circuit breakers that stop calling a failing service before the caller's resources are exhausted. Timeouts aggressive enough to shed load before queues back up. Chaos engineering practices, like those Netflix pioneered, test whether these boundaries actually hold.

The third is reducing interconnections. Ronan Point's panels were tightly coupled. Every panel needed every neighbor. Newer designs use redundant load paths. If one element fails, others can carry the load through alternate routes. For distributed systems, this means questioning whether every service-to-service call is necessary. Do you need synchronous communication, or can you tolerate eventual consistency? Does every request require a database round-trip, or can you cache more aggressively? Each connection is a potential failure propagation path.

The operational cost of resilience

Newman's framework is elegant, but implementation is expensive. Isolation requires duplicate infrastructure. A service with its own thread pool, its own connection limits, its own circuit breaker consumes more resources than one sharing a global pool. Reducing interconnections often means accepting weaker consistency guarantees, which complicates application logic and introduces new failure modes.

Industry estimates put the global cost of IT downtime above $300 billion annually, though methodology varies wildly between studies. That figure argues for investment in resilience. But the cost of over-engineering is invisible. Nobody counts the developer hours spent maintaining redundant systems that never activated, or the latency added by unnecessary indirection.

The practical question for engineering teams isn't whether to apply these patterns. It's where. Which failures cascade? Which components, if they fail, take others with them? Newman's presentation doesn't offer a formula. It offers a lens: trace the load paths through your system the way a structural engineer traces load paths through a building. The components that carry traffic for many others are your single points of failure. Strengthen them, isolate them, or find alternate routes around them.

Why this matters now

Newman's forthcoming O'Reilly book, Building Resilient Distributed Systems, expands on these themes. The timing aligns with a broader industry reckoning. Cloud architectures have grown more complex. Kubernetes, service meshes, and multi-region deployments add layers of abstraction. Each layer introduces new failure modes. The 2024 CrowdStrike outage, where a faulty sensor update bricked millions of Windows machines, showed how a single trusted component can propagate failure across an entire ecosystem.

The lesson from Ronan Point isn't that prefabricated buildings are bad. It's that speed and efficiency create hidden fragility. The panels went up fast because they fit together precisely. That precision meant zero tolerance for deviation. When a deviation happened anyway, a gas explosion of all things, the system had no slack to absorb it.

Distributed systems face the same tradeoff. Tight integration means lower latency and simpler operations. It also means that when something breaks, everything breaks at once.

What Newman's framework doesn't solve

The structural engineering metaphor has limits. Buildings are static. Once constructed, their load paths don't change. Software systems reconfigure constantly. Deployments shift traffic. Autoscalers add and remove instances. Configuration changes alter routing. A system that was resilient yesterday might not be today, because a deployment changed which services depend on which.

This dynamism makes the third strategy, reducing interconnections, particularly hard to maintain. Teams add dependencies faster than they remove them. A new feature needs data from another service. A performance optimization introduces a cache. Each addition is small and justified. The cumulative effect is a system where nobody fully understands the dependency graph.

The honest conclusion is that progressive collapse in software is easier to describe than to prevent. Newman's framework provides vocabulary and direction. It doesn't provide guarantees. The question for engineering leaders isn't whether their systems could cascade. It's whether they know which components would cascade first.

Source: InfoQ