The 7 quests of resilient software design
28 likes8,367 views
The document outlines seven quests vital for resilient software design, emphasizing the importance of understanding business cases, embracing distributed systems, and avoiding the trap of aiming for 100% availability. It highlights the necessity of establishing effective feedback loops between development and operations, mastering functional design, knowing various resilience patterns, and preserving collective memory within the software engineering community. The guide addresses the complexities of designing resilient systems and encourages a shift in mindset regarding non-determinism and failure management in distributed environments.
The 7 quests of resilient software design
- 1. The 7 quests of resilient software design A guide for the adventurous software engineer Uwe Friedrichsen – codecentric AG – 2012-2017
- 2. Uwe Friedrichsen IT traveller. Connecting the dots. Attracted by uncharted territory. CTO at codecentric. https://www.slideshare.net/ufried https://medium.com/@ufried @ufried
- 3. You want to do resilient software design ...
- 4. ... and you expect everything to be like this
- 5. But somehow it feels more like that ...
- 6. ... or even that
- 7. What the **** went wrong?
- 8. The road to resilience is a twisted one
- 9. “7 quests you must complete!”
- 10. Quest #1
- 11. Understand the business case
- 12. “How much money will we earn with it?”
- 13. “Does it improve our velocity?”
- 14. Resilience is not about making money Resilience is about not losing money
- 15. Lack of resilient software design Reduced system availability Users cannot do what they intend to do Less transactions per time period Immediate lost revenue Users get annoyed Churn rate increases Delayed lost revenue Due to non-determinism of distributed systems This is at most your resilience budget
- 16. Quest #2
- 18. Everything fails, all the time. -- Werner Vogels
- 19. If X then Y What we learned in our IT education If X then maybe Y This changes everything! What we need for distributed systems We are good at this (due to how our brains work) Inside process thinking Reasoning about deterministic behavior Designing a complicated system We are poor at that (due to how our brains work) Reasoning about non-deterministic behavior Across process thinking Designing a complex system
- 20. Yet, we usually use deterministic thinking to reason about distributed systems
- 21. Failures in distributed systems ... • Crash failure • Omission failure • Timing failure • Response failure • Byzantine failure
- 22. ... turn seemingly simple issues into very hard ones Time & Ordering Leslie Lamport "Time, clocks, and the ordering of events in distributed systems" Consensus Leslie Lamport ”The part-time parliament” (Paxos) CAP Eric A. Brewer "Towards robust distributed systems" Faulty processes Leslie Lamport, Robert Shostak, Marshall Pease "The Byzantine generals problem" Consensus Michael J. Fischer, Nancy A. Lynch, Michael S. Paterson "Impossibility of distributed consensus with one faulty process” (FLP) Impossibility Nancy A. Lynch ”A hundred impossibility proofs for distributed computing"
- 23. Embrace distributed systems • Distributed systems introduce non-determinism regarding • Execution completeness • Message ordering • Communication timing • You will be affected by this at the application level • Don’t expect your infrastructure to hide all effects from you • Better have a plan to detect and recover from inconsistencies
- 24. But do I really need to care? (The system, I am working on, is not a distributed system)
- 25. (Almost) every system is a distributed system -- Chas Emerick http://www.infoq.com/presentations/problems-distributed-systems
- 26. … and it’s getting “worse” • Cloud-based systems • Microservices • Zero Downtime • Mobile & IoT • Social Web
- 27. Quest #3
- 28. Avoid the “100% available” trap
- 29. The “100% available” trap, version #1 You: “How should the application respond if a technical failure occurs?” Business owner: “This must not happen! It is your responsibility to make sure that this will not happen.”
- 30. The “100% available” trap, version #2 You: “How do you handle the situation if the service you call does not respond (or does not respond timely)?” Developer 1: “We did not implement any extra measures. The other service is so important and thus needs to be so highly available that it is not worth any extra effort.” Developer 2: “Actually, if that service should be down, we would not be able to do anything useful anyway. Thus, it just needs to be up.”
- 31. The question is not, if a failure will happen The question is, when a failure will happen
- 32. A short note about availability Assume a service availability of 99,5% (incl. planned downtimes) • 10 services involved in a request à 95,1% probability of success • 50 services involved in a request à 77,8% probability of success
- 33. Quest #4
- 34. Establish the ops-dev feedback loop
- 35. The big wall between Dev and Ops
- 36. In a distributed environment, you cannot solve availability issues on an infrastructure level only
- 37. Dev Ops “I implemented something to improve production availability” “Here are the figures how it worked” Continuous improvement cycle of resilient software design Dev is where you implement your resilience measures Build Measure Learn Ops is where your resilience measures take effect
- 38. Dev Ops “I implemented something to improve production availability” “Here are the figures how it worked” Continuous improvement cycle of resilient software design Dev is where you implement your resilience measures Build Measure Learn Ops is where your resilience measures take effect All developer activities towards improving robustness are basically “shooting at the dark” which is neither effective not sustainable Having a wall between Dev and Ops breaks the cycle required to implement effective robustness measures
- 39. For effective resilient software design you need a working ops-dev feedback loop
- 40. Establishing the feedback loop • Adopt DevOps • Adopt Site Reliability Engineering (SRE) • Or do it your own way if you know a better way ... • ... but make sure you establish the required feedback loops!
- 41. Quest #5
- 43. Without proper functional design nothing else matters
- 44. Isolation • System must not fail as a whole • Split system in parts and isolate parts against each other • Avoid cascading failures • Foundation of resilient software design
- 45. Bulkhead • Bulkheads implement the “parts” that need to be isolated • Core isolation pattern (a.k.a. “failure units” or “units of mitigation”) • Diverse implementation choices available, e.g., (micro)services, actors, SCS, ... • Shaping good bulkheads is a pure functional design issue (and extremely hard)
- 46. Hmm, sound easy. Why should that be hard?
- 47. Service A Service B Request Due to functional design, Service A always needs backing from Service B to be able to answer a client request, i.e. the isolation is broken by design How do we avoid this …
- 48. Service Request Due to functional design we need to call a lot of services to be able to answer a client request, i.e. availability is broken by design ... and this ... Service Service Service Service Service Service Service Service Service Service Service Service
- 49. Mothership Service (a.k.a. Monolith) Request By trying to avoid the aforementioned issues we ended up with cramming all required functionality in one big service i.e. the isolation is broken by design ... without ending up with this?
- 50. Let us apply our well-known best practices • Divide & conquer a.k.a. functional decomposition • DRY (Don’t Repeat Yourself) • Design for reusability • Layered architecture • …
- 52. Service A Service B Request Due to functional design, Service A always needs backing from Service B to be able to answer a client request, i.e. the isolation is broken by design ... this usually leads to this …
- 53. Service Request Due to functional design we need to call a lot of services to be able to answer a client request, i.e. availability is broken by design ... and this ... Service Service Service Service Service Service Service Service Service Service Service Service
- 54. Mothership Service (a.k.a. Monolith) Request By trying to avoid the aforementioned issues we ended up with cramming all required functionality in one big service i.e. the isolation is broken by design ... and in the end also often to this.
- 55. Welcome to distributed hell!
- 56. Caches to the rescue!
- 57. Service A Service B Request Due to functional design, Service A always needs backing from Service B to be able to answer a client request, i.e. the isolation is broken by design CacheofB Break tight service coupling by caching data/responses of downstream service
- 58. Caches to the rescue?
- 59. Do you really think that copying stale data all over your system is a suitable measure to fix an inherently broken design? * * Side note: Caches are a very important and powerful measure in many places. But they are not suitable as a cheap fix for a broken functional design
- 60. We have to re-learn design for distributed system
- 61. No silver bullet
- 62. Yet, a few guiding thoughts ...
- 63. Foundations of design • “High cohesion, low coupling” & “separation of concerns” • “Crucial across process boundaries • Still poorly understood issue • Start with • Understanding organizational boundaries • Understanding use cases and flows • Identifying functional domains (à DDD) • Finding areas that change independently • Do not start with a data model!
- 64. Short activation paths • Long activation paths affect availability • Increase likelihood of failures • Minimize remote calls per request • Need to balance opposing forces • Avoid monolith à clear separation of concerns • Minimize requests à cluster functionality & data • Caches can sometimes help, but stale data as trade-off
- 65. Be (extremely) wary of reusability • Reusability increases coupling • Reusability usually leads to bad service design • Reusability compromises availability • Reusability rarely pays • Do not strive for reusable services • Strive for replaceable services instead • Try to tackle reusability issues with libraries
- 66. Quest #6
- 68. Core Detect Treat Prevent Recover Mitigate Complement Supporting patterns Redundancy Stateless Idempotency Escalation Zero downtime deployment Location transparency Relaxed temporal constraints Fallback Shed load Share load Marked data Queue for resources Bounded queue Finish work in progress Fresh work before stale Deferrable work Communication paradigm Isolation Bulkhead System level Monitor Watchdog Heartbeat Either level Voting Synthetic transaction Leaky bucket Routine checks Health check Fail fast Let sleeping dogs lie Small releases Hot deployments Routine maintenance Backup request Anti-fragility Diversity Jitter Error injection Spread the news Anti-entropy Backpressure Retry Limit retries Rollback Roll-forward Checkpoint Safe point Failover Read repair Error handler Reset Restart Reconnect Fail silently Default value Node level Timeout Circuit breaker Complete parameter checking Checksum Statically Dynamically Confinement Acknowledgement
- 69. Using resilience patterns • Patterns are options, not obligations • Don’t pick too many patterns • Each pattern increases complexity • Complexity is the enemy of robustness • Each pattern costs money in dev & ops • You only have a limited resilience budget • Look for complementary patterns
- 70. How other people did it
- 71. Core Detect Treat Prevent Recover Mitigate Complement Supporting patterns Escalation Communication paradigm Isolation System level Monitor Heartbeat Either level Hot deployments Restart Let it crash! Node level Actor Messaging Erlang (Akka) Core patterns
- 72. Core Detect Treat Prevent Recover Mitigate Complement Supporting patterns Fallback Share load Bounded queue Communication paradigm Isolation System level Monitor Either level Error injection Retry Limit retries Node level Circuit breaker Timeout Zero downtime deployment Canary releases Redundancy Several variants (Micro)service Request/ response Netflix Core patterns
- 73. Quest #7
- 74. Preserve the collective memory
- 75. We face a new generation of developers every 5 years
- 76. We loose our collective memory every 5 years * * Mean time until a topic discussion in the community starts over form scratch
- 77. Time working in IT Growth of knowledge Depth of insights
- 78. What do we do to compensate this effect?
- 79. We look for the new & shiny stuff ...
- 80. ... as anything not new must be useless crap!
- 81. We need to rediscover our insights every 5 years
- 82. In IT, we suffer from continuous collective amnesia and we are even proud of it
- 83. How can we become better?
- 84. Wrap-up
- 85. The 7 quests at a glance
- 86. Wrap-up • The road to resilient software design is a twisted one! • Most challenges are only indirectly related to RSD • Most challenges are not coding related • Mastering functional design is extremely hard ... • ... while learning the patterns is relatively easy • How do we preserve our collective memory?
- 87. Uwe Friedrichsen IT traveller. Connecting the dots. Attracted by uncharted territory. CTO at codecentric. https://www.slideshare.net/ufried https://medium.com/@ufried @ufried