Chaos Engineering is a disciplined approach to identify potential failures before they become outages.
Chaos Engineering is a disciplined approach of identifying potential failures before they become outages. Ultimately, the goal of Chaos Engineering is to enhance the stability and resiliency of our systems.
Creating reliable software is a fundamental necessity for modern cloud applications and architectures. As systems are increasingly being distributed by design, the potential for unplanned failure and unexpected outages increases significantly. Thankfully, Chaos and Reliability Engineering techniques are quickly gaining traction within the community. Many organizations – both big and small – have embraced Chaos Engineering over the last few years.
NOTE: The articles in this guide will use the term team to indicate a singular group that is responsible for an application that you are considering testing or actively testing using chaos experiments like those described in this guide.
Chaos Engineering is a new practice within the realm of DevOps and Site Reliability Engineering. There are a variety of thoughts and opinions about what it is or what it should be, mostly from a high-level. Our goal is to help give some clarity about how to proceed beyond theories and concepts into practical steps.
This guide was created to give some specifics. Details. Concrete examples and direction for those who have bought into the idea and want to know what to actually do to get started.
This guide is for Site Reliability Engineers (SREs), DevOps practitioners, Platform Engineers, and anyone else thinking about how to enhance the reliability of their computing systems, especially by enhancing those systems’ abilities to stay up and running and providing a good experience for end users even when problems like component failures arise. It is specifically for people who want guidance without a lot of marketing and sales verbiage. At Gremlin we have created what we believe is a user-friendly and powerful means of implementing Chaos Engineering and we hope you will consider and ultimately use it. At the same time, we have intentionally written this content in a platform-agnostic way so that you can see the value of what Chaos Engineering offers.
We want to build upon the introduction we have created in our Gremlin introductory content across our website and in our presentations. Those high-level views are intended to whet the appetite. Here we flesh out the idea with much greater detail, including implementation examples and a precise definition of what is needed before you start and while you implement in your current setting. Then, we follow that by including some extra ideas to spark your imagination for the future.
There are two main sections to our Chaos Engineering content. First, we start with a series outlining the stages of reliability within an organization as you progress from first learning about Chaos Engineering to preparing to implement and on to increasing adoption. Second, we have a series of individual posts which each illustrate how one might choose to implement Chaos Engineering with specific technologies, architectures, and approaches that may be a part of your stack or something you are considering for the future.
In his ChaosConf 2018 talk titled Practicing Chaos Engineering at Walmart, Walmart’s Director of Engineering Vilas Veeraraghavan outlines how he and the hundreds of engineering teams at Walmart have implemented Resilience Engineering (which we will refer to as the pursuit of reliability within SRE). By creating a robust series of “levels” or “stages” that each engineering team can work through, Walmart is able to progressively improve system reliability while dramatically reducing support costs.
This series expands on this model by diving deep into the five Stages of Reliability. Each post examines the necessary components of a stage, describes how those components are evaluated and assembled, and outlines the step-by-step process necessary to move from one stage to the next.
This series also digs into the specific implementation of each stage by progressing through the entire process with a real-world, fully-functional API application hosted on AWS. We’ll go through everything from defining and executing disaster recovery playbook scenarios to improving system architecture and reducing RTO, RPO, and applicable support costs for this example app.
With a bit of adjustment for your own organizational needs, you and your team can implement similar practices to quickly add Chaos Engineering to your own systems with relative ease. After climbing through all five stages your system and its deployment will be almost entirely automated and will feature significant resiliency testing and robust disaster recovery failover. An additional tool to help you get started is Gremlin's reliability calculator.
Stage 0 is all about implementing good site reliability engineering practices, laying the groundwork for Chaos Engineering. The steps outlined in this post aren't necessarily prerequisites, but instead will evolve naturally alongside your Chaos Engineering practice.
Stage 1 describes the early stages of implementing Chaos Engineering, where you begin to inject failure into non-production systems and establish good practices for documenting what you learn.
Stage 2 helps you take your first steps into automation and testing in production.
Stage 3 is where you implement fully automated testing in your non-production systems and begin figuring out how to automate disaster recovery failover.
Stage 4 is a fully mature implementation of Chaos Engineering where you begin to have ideas of your own to add to and expand your testing plan.
This is a series covering interesting technologies, architectures, and approaches that companies are using today or considering for the future.
Serverless deployments are becoming an important facet of many companies overall application architecture and must also be tested with Chaos Engineering experiments to enhance reliability. Here is how.
This article describes some of the common tools that the Chaos Engineering community considers when starting to implement the practice in an organization. The goal is to give a high level introduction to some frequently mentioned options and list some of the strengths of each using a brief table and then an annotated list.
Istio is a popular, open source cloud-native service mesh management application with freely available source code. This article demonstrates how to perform a few Chaos Engineering experiments using features already available in Istio.
With the rise of microservices and distributed cloud architectures, the web has grown increasingly complex. We all…
Chaos Engineering is a practice that is growing in implementation and interest. What is it and why are some of the most…
