Backcountry Validates Black Friday Readiness

During Black Friday 2017, traffic loads impacted Backcountry's SLOs, despite following preparation best practices. To prevent future issues like this, their forward-looking engineering organization led by Director of Engineering, Gustavo Leiva, and Principal Software Engineer, Jose Esquivel, sought a new approach for the organization to test for potential Black Friday incidents.

There was a strong business case for Chaos Engineering as a potential solution because it gives teams the ability to test for real world outage scenarios and also gives the business confidence in production system resilience. Despite this business case, Backcountry had no previous experience with Chaos Engineering and would need support to successfully introduce it into their engineering culture.

And because the testing would be in production, Backcountry needed software that they were confident could run safely and securely.

Backcountry’s search for Chaos Engineering software included looking at open source options, building their own solution, and using enterprise tooling. They ultimately elected to use Gremlin’s hosted solution. Building their own tool would take away from their existing feature roadmap, and they wanted to begin their Chaos Engineering journey as quickly as possible. Their requirements for rigorous safety and security quickly ruled out open source options as the current offerings lack security features and support.

Gustavo and Jose worked with Gremlin's success team to plan a GameDay that would recreate conditions from Black Friday 2017, and proactively look for other gaps as well. Gremlin was easy to install and configure and allowed their team to get up and running very quickly. The plan also included specific SLO abort criteria that if reached would take advantage of Gremlin's Halt All Attacks feature to restore their warehouse operations to steady state.