Security

Overview

Introduction

Gremlin’s “Failure as a Service” makes it easy to find weaknesses in your system before they cause problems for your customers. Gremlin is a simple, safe, and secure way to use Chaos Engineering to improve system resilience.

Gremlin attacks are generated on the control plane. Clients make outbound SSL calls to poll for attacks. Gremlin provides secure command execution, security auditing, multi-factor authentication (MFA), and SAML SSO.

Least Privilege Setup

Gremlin does not require root privileges to any machines in your infrastructure. When installed on the host, gremlin operations are run via a gremlin user created with default Linux privileges.

Linux Capabilities

Gremlin does not need root privileges because the installed binary is granted the following capabilities to carry out attacks:

capability	purpose
`cap_sys_boot`	used by shutdown to shutdown (and optionally reboot) your hosts
`cap_sys_time`	used by time travel to move your hosts forward and backward through time
`cap_net_admin`	used by the network gremlins for all network attacks
`cap_kill`	used by process killer to kill requested process(es)

Network Access

Gremlin never intercepts the content or payload of any network traffic. Gremlin only looks at routing information in order to apply its impact to the intended network traffic.

No Ingress Ports Required

All communication between the Gremlin daemon and our service is initiated by the Gremlin daemon. For this reason, the daemon must have an outbound network path to the Gremlin service (api.gremlin.com). Additionally, since all connections from the daemon are established outbound, it is not necessary to open ports in your security groups or firewall to allow inbound communications to the daemon.

Proxy support

The Gremlin client supports http/https proxies via the Linux environment variables http_proxy and https_proxy. These should be set to use a proxy server via HTTP and HTTPS traffic, respectively. Values used should be of the form http[s]://[username:password@]address:port, such as export https_proxy=https://proxy.your_company.com:8080 or export https_proxy=https://your_username:your_password@proxy.your_company.com:8080.

The Gremlin daemon, which is typically run as a service, requires these environment variables be set in /etc/default/gremlind:

echo "https_proxy=https://localhost:8888" | sudo tee -a /etc/default/gremlind
sudo systemctl restart gremlind

Note that the Gremlin Service will only function via encrypted communication (HTTPS), attempts to connect to it via unencrypted channels (HTTP) will be denied.

Secure Command Execution

The Gremlin daemon periodically communicates with our service over a TLS-protected channel which is authenticated using your organization credentials. Once authenticated, the daemon sends heartbeat messages to the service and receives instructions from the service as responses to the heartbeat messages. If an attack has been scheduled, the daemon receives the instructions for executing that attack. Each instruction action is pre-defined within the daemon and arbitrary instructions cannot be executed.

The service API only supports TLSv1.2 connections.

Security Auditing

The Gremlin client, daemon, API, and website undergo regular security auditing, including penetration testing, by an external security auditor, Bishop Fox. All identified vulnerabilities are remediated promptly and confirmed via remediation testing by our auditors. We can provide a Letter of Assessment from our auditors outlining our most recent audit findings and remediation results upon request.

Two Factor Authentication (MFA)

Gremlin offers Two Factor Authentication. See User Management.

SAML SSO

Gremlin supports SAML SSO. See User Management.

Docker

User Namespace Isolation

Gremlin currently uses the host's file system to store temporary log and state information about attacks that have been run on the system. When running Docker with user namespace remapping (userns-remap), Gremlin needs to assume the user namespace of the host. This applies for both the gremlin daemon container as well as when running gremlin attack-container. Note that by assuming the user namespace of the host, we are creating an exception to backspace isolation for the Docker containers running Gremlin.

For running the Gremlin daemon in a container

docker run -d \
    --userns-remap=host \
    -e GREMLIN_BYPASS_USERNS_REMAP=1 \
    -v /var/lib/gremlin:/var/lib/gremlin \
    -v /var/log/gremlin:/var/log/gremlin \
    gremlin/gremlin daemon

For running the Gremlin daemon on the host

echo "GREMLIN_BYPASS_USERNS_REMAP=1" | sudo tee -a /etc/default/gremlind
sudo systemctl restart gremlind

For running a Gremlin attack from the command line

export GREMLIN_BYPASS_USERNS_REMAP=1
gremlin attack-container 38dbd9016529 cpu