Application Layer

Code Integration

Get a 'GremlinService' instance

The main class you will interact with is com.gremlin.GremlinService. This class abstracts all of the machinery to register with the Gremlin API, find and cache experiments, and report success back to the Gremlin API. An instance of this class is used in each place in your application where you would like to have the option of creating an attack.

IMPORTANT : This class is designed to be a singleton.

To create an instance of this class manually, create a GremlinServiceFactory, then get the GremlinService from that:

import com.gremlin.GremlinService;
import com.gremlin.GremlinServiceFactory;

private final GremlinServiceFactory gremlinServiceFactory = new GremlinServiceFactory();
private final GremlinService gremlinService = gremlinServiceFactory.getService();

To enforce the singleton-ness of this object, you may also use dependency-injection (DI). The following example uses hk2, but any library can represent this object graph.

import com.gremlin.GremlinService;
import com.gremlin.GremlinServiceFactory;
import org.glassfish.hk2.api.Factory;
import org.jvnet.hk2.annotations.Service;

import javax.inject.Inject;

@Service
public class GremlinHk2ServiceFactory extends GremlinServiceFactory implements Factory<GremlinService> {

    // Cache `GremlinService` as a singleton here
    private final GremlinService gremlinServiceSingleton;

    @Inject
    public GremlinHk2ServiceFactory() {
        super();
        this.gremlinServiceSingleton = this.getGremlinService();
    }

    @Override
    public GremlinService provide() {
        return gremlinServiceSingleton;
    }

    @Override
    public void dispose(GremlinService instance) {

    }
}
import com.gremlin.GremlinService;
import org.glassfish.hk2.utilities.binding.AbstractBinder;

import javax.inject.Singleton;

public class GremlinDiBinder extends AbstractBinder {

    @Override
    protected void configure() {
        bind(GremlinHk2ServiceFactory.class).to(GremlinHk2ServiceFactory.class).in(Singleton.class);
        bindFactory(GremlinHk2ServiceFactory.class).to(GremlinService.class).in(Singleton.class);
    }
}

Initialize ApplicationCoordinates

An important concept in ALFI is that each application has a set of identifying attributes. This set of attributes is named ApplicationCoordinates and is used to determine when an application matches an attack. Some example sets of ApplicationCoordinates are:

  • {"type"="AwsLambda", "region"="us-west-1", "name"="event-handler"}
  • {"type"="MyServiceType", "region"="us-east-1", "service"="recommendations", "criticality"="2", "userfacing"="true"}

alfi-aws includes integrations for running on AWS Lambda and EC2. In the case of AWS Lambda, the attributes type=AwsLambda, name, and region can be set for you. In the case of AWS EC2, the attributes type=AwsEc2, region, az, and instanceId can be set for you. These attributes are inferred from the environment using AwsApplicationCoordinatesResolver.inferFromEnvironment().

If you have other facets of your application that would be useful for targeting, you may also create your own attributes. To do so, you need a subclass of GremlinCoordinatesProvider. This abstract class has 2 methods. To create your own ApplicationCoordinates, override initializeApplicationCoordinates(). The auto-generated ApplicationCoordinates (if any) are supplied as an argument to this method, so you can append to those if they exist. Here is an example of creating your own ApplicationCoordinates:

import com.gremlin.ApplicationCoordinates;
import com.gremlin.GremlinCoordinatesProvider;

public class MyCoordinatesProvider extends GremlinCoordinatesProvider {

    @Override
    public ApplicationCoordinates initializeApplicationCoordinates() {
        return AwsApplicationCoordinatesResolver.inferFromEnvironment().map(c -> {
            c.putField("userfacing", "true");
            return c;
        }).orElseGet(() -> new ApplicationCoordinates.Builder()
                .withType("MyServiceType")
                .withField("name", "recommendations")
                .withField("userfacing", "true")
                .build());
    }
}

This set of ApplicationCoordinates are then used to match attacks. So if you create an attack that matches userfacing=true, this application will be included in the attack. This is how you can start narrowing down the scope of your attack to only those applications which are useful in your scenario.

Configure TrafficCoordinates

Once you've described your application in terms of ApplicationCoordinates, you can then start targeting individual requests within your application. This will allow you to further refine your attack. Any piece of data may be used as a facet, and therefore as something to match on when constructing an attack. Here are some example TrafficCoordinates: {"type": "OutboundHttp", "name": "customer-api", "verb": "GET", "customerId": 456} and {"type": "AwsDynamo", "table": "AuditLogs", "operation": "PutItem", "deviceType": "iPad"}

Gremlin provides a way to automatically construct TrafficCoordinates for common request types, as well as a way to create your own from scratch.

Apache HTTP Client TrafficCoordinates

Here is an example of integrating fault-injection around Apache HTTP Client. This will populate TrafficCoordinates with "type" = "OutboundHttp", "verb", and "clientName".

import com.gremlin.http.client.GremlinApacheHttpRequestInterceptor;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.client.HttpClient;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.impl.client.HttpClientBuilder;

final HttpRequestInterceptor gremlinHttpInterceptor =
    new GremlinApacheHttpRequestInterceptor(gremlinService, "sample");
final HttpClient outgoingHttpClient = HttpClientBuilder.create()
    .setDefaultRequestConfig(RequestConfig
        .custom()
        .setConnectTimeout(500)
        .setSocketTimeout(1000)
        .build()
	)
    .addInterceptorLast(gremlinHttpInterceptor)
    .build();

DynamoDB TrafficCoordinates

Here is an example of integrating fault-injection around Dynamo DB. This will populate TrafficCoordinates with "type" = "AwsDynamo", "table", and "operation".

import com.amazonaws.ClientConfiguration;
import com.amazonaws.handlers.RequestHandler2;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.gremlin.db.dynamo.client.GremlinDynamoRequestInterceptor;


final RequestHandler2 gremlinDynamoInterceptor = new GremlinDynamoRequestInterceptor(gremlinService, 1500, 500));
final AmazonDynamoDBClientBuilder builder = AmazonDynamoDBClientBuilder.standard()
    .withRegion(region.getName())
    .withClientConfiguration(new ClientConfiguration()
        .withClientExecutionTimeout(1500)
        .withConnectionTimeout(500)
        .withMaxErrorRetry(2)
    )
    .withRequestHandlers(gremlinDynamoInterceptor)
);

Custom TrafficCoordinates

You may also create your own TrafficCoordinates from scratch. In this case, you completely define the type of the TrafficCoordinates and all attributes. For this code example, assume that the code looks like this prior to Gremlin integration:

final Customer leader = redisClient.getLeader(contestId);

You may want to simulate failures of the Redis client per-contest. That way, you could verify how your UI/monitoring tools/operators react to this situation. To accomplish that, a Gremlin integration would look like this:

import com.gremlin.TrafficCoordinates;

final TrafficCoordinates coordinates = new TrafficCoordinates.Builder()
    .withType("Redis")
    .withField("callType", "getLeader")
    .withField("contestId", contestId)
    .build();
final Customer leader = this.svc.execute(coordinates, () -> redisClient.getLeader(contestId));

Once you have written that code and deployed it, you may create attacks in the UI that fail specific Redis calls and pick out specific contestIds.

Extend TrafficCoordinates with request-level attributes

Often, companies set up their infrastructure to maintain a per-request data structure and use this information to provide logging, monitoring, and observability data points. A common pattern is to set up a RequestContext and have authentication filters put in information like customerId or deviceId into the RequestContext object. This object then permits access from any later point, so that those attributes are easily available. These are often excellent facets on which to create attacks. If your system operates in this way, then you can set up a mapping to populate these values on all TrafficCoordinates. This code lives in a concrete subclass of GremlinCoordinatesProvider, which you've already seen in: Initialize Application Coordinates.

import com.gremlin.GremlinCoordinatesProvider;
import com.gremlin.TrafficCoordinates;

public class MyCoordinatesProvider extends GremlinCoordinatesProvider {

    @Override
    public TrafficCoordinates extendEachTrafficCoordinates(TrafficCoordinates incomingCoordinates) {
        incomingCoordinates.putField("customerId", MyRequestContext.getCustomerId());
        incomingCoordinates.putField("deviceId", MyRequestContext.getDeviceId());
        incomingCoordinates.putField("country", MyRequestContext.getCountry());
        return incomingCoordinates;
    }
}

With this code wired into the construction of your GremlinService instance, all TrafficCoordinates will now get those 3 attributes and they are eligible to be matched for any type of traffic you'd like to attack.

Alternate configuration mechanism

As described above, the default configuration resolution mechanism is to use either properties defined in gremlin.properties, or in environment variables where your application runs. If those don't fit your needs, then you can provide an alterate mechanism by subclassing GremlinConfigurationResolver and supplying it to GremlinServiceFactory at construction-time.