Boost up development with a local sandbox server
Quickly constructing a local sandbox server mirroring production web services for faster app development

 Posted on Mar 24, 2015 by Ha Duy Trung

• android
• ruby
• api
• sinatra
• testing
• sandbox

When we are doing unit testing, more often than not we will probably stumble upon external dependencies that we wish we could just get them out of the way to focus on the testing task at hand. That is when mocking, stubbing, faking comes in handy.

If we apply these techniques in a broader context, says for app testing, where our external dependencies are web services, how do we consistently and reliably test our app against the ever changing data returned by real API calls? That is when we need a sandbox server.

The term sandbox is commonly used for the development of Web services to refer to a mirrored production environment for use by external developers. Typically, a third-party developer will develop and create an application that will use a web service from the sandbox, which is used to allow a third-party team to validate their code before migrating it to the production environment.

– Wikipedia

Now, when we talk about sandbox here, I mean our own, self-developed, local sandbox server, not an external sandbox server as provided by our service provider (e.g. Paypal, eBay, Google sandbox environment). Why? Because if we use an external sandbox server, it is still an external server, populated with test data from other users. We are often forced to use one so that we do not polute the service provider’s production data, at the cost of real network latency (if not worse).

Eat your own dog food

The fact that we depend on someone else’s service does not mean that we have to depend on it during development as well, at least for sections of app that consume data versus those that produce data. Well, we may still need it at some point before migrating to production; but for day-to-day development, one can totally make do without it.

We start by mirroring the real service, by constructing a fake one, producing just enough information that our app needs to consume to be functional (not essentially in a logically correct way).

Let’s take an example of a social news reader app that uses a few services from the public Hacker News API:

• GET /v0/topstories.json - to retrieve list of top stories
• GET /v0/item/<id>.json - to retrieve a specific story with id

I like Ruby, it makes rapid prototyping easy; and sinatra seems to be the perfect pick to quickly construct a dumb sandbox server. But any web frameworks in any languages should be capable of doing the same thing here.

require 'sinatra'

get '/v0/topstories.json' do
  status 200
  content_type :json
  %/[ 1, 2, 3 ]/
end

get '/v0/item/:id.json' do |id|
  status 200
  content_type :json
  %/{ "by": "hidro", "id": #{id}, "title": "Hello World", "type": "story" }%/
end

Done and dusted! Now every GET request to /v0/topstories.json will give us 3 story IDs, and every /v0/item/<id>.json will give us the same ‘Hello World’ story.

Let’s make it a little bit smarter in the next prototyping iteration.

require 'sinatra'

get '/v0/topstories.json' do
  status 200
  content_type :json
  request.path_info = '/v0/topstories'
  mock_response
end

get '/v0/item/:id.json' do |id|
  status 200
  content_type :json
  request.path_info = '/v0/item'
  mock_response %/#{id}/
end

def mock_response json='default'
  dir_path = %-#{Dir.pwd}/responses#{request.path_info}/-
  file_path = %-#{dir_path}#{json}.json-
  file_path = %-#{dir_path}default.json- if not File.file? file_path
  File.open(file_path)
end

Now whenever the sandbox server receives a request, it will return a JSON response as saved under the /responses/ folder:

GET /v0/topstories.json
-> /responses/v0/topstories/default.json

GET /v0/item/1.json
-> /responses/v0/item/1.json
or /responses/v0/item/default.json

What if we want to simulate different scenarios, e.g. when the server encounters high traffic load and fails to return proper response?

require 'sinatra'

$responses_path = %-#{Dir.pwd}/responses/default/-

...

get '/_sandbox/:dataset' do |dataset|
  $responses_path = %-#{Dir.pwd}/responses/#{dataset}/-
end

def mock_response json='default'
  dataset = $dataset
  dir_path = %-#{$responses_path}#{request.path_info}/-
  file_path = %-#{dir_path}#{json}.json-
  file_path = %-#{dir_path}default.json- if not File.file? file_path
  File.open(file_path)
end

Now when we want to test scenario ‘empty’, send a request to /_sandbox/empty, the subsequent requests will look for response under ‘empty’ dataset:

GET /v0/topstories.json
-> /responses/empty/v0/topstories/default.json

GET /v0/item/1.json
-> /responses//empty/v0/item/1.json
or /responses/empty/v0/item/default.json

To revert to the normal ‘default’ dataset, send a request to /_sandbox/default.

This can keep going until we have a sandbox server that balances our needs and its simplicity.

###Switching to local sandbox server With our sandbox server ready to be used for development and testing, we need to find a way to build our app against different servers: a sandboxed one for debug build, and a real one for release build. Here http://localhost:4567 is the default sinatra host.

build.gradle

apply plugin: 'com.android.application'

android {
    ...

    buildTypes {
        release {
            ...
            resValue "string", "service_host", "https://hacker-news.firebaseio.com"
        }

        debug {
            ...
            resValue "string", "service_host", "http://localhost:4567"
        }
    }

    ...
}

...

RestServiceFactory.java

import com.squareup.okhttp.OkHttpClient;
import retrofit.Callback;
import retrofit.RestAdapter;
import retrofit.client.OkClient;

public class RestServiceFactory {
    interface RestService {
        @GET("/v0/topstories.json")
        void topStories(Callback<int[]> callback);
        @GET("/v0/item/{itemId}.json")
        void item(@Path("itemId") String itemId, Callback<Item> callback);
    }

    public static RestService create(Context context) {
        return new RestAdapter.Builder()
                .setEndpoint(context.getString(R.string.service_host))
                .setClient(new OkClient(new OkHttpClient()));
                .build()
                .create(RestService.class);
    }
}

...

RestServiceFactory.create(activity);

###Conclusion A local sandbox server makes it very quick and convenient to develop/test/demo our app.

• It saves us from being blocked from development on the dark days when the internet connection is slow, or when the real server runs into troubles.
• It facilitates pipelining of full-stack development, allowing app development with the assumption that the ‘to-be-developed’ web server will function in certain ways, without having to wait for it to be ready first.

However, keep in mind that it is just a mirror of the real thing. When the real thing changes, if we have no way to get ourselves notified of the changes, we would still be working against an outdated mirror of it!

We also need to constantly remind ourselves that our local sandbox server is a dumb one. It only produces fake responses for development purpose, not executing real server logic with sandbox data like those provided by Paypal or eBay. The decision to make it smarter or dumber is in our hands, and it is a trade-off between how much we want to mirror the real thing vs. how quickly we want to get it out of the way.

Now say good bye to the real web server, and start using a local sandbox server for a boosted development experience!

A better analytics tracking for your app
Migrating from Google Analytics to Google Tag Manager to future proof analytics tracking logic

 Posted on Mar 14, 2015 by Ha Duy Trung

• google-analytics
• google-tagmanager
• android
• patterns

Google Analytics is great. I love it! People at my company love it! It tells you so many things about how your audience uses your app. But once your feet are deep into the analytics game, your application will probably end up with a bunch of clunky analytics tracking code as below:

GoogleAnalytics.getInstance(context).send(new HitBuilders.EventBuilder()
        .setCategory("story")
        .setAction("view")
        .setLabel("1")
        .build());
GoogleAnalytics.getInstance(context).send(new HitBuilders.EventBuilder()
        .setCategory("story")
        .setAction("share")
        .setLabel("1")
        .build());

Don’t get me wrong, the Google Analytics API is in no way bad, but the direct application of ‘analytics tracking’ may backfire you in the future, let’s say when you switch to another provider, or when you simply want to change the name of a category/action. This calls for a layer of abstraction to future proof your tracking methods.

Coincidentally (or not!), Google Tag Manager comes into the picture to help the abstraction of analytics tracking in a (not so) intuitive way. The idea behind Google Tag Manager is excellent, although it takes even an experienced engineer a while to logically put all its concepts in the correct place. That may explain why it does not get as popular as Google Analytics.

Google Tag Manager explained

Let’s take an example to explain how things work in Google Tag Manager. Assuming you want to repeat the Google Analytics logic above: send a request to Google Analytics, with category story, action view or share, whenever an object of type story encounters event view, or an object of type story encounters event share. So trivial!

In ‘Google Tag Manager language’, using its 3 main concepts: macro, rule and tag, this can be specified as follows:

In simple words, a tag connects to your analytics service, whenever rules that compare data sent from your app - retrieved via macros - are matched.

With this setup, an equivalent logic to the original Google Analytics example is as follows:

TagManager.getInstance(context).getDataLayer().pushEvent("view",
        DataLayer.mapOf("object_type", "story", "object_id", "1"));
TagManager.getInstance(context).getDataLayer().pushEvent("share",
        DataLayer.mapOf("object_type", "story", "object_id", "1"));

Abstraction, Abtraction, Abstraction!

So now we have made some progress, converting our Google Analytics logic into Google Tag Manager logic, adding tons of complexity. But for what?

At a glance, the two code blocks before and after look almost identical. But looking closer, you can notice that we have managed to put an abstraction layer between our ‘interaction definition’ and Google Analytics ‘event definition’.

We don’t push an event to the tracking service anymore. We push our data model and its interaction to a ‘data sink’, that will decide how this information will be processed and sent to the tracking service. In Google Tag Manager, ‘data model’ is defined in the form of DataLayer.

If we generalize our data model and its interaction definition, treating anything that can be interacted with as a Trackable object, then we can just simply dump any interaction with a Trackable to Google Tag Manager, or an in-house data sink, delaying the decision of what to track for another day, and move forward with the project, without having to worry about fixing the code later to change tracking logic.

Trackable.java

1
2
3
4
5
public interface Trackable {
    enum ObjectType { story, comment, ... }
    String getTrackedObjectId();
    ObjectType getTrackedObjectType();
}

Analytics.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
public final class Analytics {
    public enum Event { view, share, ... }

    public static class Builder {
        private final Trackable mTrackable;
        private final List<Event> mEvents = new ArrayList<>();

        public static Builder createInstance(Trackable trackable) {
            return new Builder(trackable);
        }

        private Builder(Trackable trackable) {
            mTrackable = trackable;
        }

        public Builder add(Event event) {
            if (event != null) {
                mEvents.add(event);
            }

            return this;
        }

        public void send(Context context) {
            if (mTrackable == null) {
                return;
            }

            for (Event event : mEvents) {
                TagManager.getInstance(context).getDataLayer().pushEvent(
                        event.name(),
                        DataLayer.mapOf(
                                "object_type", mTrackable.getTrackedObjectType(),
                                "object_id", mTrackable.getTrackObjectId()
                        ));
            }

            // some other in-house data sink dumping
        }
    }
}

Using the above Trackable interface and the fluent-API Analytic.Builder class, one can dump interaction data to Google Tag Manager as follows:

Analytics.Builder.createInstance(new Trackable() {
            @Override
            public String getTrackObjectId() {
                return "1";
            }

            @Override
            public ObjectType getTrackedObjectType() {
                return ObjectType.story;
            }
        })
        .add(Analytics.Event.view)
        .add(Analytics.Event.share)
        .send(context);

Of course, generalizing things and introducing abstraction layers will make things less flexible for some special cases. But in general, I always find that with some smart combinations of Google Tag Manager’s tag and rule, the same can be achieved without losing the generality. And even better: the real tracking logic is now essentially broken free from our app!