Testributor Blog parallel posts, distributed thoughts and more

Testributor workers on AWS spot-or-not instances

testributor plus aws

A couple of days ago a user asked us if it would be possible to spawn workers using AWS spot instances. We found this idea very interesting. Combined with some always-on workers, spot instances could give a cheap but important speed boost to the tests.

We tried it and we liked it. Here are the steps if you want to try it yourself.

Create a worker on an EC2 instance

Our final goal is to create and AMI which we can use to spawn new instances.

We will assume you already have an account on AWS and you know your way around the Amazon console.

We used Ubuntu 15.10 as our base AMI but you can choose an other Ubuntu image here.

Clicking on a the provided link on the selected image will take you to AWS console where you can spawn an instance with that image. After succesfully launching, use your key to SSH to this instance.

Setup docker on the instance

Now you have to install docker and docker-compose. The process is described in detail on Docker documentation.

If you selected Ubuntu 15.10 the following commands should get docker and docker-compose installed.

$ sudo su
$ apt-get update && apt-get install apt-transport-https ca-certificates
$ apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
$ touch /etc/apt/sources.list.d/docker.list
$ echo "deb https://apt.dockerproject.org/repo ubuntu-wily main" >> /etc/apt/sources.list.d/docker.list
$ apt-get update
$ apt-get purge lxc-docker
$ apt-get install linux-image-extra-$(uname -r)
$ apt-get install docker-engine
$ service docker start
$ apt-get install docker-compose

Now make sure docker starts automatically after booting the image:

$ update-rc.d docker defaults

Create an init script

One way to go is to use the docker-compose.yml from Testributor, define the restart: always setting on every image (see the docs) and create the containers.

This would work but it has a problem. We would have to build a new AMI every time we change our docker-compose.yml. We want a more flexible solution so we decided to host the docker-compose.yml on Dropbox and use the share link to fetch the docker-compose.yml and build the containers after boot. You could choose to host the file somewhere else, just make sure that the file always lives on the same url, even after changing the contents.

Create a script like the following and save it on the home directory of your user. The commands on the previous steps were run as root so run exit to return to the default user if you haven’t done so already.

testributor_start.sh

curl -L -o /home/ubuntu/docker-compose.yml dropbox_url_goes_here
sudo docker-compose -f /home/ubuntu/docker-compose.yml up -d

(Change dropbox_url_goes_here with the url of your docker-compose.yml file.)

Now make this file executable:

$ chmod +x testributor_start.sh

Make sure this script is executed after booting the image. Open /etc/rc.local and add this line before exit 0 line.

./home/ubuntu/testributor_start.sh

(This should be the path where you saved your file. The dot on the beginning of the path is needed.)

You can try the command manually to make sure it works before exiting the instance. If you run it, you should see your containers created and after a while, you should see the worker added on Testributor.

If everything is ok, exit the SSH session (type exit on the console).

Create an AMI from the instance.

Go to the AWS console, on the instances page and select the instance you were logged into previously. From the “Actions” menu, choose Image -> Create image. According to the AWS documentation it could take up to 24 hours for you AMI to be created but in hour case it took around 5 minutes.

After your AMI is created you will be able to use it to spawn EC2 instances or spot instances which will automatically download the docker-compose.yml file and start a worker. If you change the docker-compose.yml file you can rebuild the workers by simply restarting your instances.

Bid wisely

If you are not familiar with AWS spot instances, it would be a good idea to start with the docs. The concept is, that Amazon it putting its unused EC2 instances in auction. When you bid is high enough a new instance is spawn (with your selected AMI) for as long as your bid price is higher than the price of the instance.

It is probably a good idea to have at least one “permanent” worker, so your tests will always be run, but you can use cheap spot instances to speed up things when they are available.

The AMI we created previously can be used to spawn “permanent” instance too so these steps should be everything you need to start using AWS for your workers.

If you made it this far, then you probably found this article interesting. If you find something wrong or missing or if you just want to share the love, send us an email.

Would you like to give it a go yourself? Start by signing up for a free Testributor account HERE.

*Amazon Web Services, the “Powered by Amazon Web Services” logo is trademark of Amazon.com, Inc. or its affiliates in the United States and/or other countries