At mytaxi, we’re currently using Amazon ECS to run our production environment. To save money we decided to go all in on spot instances.
For those not familiar with how spot instances work: you tell Amazon what you are willing to pay for a certain instance type on AWS. The cost for that instance type depends on supply and demand and can, in theory, exceed the on-demand price. If the price you set is lower than the current price of the instance type your instance gets terminated.
Since this year you can subscribe to events on AWS telling you about spot instance terminations. You get a notification three minutes in advance. We’re using these to set cluster nodes to
DRAINING state which tells them to reschedule their containers to different nodes. A graceful shutdown so to say.
This is implemented as a lambda function written in Python which is getting triggered by Cloudwatch events and can be deployed using Terraform. For more information and the source check out the Github repo at mytaxi/terminationspotter-lambda, for anything else feel free to say hi on Twitter.