Using Capistrano For Configuration Management

As I mentioned in my short comparison of Puppet vs. Capistrano, we’re currently using Capistrano not only for deployment and live monitoring of our application, but also for configuration management.

Using How-To Documents For Configuration Management

Before getting started with Capistrano, I used to write very detailed how-to documents to make sure that I didn’t need to re-learn any installation or configuration procedures I had already done. In fact, I got bit every time I slacked off and failed to write down what I did. Yes, I regretted every single time I tried to take the seemingly “fast” route, having to google my way through a complex and nasty configuration again. I can’t imagine a worse way of wasting your time…

My how-to documents usually contained a copy of every command I ran on a terminal plus links to the relevant sources. I usually documented deviations of given tutorials and emphasized our custom setup. Whenever I needed to re-do the same or a similar installation I was able to walk through the given steps more or less by only copy-and-pasting what I wrote down in the how-to. That way, these documents acted as our baseline for configuration management: Describing all detailed configuration parameters and documenting each and every change on the system.

Automated Configuration Management

As nice as it was to have such a collection of how-to documents describing the detailed setup of every aspect of our systems there was still room for improvement. The installation and configuration tasks were, on the one hand, very well defined, but, on the other hand, still quite complex and tedious. Executing all the steps manually caused mistakes like forgetting a step or mistyping a command (sometimes with serious consequences, like typing a rm -rf * command in /usr/local instead of /usr/local/anyapp). A natural next step seemed to be the automation of all those tasks, but it took some time until I came across the right tool for the job: Capistrano. As you might already know, Capistrano was originally built for deploying rails applications. But, as it’s mainly a tool for running scripted commands on a multitude of servers via SSH, it is very well suited for implementing automated configuration management, as shown by deprec, a suite of Capistrano recipes to deploy a full Ruby on Rails stack on Ubuntu.

When setting up our new production environment for our Ruby on Rails app, I wrote Capistrano recipes for every step in the process: Adding our users and their keys to the physical servers, installing Xen and xen-tools, setting up disk images as physical LVM volumes, configuring Xen VMs to run our web, app and db roles, and even installing a complete Ruby on Rails stack on top of our virtual machines. I created quite a verbose set of Capistrano recipes for every task in operating our environment: Adding new disk space to a virtual machine, moving a virtual machine to another physical server, installing a new application server and adding it to the mongrel cluster, and changing the number of mongrels on every application server.

Difficulties In Setting Up Automated Configuration Management

As nice as it sounds, all the advantages come at a price. It is much more complex to write idempotent scripts, which work automatically without user interaction in any given situation than just copy-and-pasting a few commands from a wiki page. Instead of setting up a virtual machine within a couple of hours it could take a couple of days to script all the steps. And even after spending such a long time, not all scripts were 100% stable. Sometimes they didn’t run but only acted as a step-by-step description of the desired steps. Sounds familiar, huh?

Another difficulty was testing of our recipes. As they usually contained a mix of locally executed ruby code, and bash scripts running on the servers, I didn’t manage to write a consistent suite of automated tests for the whole pack. So we had to treat our scripts with care. And refactorings often led to some undesired side effects. But, still, the suite of recipes was valuable enough to be worth the time and the occasional problems they caused. Especially having to re-install an application server at 2a.m. in the morning was so much more robust using the recipes than doing all the required steps manually. We really brought our configuration management process one step further by using Capistrano for automating our how-to documents.