Configuration Management remixed: Introducing Carpet

Migrating our production environment from debian to OpenSolaris I wanted to simplify our configuration management recipes along the way. What I came up with is a mixture of Puppet style manifests and Capistrano backed ease of use in a new open source project called: Carpet.

Building Your Infrastructure With Plain Capistrano

Initially we’ve built our debian environment using a set of custom made Capistrano recipes. We had various tasks like these:

• cap app:install
• cap app:install_memcache_client
• cap app:install_rails_stack
• cap app:setup_images_rsync
• cap maint:migrate_image
• cap maint:add_disk_space
• etc

All in all, there were around 150 custom tasks for installing and maintaining various parts of our environment. It was great to have automated Capistrano recipes for all those tasks as it really simplified the administration of the production environment. But, there were a couple of drawbacks which I planned to address with Carpet:

Drawbacks of a Plain Capistrano Based Solution

• It was difficult to remember the exact order of task execution needed to transfer the system from one state to the other. Often, adding some functionality required a new gem, a cron job and a config change in production.rb. Preparing the release for this feature meant we had to recall exactly all the steps necessary to setup everything.
• Some of the Capistrano recipes were quite brittle. While the main reason was the complexity of our setup (we were using Xen virtual machines with LVM and disk images connected via virtual bridges and a VPN between the physical boxes…), another reason was the lack of unit tests for the recipes.
• Our recipes were a wild mix of configuration and library code encapsulating commands e.g. for dealing with LVM volumes or XEN VMs. Separating configuration from implementation was one of my concerns for our new configuration management solution called Carpet.

Puppet to the Rescue?

Fortunately, there was already one solution for my set of issues: Puppet by Luke Kanies. Puppet implements a declarative approach rather than the imperative approach usually implemented using Capistrano. For a more in depth comparison check out my post about Puppet vs Capistrano.

Don’t Fight Too Many Fights at Once

I was evaluating my options in depth: I wanted to switch from Debian + Xen + LVM to OpenSolaris + Zones + ZFS and I wanted to make my scripts more robust. I had gained a fairly deep understanding of Capistrano by that time, but had no experience with Puppet whatsoever. To avoid learning new things on all fronts (OS and our configuration management solution), I decided to stick with Capistrano instead of Puppet. Yet, I still yearned for the beauty of Puppet’s declarative approach.

Mixing Capistrano and Puppet: Carpet

The urge to re-use my experience in Capistrano combined with the urge to have some of the beauty of Puppet made me work hard on a concept and solution for making my Capistrano recipes more declarative. I wanted to focus on defining the available system services rather than describing how to build them.
This was the starting point for Carpet, a set of Capistrano extensions and pre-defined recipes for building and maintaining your infrastructure. Introducing a couple of new keywords to Capistrano and defining a few ready-made recipes for building appliances made my life much easier. Now, it’s really simple to define your infrastructure with a few lines of ruby code.

Here’s a minimal Carpet definition of a web server:

type :web do
    needs :apache_lb
end

node "web1", :web, {
    :ipaddress => "10.0.0.10",
    :interface => "bnx0",
    :mem => "768m",
    :swap => "768m",
    :disk => "20G",
    :domain => "example.com",
    :name_server => ["ns1.example.com", "ns2.example.com"],
    :user => application_user
}

That’s all it takes to define a load-balancing apache web server using Carpet. And, because Carpet blends seamlessly with Capistrano, all your existing deployment scripts like cap deploy:migrations still work. Let me show you in some more detail how Carpet simplifies Capistrano.

Carpet Introduces New Keywords to Capistrano

The Carpet gem makes a few more keywords available for writing Capistrano recipes. Instead of defining your roles with the role keyword you use type. And instead of defining a server using the server keyword, you use node to describe an instance of any type. The definition of types and nodes is the foundation of Carpet’s power.

Define Types by Describing Their Dependencies

In the Carpet web server example above, we define a type (aka ‘role’) with the name ‘web’. Within the block describing the type you find the third keyword introduced by Carpet: needs. needs describes dependencies. When describing a web server we say, “a web server needs an apache load balancer”.

Carpet Comes With Pre-packaged Appliances – Recipes to Build Servers

apache_lb is nothing more than a Capistrano task which comes with Carpet. It’s a recipe to build a basic apache based web server which can act as a front end for a bunch of mongrels running on app server boxes. Carpet comes with a complete set of pre-packaged recipes like apache_lb, mysql_master, memcached and rails_22. I used to call such pre-packaged recipes appliances as they’re really turn-key solutions for typical roles in a Ruby on Rails stack.

Describe Nodes With All Their Attributes

A node is an instance of a type – a real (physical or virtual) server. The Carpet keyword node defines the name of the server, its type and all the attributes for that node like IP address, disk size, memory size, etc. And, of course, you can have multiple nodes of one type like multiple application servers.

The Beauty of a Declarative Approach

If you examine the above description of your web server you hardly see any Capistrano tasks. Instead of remembering to call tasks like

cap configure_ipaddress
cap configure_disk_space
cap configure_mem
cap install_apache
cap setup_vhost
...

you simply type:

cap web1

and Carpet will use Capistrano to install a complete web server for you.

Read on how to get started with Carpet or check Carpet out at: github.com and subscribe to our RSS Feed to stay tuned for more in-depth explanations and real world examples on how to weave your infrastructure with Carpet!