State of Development: Annual Address on How We Ship Software

It’s been a while since I talked about how we develop and deploy software at my current job. It’s come a long way from the “Good Ole Days”, when cowboy coders manually FTP’d their changes to the master server and rsync came along 5 minutes later to replicate the changes to the slaves *shudder*. Keep reading below for the details.

github repository

We migrated to github from subversion about a year ago. Even though we don’t use that branches so often, I’ve found the speed & stability of git greatly improved and the amount of frantic recoveries due to user error (i.e. copying directories but forgetting to delete the damned .svn folder) dropped to zero.

As we’re using an Organization, we have a development team with access to the website code repository, and a sysadmin team with access to the puppet server configuration repo. What – no DevOps? As I mentioned last month, we’re not quite at the “Continuous Deployment” phase. Plus, I don’t think DevOps means giving sudo access to developers!

Hudson build server

Hudson is the core of our build environment. Every 5 minutes, it checks out new changes from github, runs a rake db:migrate and starts testing. First, ~250 phpunit tests comprising 30% of the codebase runs for 30 seconds. Concurrently, webrat runs ~70 tests with over 650 assertions for 2 minutes. These mostly ensure SEO relevant tags & text appear where expected.

A successful phpunit build triggers four selenium tests (hosted by SauceLabs) – primarily form-based, user-content generation tests, these are our most basic “smoke tests” on pages that use javascript (something webrat can’t do) and take about a minute each. We even do a Facebook Connect login & registration test using selenium.

A successful webrat build deploys the new codebase to staging, including synchronization of new static files to our test Amazon S3 bucket and purging of changed static files from our test Amazon Cloudfront instance.

Once a day, we have a “code quality” build. PHP PMD and Checkstyle tools do a sanity check on everything from class complexity to code syntax keeping a nice overall trend graph that helps us quickly identify bad commits. Admittedly, we use the information from this build as a “rainy day” refactoring backlog.

PivotalTracker integration

PivotalTracker (PT) is my favorite tool for Scrum based project management. What, at first, seems too simple for professional use, quickly becomes an organizational focus point for sprint navigation. Since I first introduced Scrum, we’ve stuck with weekly sprint iterations. The main reason being better able to focus on the tasks at hand – it’s easy to say you’re busy until next week. 95% of folks can live with this and, it also gives them motivation to show up for the sprint meeting!

Product management manages and prioritizes the Backlog with other stakeholders in the company during the week. In our weekly sprint, developers commit to delivering X amount of stories/tasks/bugs relying primarily on PT’s built-in Velocity calculation (an average of the last 3 sprints).

As stories are committed into github, developers mark them Finished. This cues the corresponding product manager to have a look on the staging environment (where caching, etc. is enabled to make it one step closer to production) for the final sign-off for release.

Capistrano release

We typically do 3-5 releases a week – even more for smaller, pixel-pushing type changes. When we’re ready to push live, we grab the last, accepted git commit revision id from Hudson (double checking everything’s green) and do a cap release c=|git_id| h=prod from our desktop.

Capistrano then logs into all the servers and reports the currently running revisions (in case of an emergency rollback), runs rake db:migrate for any database schema enhancements, pulls down the code from the given revision id, and does the same sync/purge of static files on Amazon.

It then sends out an email to the dev and admin groups noting the released revision and relevant environment(s). And, finally, it even goes into PT (making use of the API) moving all “Delivered” stories to the “Accept/Reject” phase and creating a release “Milestone” again with that git revision id as title.

As an added bonus, it even pushes all these stories up in the Current iteration just below the last “Accepted” story. Now, product management has a “speak now or forever hold your peace” chance to review the story live.

Testing and automating builds and releases is the quickest way to ship quality software. Like Snoop says, you gotta “drop it like it’s hot” – don’t let inventory code clutter your warehouse!