Test driven Apache configuration?

Creative Commons License The U.S. Army
I know I might be going out on a limb here, but I recently got burnt pretty badly by _not_ having any test harness around my Apache configuration.
We maintain a database of some 15K rewrites and I accidentally blew this away. But it gets worse – we didn’t even notice for a whole week!

Of course our Google ranking tanked, we lost a lot of visits and I’m left scratching my head. How did this happen? And, more importantly, how can I make sure it never happens again?

As usual the root causes for such a catastrophe are manyfold. The environment was extremely fragile with zero checks in place to verify correct operations. First thing I did was setup some pingdom checks to alert us if anything is amiss with our rewrites. But a few checks performed by some (albeit very professional) external company isn’t enough to safefast a critical business asset.

So I decided to make a transaction safe wrapper script for modifying rewrites. After adding/modifying any data, there are quite a few manual steps involved in deploying new rules. Of course, our environment is load balanced which multiplies the chances of making manual mistakes.

The new script handles deploying and testing a subset of existing and new rules for all the given servers. If any test fails, the old rules are “rolled back” and retested to ensure minimal business impact.

What used to be a high risk maneuver is now fairly routine and much more transparent!

How do you handle business critical, Apache configuration changes?

2 thoughts on “Test driven Apache configuration?

  1. Hi,
    Have never had to manage rewrite rules in a database before, however, we did implement some solutions both for apache configs and for rewrites in files.
    For one project, we used rpms that contained the Apache config.This allowed easy config tracking and rollback but this was somewhat heavy process though it didn’t cause much issue because there were infrequent updates.
    Another solution had the webapps contain their own apache config files that were versioned, packaged and deployed with the apps. This enabled the developpers to be autonomous with their apache config but it didn’t enable day-to-day management of it.
    We typically have monitoring on the various URLs so we can notice if we break anything. I could see also a solution where tests are populated from the apache config (additive, not substract from it and then we can acknowledge any issues)
    Tackling a different situation but with similar problems now. We’re trying to maintain consistency of our apache configs through our various environments (from development, test, load test, production), version controlled but also customized for these environments.
    Suggestions are welcome 😉
    Thanks

    Like

  2. We have unit tests around our Apache config…as you indicate, it’s just as much a part of the application as anything else. Apache is an easily started and killed process. Our apache config is a template, so we just gen the conf file, start apache pointed at that file, hit it with a few tests (urls can’t go beyond a certain length, for example), and then kill the process.

    I wouldn’t hesitate to do testing on daemons like this. The main practical question is speed – if the daemon is quick to start and kill, and it’s not flaky for some reason, it’s totally testable.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.