The U.S. Army
We maintain a database of some 15K rewrites and I accidentally blew this away. But it gets worse – we didn’t even notice for a whole week!
Of course our Google ranking tanked, we lost a lot of visits and I’m left scratching my head. How did this happen? And, more importantly, how can I make sure it never happens again?
As usual the root causes for such a catastrophe are manyfold. The environment was extremely fragile with zero checks in place to verify correct operations. First thing I did was setup some pingdom checks to alert us if anything is amiss with our rewrites. But a few checks performed by some (albeit very professional) external company isn’t enough to safefast a critical business asset.
So I decided to make a transaction safe wrapper script for modifying rewrites. After adding/modifying any data, there are quite a few manual steps involved in deploying new rules. Of course, our environment is load balanced which multiplies the chances of making manual mistakes.
The new script handles deploying and testing a subset of existing and new rules for all the given servers. If any test fails, the old rules are “rolled back” and retested to ensure minimal business impact.
What used to be a high risk maneuver is now fairly routine and much more transparent!
How do you handle business critical, Apache configuration changes?
Agile Web Operations 
Hi,
Have never had to manage rewrite rules in a database before, however, we did implement some solutions both for apache configs and for rewrites in files.
For one project, we used rpms that contained the Apache config.This allowed easy config tracking and rollback but this was somewhat heavy process though it didn’t cause much issue because there were infrequent updates.
Another solution had the webapps contain their own apache config files that were versioned, packaged and deployed with the apps. This enabled the developpers to be autonomous with their apache config but it didn’t enable day-to-day management of it.
We typically have monitoring on the various URLs so we can notice if we break anything. I could see also a solution where tests are populated from the apache config (additive, not substract from it and then we can acknowledge any issues)
Tackling a different situation but with similar problems now. We’re trying to maintain consistency of our apache configs through our various environments (from development, test, load test, production), version controlled but also customized for these environments.
Suggestions are welcome 😉
Thanks
LikeLike
We have unit tests around our Apache config…as you indicate, it’s just as much a part of the application as anything else. Apache is an easily started and killed process. Our apache config is a template, so we just gen the conf file, start apache pointed at that file, hit it with a few tests (urls can’t go beyond a certain length, for example), and then kill the process.
I wouldn’t hesitate to do testing on daemons like this. The main practical question is speed – if the daemon is quick to start and kill, and it’s not flaky for some reason, it’s totally testable.
LikeLike