Most of us have seen, or at least heard of, the Mel Gibson movie “What Women Want”. It certainly is a tantalizing idea – being able to read the thoughts of other people. Now being able to read your sysadmin’s mind may not be at the top of your wish list, but, I bet there were a few times it sure would have helped fathom the meaning of that chill stare of contempt from across the table.
Or, I could just tell you what they’re thinking. For starters, what will be the production usage of that new feature you’re developing? Now, I know you’re not Nostradamus – but, you’ve thought about this, right? Any idea how many additional web server requests per user or session? What about the performance overhead of those additional database queries — what’s the longest running query now? Will you need to run that cronjob every five minutes or is once a week fine? These are just a few of the everyday concerns your sysadmin has while operating your company’s website. Have enough respect for his work (and yours) to do a bit of homework and come up with some numbers here.
Speaking of numbers, what are the common bounds of this feature? Is it CPU bound, sucking every last cycle out of that new dual, quad-core Xeon processor machine? Or maybe I/O bound, happily thrashing hard drives? What about memory? If you know you’re going to need a couple more GB, place the order. How about ports? Please, don’t expect port 10537 to be open in-bound in production so Fly-by-Nite industries can call your homegrown web service. These points may seem to technical to discuss, but, trust me, the web operations folks are exactly the right people with which to discuss technical details.
The next gotcha – building and deploying your code. I know you do it 20-30 times a day, but your sysadmin hasn’t had the pleasure of doing it a single time. Shoving the branch name and revision number over the fence with an email asking them to “just run build.bat” probably won’t make you many friends on the web operations team. Instead, spend a day writing up the particulars of building and deploying your application on the wiki. You’ll probably find a whole handful of serious problems, and all because you’ve forced yourself to think about how to run this on a box besides the ones sitting in front of you.
Ok, let’s say you got the basics covered and the sysadmins want to release your code. The next question will be “How do we tell if your application is in trouble?” Just providing a simple monitoring hook within your application may make the difference between restarting your application or rebooting the entire server. For example, a basic screen performing a couple of database queries effectively tests your entire stack. Have one db query written as a raw connect, and the second one using the provided connection pool (if you’re using one). Being able to see at a glance if the database connectivity is gone (db problem), or the connection pooling is locked (app server problem) or if the screen itself can’t be served (web server problem) saves operations a lot of time and builds trust (not only in your application, but in you, the developer).
Speaking of saving time, standardized logging with senseful production verbosity is another great way to give you crucial feedback as to how your code is functioning in the wild. Use a log file format which is easily parseable (CSV or other special character delimited file format). Not only does this make your log files easy to read, they are great for producing graphical reports. Beware of excessive chatter in a log file to avoid the lesson of “The Boy Who Cried Wolf”. I don’t particular like seeing anything below ERROR level in an production application log file. If needed, you can always turn the logging level down to INFO or DEBUG if you detect a problem which needs closer attention.
Now that I’ve given you a glimpse into what you thought were the stark, raving mad desires of most sysadmins, don’t forget to add these operational user stories to the project backlog! If at all possible, get the sysadmin into all your project meetings. Given the right forum, and opportunity, he’ll be sure to bring up all these issues (and possibly quite a few more).
Agile Web Operations 