Visible Ops: Rolling Out Change Management

pingnews.com

Last week, I introduced you to “The Visible Ops Handbook” and their 4 Agile Steps to ITIL Compliance. While there is no silver bullet for your particular problems, these steps should serve as a good starting point. Today, I’d like to go into a bit more detail regarding what the Visible Ops folks call “Phase One” of getting a grip on your operations environment. Remember, this first step can take many months to roll out so if you want to do this in an agile fashion, start small and gain forward momentum (and the rest of your team’s confidence) with early, measurable successes.

Stabilize The Patient

I’m sure you know exactly which applications always seem to give you problems whether its routine maintenance or *shudder* a big upgrade. Whether its because of fragile architecture or “too many cooks in the kitchen” these applications probably account for the majority of your team’s effort and probably a lot of your site’s downtime. Logically, it follows that these should be the systems you lock down first. Who has access and why? Are they really responsible for the system or is their access just historical convenience? Pick a few of the most troublesome servers to start off your ITIL initiative with the following steps in mind:

1.  Reduce or Eliminate Access
   •     grant access only to those directly responsible for running the system in production!
   •     if more than 20% of your developers can commit directly to the release branch, introduce a Committer Role!
2.  Document the New Change Policy
   •     establish clear rules and processes about how changes shall be made and documented
   •     be prepared to deal with those who break the new rules!
3.  Notify Stakeholders
   •     ensure you have the understanding and approval from business
   •     you’ll be having fewer releases in the short-term so clearly communicate your goals to everyone
4.  Create Change Windows
   •     regularly scheduled changes create nice “Release Trains” that everyone in the company knows about
   •     “I have to get this feature signed off by QA this afternoon if I want to make tomorrow’s release!”
5.  Reinforce the Process
   •     verify that every change is made by authorized personnel and documented accordingly
   •     be prepared to deal with those who break the new rules!

Electrify The Fence

You need to be able to detect any unauthorized changes made to the “critical patients” you’ve identified above. There are a lot of excellent configuration management tools on the market (check out my introductions to the open source CM tools Cfengine and Puppet) and spending a day or two installing and configuring these will save you a lot of headaches and wasted work in the future. These tools can automatically discover undocumented changes made to your systems, notify the relevant change managers of the problem, and even revert the system to its previously known (and hopefully correctly functioning) state. To help you mitigate these incidents, here are some simple questions proposed by the IT Process Institute in their “Visible Ops Handbook”:

 Who made the change?

 What did they change?

 Should it be backed out? If so, then how?

 How do we prevent it from happening again in the future?

Modify First Response

Even with all the above processes in place problems will still arise. Once you’ve been alerted to a production problem don’t immediately login to the servers and start poking around. This is a knee-jerk reaction that will often cause you to simply restart the service in a fit of blind panic. Instead, go to the change logs created above and try to figure out what happened from a more analytical perspective. The insights gained from spending these extra ten minutes are well worth the downtime. If you would have just restarted those servers within 2 minutes there’s a good chance nobody else even noticed the problem; while this may sound perfect, if there’s no problem how can you justify spending the time and money fixing the root cause? Remember, if this happened at 2am (and your site is accessed globally) this outage would probably be measured in hours. Ensure you’ll have all the information needed to diagnose the problem before you restart the application(s)/server(s)!

5 Reasons for Change Management Failures

Congratulations! You’ve just implemented the foundations of a professional change management environment. If you’ve made it all the way here, you’ve probably ran across some of the following (unfortunately, all too common) excuses why this can’t possible work in your organization:

 “We can’t – we won’t be able to get anything done.”

 “The business doesn’t pay us to not make changes.”

 “We don’t need to – we trust our own people.”

 “Our people are professionals and don’t need constant micromanagement.”

 “We already have a policy and we believe there are no unauthorized changes.”

Don’t be discouraged! Change is hard (and change management may seem down right impossible at times) and convincing people takes a lot of effort. No matter what the naysayers are complaining about, you are doing the right thing and time will validate your decisions. In fact, my experience is that within another month or two, folks will wonder how they ever managed the old way! And between you and me, we both know they weren’t really managing at all.

Check in next week for second step to getting your company ITIL compliant!