Last week I made an honest mistake that caused severe downtime on our website. Of course, I spent a lot of time thinking about all the contributing factors which ultimately caused such a nexus in the space-time continuum to appear over my keyboard at that crucial moment when I hit the enter key. It didn’t take long to realize what I had done – within a few minutes I heard the first murmurs of “WTF?!” coming from my colleagues.
It turns out that one of the factors which caused the critical failure had been implemented a good year before I had even started working for the company. And of course, there were multiple people involved and of course all of their implied, silent assumptions about how others worked. But regardless of all these “setups for failure”, at the end I was the one that did it – I was ultimately the one responsible for pulling the trigger which obliterated our site within minutes.
Projects vs. Adhoc tasks
Like most companies we have a few parallel projects running, a few more in the planning stage and a whole slew of adhoc tasks. Planning project tasks is easy – you maintain an up-to-date backlog, slowly working the list down in order of priority and conducting at least weekly status meetings with all the involved parties.
Planning ad hoc work, however, is an oxymoron. It’s not called ad hoc for nothing. ad hoc is Latin meaning “for this purpose” but in my field, it has a negative connotation associated with “this is super-urgent and needs to be done now or we’ll go bankrupt.” I’ve also heard it described as a “sprint breaker”, “velocity muzzler” and even “soul sucker”. When you’re the only technical guy in your company, your schedule can fill up with this kind of work in no time at all.
Pair-operating helps maintain focus
They say “Fool me once, shame on you. Fool me twice, shame on me.” I know better than to lose focus while working on a production environment, but the difference this time around was I didn’t have a spotter. Pair-programming is incredibly powerful way to improve code quality, distribute knowledge and foster team work.
I’ve taken this one step farther and done pair-operating at a past job with really great results. Two pairs of eyes are more than twice as good as one. You automatically take more care with your commands because you don’t want to look like an untrained chimp in front of your colleague. You usually talk about what you’re going to do before you do it. And, last, but certainly not least, if you’re typing something stupid, your partner will holler at you. I believe I’ve heard “What the hell are you doing that for?” more than a few times.
Ad hoc work must be planned
So what does all this have to do with agile? Simply stated, don’t allow ad hoc work to be – well, ad hoc. We accept that business changes their minds and priorities from time to time, but it doesn’t mean you have to turn your back on one customer to help the next.
Instead, setup a simple ticket system that everyone has access to which clearly shows all ongoing and planned tasks (and their priorities). They’ll see that their complaints or concerns are not discounted out of hand (and this is already quite sufficient for most folks) and you’ll be able to again plan your work.
This is especially crucial if you are working in parallel on your production servers. Taking on ad hoc work while you’re in the middle of trying to rsync live data is career suicide. Color the terminal background a nice blood red and tell everyone that as long as they see you working with such a window up, they’re not to disturb you (unless of course the building is on fire).
If I had planned my work more carefully and not allowed as many interruptions I would have been more focused on my planned tasks and felt less hurried to complete them as quickly as possible. Don’t try to please everyone all the time – this is a recipe for certain failure.