Devops: cleans and shines without harsh scratching

This is a guest post by Julian Simpson “The Build Doctor” (@builddoctor)

Out of the frying pan

Just as the .com boom turned to bust, I worked with a guy who had developed his own system for managing change:

• You backed any file youʼd change, with permissions, and a particular file extension.
• You copied the block that youʼd edit and commented it.
• You added a comment to explain who commented and why.

I admired the rigour that was being applied but had no desire to be a human version control system.

So I took the wise step of moving to a major hosting firm weeks before the .com crash. There I watched a service grow from bare Sun metal to a working application cluster over weeks. We never saw a single employee of the client; an army of consultants built the thing up. Every time a new a consultant did something, something else broke. The Veritas guy broke Oracle; the Oracle guy broke the HA application; and so on. They eventually gave up on the project.
It seemed that building infrastructure wasnʼt like civil engineering after all. You need to build systems lots of times to build them well once.

Discovering jam at the bunfight

After the crash I took the first job I could get: systems administrator in a real estate company. When we werenʼt arguing with the developers, and the BAʼs and the helpdesk we were trying to stop the development manager in his quest to be our boss. Fortunately , I had already latched on to the teachings of infrastructures.org and used cfengine to get the grunt work done.
Cfengine and perforce (hey, Subversion wasn’t at version one!) actually proved to be quite the combination. You could have the rigour that my dot-com colleague wanted, without pain. We could also rehearse our installs. Before they replaced our million pound a year application with an off-the-shelf package, I was able to build out any node I liked.

New strings to pull

Sometime in 2006, ThoughtWorks Colleague Ade sent me a link to Puppet. I started rolling puppet manifests and testing them, even though it would have been far easier for my own needs to do everything by hand. Applying testing discipline to infrastructure was fascinating, great fun although I never really got to apply it to a day job.

Interest in applying Agile methods to systems administration grew, and I had a conversation with Patrick Debois at the end of 2008 about how you could do this stuff.
Conversations about Agile and Systems Administration continued on mailing lists and Twitter, with Patrick forming a kind of nucleus. There was enough interest in a conference, and the name ‘devopsdays’ was eventually chosen. Patrick wrote:

still hesitating between naming it: “Agile Sysadmin” or to make it broader “DevOpsDays (D.o.D) Europe”

Agile Systems Administration was accurate if a little narrow in scope, so devopsdays was settled on as a compromise.

A cultural change

As such, DevOps is more a shift in culture than anything else. Why should we stick to the stereotype and refrain from coding? How can we use software to solve our problems, not just business problems? Can we let go of the BOFH? And will everyone please start using the same toolset? Java developers gained a lot by discovering JUnit and writing automated unit tests (yes, they did steal it from Smalltalk).

But for who?

I canʼt help noticing that the European chapter of the DevOps movement seems chiefly composed of sysadmins. Scarily clever sysadmins who hack fearlessly away at problems, but mainly people with a systems background. The ideal of getting some developers in to hack with hasnʼt come to pass. Sysadmins revolutionising their craft can only be a good thing. I think weʼll have been successful when itʼs commonplace to see developers and sysadmins pair programming.
Predictions

1. It will take years to reach admins in successful businesses. This change has been driven by two things: the need to scale, and startups that donʼt want to have Pete Townsend at their launch party. Late adopters donʼt face this pressure and have less need to change.
2. There will be hype. There will be attempts to sell many DevOps products. There will be yet more dumb press releases. You havenʼt seen anything yet.
3. Prepare for the Trough of Disillusionment, where people get DevOps very, very wrong. “Right, the developers have written the Puppet Manifests. Now make them work! No excuses!”
4. Tools will mature, snake oil salesmen will move onto the next Big Thing (I hope we finally get jetpacks) and people will finally begin to grok this.
5. Eventually itʼll be normal. Weʼll be delivering code faster than ever and wondering how we ever used to cope.

Sound familiar? The Agile Manifesto will be a decade old in three months. Prepare for the long haul.

Systems administration

We all lost something when you could do q vocational course in Java or .NET and be unleashed on the world with little insight into how production systems actually worked. Systems administration used to be a legitimate way to enter the industry and then (armed with very real lessons in computing) move on to other things. When it became a hedge against inexperienced developers, we lost.
Perhaps because we flooded our industry with developers who knew little about systems (i was recently told that london has over 50,000 Java developers), we had an unhealthy response: the slashdot and BOFH cultures. My personal belief is that devops is a rejection of conflict between two groups who attempt to achieve conflicting goals, and an attempt to be good at whatever pigeonhole you’re in.

About the author

Julian is Build and Release Consultant at The Build Doctor Limited, Systems Administrator, Devops enthusiast and Blogger.