Spliffs and Submarines: The Two Cultures and the State of Devops

This is a guest post by John Arundel (@bitfield)

In answering the question: ‘What does the red spectrum tell us about quasars?’, there are various words that need to be defined. What is a spectrum? What is a red one? Why is it red? And why is it so frequently linked with quasars? …What the hell is a quasar?
— Rimmer, “Red Dwarf”

What is Devops?

Everyone knows what they think it means, but they all have something different in mind. Not surprisingly, many of the conversations about Devops end up being arguments at cross-purposes.

Things people think Devops means:

• Developers who can install Linux
• Sysadmins who can write code
• Connecting operations to business
• “1. The cloud. 2. ??? 3. Profit”
• Infrastructure as code
• Agile operations
• Cross-pollination of skills between coders and systems hackers
• Continuous integration and deployment
• A weekly hostage exchange programme between development and operations teams
• A vague, warm, fuzzy sense of co-operation. Can’t we all just get along?
• A pragmatic recognition that we need to work together to solve common problems, in order to achieve common goals

Some people see it as a revolution in the way we do operations and development. Other people talk it down as something that good sysadmins were doing anyway, and that the technical proletariat have only lately caught on to. For many small organisations, what they’re doing must be Devops because there’s only one person doing all these jobs anyway. For big organisations at the other end of the scale, where the team that manages switches must fill out an online request form to communicate with the team that makes firewall changes, Devops-style collaboration seems like an impossible dream.

And it is a dream

The dream is that one day we as a technical people will rise up and live out the true meaning of our creed: “We hold these truths to be self-evident, that all geeks are created equal”.

To the progressive optimist, the idea of programmers and sysadmins batting for the same team seems obviously desirable. To the conservative cynic, it seems obviously doomed. Both are half right. The short history of computing has seen the rise of two very different cultures: the people who program the machines, and the people who keep the machines running sweetly.

My first IT job was in the kind of small and supremely relaxed Web software company characteristic of the dotcom boom. Developers wandered into work when they felt like it and left when the pizza ran out. They were artists, a creative élite. Many liked to code high, smoking spliffs at their desk, and the biggest clouds of smoke came from under the door of the CTO’s office. It was like a cross between the movie ‘The Social Network’ and being on tour with the Grateful Dead.

Fast-forward to a few years later, with suitable cinematic effects, and I am working for a large and respected US enterprise hosting firm, staffed chiefly by ex-military types, specifically veterans of the submarine service. Everything runs by the numbers. There is a procedure for everything, and everyone knows their place. If you break military discipline, you can expect a short, sharp shower of rebuke. Innovation and original thought are frowned upon, and spliffs are about as welcome as they would be on a nuclear submarine. Light up a reefer in the data centre, and Halon asphyxiation will be the least of your problems.

The culture clash is striking

In the software studio, we were sculpting something creative, unique, and original, and we were pampered artists. In the server mines, we were running a large and expensive commercial operation, and we were what the hosting industry sarcastically calls ‘intelligent hands’.

The world has changed. People will no longer cut you cheques in exchange for Internet dreams and hash-scented promises of code that will ship ‘real soon now’. Virtualization of the hosting industry, combined with ruthlessly dwindling profit margins, has largely eliminated semi-skilled data centre jobs that require only the ability to rack servers at 4am and survive on bad vending machine food.

Anyone with a pinch of Rails-fu can hack together a social web app with a big colourful icon, push it to Heroku before lunch, and spend the rest of the day wondering how to monetize it. Suddenly you don’t need to be a big company to make software, or to put it in front of millions of people. The software business has moved from the old days of big Everest expeditions with hundreds of Sherpas and tons of supplies, to an Alpine-style business model: small teams, moving fast, with minimal support and overheads, are the quickest to the summit.

So now the programmers, the builders of castles in the air, and the operators, who make sure the castles don’t fall down, find themselves all in the same tent at 29,000 feet wondering whose job it was, metaphorically, to bring the tin opener. Coders whose idea of deployment is ‘well, it works on my Mac – I’m off snowboarding’ now have to talk to Bastard Operators From Hell who previously regarded themselves as exclusive guardians of the secret flame of Unix. Technical people everywhere are starting to find that ‘they’ aren’t so very different from ‘us’ after all.

We have useful things to learn from each other

Talking together is a great start, and working together is even better. Thinking together is best of all. The discipline of pairing, where two people share a screen and pass a keyboard back and forth, coding, designing, building, automating, configuring, scripting, testing, fixing, deploying, monitoring, releasing, is probably the most efficient way to share knowledge short of a Vulcan mind-meld.

If you’re really doing Devops, integration shouldn’t begin and end with the org chart. Exchange programmes merely reinforce the notion that the other team is a foreign country. We should stop thinking of Dev and Ops as rival tribes, and pool our resources to defeat the real enemy (Marketing).

When coders and sysadmins work together routinely to get their jobs done, and when teams include multi-skilled people who like to learn as well as teach, and when people no longer say ‘That’s not our problem – talk to the sysadmins’, or ‘Don’t blame us – the programmers screwed up’, and when management starts to recognise that joined-up thinking equals good business, then we will at last be able to say that we truly live in a State of Devops.

About the author

John Arundel is a sysadmin, coder, and confirmed geek. He enjoyed his time in submarines, but now works as an independent consultant helping small companies build big infrastructures, talking and writing about Devops, solving interesting problems, and building castles in the air.