Devops Demystified

Last week, my friend Scott Elliott did a great guest presentation at the college where he talked about, amongst other things, devops. Moreover, I’ve been involved myself in the setup and configuration of two software development workflows (which technically makes me a devop); one was in 2017 for a tech startup in Kitchener, and the other one was earlier this year for a much larger project at the University of Waterloo.

So what is a devop, and what is a workflow?

To answer those questions, we first must talk about some specific needs that arose for many software development projects over the past few years that led to the creation of both workflows and the devops that create and manage them.

Many software projects today involve creating Web apps that run on a server in a datacenter. This datacenter can be within the company itself (called a private cloud) or on the Internet (called a public cloud). The servers that host these Web apps are almost always virtualized; either an operating system running within a virtual machine, or an operating system container that contains the software needed by the Web app. Netflix and Facebook are examples of Web apps hosted in a public cloud that everyone should be familiar with, but most companies today host their own Web apps in the cloud to drive their own business processes.

Software developers create these Web apps on their own developer workstations. However, due to the nature of software nowadays, testing a Web app on a developer workstation is NOT the same as testing it on a production server in a real cloud environment. So, bottom line is that software developers have to be able to send their Web apps to a production server in the cloud for testing each time they make a revision to the Web app itself. After testing it on a production server in the cloud, they may need to make several revisions to it until it works as expected…. and each revision will also need to be tested on a production server in the cloud. This production server is only used for testing; when the Web app works as expected, it can be copied to another “live” production server in the cloud that is made available to other computers across the Internet.

In short, software developers have to continually run and test the Web apps that they are creating on servers in the cloud. For a large software development project, this may mean pushing different versions of a Web app to a production server in the cloud over a dozen times in a single day.

The process I’ve just described is called a workflow, and the IT people who make it possible are called devops because they are essentially system operators (ops) that focus on supporting software development (dev).

Let’s see how it actually works by examining an example workflow:

The first step is to get the Web app code into the cloud. Since many developers publish their code to a code repository such as Git or Github, this is the most common way to copy the code to the cloud environment.

Next, a workflow orchestration tool such as Jenkins will grab that code as soon as it reaches the code repository and build it using a software build tool (e.g. Maven for JAVA code).

The workflow orchestration tool will then create a brand new virtual machine or container that meets the exact needs of the Web app using an automation tool (e.g. Ansible or Puppet) and then copy the compiled code to that virtual machine or container and execute it.

At this point, the Web app is now running on a server in the cloud and can be tested. Some companies actually use software for testing too, but in my experience, it’s done manually by a software development team. During the testing phase:

• If they find problems, the virtual machine or container is destroyed by the workflow orchestration tool and the whole process is repeated.

• If the software is ready to go, then the workflow orchestration tool creates a brand new “live” virtual machine or container using the automation tool with the new version of the Web app and replaces the old “live” virtual machine or container with it. Boom! Now you have Web app 2.0 “live” on the Internet.

Seems easy, right? Wrong.

Let me tell you a little secret: being a devop sucks. It’s a lot harder to make all this stuff work properly than you can imagine, and it involves a lot of tinkering under the hood with several different pieces of software that don’t work perfectly with each other. Jenkins requires a lot of customization (plugins help this), and everything in Puppet is scripted in their own language or Ruby (Ansible uses Python, which is a bit easier) with a lot of the work for creating cloud VMs and containers done with templates (e.g. YAML templates for Kubernetes to create containers). You have to write and debug a lot of scripts, often several times just to get everything to work. And when it works, something will happen that will require you to do it all over again. Luckily devops are paid very well (therapy isn’t cheap, and they need lots of it).