Tales From the DevOps Loop: 4 Teams Approach to 1 Common DevOps Framework

Hi, and welcome to Tales from the DevOps Loop. We're looking at four teams and one framework.

Before we get started, I'd like to introduce myself. My name is Denver Martin. I am currently the director of DevSecOps at Thread Research. Thread Research is a clinical trial company. We specialize in doing distributed clinical trials for pharmaceutical and medical devices. We have participants and studies that happen all around the world. So we are continually improving and looking to make DevSecOps the best practice to follow for this type of work.

Additionally, I am an adjunct professor with Purdue University Global. I've taught IT courses there since 2005. It is Purdue's online institution for delivering high-quality education to adult learners, as well as traditional college students who prefer online or distance education.

I've also started work on my Doctorate of Business Administration. It is an executive MBA program from the Crummer Graduate School, part of Rollins College in Winter Park, Florida. My dissertation topic will include information surrounding DevOps, the implementation of DevOps, and the use of disruptive technologies like cloud technologies, SRE teams, and containers with Docker and Kubernetes. It is a great program if you have an idea of how you would like to add to and continue to improve business across your organization, or across business in almost any industry and landscape.

Having said that, I am coming to you live tonight from one of the classrooms that we have available for students and instructors to use.

Let's get started and look at one of the implementations that I've been part of. I've been part of several implementations of DevOps at several companies. This is not my current implementation that I'm working on; this is a previous one at one of my previous employers. It successfully shows how a movement into DevOps can be done. I'm not saying this is the way to do DevOps at every institution. I've implemented it in different ways at each of the institutions that I've worked on, based on their company culture, the individuals assigned within the teams and at various roles. You do need to do an assessment, if you will, as to what you think would work in terms of adoption.

With that, let's look at this particular implementation of how DevOps was successfully rolled out within an organization. I'll go ahead and show you some of the information I have.

We'll start off with a little bit of background in general. Then we'll talk about the leaders and teams, and then we'll come around to ways of working. These are the ways in which we had to change so that we were able to have success happen among the four different teams, and ultimately they all ended up being part of the same enterprise framework for the way that we implemented DevOps.

The four teams started off as almost like an independent journey. We ended up looking at three operations teams and one development team. Not to say that the development team was just one group of development. The development team actually included all of the development groups inside that area, so that became a pretty large team.

Development started its own journey of going through DevOps, ironically, without operations involved. However, operations did quickly catch up, and we did end up being able to share common tools and move forward together. But it started off very much like a silo, with each of the groups doing their own thing.

At that time, operations included working long hours, being on call, having people very burnt out, running from one fire to the next, and really no relief in sight.

We had the fortunate good luck of an executive member, a senior director who eventually became a VP, who had the idea of moving all the production systems, all the development systems, and all the operations systems from an on-prem, self-hosted cloud into AWS cloud. That gave us an easier time, better standards of quality, and relieved us from the day-to-day operations of having to maintain our own cloud fleet in regards to the hardware. This was a very wise move on our part. You could go to other cloud platforms if you felt you had enough skill set in that. That could include Google Cloud Platform, Oracle Cloud Infrastructure, or Microsoft's Azure product if that is the one you like. We settled on, or agreed to, AWS as our cloud platform of choice.

We'll now step through this a little bit and talk more about the individual teams. The slides do have a little bit different information; I didn't want to read those verbatim to you. It is good information if you wanted to share, but actually listening and observing this is going to be something that you'll take away a little bit more.

If we look at leaders and teams, Team One is how I'll identify them. They were an operations-based team. They predominantly worked on the network infrastructure. Their team size was nine members: seven in the U.S. and two in India. They did provide worldwide coverage.

Prior to DevOps, they were working 80% of their time firefighting, both in India and U.S. hours and even after hours. Only 20% of the time was available for them to work on 30-plus projects. They had a lot of work piling up on them, and very little time to do it. Break/fix took up a large amount of work. The on-call rotation was pretty much treated like a hot potato. Nobody wanted to take it, and when you got off it, you were very relieved and very happy.

The skills on this team were good. They moved to AWS cloud from a traditional physical data center. They understood their craft really well. They understood networking very well. They understood how networking impacted other teams and other groups, so there was a good foundation in that regard.

They were responsible for the work centers. The work centers they had in place were building out the environments, connecting to remote clients, and doing basic operations. If something broke along the path or the connection end, they would go through and work on that.

Before the new leader joined, the prior leader had a very command-and-control type view. They handed out individual task assignments. They pretty much siloed each of the resources. The resources did not work together as much as you would think. They each had their own projects or their own actionable items that they were working on and hardly ever talked together. There were never any team meetings. It was a very heavy, leader-laden type organization.

When the DevOps leader joined and subsequently replaced them in the prior manager's role, they ended up starting a delegated workflow. They figured out what the workflow was for each of the different work centers, then figured out a way to delegate authority, moving it closer to the individual team members, and encouraged the team members to work together as a team as they worked through the different components and pieces.

They also looked at setting up ways of doing continuous learning, which included doing a book club to introduce them to the DevOps mindset. Already in the cloud, they had some advantages they were not able to take advantage of because they were still working in the older IT shop mentality of: if it's broken, fix it, and then whenever you have time, work on other projects. That was a tough way to look at it.

There was a transition plan put in place to have them adopt more of a DevOps mindset. I fully believe DevOps, while it has tools and principles and philosophy, more importantly has a mindset for people who have worked in DevOps and continue to work in DevOps.

Team Two is still an operations-based team. They were more or less platform infrastructure, so they would deal with S3, the storage within the cloud. They also worked on EC2 instances and OS-level patching, and all the things that are more platform-centric. RDS database would roughly fall under their purview, as well as making sure the instances supporting RDS were doing well. They may not have been database administrators, but they were able to make sure the foundations and functionalities were there.

Again, they helped move to the AWS cloud from the traditional physical data center, so they had much of the same skill set as Team One did.

Their prior leader was again command-and-control task assignment, siloing all the resources. It was the way of working at this company originally.

They actually had to transition through two DevOps leaders. The first one did introduce them to tools like Terraform and pushed automation in the form of scripts. They increased project workload to give every engineer their own little pet project. They assigned tasks after reviewing every ticket with the team. They reviewed the queue together, and then he ultimately assigned out the work. While it was a step in the right direction for DevOps, it wasn't truly embracing where you would expect your leadership to be at that level.

There was a need for another DevOps leader to be put in place, and this DevOps leader was actually promoted from Team One. They had a really good grasp of the idea, had previous management experience, and saw all the advantages that were happening in Team One during the revolution, if you will. They took a lot of the same ways of thinking and working from Team One into Team Two. Having come from Team One created an immediate feedback loop between the two teams, which had a lot of work that was interdependent.

Originally, Team Two was 85% to 90% break/fix firefighting and only 10% to 15% planned project work. They were in more of a firefight than the network team was, so they definitely had a need and a want to move to DevOps.

We'll take a break from operations teams and look at the development team. Team Three is our DevOps team for development. This was a full-on DevOps shop. They had multiple streams. They were doing agile project management for development efforts and for doing major releases. All major releases were happening, instead of iteratively, as large big-bang releases of the code software that the company was producing.

It is kind of interesting that they were doing DevOps on the actual development of the code, but the code itself got major releases. Basically, code was being released two to three times a year. If you were lucky, you might get four in a year, depending on whether they were doing the points poker correctly and whether they had enough to make it a major release. Instead of following the iterative approach that you typically would think of, making feature requests sooner, faster, and quicker, those seemed to go a little bit slower in that regard.

They did have a mastery of the DevOps tools, repos, and source control of the code. If you were looking at people who could execute DevOps really well in terms of tools and technology, this was your team. These were the people who had that down as far as the development work goes. But there weren't any real feedback loops from operations. After they developed it, they released it, often telling operations and the customers at the same time that the code was ready. Operations had no time to prepare their scripts for install or upgrade. There was no mechanism saying, "Hey, we're working on this particular piece," or any feedback from operations letting them know something was broken. It was definitely a troublesome way to look at having DevOps implemented in half of your organization.

In this one, we did have a DevOps leader who pretty much saw that the tools belonged to his group and that he would let us, on occasion, use those on the operations side. As you can probably tell, I was part of the operations group within this movement. We often had to have special tickets open, special firewall rules opened up between the organizations to allow us to have access to the repos, to be able to do source control, or even to access Jira and Jenkins, part of our tool stack, for creating automation and looking at doing operations. It was quite a contentious undertaking. Basically, they were the DevOps shop and we were the interlopers.

Eventually, that leader was replaced with another DevOps leader. In this case, they partnered with us. They actually worked with Team One and Team Two, and eventually Team Four. We invited them to the BPMs, the blameless postmortems, to talk about things we recognized in operations. They took the feedback back and looked at ways to improve the stability of the product and applications based on what we were seeing in operations. They looked at how we were doing our deployments. They started working on code to have one common pipeline between operations and development.

On top of that, believe it or not, development was actually on-prem. They weren't doing any of their development work within the cloud, and all of the operations team was running in the cloud. That created some interesting struggles as we built out a common pipeline.

If we look at Team Four, Team Four is an operations-based team again, but this is your application and product support. These are the people who actually worked with our clients. They were on calls to help clients if they were running into issues or having trouble using the product. If there was stability trouble within the inherent AWS environment, they would be able to get Teams One and Two involved if needed.

Eventually, they did work with Team Three, getting more information on how the customers actually used the product and what issues they saw, to have a better understanding of what feature requests should come about based on actual customer feedback. They knew the product inside and out. Many of these people had no experience within the other technology stacks, so they didn't necessarily understand how cloud worked really well. They didn't know the tools that were there. They didn't understand some of the automation they could put in place while helping support clients. They were usually the ones working tirelessly to do a deployment or an upgrade of a new release that they were not necessarily aware of: what was going to be in that release, what things would break, or how to get those things resolved.

Getting feedback within development, being part of the development cycle, and being able to influence what features got implemented and how they got implemented became very important.

In this case, we had a leader that was command-and-control task assignments, focused on the quick fix and not really the long-term fix. They believed that as long as the problem was not happening, that was good; just move to the next thing and work on the next problem.

When we implemented a DevOps leader within that group, they focused on finding and fixing the root cause. They set up a book club so they could review getting the DevOps mindset within their team. They set up training goals to learn new ways of working. They developed team-internal BPMs. Even the ones that didn't make it as a true outage, if they recognized something as a team that didn't quite work right, they created their own BPMs to work on within their team. They took that to heart to understand how to improve.

They also started working with development and giving them direct feedback, as well as giving feedback throughout the BPM process or the P1 review process that we implemented later on.

I've kind of alluded to it. Some of the major influences here are that they did have executive sponsorship, and hiring managers who understood DevOps not just as tools, but also as philosophy, mindset, and way of working. They understood that they had to change culture. They had to change the way the teams worked, and this had to be something that wasn't just in transition or a fleeting thought. It needed to take root and survive people moving in and out of different roles throughout the organization. There was really good recruiting done to find those right leaders and then have those leaders bring their teams up to understanding what DevOps is.

Team One created a network summit. Each year, they would have a network summit and bring everybody together, even folks from India, to work for a week doing a retrospective of the year and major planning for the coming year. They figured out what worked and what didn't. Purposely, we brought them into a small conference room where they had to actually talk to each other. I don't know that we would do that now in the COVID timeframe, but we would definitely get them into a Zoom or MS Teams call together and schedule a lot of time together, giving them some white time or white space to chit-chat and work together.

During that first summit, they talked about the four types of work, the three ways of DevOps, and how to prioritize. That was a very interesting process as we worked on the best way to decide which project to work on first. The team originally decided to work on all the big major business projects, and quickly learned that more important was learning how to eliminate unplanned work, as it is the true time thief. From there, they learned how important scheduled maintenance was, and how to understand, if you looked at your own work center, how you make it more efficient so it doesn't become the constraint.

All of that was really good learning. They gained that through the book club, where they reviewed The Phoenix Project and The Unicorn Project. They looked at Goldratt's The Goal, Beyond the Goal, and It's Not Luck, as well as eventually doing some Decker series work, looking at just culture and drift into failure. They became a well-rounded group.

With Team One, over the course of the next six months, they set up best practices. They started creating standards and deploying the standards. They looked at tools and automation. There was a CMDB that was always two years away, so they created their own database for use for automations. They started to move from break/fix work that was unplanned to more project and planned work.

They had a huge win: they developed a network router script that could run and populate knowledge automatically. It reduced their average troubleshooting time from 90 minutes to five seconds, just for gathering information from all the different sources and having it in a standard format to start troubleshooting quickly. They started working on automations for how to synthesize that information and then write automatic scripts that could self-deploy and self-heal based on the readings they had within the automation database. This was truly looking at the ways of working. This is also where we saw the first emergence of a Kanban board within the operations side.

They developed a way of working that was different than before, which had been individual tasks that were never seen together, didn't have swim lanes, and weren't sure where they were in the process. This was a really good way of doing that. On-call went from being a hot potato to being something they enjoyed doing and finding new ways of working. That was a very interesting change in that atmosphere.

Team Two was a little bit larger. They were moving forward on more projects than Team One, but they were really focused on tools and projects, not focused on the why of DevOps or the how of DevOps. The team had a solid adoption and borrowed the development tools initially, then shared those with Team One, the network team. They were great with automation and scripting, but they were not seeing it as true orchestration or self-healing. They had a lot of manual intervention to push those off. Again, they saw on-call as a hot potato.

The manager became frustrated and eventually left the role, and then we implemented another DevOps leader. The new manager was promoted from Team One, had prior platform experience, and looked at the projects and the WIP. He deprioritized a lot of the work that didn't hit the core requirements in terms of the company goal or the team goals. He also brought a new level of automation, having them look at it as a service catalog and think about pipelines they could do that would take work away from Team One and Team Two and have it as one single push-button flow. They also implemented a book club where they did continuous learning, including The Phoenix Project.

Team Three owned the DevOps tools. They controlled the access. They had firewall rules to keep people away from them, and then would only allow access on an as-needed basis and have that access removed whenever they thought they should. They also didn't let anybody know when they were scheduling maintenance on the tools. If operations needed those tools for supporting, rebuilding, or doing something with a client, they sometimes weren't available during key times. It was tough to look at the way the two teams, or three teams at that point, weren't talking well together. They were definitely their own independent DevOps group that didn't really include ops.

There was a change in management. They started attending the blameless postmortems, as noted before. They helped pay down some technical debt, and new releases started to become more stable and easier to deploy. They started working on a common pipeline with developers and operations working together on that, and then actually moved their development environment to the cloud, which is where operations had all the environment already set up. They were able to take advantage of all the pre-work that was done and get environments stood up that were very much like operations, which the development teams could work on. Once we got the teams working together, it became a really good way to move fast toward having complete operations within DevOps.

Team Four was an application support team. They knew their product really well, but they needed help getting into the modern tools to understand how it is. They treated the application stack more like pets than cattle. We had a new leader come in, and they started a book club to understand the difference and the way that Teams One and Two were working, and then try to get them involved with Team Three.

With that, the new manager emerged. They transformed from the transformation from Team One and Team Two. The application team started to join inclusion in BPM calls. They relaunched the book club with real success. They also started to work around the five ideals of The Unicorn Project, and they worked to develop a safety two culture. This team was fearful of being fired every day, and what we needed them to do was feel secure in their role and know that we needed their feedback. We needed them to move things forward and do really well.

This team moved to 60% scheduled maintenance and 40% firefighting in one month. That was a major transition, and they had the support from Team One, Team Two, and Team Three to make that happen.

Overall, our customers are our winners when it comes to moving to DevOps. Our second winners are our employees, who have a greater sense of work-life balance. Generally speaking, I've been told many times that after a team has moved to DevOps, they feel like they are now in therapy together: they can talk things out, they can work on things together, and it is generally a safe, happy place to share the experiences they are having at work.

Hopefully, if you're moving forward in a DevOps opportunity, you can do that. This is just four tales, or one tale of four teams moving into one common framework. I'd be willing to talk with you more at some of the Birds of a Feather or Lean Coffee gatherings. If you see me in any of the places, reach out. I'd be glad to talk more about these tales and others. Thank you, and have a good day.