Got Score? Scaling DevOps Adoption (US 2021)

• The usage of your DevOps tools tell a story • Elicit this story to identify insights • Present these insights into opportunities for improvement • Finally, wrap these opportunities into initiatives and gamify adoption We started our journey in 2016 intending to reduce wait times and manually-intensive, error-prone, repetitive handoffs across different departments. To accomplish this, we branched into a 'Center of Excellence'. Tasked with supporting up to 6 LoBs, each having multi-year, multi-million dollar initiatives, we designed the DevOps CoE with various arms - one tasked with Platform Engineering, while the other for Adoption, while still another for exploring emerging technologies. With support at all levels, we also were able to conceive various forums to generate grassroots enthusiasm while strategically employing some top-down levers to ensure, subtly, that everyone had some skin in the DevOps adoption ‘game’. And our entire adoption approach is extensively centered around metrics. We elicited data from our platform, wrapped these into many insights, packaged these insights into initiatives and are now well on our way in 'moving the needle' - on a near-real time basis - all the while, while having fun doing it. And in doing so, we’re leading the way for other business functions to follow, and adopt the ‘data-driven’ way of thinking.

breakoutlas vegasusvegas2021
RS

Rajat Sud

Principal Engineer and DevOps Evangelist, CareFirst

DA

Don Almeida

Manager, DevOps COE, CareFirst

TRANSCRIPT

00:00:12

Hello, everyone hope you're having a great time here at this summit. And thanks to Jean for weaving in a great group of presenters at this year's DevOps summit, uh, some really high quality information and people, people who are changing minds and hearts and the very fabric of how we work. We are thrilled to share our journey, uh, uh, on dev ops journey. My name is Don Donald Meda, and I'm the manager at the service benefit plan administration services corporation, which is an affiliate of CareFirst, uh, the largest insurer in the state of Maryland. I'll be tag teaming with rugged suit, our principal engineer, and chief evangelist at a dev op center of excellence

00:01:01

To help set the context of our journey. I thought I would let you know something about a business. We are a healthcare technology organization providing centralized operations and technology solution for a national health plan with a 5.5 million members globally. We are headquartered in Washington DC area and our team includes 1300 individuals, and we are all dedicated to transforming the healthcare technology in service of our customer's goals, by improved member experience and health outcomes, the SBB FC and chore seamless delivery of claims enrollment customer service capabilities, and we deliver data and analytics that enable critical business decisions. Now we do live in a heavily compliant centric industry where there have been specialist roles carved out to support various business functions, even within the business of software production and delivery from engineers to analysts, to configuration management and environment management specialists and InfoSec management specialists, all working in concert to orchestrate the release all the way to production.

00:02:12

So very early on in our journey, we understood and charted out what is now become a value stream analysis, where we captured multiple handoffs and wait states. And as you can see, they required a few each burden with the creation of management of auditable artifacts in various ticketing systems. And what's even more noticeable are the amounts of wait states where individuals are waiting on others in other job functions to complete their tasks while they embarked on their own to streamline. This was a challenge. And that proved to be the Genesis of our dev ops central excellence, essentially to reduce wait times and handoffs along realizing goals through focus areas, essentially go from what you see right now to this

00:03:11

Rare. We could leverage automation to reduce hand-offs and wait states, and more importantly, promote this then novel concept of delivery team instead of siloed roles and job functions. So, as we just mentioned, our central for excellence was established to streamline the software delivery lifecycle, thereby helping the organization scale effectively scale effectively by delivering features to our customers with improved quality baking in automation in order to reduce wait times and break down silos. And we did this by focusing on these areas on the right side, consistent and repeatable practices, for instance, leveraging automated unit testing, Celia as a practice to create builds once and deploy anywhere, providing the ability to have effective branching strategies for parallel development and fostering the treatment of everything as code, which means applying consistency and portioning controlling all artifacts. In addition to source code, then compose an application and then dig these artifacts and subject them to optimize and predictable flow streamlining processes and automating handoffs with the goal to eliminate them.

00:04:31

And this facilitated via improved feedback loops. We want to make it safe for teams to make changes, and that can be done by providing feedback faster. For instance, baking and court quality gates into the CII bill process itself to hold the process. If the thresholds are not met once deployed functional test suites, automated and executed on a cadence, and most importantly features are made visible to teams so that they can react. And the culture part, we continue to evangelize safe zones where we encourage continuous learning and experimentation. We are programmed level retrospectives, blameless, postmortems, and leadership actions. And then tying this into our discussion today, how we have empowered teams to make good on these initiatives by in effect, shining on a spotlight on their practices, showing them introspect, adapt and improve.

00:05:34

As we share an excerpt of one such insight for us, a few details about a central excellence empowerment engagement model. First things. First, we, we are obsessive about our platform, which is composed of industry standards, tools that provide CIC D code quality scans, testing, binary management, planning, and source code management. We provide a guaranteed uptime to our customers, the various lines of businesses, and we are well-structured training guides and enterprise DevOps playbook provide easy how to use for everyone interested in learning about adopting and even contributing to the betterment of a platform. Given the scale of a work that we perform and the new initiatives that the business undertake. The other arm of our center of excellence is arguably the busiest from rapidly onboarding new teams that are deploying using our existing tech stack to walking alongside others that are adopting new novel technologies, such as low-code no-code platform.

00:06:43

We regularly apply dev ops and system thinking to planning, development, deployment, testing, and monitoring practices. Whether we have to understand and configure the technology native DevOps capabilities or whether we have to customize our platform. The goal is to foster consistent and repeatable practices, what educating and empowering teams to improve the dev ops maturity. And as we deploy these capabilities, we turn around and ensure make metrics insights, and these are being effectively used. And also there are any gaps and opportunities for improvement. Finally, these oppurtunities are wrapped into an enterprise wide initiative initiatives, such as these surrounding the improvement of the stability of a pipelines and plugging in any gaps that may exist in them all with all the intent to make for consistent and repeatable practices while reducing manual time consuming, wait times promoting improved development practices, such as poll requests, decorations, and increasing the scope and occurrences of peer reviews, increasing port quality awareness for immediate field quality gates and improve a unit test and code coverage. These are just a sampling of our initiatives that we have embarked upon in the past and present. And to a large extent have demonstrably moved the needle, which brings us to a keynote of this conversation, which I will hand over to Roger.

00:08:22

Thanks, Don. So as we embarked upon maturing our practices and what just with improving the adoption rate of the various capabilities that our DevOps platform offered, we wanted to approach it in a very deliberate, scientific manner. And to do this, we really started analyzing the various scan reports offered by a disparate tool chain and realized that this required a fair bit of manual processing to really convert them into actionable insights that our teams essentially our customers could availa. So fortunately, given that all our tools expose all of their data via API APIs, we will able to probe and ingest them into a BI tool that allowed us some degree of creativity on how best to construct the insights, how best to visualize these insights into opportunities. And most importantly, in crude dev ops fashion switch from once quarterly or once monthly insights that required two to three person days to produce to once daily.

00:09:37

And in some cases, even in real time sets of insights, we'll talk about one such set of insights and how we've been able to wrap them into company-wide initiatives all the while, having some fun by doing it in terms of drives events and dev ops stairs. But before doing that, we wanted to share some of the initial challenges. I mean, you publishing insights are fine and dandy, but if there is no one looking at the other end, then guess what? You're just publishing yet another pretty report. And just like many of you having been in the business for many decades, we don't consider ourselves unicorns, but what we are are thoroughbreds, oh, well seasoned professionals who have gotten us this far and with the right levels can propel us even to new Heights. And as a rather large organization, we do have an influx of well-experienced Allen from various parts of the country.

00:10:43

So it was imperative that we conceive and publicize forums where our team members new and old are in a position to share their experiences. So we leveraged the levels from the bottom up in terms of adoption drives, building a dev ops community of practice in a speakeasy format for folks to contribute and discuss in a non-judgment. So all the while making it fun, all facilitated by our dev ops center of excellence. On the other side, getting the alignment of up years in leadership positions was also critical. So we also conceive of a gentle top down level to really up the ante in adoption, in baking into individual performance reviews up and down the organization chain the responsibility to improve the dev ops adoption rate. And once these levels were in place, so began a cultural shift led by the production and distribution of actionable. Just-in-time just enough insights providing clear direction on where the current maturity level lead, where it should be and what steps needed to be accomplished and more important that you need habits out of to improve. And all this in conjunction with providing forums for expressing feedback, opportunities, challenges, and general top leadership.

00:12:23

So enough doc, here's how we make the rubber hit the road onto our headline. One such initiative that got the ball rolling and put us on a fast track of maturity got score. Essentially what we built is a score, a single score associated, but every application based on how that application implemented the various DevOps capabilities, capabilities, such as producing automated bills, deployments, tests, good quality scans, et cetera. And just as importantly, how stable those capabilities for, and then wrapped those insights into a single aggregated numeric score. This score at a glance provided an objectified indicator on the DevOps disposition, call it, the DevOps helped off an application, which in dune was reflective of the team that contributed towards it.

00:13:34

The instant we published this, we noticed some may say a visceral reaction to those with scores on the extremities. The ones with the highest score of course walked with an increased swagger while the low scores instantly wanted to ask why. And in doing so rapidly explored our dev ops offerings and began onboarding there and about the leadership. Let's just say who doesn't love a good old fashion, healthy competition among teams and lines of businesses to get the highest score, right? And use an XO of the methodology on how we derived the scores. Like we just mentioned, we automatically award points to every application based on their adoption of our platforms, dev ops capabilities. So for instance, if they have implemented automated desks to execute as a bot of their pipelines, they get 10 points. And from a practice perspective, implementing the capability is not enough, but they have to effectively use it and is measured and reported in terms of stability. If their builds and deployments and test stability exceed 80%, they get an additional five points. If between 50 and 80%, they just get three points, an unstable build or deployment will not net them any points and will represent an opportunity for improvement. And as we continually add new capabilities, we've we factor this methodology to provide that insight to our team, to ensure they've implemented it. And more importantly, how effectively they've implemented it.

00:15:39

And this is where all the actionists, as you can see, we first drop the composition of our various lines of businesses, how many applications, each sport, and then on the dock rate or the aggregated scores, remember we spoke about promoting a healthy competition between lines of businesses. So I would imagine this is where our senior leadership spends that time in the lower left are the details for every application. We do apologize for the office location of company specific proprietary information, but hopefully you can imagine that every line represents an application and what its score is. And clicking through the line, refreshes the widgets on the lore, right, to reflect the application score and the health of the various dev ops measures. And this dashboard specifically the health of the C CD measures along with the code quality measures at the bottom complete with actionable letter grades.

00:16:55

And this translates into an actionable quantifiable plan with easy to digest insights that have paved the way to create action plans and facilitate conversations about how best to improve the applications. Thereby the teams poke leading down to the people, composing them and their dev ops capabilities, and also do big in that oft updated feedback loop where improvement initiatives can be recognized in various organization wide forums while degradations can be course corrected. And what mentioning again, that behind the scenes, as we onboard new DevOps capabilities, most recently, especially in the area of automated securitization, we are constantly refactoring methodology to account on a score for all of those.

00:17:57

Not sure if you picked it up in deal, your screenshot, but our maximum attainable dev ops score across all applications currently stands at a maximum score of 90. So hopefully this illustration has given you some food for thought on how to effectively implement and scale existing and new DevOps capabilities across your portfolio of applications. While also having some fun along the way, by appealing to the competitive streak in young colleagues, and more importantly, educating them and bringing them along for the ride. And now this provided somewhat of a blueprint on how other business functions have used the similar model to further their maturity efforts and enable a somewhat apples to apples comparison of teams with the intent to identify opportunities or hail and share successes and share them with the broader organization.

00:19:03

Essentially start with the least common denominator, the deployable unit to compute the effectiveness of adoption, then aggregated up to the team to gauge efficiencies, the product, to gauge predictability and quality the portfolio or program to determine effective resourcing. And then finally all the way up to the enterprise for quantifiably demonstrating maturity and in doing so periodically, be it yearly monthly, or again in crude dev ops fashion in real time, demonstratively moving the needle in the positive direction. So in conclusion, yes, what we are seeking from this wonderful community, even though a data-driven approach brings about a ton of insights. We do recognize the human aspect of dev ops and that a lot of opportunities are right for the taking in Nunes conversations. We'd love to hear about how you've elicited those insights about those conversations. What forums have worked in your experiences to facilitate those insights, especially centered around bringing your colleagues along for the ride, especially well tenured employees that have been consistently performing in the same role over the past many years.

00:20:33

We also have a keen interest in Yering about tackling the burden of technical debt. Especially for older applications develop an in production the past five or six years. Every time introduced a new tool with that comes new sets of insights. How have you effectively prioritize these insights? And finally, we'd especially love to your out of vested interest from those of you in regulated industries. How do you move the needle for processes and policies, especially those inherited from pre dev ops world. And with that, we thank you for lending on you and look forward to learning even more from your experiences.