Europe 2021
Beyond Firefighters vs. Safety Matches - Growing the DevSecOps Talent Pipeline
Three years ago, I presented a talk on the skills pipeline myth at ShmooCon, denoting that our ability to overcome our staffing and skills shortage is to look more broadly at the available talent and development channels that currently exist to address our shortcomings. Three years later, as the rise of DevSecOps out of DevOps has taken hold, an alternative model has adjust how we need to view the talent and skills pipeline to ensure we don’t put ourselves into a bigger hole and create a larger problem.
My first talk equated the typical mentality of technology staffing efforts to focus on direct or specialty skills to address each niche of the need for various security roles, rather than look at the needs based on the design of the complete ecosystem. My term of “firefighter”, were those specialists brought in to do a singular focused task, who were difficult to find for the role and resulted in premium pay and tightly scoped expertise. These are individuals brought in, often after an event has occurred or the organization has been told they need these skills or competencies.
Conversely, addressing the problem in the design phase, would expand the possible pool of talent to address issues that may not be specialists in a sole topic, but may encompass a broader range of skillsets. In this case, preventing accidental ignition through the implementation of controls, such as “safety matches” or other prevention and mitigation techniques. In this case, while solving a narrowly scoped issue can be viewed and “attacked” as a solution from different points of view and available resources.
With the movement of DevOps to DevSecOps, the model for both is a shared skill AND responsibility model. It is a reversed mindset to where the title of a DevOps or DevSecOps engineer relies on being a generalist to be able to be plugged in nearly anywhere, but may swing this concept to the extreme the other direction. In this case, nobody is potentially really good at anything, and thus impacts quality, reliability and security by expecting these roles to service every need.
In this talk I plan to see how we can find the middle and work towards solving our skills pipeline issues, but also adapt a successful shared responsibility model that adequately addresses the needs of modern architectures and service use models.
AK
Amélie Koran
Senior Technology Advocate, Splunk
Transcript
Hello. All welcome to my talk today. Beyond firefighters versus safety matches, where we will talk about growing the DevSecOps pipeline, mandatory forward looking statements, to be honest, being required to show these during a presentation for a few seconds is a lot less painful than having to get a talk to prove as a government official. So I'll take the Mulligan for you all. So hello, I'm Molly Kranz, senior technology advocate here at Splunk. I've been with them since 2019. Prior to that, I spent a decade as a public servant split up by a year at Disney. It happens. I literally needed a vacation, I guess, and by vacation, I mean working on the future of technology for one of the largest media conglomerates in the world bonus points for being able to say, I went to Disneyland for work as research during my time in the government, I worked for the interior department, treasury department and left service as a CTO for the us department of health and human services office of the inspector general.
Oh, and I helped start the us digital service while on rotation at the office of management budget at the white house. So that two, I'm mainly a security person in my career and tend to circle a lot in that community, including speaking and volunteering as a staff at a number of conferences, including multiple security beside conferences and as a Def con goon, which brings me to my topic today, three years ago, I stood up at ShmooCon in Washington, DC, adjusted my soapbox and told the general establishment that the way they were trying to handle an important shortage or shortcoming was wholly wrong and why they should listen to me, it made it easier that I was surrounded by friends and colleagues when I essentially did my best impersonation of Martin Luther nailing my treaties on the door of the Catholic church in Washington, DC. Most of that establishment tended to say that what you needed to do was to wave a magic wand and you can create or find talented security people for every organization and that everything was fine, or that you can fix this. My by adjusting curriculum in schools asking a passing UFO, if they had any security pros, they could lend planet earth or just wait it out until the rest of everybody's demand was met. And start from there. If you didn't want the fight for the talent, this in fact was not fine. I thought cooler heads would prevail. People would think a bit deeper about this problem and maybe gets to solving some of those issues either through policy or direct action.
please.
I know this is odd, but the more we force folks down a given and I'll be at narrow path, there's going to be some pushback and, uh, some pressure differential and kind of a back pressure. If you will, some will consider this some level of gatekeeping, but it's more of an ensuring that those with, for which our recent policy to get as many people in the cybersecurity to solve the pipeline problem may hit kind of a bottleneck. We may need some specialized talent. We may be forcing too many people down, one path, creating some resistance and possibly lowering bars for skills and salary that companies are willing to offer for some of the important expertise. It's good that we are catching up a little bit to the perceived talent shortage by addressing this earlier in education training and transitioning from other roles, but we need to do it smartly.
This is similar to my astronaut problem from my prior talk, where the prestige role may have a limited payoff in a short lifespan, even after taking a lot of work and effort to get there. So conversely, with the rise of DevSecOps, which is not dev dev ops plus security, the idea that we may be putting too much on one skill set or role may also prove to be somewhat problematic while there may seem to be limitless supply of folks who say they can do this job or want this job, this actually isn't the case. Plus the same skills gap we find in security. We also find in these roles and with that, are you then putting water or gasoline on that fire? It's not like there's an endless supply of these folks either, but many may find it easier to put a cap on than claim specialty in the particular role.
So we want the skills pipeline to ever be refreshing sustainable, but we don't want too much generalization to lose the depth. So some skills and overloading of some roles, how do we find that happy, medium? Or do we want to, is DevOps and DevSecOps something that's realistic and sustainable? Is it at the very least what we want and actually need? Or is it just a fat everybody wants to do like look at Tamagotchi and beanie babies. What's the care and feeding required to make this work? Namely, what are the skills we need to generally focus on? Or are we running down the wrong path? Just like dev ops isn't stacking development on top of ops and the relation to the workload and responsibility, but it's sharing and enabling some of these capabilities in a single role DevSecOps is moving and sharing skills and responsibility among a single role.
But where does all of this, the rest of the stuff that got peeled off of the traditional stack end up, this is where you allow for some specialization and the ability to focus and optimize and make efficient rather than having to build it all into one role person or responsibility in short it's. Okay. Not to be the wizard. It's okay to be an apprentice as well. Dev sec ops is again, not one plus, but also a balance of the various forces at play within the organizations for technology, part of the finding of the balance between the as-is and the, to be, or the president of future state as an assessment of the talent and your skills within the organization, and what parts of it exists to make the transition easy, and which gaps are there that need to be supplemented or built up. Sometimes this is merely training.
Those who are already there in some new skills, others it's finding entirely new resources to augment your team. The challenges with the skills augmentation is finding ways to add and plus up the folks who already have without overloading them with work and re responsibilities DevSecOps is about shared and balanced, not less than overloaded. So think of it this way. As you see here with the diagram, being a tight Venn diagram where the skills and responsibilities generally overlap fully, there's still some specialty roles within each main category, but it's a tight weave elsewhere. You can approach a more comprehensive overlap as it begins to appear as just one circle, but there will always be just a few responsibilities that the original team may need to solely own or as such, but may not have to actually do too regularly. Those diagrams on the right are typical organizational patterns.
I've observed in my multiple orgs that I've worked at four and N each are not quite at equilibrium, which is a goal here, but where are the forces from each team are balanced, but as well as the motivations of the team and overall strategy and organizational leadership, or in poll, some are led by a strong development arm. Uh, but others may have a subservient operational and security teams and other cases, there's some movement and conjoining where they touch, but not a core shared DevSecOps culture. This is as much a process and strategy issue then as well as much of a cultural and personnel issue. So anybody who might recognize these classical look and fellows Tantalus and specificity, you probably have heard of the ladder, but maybe not so much about the former. We often talk about how tasks seems Sisyphean all the work dedication effort, but we don't get our payoff to know this was a penance that Sisyphus by hate was put on to sophist by Haiti's as he cheated death, which in itself has a good story.
I highly advise reading the backstory, the former Tantalus anybody see here where the name of the root comes from tantalize, his penance, which wasn't it from exactly as doing something as appearing, as noble as this a fist was doing an, uh, being placed in a pool of water, neck deep, and when thirsty having the water move away from his mouth. So he couldn't drink and above the pool was a tree with fruit from which he wanted to grab and eat, being moved out of reach every time he grabbed, by the way, his offense was really revealing the secrets you learned while hanging out with the gods, uh, to the mortals, uh, killed the son, fed them to the gods to see if they knew if they were eating his son kind of a nasty thing, and then stole Ambrosia the nectar of the gods, and also gave that to the mortals kind of like sounds like an it vendor, right?
Uh, texts seems a lot like these two, the are regardless. So either a lot of work to see it just rolled back to zero or being tempted by a thirst or hungry, just to see it always out of reach. And these were penances. It seems like every day in tech with the move to dev sec ops or even dev ops, for that matter, you're using the desire for speed and optimization to go to, to play a little bit with your ego. You know, as mentioned earlier, our pipeline for tech is still a constraint workforce, but market demands put the desire from orgs and their leadership to put more features and capabilities out quicker and quicker. So they see the adoption of automation in CIC D methodologies in their org as a way to increase and satisfy those demands, but not realizing that you need the right folks, setting it up, running it and monitoring it to not bring out the proverbial, sacred horses out of the stable.
But if the lessons from the mythical Man-Month have taught us anything, you can't throw more people at a problem or expertise and have it sped up or solve properly. Yeah. It's a matter of matching needs with the staff and ensuring that you will find a balance, allows them to best apply their skills to a problem. So the push for a universal SRE or even a pluggable developer to take on every aspect of the design development delivery and operations of a service or application is only going to get you tired, SRS and developers work the strengths, but not the strengths to death, develop the teams of talents and cross train where possible, or have complimentary skills, but never ever throw it on the backs of small teams to build secure and operate. Okay. It's great to be all soft and conceptual, but where's the data to back up this warm and fuzzy talk up.
Okay. During my initial presentation, I found a major gaps between the bureau of labor statistics, BLS estimates for roles and expertise, given a job category and their occupational outlook handbook, which is for looking supposedly, but maybe misses the actual realistic demands versus what the economy actually is supporting day to day to know my ex worked at BLS. So I have a little idea how the place works after 10 years of marriage and half a decade of them coming home and sharing their own frustrations and questions. If anything, I look around my neighborhood in Northern Virginia and say, well, there's a lot of data centers being built, people doing a ton of online shopping, and really wonder if wind turbine service technicians and solar and stars. Installers are the actual jobs of the future. In the top 10, you do have information security analysts. So they may not be too far off with a meeting income of a hundred, 103 K per year.
And those installers ranging from 46 K to 56 K per year. So I wonder, what's going to draw more folks into a career when it comes to one new to secure their future income. When I researched this three years ago, just before they updated their job outlook, it was barely above a hundred thousand people with this job. Now it's listed 130, 1000 with a decade growth of about 41,000 more folks, a 31% increase kind of. So it gives you a pause for a moment. Uh, it was a hundred K in 2016 to 120,000 five hundred and twenty twenty six. So it's not a lot. Now BLS has computer programmers and software developers in two different jobs series. But for simplicity, I will focus on the latter while they aren't even listed in the top 10. They supposedly make up over 1.4 million or one, a 1.4 6, 9, 200 jobs with a suspected 316,600 expected job growth in the next decade, which is a 22% increase unless software developers subsumed the security folks.
Not only can you see a growth issue with the needs for supplies, but a major difference in skilling like nearly a one to 15 ratio, slightly larger. If you want to toss a computer programmers, which brings it to nearly a 1.1 w 17 ratio. So that's a lot throw in what's termed as a network and system admins, and you get 374,000 jobs with a 16,000 growth in the next decade of about 4%. There's some overlap of all these with computer systems, analyst, job descriptions, but these were most maybe be most likely be your architecture business consultants and some of the testing and requirements folks, or even agile coaching types of skills, which are 630, 2000 people with nearly 47,000 improvement on employment for an outlook of about 7% growth. There's definitely not going to be enough security people according to government outlooks, to keep pace with all the folks building and deploying apps, all of this will ride on software developers. Is this a good or a bad thing? And it's a lot of numbers. I think it's incredibly bad for many reasons. So what does our industry actually have to say about this? Just because of cursory look at what the government thinks we need. And the actual reality of the situation seems way, way, way out of whack my initial talk source, a really cool site that also sourced our numbers based on the actual open requisitions for security positions.
I think it's reliable. It's been cited by a number of folks in industry and academia before and controlling for the unicorn clause, which I dictate is having one person for a wreck with every skill under the rainbow and is going to conferences and speaking with peers, students, and others. And it's at least grounded in some sorta reality. Oddly, this project cyber seek is also supported by NIST and DHS through. Nice. Uh, so there's a lot to ask why the government stats and tools differ so greatly and what they are representing. Um, so let's see. Number slide. Yeah, there were a number side. So now besides needing bodies for roles, the other ask of executives and other leaders is the skills gap. So what does that all mean? In short, you can get the job, but can you do the job sort of what the question that's asked, uh, in Joe versus the volcano?
So while you can fill a slot, will you be actually able to do the job effectively incorrectly? If we already seen a massive skills gap, what's the say, asking the developer and operations person to suddenly take on those roles, they can learn it, but with how much rigor then they can actually have, may have their skill overlap, their original skill sets. Well, we expect a junior, a newly minted dev ops or dev sec ops engineer to provide the same level of expertise as a senior one. And with DevOps and, uh, security folks making of, uh, three sliders on their expected work, uh, coverage are all the lever levers at the same level. And you kind of plan and compensate for that also for all the new and awesome stuff that dev ops folks are asked to add such as AI ML and other bits of automation.
You can kind of assume that everybody in the leveling leveling for junior mid and senior are going to be able to grasp and explore these new demands and let them alone secure them. Uh, the last numbers from 2019, uh, in my talk, there were 716,000, uh, employing security people with 314,000 openings. Uh, the Evans data, 2020 worldwide developer population and demographic study notes that the total number of software developers globally is about 24.5 million. This graph here demonstrates a similar stratosphere growth and employment. Um, just with the kind of growth expected in the next couple of years. Now, it's nowhere in the realm of the number of people whose main source of income is retail sales, farming, or other professions, which are some of the larger employed roles. Uh, people maintain globally, but it's still a pretty high percentage. That's still 3.1% of the work, the entire global workforce, even then this encompasses career contract self-employment subsistence and other temporary work.
When you look at the workforce. So Ian Malcolm says, I'll tell you the problem with scientific power that you're using here. It did require any discipline to attain it. You read what others have done. You took the next step. You didn't earn the knowledge for yourselves, so you didn't take any responsibility for it. You stood on the shoulders of geniuses to accomplish something as fast as you could. And even before you knew you had patented packaged it and slapped it on a plastic lunchbox, and now you're selling it. You want to sell it. We're told as individuals or executives that we will resolve our problems through the application of fancy new technology, or is exploiting new advances. If you only buy this product or service, kind of, it sounds a little bit like drastic park. No, just stop it. It doesn't work that way. It never has never will.
If you're traveling full cure seller heading from town to town, or in this case company to company, industry, to industry peg pedaling, the new thing, they can make some money off of, but you're left wondering what is actually does once it's in your hands, because they didn't tell you there are extra steps that were needed or didn't even work actually in the first place. Now I know this is virtual, but I bet three quarters of you are nodding your head and cheapest agreement to that statement. The CQL film, the lost world. I'm sure you've all seen it. Seen it. Uh, also has another Gemma wisdom from doc Dr. Ian Malcolm, when they Nively wonder in the magnificence of what they've been demoed or promised when your plans go awry. And we're left with our consequences at odds, with our wonders and expectations with only folks who've been through the proverbial shit to offer the warning.
Ooh, ah, that's how it always starts. Then later it's running and screaming. So what are we supposed to do? Well, take the word from somebody that says our problems will be solved with an elixir or actually think deeply about what we're about to get ourselves in. Who would you trust? So my experiences, wow, 15 slides in I'm actually sharing, you know, IRL, uh, experiences. This talk was built nearly on a quarter century of, uh, you know, experience doing a lot of this stuff. Um, so you know, you, you can listen to me, you can ignore me whatever, but here's some nuggets of what has, and hasn't worked. I don't have all the answers let alone that I can actually give you an answer, but I'm giving you a heads up here, consider what we're doing, essentially, you know, kind of down the same path. You're looking at the problem, thinking we can just do the opposite of what we've done wrong in the past.
It needs something new though. Now I don't mean to nuke it for more, but meaning is the only way to be sure, however, the issues with, you know, that being organic and short, if you can't shoehorn a change in and you don't have the right option, is that it's going to grow to the environment, kind of finds itself in. So if you're not fostering a good environment, it's going to just be a disaster. If your culture policies and procedures, technology, and business itself, isn't set up to allow you to grow Prize-winning flowers. You're going to end up with ditch weed, quality dandelions. If you don't have an organization that is willing to change and recognize that they need to change and has the patience for it. I bet that is just going to be sadness for an overarching case. I will take my immediate past the federal government that primarily everybody sees us as antiquated with legacy systems and thinking that's been most receptive to change the monetary modernization.
Actually it came down from two directions that the actual fact that there were many systems and solutions that were just at or rapidly approaching end of life and even some well beyond that, the nature of this came from how the government tended to work with changing leadership. We're budgeting, rotating crops of contractors. And if it's not broke, don't upgrade it. Conversely, if you want to give the Obama administration some level of well-earned kudos, uh, it's introducing of the digital services playbook, which definitely leaned into agile and DevOps ideas and methodologies. Some contractors for projects were doing it already, but it was never really kind of formalized like previously with cloud first as a mantra and initiative from within government. Uh, we saw the advent of 18 F born out of the presidential innovation fellows and the us digital service, which was a joint initiative between OMB and the office of science and technology policy within the white house to approach getting UX dev ops and agile into the top levels of project initiation and management were 18 F we're more of kind of the consultative doers and initiators of such stuff.
Some agencies were very resistant to these efforts. Uh, sometimes, uh, a little warm tongue, if you're a Lord of the rings person from contractors who had invested interest and keeping the status quo, but also knew they didn't have to bench talent to keep up with sea change. So here's a brief on the several problem models I proposed in my last talk and how they were applied to DevSecOps. So the accountant problem, the commoditization, your career or job, or the value that companies and recruiters put on the skills required and the expected availability of these candidates. This is the point where the market feels that there's no need to fight for the skillset. This has happened in many tech roles, but for different reasons, but namely due to disconnect between hiring managers, HR, and understanding of required skills, the seal problem, this is also titled as the rockstar ninja problem as well.
Uh, there's been an additional shading to this, especially in the security realm, uh, over professionalization of the career space that isn't quite mature enough, uh, to realistically support it. This is often a system that's gamed or where people are more bolstered by asserts and credentials without a demonstration of applicable skills. So it's juggling and diluting the market for talent, confusing HR and managers, and kind of creating an arms race to be noticed by professionals and job hunters. So trying to stand up often elongates the hunt for roles, but awesome, uh, makes employers gunshot. If they end up having a spate of bad hires, simply only looking at credentials rather than the history of delivery and achievement beyond the paper. The astronaut problem is a bit more exacerbated in the DevSecOps world as compared to the security only world we're being perceived. An expert in a particular skill set is then burdened by which specific tool or platform you're expert in be it tools offered by Google, Amazon, Microsoft, or others.
The uniqueness is only as good as its application and in the pace of how quickly technology and any technology changes and matures to be extremely deep and not particularly broad may give you a career of reinvention or the possibility of being a one trick pony. And then the firefighter problem. This is ultimately where we all go in the best of intentions, work ourselves out of a job, either due to stress or boredom after a while. It takes a toll on us, but eventually we need something else. Some will leave, some will pivot, but often a lot of what we do is being pushed to automate things. However, as we start to do these savers, we still need people with these skills and minds to do all this. And it's a little unique. Not everybody can be an SRE, just like not everybody is a malware analyst, but not everybody wants to be those either.
Uh, but those skills are in need. And there's very few times you're going to ever be able to catch every case and give it an auto handler mission impossible. Now move fast and break shit. No on that as well. This is gap analysis, capture the desire to make change, do good and see improvements, but also know that we need to ensure that we aren't just going through the motions, burning people out and not delivering quality solutions. The way could be to look at this as a step wise maturity model of which there are some roadmaps and decide if this is prescriptive or advisory, oh, some orgs do not take well to rote checklists, but as part of Maturi having an undocumented and repeatable process that folks can follow and refer to, you know, especially in cases where stuff may go sideways as a sign of maturing and the way to take the next step up.
What makes up the right talent for you objectives? Well, do you need a generalist? Do you have highly specific problems solved? One of the ways to bridge some of these self assessment gaps is to look forward to short-term or contract work. In some cases there's plenty of guns for hire as at work, those that have specific skill sets and are being well-regarded for being good at it. Uh, but also realize there are scalpel in this monitorization surgery, and we'll be fine with moving on after their initial work is done. Just make sure things are well-documented before they're handed off and that your regular staff know how to take it from there. I know this from experience, it's one of the largest worries in exec for an executive taking on much of this transformational work is wondering if they need to find the unicorn and try to woo them on staff or build actually a solid team of folks that can handle those things and do a lot of the others.
But highly specialized work is on somebody else who may be more transient and a little bit more expensive. It's also up to leadership to be very engaged and box a lot of this work and set up expectations between you and that specialist as part of what you need to do to make sure you're balancing the work and talents is to look at where we look at, where you are and where you need to go. As mentioned before who's, uh, noses from a, you know, kind of fear of missing out fat or that it folks are scrambling in order to match, uh, another industry or sector in competition, or it's actually making things better, given resources, talent technology, and advances, and so forth. Not everything has to, or we'll move over to DevSecOps. There's never a perfect end game plus not everything fits well in a shared management responsibility model.
It's like asking the tiger cage cleaner to also be their veterinarian, uh, rarely to the skills and expertise cross over the same responsibilities in real life. For sure they can help and say, if that cage cleaner was interning, and this was part of the either hazing or part of the intern's responsibilities, but it would take some time to spin up that level of expertise where that task was easily interchangeable. This is also why there's multiple tiers and help desks and socks. The scaling and cost at certain levels do not be get everybody being a tier three grade expert or needing to be on hand for all the harder issues or events to handle. There's also some issues that we'll just hang on until they die, such as having an application portfolio that the sunsetting of an app or service isn't refactoring replatforming or rebuilding, but just setting the sunset date and letting it die with decreasing DME and support until folks are weaned off of it. It's not pretty, you can tie it up into you, can't tie it up into the boat, but you will need to analyze what work streams can be shared and which ones will and should remain on their own individual ones.
This is a jumping off point while I've been around a bit seen different companies, different sectors and types of products and services they offer the way to address this as highly individual, as a Highlander. When the Venn diagrams of doom earlier, where you start from is highly individual, both the size of your work, its maturity and the cultural structure of them, how you navigate that is up to you, your teams and leadership. The speed is also part of how the organization will adjust these bubbles and circles and their interactions. This may be both cultural, uh, having the right people in place, but also risk tolerance for the change that they may have getting you to a proper staffing and talent in each of these areas will help you reach that mark, getting your ops dev and security teams working closer and better together, but it's a long road.
It may branch out. It may change. It may just be a step into something else we know, especially in tech, that nothing is static and there will be something after DevSecOps that we'll be asked to chase. It's an imperfect model and those models will change along with society, cultural and technology, as well as our own education pipeline, our goals, and the desire for folks to want to be in tech overall market demands will offer branching opportunities, but you will get to a point once groups are resourced and align properly, or at least in a working capability and I'll make it easier to address what's next for some, it can be growth, others positioning for an M and a activity. Others looking to split functions or just plain stabilization. Oddly, the last bit never gets enough credit as sometimes just getting to a place where it's not always a fire drill. It can be an admirable thing in itself. So where do you go? It's up to you. So thank you. Be safe, stay healthy, and always be learning and exploring.