00:00:05

I've been an admirer of the work of Shannon Leeds for nearly a decade because she has helped pioneer and codify so much of the work that has become known as DevSecOps. What I've always loved about her work is how she's been continually seeking how to better bridge the world of information security and what actually needs to be secured and who needs to be doing the securing. So what is some evidence that people who matter actually value her work? In 2014, she won the coveted Scott Cook Award for innovation, for the work that she did, building the cloud security program at Intuit, which is only one of the many amazing and pioneering things she did there. So earlier this year, I was so delighted when I learned that she became VP of security at Adobe, which is the fifth largest software company by market capitalization. So with the term DevSecOps being used so widely in our industry already, you'd think that her work is almost done right. But after she shared with me what her aspirations truly are and what she's been working on right now, it made me realize just how far we need to go to truly create the ideal interaction between information security and the rest of the organization. So I was so happy that she was willing to share the work that she's doing right now with this community, and I trust that you'll find it as startling and as exciting as I did. Here's Shannon.

00:01:23

Thanks Jean. I'm really excited to be here at DevOps Enterprise and, um, been a lifelong dream. Super, super excited. I, um, am here to share a little bit about my recent research and current research. Uh, as you all may know, I have been in this industry for a long, long time, and I'm gonna share with you today a little bit about rave metrics. So let's rave and we're gonna measure software value and trust and talk a little bit about the overview of some of the work that I've been doing to pull it together. And then I have a ask for you at the end. So stay tuned.

00:02:02

So little bit about me. Like I said, I've been in this industry for a really, really long time, and that number on there really does, uh, mean a lot. You can see all the little gray hairs here. Um, excited to be at a virtual discussion because I really find these to be rewarding and, um, love to get connected to folks in this community. I've been in, in the industry so long that honestly, I started out as a dev. So as much as I'm a quote security professional, uh, I would say I've just really become a DevSecOps over the course of my career. I am very, um, involved in STEM and I have a couple of little girls, so they like to do ballet. I, uh, create comic strips. I'm into kickboxing. I started the DevSecOps stuff that's out there, and, um, have been working in pioneering in a lot of different things.

00:02:51

I work at Adobe. Uh, I'm a VP of security, as Jean had mentioned. And I'm just gonna share a little bit about what's going on at Adobe. So I recently joined. I've been here about, I don't know, a little over a year. And, um, we have some amazing products. I'm really, really, uh, excited about what we do as a software company. Over a hundred products. We have 4,500 developers plus, and, um, we have about 15 billion in annual revenue. So pretty big large company, uh, essentially investing in the software of all of our future. I remember when I first got to Adobe, again, another lifelong dream. I started my career using Adobe products. I remember like some of the first stuff I did was create a logo for a company that I created back in the day. And, uh, I really, um, am a big fan.

00:03:47

And so I'm so excited to really be here and start this journey. So what does this really mean? I do at Adobe. Um, so I am the VP of Adobe Security. Uh, I run a group that essentially is focused on two things, product security and adversary emulation, and understanding how you build resilient products. So this notion of shifting left as well as how do you really learn from adversaries for investment purposes to make it so that we can learn from those adversaries and pull together that total story of, uh, security being added into software. So maybe you've heard about DevSecOps. Um, I always ask people, have you heard of it? And, uh, what I find over the years is it went from, uh, a conversation at an ISAKA presentation. You may not know what that is, but basically it's a group out there that does information security and it's a, a practitioner's group.

00:04:52

And I remember my first slides, I actually had like a, a tortoise and a hare, and I was explaining how, uh, the security industry needed to evolve. And so as much as, um, you know, folks here are really involved in DevOps, I I believe that a lot of us are really trying to bring security into the forefront of software and make it much easier as well. So, I'm sorry about DevSecOps 'cause the name does have a whole bunch of folks in DevOps kind of worried about that. But at the same time, I'm also not sorry because I am finding that over the course of the last 10 years, software has been increasingly getting better at doing security before it gets released. And that means that we're checking our software and making sure that it's as resilient as possible. So that's my plug for security. Uh, I actually have an interesting take on what I think has been happening in the security industry and DevOps and all these things.

00:05:47

And so this talk is really gonna be focused on more about measurement and some of the things I've learned. But I'm gonna start you off with sort of the journey towards how did I get to this point with this software metrics conversation? And really, if you look back in 2014, the essence of all of this came from sitting down and realizing like we were security folks and the developers were basically moving around some of those, um, clunky, uh, you know, clipboards that we had. And so we finally sat down and wrote down really what we needed to do to change. And it was like, we need to lean in instead of say no. Um, I think at one point I got a, um, a t-shirt sent to me that said the word no on it with a period. And it was like, uh, you know, that was the brand that was being sent around was security was really all about, it had to be our way or the highway.

00:06:37

And in, in software development that that can be really a, a pain, if you will. Being a developer myself, I would say, um, I know what it's like to be sitting there trying to make something work and then being told, oh, by the way, you have to layer on all these different things to be able to make your software, uh, secure, capable, available at the same time that you're just trying to figure out how to make it work for somebody and make it useful. So in 2014, I would say, you know, we kind of put this down and it was a little bit of an experiment, and at the same time it was pretty good experiment. I think. Now what was also really interesting is in 2014 we knew we needed to get data and security science. It was actually pretty expensive back then to try and do what people are doing these days with security capabilities and security science, if you will.

00:07:23

Um, big data was pretty expensive. It's actually gotten way cheaper. It's become more commoditized over the years. And then this notion of bi business driven security scores really getting to the point where we weren't just saying, here's a list of things you have to do, thousands of things you have to do, but really getting to the point where, how can we summarize it and make it much more simple. And so I would say, you know, what came out of that 2014 realization has been many, many years, but one of the big most critical things is we've been putting all of the security capabilities into the C I C D program. We've been actually, you know, building things into the software pipeline. And at the end of the day, I think it's been measurement that's been the most critical and most necessary element of pulling people together.

00:08:09

So, um, why do I say that? And by the way, I find it to be extremely challenging to talk about security measurement and how you're gonna actually be part of the conversation of building software. So, you know, when it comes down to it, um, the reason behind this is metrics really create a little bit of that guiding light that North Star, uh, measure what matters. The John Doer book, how do we actually start to determine what we're gonna do and how we're gonna solve problems? And so at the forefront for a security professional, we're all trying to figure out how do we actually make this thing that's really important, getting rid of adversaries in our software, uh, taking away their advantage, taking away the attack surface. How do we really make that that come to the forefront? And the idea behind this is like, um, we could go focus on what we can measure.

00:09:02

Uh, you know, the idea behind metrics is really interesting and unique, and what we can measure is often not what, what we should measure. It's not necessarily what matters most, but it's got numbers associated with it, and they're actually something that's relevant. So it can help the conversation. And, and what I find is that, um, commonly those metrics are actually around value creation. So there's been a lot of amazing work out there. So don't take any of this as being pejorative towards the work that's been done out there, because it actually isn't. In fact, I'm a big fan of the work that's been done out there. I just have a different take on how I think it needs to come together to truly take us to that next level of how do we actually all work together to inspire a community of software development that I think is at the forefront of what people thought they were putting together when we, we built out things like, uh, you know, software as an industry.

00:09:54

So we have a ton of stuff out there, the DORA metrics, the quants metrics, this s r e metrics really all centered around this notion of developer productivity. And some would say, as developers, the last thing I want is for somebody to measure out how I'm doing in terms of productivity. And then others are really keen on it. Like, we really need to get to the point where we're measuring out our wellness, um, how we're actually doing what we're doing. Are we getting the help we need to be able to build software, um, more complete if you, if you will. And over the years, I've also heard that, uh, a developer has to do everything. They have to be able to make business decisions, they have to be able to make security decisions, they have to be able to make all these decisions, and they have to be able to write the software at the same time.

00:10:37

And, and maybe that context is something we need to change, is that it's not necessarily that they have to do all these things, it's that as a community, we all have to do enough software development ourselves. So this notion of maybe being, you know, security, um, uh, part of the security community, but creating simplified capabilities that could be included in software is something that is part of the paved road path of where developer, uh, developer can be productive and solve for customer problems. So, you know, a long while back, uh, I started out trying to map out what did I mean by DevSecOps and, and what did we mean as a community about what it was gonna do to fit into the process. And at the time there was a little bit of work done out there around software development, life cycles and security folks were showing up and trying to leverage their tools during whatever part of the process.

00:11:30

So, hey, I need access to GitHub to be able to look at your code and, and then come talk to you about how that might be working. And ultimately, um, value and trust was something that I think, uh, came to the forefront for me is it wasn't that we just needed to do this notion of measuring value creation, but trust. And so, you know, I I love this picture because it's really about if you have value and trust being measured, this notion of software being delightful is something that I think comes to the forefront. So I, I took this, um, picture to heart. I actually am a big fan of, uh, some of the, the products we have and, uh, one of the ones that I spend some time on is stock. I, I do tend to love to try and visualize and narrate how I'm thinking and feeling about the journey that I'm, um, funding with my research and my time.

00:12:24

And so, uh, I did learn a lot at Intuit previously where we talked about things like design for delight and having the principles of designing for delight. And really that customer understanding of software that delight that's being created is I think the driving force that we're all trying to measure. But by the way, measuring that is hard because measuring trust is hard and measuring security is hard and measuring all these things that are a little bit more, I think not necessarily part of just taking code and putting it down and putting it onto a system. And so, you know, today I'm gonna talk to you a little bit about how we actually bring those all together. And another one had that has recently inspired me has been Scott Belsky, uh, who works at Adobe. And so if you take, um, Scott Cook and Scott Dsky, I, I have definitely been inspired by both of these gentlemen to really think past the notion of what we do day to day and more about how do we actually take this as a community, a journey together, and how do we really bring this, uh, complexity of software to the forefront where we could solve human problems and simplify that for our customers.

00:13:33

So, you know, what, what does it all result into? Well, if you hunt down both of the inspirational folks and you look at the basis of their work, some of it is really about what makes someone a net promoter. I mean, ultimately, if you think about it, what products do you leverage? Which ones make you happy? What are you buying out there? Like, what products do you have sitting in front of you? Those are things that, um, if they were to go away, I think somebody mentioned to me that Superhuman said that if their stuff was to go away, they actually measure the notion of would you be disappointed? And I think it's a really good measure because it helps to understand a little bit about that trust algorithm that we're talking about here. So you know, what makes someone rave about your products to their friends, meaning what makes them put their credibility on the line and really talk about your software in terms of it's the most amazing thing.

00:14:28

It does this, it does that. And oh, by the way, it's always up, it's always online. That's how we got to the cloud in the first place is we were just unhappy with like, hey, this system isn't scaling, there's error rates associated with it. And as we all research really what the expectation is of the customer, we invent all these amazing capabilities that make it so we can get to that potential of what people dream about in terms of using software products. I think that's the most amazing clearest vision I've ever had about why software is something that speaks to my heart. And so to me, I wanted to sit down and really say, all right, what is that algorithm that makes me rave? And all of a sudden it was like I put two and two together. I'm like, well, maybe there's something here that'll make me remember all these metrics and have a harness for being able to do so.

00:15:14

So I'm gonna talk to you a little bit about, um, the four things that I think are important for this trust algorithm. And one of them is, you know, as a customer I can rely on your product because it's resilient. And so resilience is an interesting part of this journey. Um, I picked out that word very specifically. I didn't talk about availability, I didn't talk about, uh, some of the reliability secure ability as being that forefront, but for the, for the customer really having resilience software that rugged software, if you will, it's something that I think they deeply care about because it is part of that algorithm of why they come back for more. Another one is, you know, why do I wanna do things with your software? So it having the ability to say, I want to adopt your product because it helps me.

00:15:58

Um, I think software has to help you for you to want to, uh, use it and to value it. So that utilization, uh, of software is a really critical element. And then another one is, your product has high velocity feature evolution, meaning it's changing and evolving and it's taking my feedback and I'm getting more out of it over time, so you're earning my trust continuously from this part of the algorithm. And then the last one is like, your product is a low error rate. I'm not disturbed by constant interrupts. I I, when I click on something, it's not going to an error page all the time. I'm actually seeing that you put some time and effort into the experience and that your error rate is actually pretty low. So I call it rave. Uh, I bring all these measures together. I like to build upon what other people are doing, not just reinvent the wheel.

00:16:50

I coined this phrase because I think that, um, branding, bringing something together and really focusing on the most important elements of what we're all trying to do is how we're gonna get to that point where we're all talking about it in a way that makes it forefront. And, um, we're gonna solve that big problem, which is humans have better software to solve problems that we all need to solve. So, uh, what does it all mean? So rave is really resilience, adoption, velocity and errors. Now I'm not gonna predict, uh, every single metric that's gonna be at the forefront, but I am gonna try and get us to start talking about and debating which metrics matter most in the community. So using the rave framework, what I, what I'm really trying to get out there is, um, that we're bringing not only developer productivity to the forefront, but we're also adding in this notion of customer delight.

00:17:40

And so, um, lemme talk to you real quick about the resilience that's out there. You know, software's designed gracefully and um, it helps us to establish an operation with thresholds. If you think about it, uh, resilience is really about saying what you would like to achieve and then building your stuff so that it eventually achieves that outcome. I believe thresholds are critical in establishing good resilience. And the way you do that is really to look at the metrics associated with it that are out there. So are you willing to put down what your service level objectives are? Do you have availability with five nines? I think that that's a threshold. It's like what is your threshold? If your software's down 1% of the time, did you meet your threshold? It's that the risk you'd like to take for your company? I've got this notion of secur ability and many people have heard me talk about it throughout the years, and if you don't know what it is, definitely look up Secur ability, look up my name and, and go take a look at what I'm driving in, in terms of trying to make security measurement come to the forefront because I do think it's critical.

00:18:41

And there, I think five nines would be probably pretty drastic because there are things we have to do to be able to balance security against all the other resilience metrics. So I do think that you could say, Hey, is it five nines or one? In many cases, if you look across the industry, adversaries are becoming much more aggressive and holding them back and creating great software is about making sure the risk is, is dealt with in the right way. These, these are really important. If you look, um, Dora is mentioned in here, so I'm giving you an idea of metrics that you could put into your resilience area. Um, I'm not telling you which one to go chase down, but I do know that when you focus on the most critical things of the resilience problem, where are you having the biggest challenges? Then you tend to put priority where it matters most.

00:19:27

And so I would say the balancing of the ilities, we've all talked about that in the past as well. The balancing of the ilities is really about these thresholds and, and how you actually focus on the most important parts of what you're developing in software adoption. You know, software is able to be actively adopted and fully utilized. It's one of the biggest things we all want in the industry is not only am I building it, but you're using it and you're getting a lot of value out of it, is the way I like to talk about it. So some of this is coming through with things like product-led growth and I think monthly active users that MAO that's out there that's coming to the forefront, TP 99 customer set, you know, uh, product-led growth is at the forefront of really helping us to think about, um, software is only useful when it's being used and when it's being adopted.

00:20:15

And, uh, when people are delighted, they really do talk about and rave about your software. So I do think that this is part of that algorithm as well. Velocity, um, you know, software achieves the, the rate of change that matches the expectations of its community. I think when we talk about developer productivity, there's a really big sweet spot here in velocity that's helping us to understand that customers truly do want us to evolve and take their feedback and change and, and do what's necessary to continue to create value. It's not just that they adopted your software, it's that they actually will stay because you continually change. And so here you can see the space metrics, the product-led growth metrics and Dora metrics are pretty forefront. The one that you choose or the two that you choose, or the 10 that you choose. It's really up to you how you wanna, um, manage the velocity part of this algorithm for trust.

00:21:06

But I do think that it's a critical element and that's why, like I said, I'm a big fan, so it's not a, it's not an or in my mind, it's an and and in a lot of ways. Um, I actually believe that there's so many great metrics out there that can really help all of us to have the right debate and learn from each other in this space. And then finally, errors. You know, software is successful in achieving quality targets when it has a low number of introduced errors. No one wants to use buggy software. It's one of the reasons why we all have things like, um, you know, bug tracking software out there because errors actually erode the experience. And so some of the great metrics that are out there, like error rates, I love the one error budgets. In fact, there was somebody talking about security budgets at one point following the SS r e metrics that are out there.

00:21:48

You know, defect leakage is important and test case pass rates. And so you're gonna see, I think if you were to look out there that there's some amazing metrics in each one of these categories that are part of rave. Okay, so what does this really all look like in reality and why does it matter to me as a security professional and the rest of us as a security community? Well, the truth is, is that security needs to be part of the conversation and part of these critical metrics. And as you saw I said something about secur ability and resilience, well, it was through all this inspiration of understanding what was truly out there in terms of the total population of metrics that changed my mind about it just being secure ability. That was my driver. And in fact, security has to be part of this conversation in a meaningful way.

00:22:40

So if you think about it, that means curability part of resilience here, it means that we have to have our software be malware resistant. It means we have to have it adversary resistant. We need to be able to focus on things like adoption and realize that in some cases the adoption may also show adversaries that are trying to adopt products that we don't necessarily want to be attracting velocity. We really do need to make it so that developers have the tools they need, the capabilities they need built into the C I C D. And part of that conversation is we should be focusing the security professionals on the velocity that developer velocity as part of the journey as well. And finally, with error rate, um, I know a lot of security professionals spend their time on errors and defects and we call them vulnerabilities and whatnot, but truly we all care about the same things as a community, whether we're security or dev or ops.

00:23:36

And this is a unifying force. In the absence of utilizing this type of framework for building trust, I think we're really gonna be missing out for many, many decades to come. And I do think this does solve a couple of things, which is bringing security to the conversation. Without it essentially we'll still be silos, we'll still be having challenges. And I think that's where we have to change. And like I said, I'm inspired to get to this journey as a security professional because really I learned about creating customer delight and I think security is part of that conversation of creating customer delight. So for me, it's all about creating the right experience. Now Jean asked me to put this slide in and I'm really excited about it. So the help I'm looking for in this is I really want to have you help me take this rave metric survey.

00:24:26

There's gonna be some questions in there about dev and security, and I do think that that data would be at the forefront of helping us to bring this rave panel together and create trust in software. Also, if you could reach out to me if you're interested in debating with me about metrics, the good ones, the bad ones, the why, and wanna do a podcast with me, please reach out. I'm totally interested. I'm gonna be doing this in 2023 and, um, something new to look forward to. With that all said, thank you so much for your time and uh, let's get back to DevOps Enterprise, let's rave as a community and bring measurement to software value and trust.