DevSecOps - The Broken or Blurred Lines of Defense (US 2021)

A classic model for risk management and control is something called “The Three Lines of Defense (3ODL).” The three lines are as follows: Line 1: Risk Owners - Front line staff and operational management Line 2: Risk Oversight - Risk management and compliance functions Line 3: Risk Assurance - Internal audit However, with the advent of modern sociotechnical systems like Agile, Cloud Native, and Event-Driven architectures these legacy lines (3ODL) are at best blurred and at worst completely broken. With the modern patterns and practices of DevOps and DevSecOps it’s not clear who the front line owners are anymore. Risk management and organizational compliance teams struggle to adapt to new cloud-native models such as ephemeral containers, functions, microservices, and event-driven architectures. Most organizations' internal audit processes today are highly toil based and have very low efficacy. This is something I have called in previous presentations “Security and Compliance Theater.” In this presentation, we are going to look at a couple of case studies that include the good, the bad, and the ugly when it comes to 3ODL. Primary topics covered will be organizational design, DevSecOps, and Automated Governance.

vegaslas vegas2021breakoutus

(No slides available)


John Willis

Senior Director, Global Transformation Office, Red Hat



Hello everybody. I'm John Willis. I'm a senior director of global transformation office at red hat. Um, this presentation is called security differently. So, and you know, my, I gotta put my shameless plug in. I started a podcast this year on, you know, really, I mean, had a passion for Dr. Deming for years. So I've been doing these like amazing podcasts for me, where I talked to people in our community and outside of our community. So I just did one with somebody who, uh, who is the history of autonomous vehicles that goes all the way back to like, um, you know, general motives and, um, and motor sports. So like, it's like really cool. It'll be up in a couple of weeks anyway, check it out. So security differently. Um, we usually what we're talking about here, right? Like this is the topic of this presentation. I started thinking about, you know, like I I've been involved, obviously I was involved in sort of early days of the dev ops movement.


Right. Um, and then I would say, give, or take five years, probably a little longer than that. The sort of DevSecOps things started sort of populating. I was very early on that I helped, uh, Sonatype build the first DevSecOps stays modeled after dev ops days. And so I got sort of thrown into, um, the, sort of the drone in the lake early on. Right. And, uh, and, um, and so, uh, but, uh, there was a point of which, and probably sometime last year where I started thinking about, um, if we just talk about their sec ops, the horizontal breadth of security is pretty long long and, you know, are we accomplishing a whole lot when we sort of broad stroke ish talk about there? And for me personally, I felt like I wasn't really being intellectually honest with myself. And so I started thinking about like, this is a very meta sort of question I pose to myself and now I'm posing it to you, which is what would dev sec ops look like if dev ops never existed.


Right. So let me be clear. Dev ops has been phenomenal. DevSecOps have been, we accomplished quite a bit, a lot automating a lot of the security things in the pipelines, but I wondering if that, that even with all that great sort of advancement and an opportunity, is it a scenario where we've been trying to force a square peg into a round hole? Right? Like what if like for some sort of strains, uh, historical fiction version where the innovation that dev ops was started as a security innovation, would we have made all the same choices as good as the choices were? And so I use this example, and this is the one that probably got me thinking about the most, which was, you know, I go to visit these companies and, you know, it would start in the premise of John. You want to take a look at our DevSecOps reference architecture and we look at it and, you know, you'd see sorta really good stuff.


You'd see they've automated things like SAS and DAS, or they've got, uh, you know, software composition analysis built into the pipeline. It's just really, really good stuff. Right. And in a deeper conversation, we talk about audit and I'd find that there was a, you know, in my mind, is outside looking in, um, as a sort of a disconnect, we're just able to say, you know, yeah, we were using it proudly. They would say using the three lines of defense model. Right. And for me it was an outsider looking in, I'd say, well, isn't that sort of three silos, you know, in fact, you know, where the, sort of the, the, the sort of virtual firewalls have been designed to break out the silos, like the third line never talks. First line, second line is a translator, which by the way, in most organizations does a terrible job.


Um, you know, the, the, you know, why isn't the third line sort of showing up zine requirements, right. Uh, it's also got this, like if the first line doesn't catch it, second line catches. I mean, it sorta immediately reminded me of the original character of dev ops, which was, you know, Andrew Clay Shafer, who I work with now, you know, did this in, um, 2009, uh, Riley conference. So the same Congress where John Haas bar and, uh, PR did that sort of like, you know, the sort of landmark 10 deploys a day quicker, that same day Andrew gave her a presentation called agile infrastructure. And what's interesting is that, um, you know, Andrew's, wasn't recorded. So it didn't become a manifest. Now they were both great presentations, but in that presentation and Andrew was the first one to sort of create this observation of this wall of confusion, which was a beautiful, um, sort of caricature of the problem.


You know, I want to change this to operations, say, you know, that he can't do anything, but it's sort of throwing it back and forth all operations and, and, you know, so, and if I think about that three lines of defense, I also think about Conway's Laura, not in the perspective of sort of, um, monolith to microservices, but really just at the time, you know, the, the Melvin Conway, the adage stating that organizational design systems mirror their communication sources. So these are, these are not just sort of technical how you program, you know, I think that was probably Cottonwood or Melvin comedy's original tent, but I think it sort of expands into your organizational design. And so then I look at three lines of innovation, I guess the question, and again, this isn't a popular question to ask in a large bank. And like, I, I feel like I'm going to get chased out with picks and, you know, and, and swords and sticks and whatever, but, um, you know, are we just creating, you know, like two lines, you know, two walls of confusion by design I've we like all of the things that we've accomplished in dev ops.


And now I would even say dev sec ops, if you're sort of adhering to this model, and now there's sort of a two line version from the Institute of eternal, but the point is it's by design siloed. And I sort of think, well, that's what we saw. There were a 10-year period with dev ops and isn't this, our square peg in a round hole. Right. So, um, and you know, over the years, uh, I, I tried to explain how I, so after I worked for Docker, I saw the coming Docker, and then I left and put my own shingle out where I, I started doing this thing where I call it qualitative data analysis for transformation, just fancy nonsense, but the point being there, qualitative data analysis, wasn't nonsense where I started, you know, part of that is a sort of a classification of, uh, of sort of, you know, different sort of, I would say patterns.


And, and I came up with these seven, I call them seven deadly sins because it sounds cool. But, um, ultimately these are the patterns that you saw everywhere and I've gone full presentations. But the one thing that sort of the unattended thing that I didn't realize at the time is that they all sort of magically fell to this seventh deadly sin, which was security and compliance theater. And, uh, you know, and, and, and what I found was like, even in most advanced dev ops dev sec ops organizations, I would sit in front of a CIO and that's who I do the final report to, you know, some companies I did a second largest banker interviewed 400 people over a whole summer, and you have to tell them, look, it's just a dangerous proposition to sit across like six feet away from a, from a CIO, with a fancy wooden desk and say, you know, by the way, you're, you know, sorta your security audit posture is just sort of theater and, you know, and you break it down into sort of cloud native world.


And there's just no connection between what the auditors are asking you to do and what you are saving, any sort of evidence related to that. And I'll, I'll break that down a little bit further. Um, the, you know, if we look at, um, how we do things today and how probably we should do a security differently, it's, there's really three questions, right? How do we prove that we're safe? How do we demonstrate it was safe? And, and they're not the same thing. And then, um, how do we do both, right? Like, which is like, we have to dwell, we can't just do one. And I think, you know, sort of, I use this phrase, PostNet cloud native modernization, our post cloud native world. Right. And, and I, I keep saying until somebody punches me in the nose for saying it I'll continue to use it.


I haven't been punched in those virtually physically yet, so I'm going to keep using it. But to me, it's a line in the sand of like, the world has changed. And so in a, in a non post cloud, native world pre continental world, right? Like how do we prove a safe, like in general, we use service. Now you change records. Bob tells Sue who tells Sally, who sells Sam. And Sam says, you can't go into production unless you do this. And, and we, we create this sort of telephone game of abstract, subjective knowledge related to incredibly complex operating environments and systems. And then how do we demonstrate we want it right? And that's the 30 days of 40 days with just clear toil and email exchanges and screen prints, and sort of long discussions about the third line, not understanding the first line or the second line, not being able to weigh in the difference between a Sonic cube, uh, and an excess log and why they appear, they do the same things.


They really have different intentions, like, so, so, um, so what, what I think security differently is about is, you know, how do you move from implicit security models to explicit proof models, more specifically, how do we change subjective to objective? Like how do we change sort of people creating narrative about the change in the success of the change, maybe having links to some logs, or maybe having a check box saying we did these six things, but in general, it's a completely disconnected model. And I would argue that that's what creates most of the low efficacy and high toil of most audits to a model where it's objective, it's digitally signed evidence that is generated by a machine. There's no human involved in that evidence. And by the way, that by itself is not good enough, even though that sounds like, you know, um, uh, sort of like fiction, right.


But I do know a couple of banks that are doing this right now. Um, that's not good enough because we know that complexity sort of demands drift. And so we have to actually constantly verification that the things that we said in the act gestational data or the configurations that drove the actor's digital data are actually accurate. Is it accurate one hour, one day, one week, one month? So this is sort of a chaos injection in a security mindset that is more than just chaos engineering, not just spiking CPU, or it is about sort of if, um, a configuration manifest or sort of configuration document that gets through the pipeline should never have this port open, then let's open up the port in live production, that kind of thing. So, as I said earlier, when I was thinking about like, how do I get my head around sort of meaningful knowledge or sort of, I consider myself more than anything else, a student of this industry, if I'm just going to talk about post cloud native things from a security perspective to have sex with, like, I don't want to talk about like the sort of horizontal, um, like all the things in security.


And I started like in conversations and working groups, I, it sort of boiled down to, I think there's really three things. I'm not saying everything else. Isn't important. I'm just saying that I can't think of anything that is more important than these three things right now. Um, and so we took it risk defense and trust. And if I use the sort of grid of subjective to objective verifiable, so we look at risk and say, you know, today, most organizations are subjective. You would, the, sort of the root evidence of our, um, of our audit is the change record. It's, it's subjective. How do we get that to sort of an engine and automation engine we've been calling it DevOps, automated governance, but a system that basically creates digitally signed evidence. I like to say, think blockchain don't use blockchain, um, of attestations and controls, and then like using some Toros continuous verification or sort of security chaos to try to make sure that like, we're, you know, even though we know that like, theoretically should have got in the system, um, looking this way or having these particular, um, artifacts or configurations, but let's just make sure that they're not right.


And then, uh, defense, right. We moved from like the classic detective respond, or even in some cases, very, um, sophisticated remediation systems to, everybody's trying to build the cyber data lake. Right. And, and that's great, but I'll tell you later on the project I'm working on where, uh, we were calling it sorta the decorator model where like, just throwing everything from a security event or security orchestration into a sort of elastic search or some, some, um, uh, cyber data lake, if you will, the, the, like you still got a lot of work to do if you haven't sorta worked in the front end. So it's sort of a shift left of the context and the meta. So we'll talk more about that. And then, you know, one of my sort of favorite mentors, brilliant woman, she spoke with DAS many times, um, Shannon Leitz, um, she talks about adversary analysis.


So like, like that's beyond the sort of objective and the correlated cyber data lake data, which is important table stakes, I think in, you know, in, in sort of post-planning world, but she talks about adversary retention rate analysis of adversaries. Like, this is the verify, like I can say I'm doing all this great stuff on a defense preferred perspective, but the true test is how do I know what adversaries are coming when they're coming? How long they're staying are the things that I'm doing that deter them from coming and staying. And then last but not least probably wouldn't have too much time to talk about this in our, I'm trying to really focus on risk in this one, but I wanted to make sure that I sort of express my ideas about, um, about these three, um, the sort of primitives is a trust. And I'll say this, I'll say that, um, if you think about what software defined networking did for, um, uh, for networking, right, it really was, you know, at the highest level, right?


The veteran more complex than this, but, um, it was a conversation about sort of thinking, changing the mindset from north south network transactions to east west, like the world had changed where, you know, you had multiple cross rack communication. The, the concept of thinking about traffic as east west was a lot about sort of the, you know, there are a lot of other permanents that sort of created the SDN conversation. I would say that I think we should be on a beginning conversation about thinking about trust the same way. And there's a couple of technologies like this. So moving from a north south trust, like you're authenticated in your, in to sort of a femoral Costa based selective node to node constructs. That basically when you're in that cluster or you're in that pod, like you're, you're automatically authenticated and there's technologies that we can do that very well.


So we start moving and that's, that includes also secrets management. Like, so moving away from, uh, no, you know, I think vault is a great product, but like, and I'm sure many would argue, but to me it is sort of north, south dish still. Um, and it's the best game. It's the only game in town right now, but, but like imagine secrets management sort of falling into that east-west pattern very much like, like authentication. So, um, you know, three novel ideas basically, um, devastated many governance. We'll talk about that. Uh, pragmatic cyber day lakes. This is a project I'm doing with an organization called oh, nog, and then, uh, really haven't done anything significant. I do think there's some certain technologies that should be explored for distributed trust, uh, models for identity and secrets, which is that sort of third, a trust model. So risk, um, what are we trying to do?


We're trying to reduce auditorium increase audit. So it's not good enough to say we're doing all these things in DevSecOps reference architectures. The real question is what is the efficacy of your audit? And like the honest efficacy, not the sort of, I I've had situations where I go into a CEO's office and I, you know, I tell them the reality of what's going on and they get furious on the audit. Like, they'll accept all the other things when the seven deadly sins, but on the audit electronic, I'm going to have to sort of disagree with you here. We want to ward, you know, Ernest and young and Oregon aerosol anybody, right. It's not nothing wrong with it was young, but like one of the big three or five or whatever they call them today, um, has said that where you are like the sort of best in finance, right.


In our audit, you know, and like, yeah, sure. But like I just talked to a team that is basically doing about 300 deploys a day, all on Amazon to dynamo DB, um, using, uh, femoral pipeline as code sort of infrastructure. Um, you know, it's like, I will tell you right now, there is zero evidence that relates to your audit and that's not even getting into sort of Kubernetes containers, OpenShift, and let's see, you can go one step further serverless and functions. Right. So, so, and, and we waste a lot of time. The more technology that we expand on the more of these sort of long winded sort of conversations we have at on a time, in fact, a common adage that I hear and I use this back to the CIO is I can tell you that what's wrong starts with, I don't know how many times your organization I heard, we don't tell orders things they don't really know.


In other words, we are not, we don't care about protecting the brand GRC, just, you know, that's just the annoying thing. I mean, if, if we're taking the brand, isn't really the most important thing and a fortune whatever level, I don't know what is right. And, and we flippantly just in most organizations, it's sort of, you know, don't, you know, we'll tell them about Kubernetes next year. So efficacy and control. Um, there's been a number of publications over the years, um, for mighty revolution. Um, really good stuff. Like in 2015, there was a paper on an unlikely union dev ops. And it's really saying how dev ops could actually be the solution for, um, segregation of duties, separation of duties, right. Um, aside, if you will, um, in 2018, a really sort of funny tongue in cheek, a palsy letter to auditors, these are all creative comments on an it revolution website.


And, but it wasn't just the sort of two page apology letter. It was another 30 pages of control rags and promises that we're going to do this, this and this in 2019, I worked with, uh, PNC capital one Marriott saber group. Um, and we, we tried to put this, um, objective referenceable architecture on paper. And it really, I, you know, it was sort of, I'm very proud of the work we did there. Um, and like, again, I won't go into gory detail, you can download this. It is the formation for, so everything I've been working on in the last two years with a couple of large companies, but you, you sort of look at it as, um, the seven stages and, you know, and w we're trying to do is create boundary points for, um, control your gate, gaining control and at the station, all data evidence.


And, and so not to go too far into this, but, um, you know, over the over time, you know, the original book, the original publication 2019, and it's probably 75 out to stations, which well-documented like where it works, where it might come from and why it's important, uh, related to some control rag. Um, but if you look at some of the stuff that you know, that people are doing with this sort of model is, you know, percentage test coverage, change, size, psychometric complexity, was there a review on the pull request, um, you know, sort of optimum branch strategy, right? Like these are things that also be gated and then also, uh, created into an immutable evidence list for that deployment. So imagine at the end of the day, um, zero days in audit, because it's just an immutable digital, it's a digital signature stepson, all the digital signatures of the things that have happened in that deployment by the computer, not by humans.


And, um, it can't be changed. Like it's not broken. Like it looked at the SOTA again, think blockchain, not use block time, but imagine sort of the, the evidence is a digitally signed event or a signature. And like, there's no debate. Like it's there, like, you know, shoe sorta dispute the engine that used it. This is so true. Not just, um, jump out a little bit in the early days of chef, I was there really early before we even had like beta, we had alpha, we had no sort of enterprise and a real large financial institution came and, and sorta visit us. And we were like, oh my goodness, why, you know, we didn't sort of ask them, like, why are you here? Like, it was one of the top investment firms in the world. Right. And, uh, they said, they felt that chef would be useful, you know, for certain things like PSI DSS and things like that.


And because if they could prove, like, if, if the assumption was the engine was certifiable that built this thing, and this was an automated engine to do it, it would save a lot of audit time to have to look at the instructions or notes of how people built servers. They would accept the fact that the chef server is secure and that what it builds is, you know, sort of the evidence itself, like it just sort of elevated everything. So it's really an expansion on that. Uh, as you can see in the build stage, you might have LinkedIn and SAS and sort of just other sort of things at the package stage, it gets a little more interesting. You have artifact versioning, a package, metadata code signing. This is interesting. Right. I'll tell you a story later about something I did with solar winds and, um, and you know, where, like a lot of times, if you sort of have a reasonably mature implementation of what we would call dev ops automated governance, um, co-signing would be table stakes.


Right. And so we'll look at that container image scanning sort of scanning, uh, pre-prod prod controls. Right. Um, and then, you know, I think over time there was sort of this maturity from the first paper of, you know, how do we start to get that third line of first-line the work together? Um, we've got to give them a collaborative way to create an artifact. So it's not good enough to just send risk to design and requirements. We should still have a communication toil, right? They don't speak the same languages. And even if you have the sort of second line or whatever, um, there might be a lot of, sort of, um, agree to not disagree, passive agree to not disagree stuff, but w where you, you elevate, um, the conversation where you have to agree on a sort of a construct or sort of a code there's an early days, people were like, there's no way rescue is going to like work with Yamhill.


It's not true. Yamhill actually is sort of universal for anybody. Really. Honestly, it seems complicated, but, and then like you could actually then have that artifact be a collaboration between risk where like the, the, sort of how this is going to show evidence, audit the both teams created together. And then there's very little confusion of why this evidence is sort of proof as opposed to a screen print or an email conversation, you know, and then, uh, at the beginning of this year, um, we decided to do a second version of the night, 2019, uh, automated governance. And, um, and we were going to put out his paper, but we decided to create, we figured that one of the things that we felt that, um, that the original paper was very technical and probably a lot of people after the third page, just like, you know, I don't have time for this.


So we decided to do it as sort of a fictional story about a bank that fills on audit and then sort of goes through all this sort of process of implementing a debit, sort of made it governance. Um, it was so well received by the it revolution and crew that, uh, you started just to create, uh, a full book out of it. So sometime next year, uh, another book based on the, uh, the sort of goal rat sorta, uh, Phoenix project style, you know, um, you know, sort of, um, you know, sort of Eric and bill, um, uh, sort of Alex and Jonah from the goal, right? Like we've got that sort of contract. So, uh, it's, it's gonna be fun. It'll be, so it was like eight of us on it now, so, but that's good. It's eight authors. I told you earlier, I did some work where I was invited into solar winds to talk to the executives about automakers.


So there was a sort of, uh, one of the large, um, uh, consulting organizations, um, like somebody worked there knew I was sort of an expert on this subject. So I went in and I talked to the executives of, um, why I thought army governance would help solar winds. So I went out and took the, sort of all the public information about sort of that sort of attack or that, um, kill chain and, and CrowdStrike was, was by far the best. And I use that to sort of present, Hey, these are the things that CrowdStrike has publicly sort of dressed that happened. And if you don't know, like they were able to sit in there for almost 18 months, sit on Ms. Build eventually hijack it and then sort of create their sort of affairs code that went out to everybody, including probably you who are listening.


Um, and, and what I was trying to say is like, there would have been a lot of red flags if you had automated governance first off, um, you know, like a model, you know, something we've written at red hat called apply ghost, but there's in Toto. There's a couple of where, like you could use, um, uh, sort of a pipeline as code. We thought about infrastructure, all pipeline is code looks just a femoral it's changing and sort of repaving constantly. So if you're sitting there trying to understand your Jenkins files and all this, and they're static for six or eight months or a year, it's only the game when there's an engine that uses abstractions to create the stuff like, uh, most likely the adversaries are going to move on to somebody else. Uh, there was a lot of LARC masquerading, um, like, uh, immutable at the station store.


You know, we use something that's built again, there are other solutions, but right out uses something called six store, which creates a Merkle tree it's immutable infrastructure. So like it can not be changed. But the biggest one that was glaring to me was in the CrowdStrike analysis, this is the MITRE attack framework. So I used the MITRE attack framework as the sort of structure for this was code signings then, which has mismatched code signings all over the logs. Right? And these things are done that would have been table stakes for a base implementation or what it means to governance. Um, so I took about some of the tools, and again, this is sort of a quicker presentation, certainly if you want to learn more, absolutely. You know, ping, but we've done some internal work. Um, this bill Benson was wondering two authors, a new paper.


He wrote an infrastructure system for, uh, the government called dead sword. And he ultimately this turned into something called it's. I mean, I can do a whole president and there are presentations on it about how it fits really well. It's a great interface definition for how you want to build your pipelines, but it doesn't have an opinion on the implementation. So it's the perfect tool for automated governance. Um, six stars, another one that I, I I'm spending a lot of time thinking about, um, certainly red hat has, I mean, this isn't a product presentation, but we do have some strength in our open shifts. So our Coumadin version has these, you know, uh, really, um, intense compliance operator, uh, events, cluster management, uh, Vance cluster security, which is basically a StackRox, which is sort of a Twistlock or a coset solution. So, um, um, just winding down, that's a quick view of sort of, uh, risk differently in sort of the security differently, um, story, the, um, defense differently.


Uh, here, I talked a little bit about this about, but again, it all goes back to reducing toil and reduce increasing efficacy. So like all the stuff that we're doing today to react sort of our Sims and sores, um, what if we took a mindset of a, sort of a shift left on our data center, um, you know, our sort of cyber data lake constructs. And so I've been working with a group, uh, called Onaga it's one of the largest, um, networking user groups. They were involved in SD wan SD SDN was based out in New York primarily, but they are like the biggest large cap banks and financial institutions in healthcare, all be involved in, in fact, our board of directors. And I was invited in there, um, to sort of look at this sort of democratic government from a cloud perspective. And this was our first version.


This is creative commons as well. But our second version, we really focused in on this problem of sort of, I would call it a shift left on sorta cyber data lake. Right. And it was two problems. One is, um, which was our original problem. So we interviewed like, I dunno, about 30 hot, large caps. And when you started trying to figure out what, what the, uh, and most of the, um, cloud security operations, where people had to deal with clusters, especially multicloud, right? The thing we heard commonly was it was, it is so hard to, um, understand the common context of an event that comes from Amazon versus Google versus your, that actually mean the same thing, but in the way they classify it, the way it's sort of even the sort of text of the event, uh, it there's a lot of toil and figure, oh, you know, those are the two the same thing, but the bigger problem that we discovered was just, um, normalization of that.


So that's a normalization of context problem that we found that normalization of metadata was the bigger problem. So when you interviewed these companies, it was just, I mean, I want to use the word comical, but how they got their metadata and where they combine their metadata. There was sorta, we went to Prisma, we went to F five, we went to spreadsheets. We went to, and, and even then it was sort of difficult and in a multi-cloud environment, like you can already start at the beginning of the sort of chain. If you look at like probably, you know, five of the most well-known Mehta fields from a cloud account resource event, name, event type, basically. Um, if you look at how is your or Amazon, or, or Google, or IBM or Oracle, they all use different sort of tags. So if you just dump all that all the way to the right, into a cyber data lake, and you don't sort of work on the, sort of the, the context normalization, if you got Mehta normalization, like, yes, like these advanced sorta data lake tools, like they can do the correlation, but like your, your percentage of accuracy is going to be lower.


And the amount of time to sort of create is going to be higher. Why don't we sort of work on like, can we do context normalization earlier? So we created this idea called a decorator. Um, there's a, um, checkout owners look at this sort of automated cloud project. And, um, it's a really cool to automate open source. Um, and so the idea was to create these sort of decorators. And today we it's for the last, uh, this year we've had Microsoft, Google, IBM and Oracle working on this project. We're still trying to recruit Amazon. So like, the idea is like, and some large like Cigna and some of the large cap banks. Um, so one of the things we're producing pretty soon as I don't source repository within SDK, that actually start the conversations, very exciting stuff. And then finally trust differently. Um, the, um, you know, if we look at that sort of third primitive that I discussed earlier, you know, um, certainly zero trust architecture is a table stakes, uh, th this conversation of most modern least privilege.


Absolutely. Um, all that stuff, you know, we need to sorta think about secrets management. Like I, you know, I think vault today is a, is an amazing, like sort of almost must have product. Um, but th but the thing I'm sort of like, I'm always sort of looking at the future of like, what are the problems that we have to solve? You know, one of the things I, when I interviewed at red hat, you know, I was asked by Jim white Harris, I asked him, so I asked him, why do you want to hire me? And he said, you know, he said, you, you, Kevin, Andrew and gang have, um, been involved deeply in the conversation of what created the first 10 years of dev ops. You know, he said, John, we point to your books, you know, we do this. And he said, well, we want you here for the next 10 years and help red hat define.


And that that's sort of, that was the sell, right? Like, you know, great sorta, uh, leaders are great sellers. Like, you know, I didn't see myself working for a big company, even with Andrew, Kevin, and Jay, but, but they, at the end of the day, um, that was like, okay, you got me. Yes. I mean, um, but you know, I used to use this sort of joke about like in 20, 25, if the, um, if the hot topic is still get ops, then we failed miserably. Right. So I'm always, I, I, you know, I just, can't, I'm not saying I'm special. I just hate status quo. I've always had it in everything I do. So I think this east west, you know, this idea that we are now working, most of our workloads are moving into cluster pods, you know, sort of a femoral, um, I would say Lamar atomic type compute structures, things like container images running as containers and, and functions.


And, you know, a venture fund architectures is a big conversation. I think we see a lot of that sort of bounded by the service mesh structure today. Most of it is it's still ongoing, but it's not the only tools. Um, and so in that world, uh, you know, there's some interesting conversations about sort of, I would call east, I've sort of made this up, but like east west sort of trust models that, that, you know, sort of, if you walk through this, there's sort of node to notice a cluster of nodes might represent a particular trust domain. And, you know, and again, because you have, uh, things like Istio a service mesh traffic management, there's a lot of clever stuff that we probably should be taking advantage. And so in the end, you know, trust, I think, you know, the age old adage, like the, you know, dev ops started with a conversation about, can you throw the server out the seventh floor window and still sort of operate, right?


Like, and, and, you know, I, I think the notion of like, there's nothing new about repave and rotate, right. But it's, but it's an important discussion in security, right? Like if we, like, if the adversaries have 18 months or a couple of years, like in the Marriott center, I think it was four years, right. To sit and learn, like they're gonna get ya. Right. But if we're sort of like dynamically, or even at different sorta like throwing them off guard by sort of repaving and rotating, um, certainly the zero trust architectures and then the emerging tech, you know, I mean, I'm a big fan of spiffy Spire. I work with insect, Andres vaguer. Who's one of the committers on our project is another off of their, of this investments on limited. Um, you know, I, I think this is some promising ground for, this is what I've call east west trust.


And then you're the sixth store is really interesting. It was, it's a red hat developed crud for, uh, uh, project in Google called trillion when truly was about, uh, certificate transparency. But it has this beautiful sort of backend Merkle tree structure that can be used for at gestational store. So, and I also think six square has the possibility properties to do east-west trust models as well, but very sort of early in that conversation. Um, anyway, um, I think, um, that's about it for me. Uh, thank you so much on J Willis at red hat, or always can be founded We're about to go live on Twitter, just loop.