Open Source and Online DevOps Dojo

Skilling and re-skilling for DevOps technology AND culture is a real challenge for the software reliant industry. We can't hire fast enough and chasing talents is time consuming and expensive.


Introducing the "Online DevOps Dojo", an Open Source project to learn and practice DevOps patterns, as made popular by the "Accelerate" book and research by Nicole Forsgren, Jez Humble and Gene Kim, from the comfort of your web browser.


In this 30-minute breakout session, we will do a walkthrough of one of the training modules of the Online DevOps Dojo: "shifting security left". If you are a developer, you will get directly reusable content. If you are a leader, you will understand the concepts, see them implemented, and get the resources to create a similar program for your teams.


After a brief history of our use of DevOps Dojos, why we moved to both a face-to-face and online model, we will spend time on the story and the characters (also Open Source) of the series.


Shifting security left is a big topic: as we walk you through the module, we will get to the why and how to implement proactive security countermeasures as part of the CI and also CD pipeline, as code.


The best is that you will be able to follow along, share the Online DevOps Dojos with your colleagues, and why not use them for your own DevOps journey.


One last thing: the learning modules are open source: we invite the DevOps community to not only use, but also contribute enhancements and new modules!

OJ

Olivier Jacques

Distinguished Technologist, DevOps, DXC Technology

CS

Chris Swan

Fellow, VP, CTO Global Delivery, DXC Technology

Transcript

00:00:07

I and welcome to our breakout on the DXE online DevOps dojo. Uh, I'm Chris I'm CTO for delivery at DXC. Uh, and so for those of you that know the equity, uh, it's the combination of, uh, service companies, uh, from HPS enterprise services and CSC, uh, that came together a little over three years ago. Now some of you might have already seen Olivia, uh, present in the path there. Olivia, do you want me to talk about what you've done before and kind of, what's different about what we're showing this time around?

00:00:41

Sure. Thank you, Chris. Yes, I'm alleviated, I'm working as a distinguished technologist at DXE and I'm a company, our customers, uh, on their DevOps transformation. So, uh, back in 2015 where, uh, Dixie was not there yet, uh, we were invited, uh, to share their journey, uh, with HP it at the time. And then I attended a talk by Ross content and, uh, uh, from targets at the time, uh, on either Mickman I believe, uh, on, uh, their DevOps dojo. And, uh, this really clicked with me as, as a way to articulate the DevOps, DevOps transformation. So really something that, uh, became that hit home, I believe. And, uh, and then we, we came up with this, um, online DevOps dojo, which, uh, as we needed to scale, um, uh, I'll DevOps, skilling and reskilling, uh, within a company and for customers, um, we turned, or we complimented all DevOps dojo, which was really a face-to-face format and still is with online trainings, uh, to support this, uh, transformation at scale.

00:02:01

Yeah. And yeah, from my perspective, there's work I'd done in that kind of year before we became DXE, uh, where we developed an infrastructure as code workshop that had reached, uh, about a thousand people by the end of the year. But then I came across, uh, the Calacatta, uh, platform and realized that we could really kind of make this thing web-scale, uh, and take it to a much larger audience. Uh, you know, we've got over a hundred thousand people in the, in the company, uh, in delivery that kind of needs to be re-skilling towards dev ops and that's what it was intended for. And subsequently, you know, what we spent the last few years working on is, uh, you know, some more great content that Olivia is with the team was put together. Uh, but also, uh, now there's open source release, which means that it's available to, uh, to anybody that wants to access it. And also because it's open source, they can build from, uh, the material that's been put together here. So Olivia, do you want to jump in?

00:02:58

Yes. Well, this is really what's new, right? So I think, uh, in the later years we were again invited, um, to, to share about our DevOps journey. And we shared about our own version of the DevOps dojo, what we have done with it. And, uh, we, we came to a realization that open sourcing it and making it available, uh, to the rest, uh, to the rest of us to do to the rest of the industry was probably something that would be best for us to do. So it is, uh, this is the online DevOps dojo, which not, is not meant to be a full DevOps or job, but covers, uh, parts of the, uh, the training and the training aspects of it, at least. So it's open source. You can see the URL and we'll post it also in the, in the video. Uh, but you go to github.com and you look for the online DevOps dojo, and you will see this repository.

00:03:48

Um, the, the intent of this DevOps, the Jewish area that's in the audio presentations by summit is really to, to guide and to go through a set of modules and a set of trainings and learnings, uh, that, uh, that our people and our customers have to go through to understand the DevOps practices. Um, so we start with a story, uh, and we believe that storytelling is extremely powerful. Um, so we have, uh, our team, which, which is the clinic team with the team as, uh, Charlie, who is a CEO of the pet clinic company. It's a, because its own story. And you will discover later in the training modules that as dogs as well, uh, churn is a DevOps coach, a change facilitator. We have Paolo or product owner. Uh, we have, uh, Adam with a CSUN, meaning we have done the developer Tina with our tester, but the intent of, of this putting together a story is to bring the knowledge home, right? So often the learnings have to come from where you are and what you know, to something new, something different. And that's what the characters are here for. It's, they're here to tell you, well, I'm Tina, I'm doing testing. So that's who I am. Let me see what is changing for me and what I need to do differently.

00:05:09

And this is a cast of characters. That's very much inspired by the it revolution books like Phoenix project and unicorn project. Uh, and also the other pack clinic kind of comes from the app that we're building the continuous delivery pipelines around, uh, which is the spring pack clinic sample app, uh, which really kind of gives us a good foundation for doing all kinds of things with an application.

00:05:33

Exactly. So the pet clinic is really, uh, an interesting, uh, I guess, showcase now I'm talking about the modules that we have released in, uh, in open-source right? So, um, the modules that we're having to go through one, because it's kind of a different presentation. It's not going to be about slides. We don't have slides. Uh, but, um, but we're having to do a kind of a live demo of, of the modules just to have a feel of, of what it is and how it works. And again, you can follow along because this, this is open source, so you can just go to the site and follow them and go through the modules at the same time as we are. Um, the modules that are there are, it's just, uh, an abstract of a set of modules that we have available and ready and to use, uh, for us, uh, they are coming from the acts from practices and patterns that have been described in the accelerate book, uh, because we believe that this study, uh, from, uh, uh, from, uh, Dora initially, and which moved, um, actually is really interesting to connect the, uh, patterns with business outcomes.

00:06:42

And that's what we want to, that we want to, to explain and, and, and share. So we have five more years. The first one is the welcome and set up, which I already went through, uh, leading change, which is really more about the control module on how to change and how to lead a DevOps transformation. So we have both cultural and more technical module, one module about veteran control and how to do that fully about version control. It's about version control, peer reviews, and, uh, and, uh, and integrating then the continuous integration together with version control and the model of the truck and to go through, uh, today is a shifting security left, uh, which is a very interesting module as to explaining what developers can do to embed more security practices as they do their development and deployment activities.

00:07:40

So here we are just clicked on that. So what you can see is that I did not have to install anything. Uh, we are actually totally in the web browser, and this is working outside of the box without any type of additional activity. And, um, this, uh, welcomes me with a worker message explaining what the goal of this module is about. So I told you it's about shifting security to the left, and I'm going to stop the scenario. So scenario starts with a story, as I said, so, Hey, oh, wow. Uh, we are making the team is the picnic team, actually Sputnik team is making the frontline news, uh, with owner's information leaked from the pet clinic. So this is kind of a panic mode, uh, in the clinic team. Uh, we have been hacked and, uh, we, uh, our characters, how, uh, who is the Acura there? Uh, we'll explain a little bit how the, the act, uh, the database with SQL injection and was able to dump the entire table with all of the pet owners hopelessly, um, not very happy about it. So I won't go through all of the, the story and the drama that, that builds there. But bottom line is that this is a high visibility issue of your sleep, and this is something that we need to, to resolve

00:09:03

Now, continuing of the module. Um, the first thing that we need to do is set up your environment. So I'm here with my own environment, uh, which is there. It's, uh, it's a, it's a window that's available. Um, now I'm green to, I'm going to set up the environment. So the first thing that I'm asked to provide is a personal access token. Um, the wall lab here that has been created in front of our eyes takes advantage of GitHub. So, uh, we are forked, um, the, a copy of the, of the fit clinic project with the picnic environment, for the team. It's now on my own GitHub profile. And what's happening right now is setting up the environment for the rest of the labs.

00:10:01

And do people have to stick with the script here or come? Now you gotta kind of try out things and explore a bit.

00:10:08

Yeah. So, well, this is the other thing. We really believe that a script are good videos are good. Um, you know, to go to a video on video or the running platforms, but they don't go far enough. And, uh, there is nothing better than actually practicing and trying out yourself. So the script is the guidance. It's there to help you go through the value steps, and I'm going to continue there. Uh, but, uh, you don't have to follow the script. You can navigate around, explore and see what you are going to do here. This is your own environment for an hour, uh, and to disappear after that. Uh, and, uh, this is, uh, this is basically training as code that has been developed and that you are going through right now.

00:10:55

Do you want to just talk a little about, what's kind of going on behind the scenes here in terms of the catacombs training environment and how we've put this together?

00:11:03

Yeah, absolutely. So, um, the is essentially what you get is, um, um, uh, uh, an environment just for you for an hour. And, uh, this is, uh, this leverage is a Docker massively. Uh, so what's what happened at the very beginning when we started, this is that for some of you, me, I recognize is that I pull the Docker image, which pitches my own tire pipeline engine Kenson environment, which is now naive. And it also connected my own GitHub repository, which is therefore for me to explore and to, to connect to champion, to show you in a second. Alright,

00:11:49

Here it is. So this is my own big clinic, uh, repository. And, uh, what happened is that, um, at the time of the setup, the pet clinic, GitHub repository was connected to the Jenkins environment, which did not exist, uh, five minutes ago, uh, which is now live and that I can connect to, right? So I'm just going to connect to this environment. All right. So I have a fully running up and running, um, uh, Jenkins books, which is not the box. It's a container, which has been connected with my credentials, which I entered at the beginning of this module to make it a repository so that we can do continuous integration and other activities.

00:12:36

So Jenkins is really, they're kind of providing the pipeline, connecting this sample application to the underlying source control and the things that we're about to, uh, interfere with, to make some changes for the security problem. Right,

00:12:50

Exactly. Jenkins is there, remember right there, the purpose of this module is about shifting left on security. So we did not want the student to do, um, yak shaving and, uh, and, uh, you know, spend the time on understanding how to provision Jenkins at plugins configure the, the, the, the repository connect to the web hooks, et cetera, uh, which are not important for this learning, right. Uh, there are the purpose of another module, but not this one. Uh, here we are trying to do everything automatically. And Jenkins is one of the mechanism to do, to implement the, you know, shifting left on security. It's a very popular mechanisms, but there are other alternatives to Jenkins like the directions or circumstance or archetype, CIM, all other mechanisms to do that.

00:13:39

So, uh, following, uh, the, now this is a, you know, after the drill, so we just set up the environment. So this is the team dialogue. Uh, so a week as past, uh, now that the fed rate is over, uh, this is the story, what, uh, we are going through. Um, we are thinking of the team, which Selma from security, they're thinking to implement, uh, more security as part of shifting the security left, uh, meaning that, um, shifting the security back in the ends of the developers. So that is, becomes a concerns very early in the, the life cycle. So she mentioned about, uh, static, and that is application security, testing, SAS and dynamic application security testing desk. Uh, and then there was a dialogue about between the, the, the characters about the concerns that this may, this may, uh, introduce, uh, also how to implement that. And we eventually, uh, our churn, our DevOps coach, um, so would suggest that, Hey, we can do that using a plugin that would be able to check, uh, if we have junior abilities in our, uh, dependencies for the picnic, which is essentially a Java project, uh, which has a lot of dependencies.

00:14:56

So we can do this some other ways. Uh, what else could we have been doing here, Olivia?

00:15:04

Yeah. So what you're going to do right now, and what, what is suggested here is to introduce, uh, this, uh, security, I mean, sorry, vulnerability analysis as part of the pipeline, which is based on, based on Jenkins. And we have, uh, what is called a Jenkins file, which describes the pipeline. Uh, in fact, um, it can be done in multiple ways. Uh, we can shift security left on with multiple ways, GitHub themselves. And I think it's true also for GitHub, but GitHub, at least they do have a mechanism where they scan your vulnerabilities, uh, or the dependencies or the support, multiple languages, uh, for that. And they will be able to tell you, uh, this particular dependency that you rely on for your project is going to be a problem. You have a vulnerability of high severity or critical severity, uh, and that's one way to do it.

00:15:54

So there are indeed many ways to implement the same need. There we are just focusing for the sake of this example, uh, leveraging Jenkins, Jenkins, phyto, Jenkins pipeline, and your west plugin that does those security analysis. So what I'm going to do, uh, right now, just for, in the steps that Dan, our developer explains, right? So we never get to the Gita copy of the pet clinic, uh, to something that is called the bonfire. So avoiding all of the details there, but the post-fight is what, describe our environmental dependencies and some of the things that you need to do. And here, uh, I'm asked to, uh, add the OSP plugin, uh, as part of the, uh, the, basically the dependencies and the bump file. So this is what I'm going to do at that after the slash plugin.

00:16:48

And do you have to run it this way? Or could you just go about that anywhere you liked?

00:16:52

Yeah, so right now what I'm doing is just simply using the, uh, the GitHub UI, the web UI, which is you, not what many developers do at the development environment that they leverage for that they may, um, but I use VM or visual studio code, whatever, and use their own good client, but just again, uh, for the example there, uh, I'm going to, uh, I'm going to use the web UI to, to, to it. So I just, I did my plugin, uh, for which is the oldest plugin. One thing that is interesting to note here is that I'm going to fail the build, uh, if we exceed, um, uh, severity of seven, which is critical, uh, which, uh, which works, which is what I want to do, I don't want to, for this example, I don't want to care about the other severities. Um, so The checks, I'm going to add that creating a branch, Uh, so that I can create a pull request for that. Okay. So this is my two requests. This is going to add. Um, Alright. Um, so, um, I did my commit, uh, what you see there is that immediately, um, now I have something that is spending, so the committee is being built. Remember we wire this repository, uh, with the, uh, with the Jenkins. So, uh, in fact, if we continue, uh, the instructions,

00:18:50

Uh, we can see that the Jenkins is already going through the, uh, the checks and we have a build ongoing, uh, which has Lots of, uh, Which has a lot of, uh, of steps going through. So right now it's all green, uh, because, well, there is nothing, we just added your west plugin, and we didn't say the Nella is, uh, this. So we can just go on with the, a step six out of eight. So now we are going to add the scanner in the pipelines of which no, what you mentioned is that we leveraged Jenkins and the Jenkins fight for that. So again, um, Dan is asking me to go and find the Jenkins file and add this step in the pipeline, which I'm going to choose what I'm going to do. So I go to the, my Jenkins file Here. It is, uh, I'm in the branch that I, where I need to do the change in the pull request. And I'm going to edit this Jenkins file again, leveraging the data where you re what I, uh, in my, my keyboard. Uh, but bottom line here is, um, dependency check in the pipeline.

00:20:15

And you just want to kind of give us a summary of what the purpose of a Jenkins file is at a high level.

00:20:21

Yeah, so the Jenkins file is, is a mean so well in the past, um, you would be creating jobs in Jenkins, which are basically a set of actions orchestrated, uh, to do, um, some, some actions, um, uh, but it was kind of configured manually in Jenkins, which is good, but as CGI and CD pipeline became more important and more, I guess, critical to our businesses, um, something came out, uh, which, which is called Jenkins file, which basically use a way to programmatically program a pipeline and, and describe the value steps in your CIO or your continuous delivery pipeline. So that's what a Jenkins file is, is basically as code describing the value steps and, uh, and, uh, and explaining, you know, what to do, uh, as part of your pipeline.

00:21:14

So it looks like something's broken. Do we want to dive in to see what's happening yet?

00:21:18

Yes, indeed. So something broke, so it was fine before, um, we have a red cross now. So something brought in several look yes. In the leader. So dependency check, we see that the very step that we just added is failing. Uh, and this must be because of, uh, something that well is not, not quite right. So let's just follow with the flow of the instruction. So, uh, indeed, uh, we are told that, uh, the dependence check should fail. So we having to switch to Jenkins classic view, uh, which I'm going to do right now, going to go to the chassis view. And then, um, then you get to the, uh, west dependency check report, which tells me exactly what happened. So increasing the size. So this is just a different check plugin that we just added when it went through 170 dependencies, 118 of which are unique. And we can see that there are multiple severities, multiple vulnerabilities that have been held here. Uh, most of them, um, medium to low, and we have one which is critical, uh, which, uh, uh, which exceeds the, the score that we said, you know, we're having to fill the bill, uh, if we do that. So that's the main for the developer to say, well, this is critical. I really want to fix that dependency and, and, uh, use a dependency that is, um, more up to date.

00:22:54

Um, so that's the point, right? So Selma from security. We continued the story here and the script, I say, well, uh, according to the OSP scan, indeed, we have one dependency that is, uh, added, which has a critical vulnerability, um, how our hacker say, well, I know in fact, it's not because you do changes in your application, uh, you, that you have to have, um, general vulnerable dependencies in our abilities can come because of dependencies outside of your own changes and from, you know, things that are not because of you. So it's important to constantly scan your dependencies, your libraries, et cetera, so that you don't become vulnerable without you knowing. And, uh, that our developers, oh, that's right. I remember, uh, we were, um, I had to pin this dependency to Nan 0 29, uh, because we had an issue, but I remember this was fixed sometime back, so we can update and go to the latest version.

00:23:53

So the next section, we are going to explain how to find a dependency that is not vulnerable anymore. So on Grinch was going to go the dependency, some cat and with core, we are, we see that, uh, there are, um, 10 X versions and also bunch of nine zero version. In fact, for this, uh, clinic, uh, we are on the men's Euro branch. So let's try and see if we can use the latest nine zero, because it's guaranteed to be, uh, obviously, uh, fully with less impact on you and without breaking wrecking interfaces. But let's try to go and adopt 9 0 36.

00:24:36

So what happened to Olivia, if there's other vulnerabilities, despite, you know, as well as the one that we've deliberately put there, that kind of crop up when people are going through the training,

00:24:47

Uh, so guaranteed, it will happen all the time. Uh, but that's part of the, a that's part of the game of the story. Uh, so that's in a subtle part. I think I really like, uh, when the, uh, when this happens, because just as we talk, uh, for dating the dependencies, um, it's, and it's, it's part of the thing that you cannot do with a very simple script, right? Um, things will happen in real life. Uh, you will have new dependency issues and vulnerabilities, so I'll, do you address those? So, um, uh, what we've done is that in the, as part of the module, we expect to at least have one, uh, dependency. So just made the change, but the pipeline may still fail and thrill fail in real life. Right. Uh, so we explain how to do that and how to fix all their responsibilities here, which wants to, um, uh, working through the, uh, the dependency tree, um, and, uh, and, and get those, uh, uh, dependencies documented and just clicking the button.

00:25:57

But I might as well, you know, type the same comment and showing the, uh, the developers in that, in that case on how to find a renewable dependency, uh, and, um, and address it, uh, because, you know, dependency tracking is not easy in that case. I just created the dependency tree so that they can see there. And if I were to spot a dependency, that was when you're a bowl, I could highlight it with this, uh, Gregg command, uh, and, uh, ensure that, so it did totally expect the, uh, new dependencies and new vulnerabilities to be found. Uh, so, uh, we also explain how to manage these.

00:26:40

Um, so we went, we actually did the change. Look, we go back to the pull request that you did my dependencies. I see. Um, well, okay, so the pipeline already as executed. So let me go there, close some other windows there. Uh, so in this case, uh, the dependency check this time is all green. We can in fact, look at the, uh, west to check. So again, I'm going, you know, this is not a video, so I can check a chicken check the report. Um, I can see now that my dependency, I don't have the critical vulnerability anymore. Um, so I can go by and just, uh, stay as I am. Or if I were to say, you know what, maybe I want to update bootstrap bootstrap. I could do that as well. I can see actually what the dependency, uh, vulnerable dependencies are see from one from 2016.

00:27:41

So from 18, and this one has a lot of vulnerabilities even drill down the time. Uh, but, uh, again, I can continue to navigate and explore and see what I could do with, uh, with Jenkins. I can see my multiple executions of the pipeline. So the first time which failed at the dependency to check to confirm which past I can see the test results. And if I go back to the, uh, blue ocean UI, just making sure, I mean, this just makes it a little bit more visible. Um, I can see all of my tests that, uh, that went through the pipeline and seeing what was kid, what was passed again, exploring the realm of the pro uh, the possible hope for this, uh, exercise.

00:28:27

So this final step, we're kind of asking people to think about deployment frequency and to automatically script this testing so that it's happening, uh, if they're not doing kind of daily pushes. So I think that's, uh, one of the best practices that we're trying to encourage here. And, you know, as we kind of get into the, the final minute of this, run-through, um, we've pretty much run through all of the steps of the module here, and we're available in slack, uh, for people to ask us questions about what they're seeing here. Uh, and we'd really love people to not only try this out, but kind of let us know through issues and pull requests about things that can be improved with the modules that we've put out there. And we're hoping to open-source small later on any final words, Olivia?

00:29:19

No, I think, uh, well it's, um, so it's something we have used internally for, for some times, uh, for ourselves and for our customers. Uh, we are super excited to make this publicly available and open source. Uh, we think that, uh, skilling and reskilling for DevOps is a key topic for our industry. Um, so we hope that this can help, uh, actually hope that indeed we can create, um, the community around this, uh, online DevOps dojo as a mean to compliment the face-to-face DevOps soldiers, uh, and that we can, uh, improve as a community, uh, the learnings and skilling for, for the people who have to do DevOps.

00:30:01

Great. Thanks for watching.