Waterfall to DevSecOps in DoD

We're going to talk about waterfall to DevSecOps in DoD. Actually, waterfall to DevSecOps in almost everywhere. It's not in DoD. And who has the waterfall in your organization besides DoD? It's a common practice. It's everywhere, right? It's not a surprise. So first of all, I would like to apologize. Nicolas Chaillan, he's not able to make the travel, and he was busy for

other commitments, and he wasn't able to travel. So I'm going to cover up his part as well, and my part, too. So we're going to be hearing me instead of Nicolas, but I can present as Nick. Let's continue. This is our legal. I have to do one, two, three, four. Everybody read it? Good. We can continue now. That's a legal requirement. That's what exactly what a waterfall it is, right? We just have to get the things done.

Before that, I would like to talk about who we are as an SEI. Software Engineering Institute, it's part of Carnegie Mellon University. That's, I'm working over there. SEI is known since 1984 and actually start from the CERT in 1988. The CERT was the first emergency response team in the world as part of CMU. SEI is a federally funded research lab,

and our mission is really help to mainly DoD or mainly whoever has any issues and problem in the software engineering context. And we have about 650 employees. So we have two main offices in Pittsburgh and DC, but we have other offices as well throughout the United States. So anybody knows the CMMI waterfall?

I'm sure all the DoD folks know the CMMI, right? Some of the people are saying you are the creator as an SEI problem in the domain because CMMI was forcing more about the waterfall-ish. I don't know, but we'll talk about it. And let's continue. And SEI is part of CMU. SEI is a non-degree program, so we're not giving any degree as part of CS or ECE. But however, we are a research

organization under CMU, and our main focus is helping any, mainly DoD, as I said, helping any big complex system to develop a software on timely and securely and affordably. So our mission is really build up the process, build up the techniques, build up the ideas, and so we can deliver the software on time.

We can deliver the software and with high quality. So that's how the CMMI actually came up in old days, and especially for the complex systems, and SEI was tasked from DoD directly, build up a module sort of process. We can build up a very complex system. That's how the CMMI starts. Now we have more... We dropped the CMMI. CMMI is not part of SEI anymore. So we kind of drop it and give it to the

institute. Now we are working DevOps. So in the DoD context, really help to build up either DHS and DoD and look to any type of hard problems that we are hearing. Hard problems are sometimes specific software component or a system level, and build up a proof of concept or prototype in using a leverage of

research. Research is basic from Carnegie Mellon University's CS departments, including AI perspective, including machine learning, including any type of complex or real-time embedded systems and real-time software, and how build up the safety critical component in the software world. That's who we are. So Nicolas, he's not able to make it, and that's his bios, and he's the Chief Software Officer of Air Force.

He has a long experience in the DevSecOps and DevOps world.

And this is me, by the way. Right. I really felt the DevOps speed. That's the true pictures. So when I say DevOps speed, and actually my journey start from waterfall. And when I write the program first time, it was 1993, I'm using waterfall. And comparing that waterfall eras, how we deploy quickly

right now with a DevOps mindset, that's the physical, tangible things I felt. So this is basically wind tunnel, and one of the vendors set up the comparing how fast we can deploy the code quickly. They put a wind tunnel. They kind of simulated the deployment frequencies, and frequencies like we are deploying one, like twice in a year, sometimes once in a year. Now we are deploying in a millisecond, and represented in a kind of a model,

the speed that you're driving a car and no window, everything is open like motorcycle, with wind blowing to the face. Now, if you open up your mouth, you're breaking the security. So I was trying to close my mouth, don't break the security. If I do, I'm going to break that. That means we're not able to deploy. That's my journey.

So I was able to success what was very hard. It took me, like six seconds. I was breathless. It was very tough. Now think about the waterfall and DevSecOps. It's even tougher than that. Great.

So let's see what is, why DevSecOps in DoD? I'm sure DoD folks know reasons, right? Why? Let's start that. Because DoD is really different. Why it's different? We are not really, as a DoD, we are not building the software every time. As a DoD, we are getting a software, and we are not able to have a controlled environment that we are acquiring a

software. So when we do that, when we try to go in more sustainment phase, more operational phase, it's becoming very challenging for us. Same is true for every highly regulated organization. Every organization are in a situation that you're not able to maintain the software you're getting it.

Like maintaining means-Patching, maintaining means like having new release. It's very challenging in DoD because we cannot release on time. We cannot find out any vulnerabilities in the systems because once you acquire the software, it is yours. It is ours. We have to really make sure that we have a right process to build up and component go further. Let's do a deep dive on it. So it's requiring a modern software

practices to establish a full sustainment pipeline. Let's look at another analogy why it's important. This is the study came from a Naval Special Warfare groups. They've kind of compared a DoD versus private sector. If you look at the number in the bottom section right here, this is kind of like a scale. So private industry has

less people, but they will produce more. So if you look at this one, this is important pieces. So as a DoD, we are not able to get the work on time with the right resources. So we have so much resources in DoD, but we cannot able to deliver on time. And we have so much bureaucracy as well. When we look at it resources-wise and process, we have so much

process burden in DoD versus less process in private sectors. And also approval process is important. It looks like we are approving everything in DoD, but indeed, we are slowing the process a lot when we try to approve the stuff. So because we are doing so much paperwork in the approval process. And these are the couple of write-ups from the code, either the

Frank Kendall and also the Ellen Lord, and we're describing we have to change our acquisition policies to make that happen. Because as I said at the beginning, DoD is more acquiring the software. That start from the acquisition path, that's the first journey in terms of waterfall to DevSecOps, we start from acquisition process. How can we change the culture and mindset we can acquire effectively, put in the right policies and practices,

then we can build up the software at the beginning.

Again, let's continue our journey. These are typical problems in DoD. Very familiar, right? See so many problems like how we are building, how we are deploying, how we are integrating. These are the numbers, true numbers, actually. I want to say the program name, these are the true numbers. I saw worse than that, like look at the big numbers. Sometimes takes integration months, even years.

It's really surprising. When we try to buy the one component from one vendor, get another vendor, try to integrate it, it's all manual process. It's error-prone. There is no any parity between environments. If there is no parity, how you do the testing? How you do integration? Because the environment is really mismatching. And most of the time, going back to the acquisition, when we acquire the software,

we are getting a bunch of documentation, a long documentation for installation, configurations and stuff. And some of the program has 3,000 pages for deployment, for installing. 3,000 pages. I'm not kidding. 3,000 pages manual process. If you try to build up, it takes years sometimes. So this is the problem what we are seeing so far with the waterfall

approach, with that mentality.

So what is the solution now? It's the DevSecOps. That's what DoD is going that direction, solve this problem with the context of DevSecOps. And everyone say why DevSecOps is important because we can go to DevOps, we can go DevSecOps, but literally we have to start a DevSecOps when we are trying to build a DevOps component together. We don't want to ignore the security as part of DevOps. It should be part of it.

Doesn't matter when you call, you may say SecDevOps, you may secure DevOps, you may call any other name, but in reality, we have to get the security into the DevOps process, into the software delivery lifecycle because, and DoD folks know specifically, it takes longer to get ATO process, Authority to Operate. If you're not able to approve any application that goes into the production environment, no matter how you create, you cannot really push into

production. To get the approval process, it's a lengthy process. It's another long talk we will be covering later on, but to get the RMF, to get the ATO, sometimes years, sometimes more than 12 months and 13 months we see that maybe some 24 months. So to achieve that, we have to put the security into the application lifecycle. That's using the DevSecOps.

But it's not that easy, right? These are the barriers we have been seeing so far. Again, so much barriers to implement it. Legacy systems, complex systems, real-time embedded systems and team culture, incentives. Every little bullet point takes hour to talk about it, but this is kind of like a very high-level understanding what are the barriers are. Legacy systems, as I said, very problematic

problem. Let me go back here again. And organizational structures, blame-free cultures, and it's very challenging implement it. So let's see how we implement it in one of the programs under DoD. So one of the program I have been working closely, then we are able to implement DevSecOps, but it took six years. Again, if you're looking for a 130 days or 180 days to get

DevSecOps, you're wrong. It will take time. Okay, don't really say that, "I'll get a pipeline. I will build up the pipeline. Everything is working great, the pipeline." Are you able to put a first Hello World in your pipeline? Then you will see a success. You will see how much problem it is, how much will take time to get a first step put in your pipeline. So don't really put yourself, "I can get in pipeline ready.

I can next day, maybe next month with the one sprint, I can do that." No, it's not the truth. It's a real story. Let's journey continue.

So the first phase that program adapted and first things understanding agile culture risk management framework. That goes really hand in hand together. Anybody knows what the risk management framework is? 800-37. So if you look at closely, 800-37 says that you have to have iterative incremental development process to achieve the RMF in place, which is six steps. So six steps clearly say you have the

incremental iterative development to understand the risk controls, and implement risk controls in your pipeline and monitor it to get data back to it. So it goes very hand in hand together. The program builds up the agility and RMF together and builds up the team culture. I kind of highlighted with you important points, build up the team culture, and then when build that team culture, the

focus was the business value. What the business value are for the component, what the warfighter end user will use at end of the delivery, end of the product. Really tied to mission specific as a business value. A requirement is going to be more business value. If the requirements is done properly with the right business value, you can deploy quickly in the production environment because people like that, which end

user will like that. And build up the minimum viable software delivery pipeline. You cannot really boil the ocean at the beginning. This organization did a first repository. Seeing a code repository was the first step to build up. And I saw many of the other programs, they have multiple code repository. The vendor has another one, integration team has another one, dev team has another

one. Build up a single code repository. Build up the first step as the build environment, the first step they have done it. And then they were focusing more agile and security as a team culture. You had the pipeline, the first minimum viable product, and also agile and secure that you can really build up the component together.

Next phase was maturing the process by putting the people and agile process according together. I know we started with agile, now how can we implement the Kanban or Scrum or Scrumban and using that in details? Expand that group of the team together and building up a right component, getting agile coaches into the team.

So again, if you're getting agile coaches in your team, you don't have a proper pipeline, you cannot build up components. That goes hand in hand together. Now you're building more integrations, more continuous integration, more continuous code repository, and more work, less documentation. Right? We don't want to create more documentation, but we will let create documentation in the context of DevOps pipeline, in the context of

the code itself. And this group specifically, they took so many notes as a Markdown language. So they were not writing documents. They were not really writing in specific documents, but what they were doing it, taking a lesson learned or even the writing requirements in a Markdown language that was in part of repositories. Why? Because almost every code repository is able to process the Markdown language,

like MD, and you can generate any type of documentation from your pipeline, from your repositories. That's how they change it. They really build up the path to production, starting from the beginning, able to push the first code into the production environment. Right.

The next step is more architectures. Now, the team literally realized that having a pipeline is not enough. You can build up something from scratch, from ground up, it is easy. When you try to go into a brownfield, you have systems, now you have to maintain the systems. It is requiring architectural changes now. When you think about architectural changes, now we are talking about the containerization concept or Kubernetes.

It's heavy dependent on architectural changes. If your architecture is not able to support for iterative incremental development, you cannot really push DevOps. Then this organization built up a modular architectures, moving to the microservices concept, and then building up SRE in the infrastructure team. Now, infrastructure team is part of that component, and they are able to build up, get the code as

the containerized version, and then push into the production environment. Basically, every contracts have SLAs and SLOs. So they build up as a team. The team is able to define, "Here's my SLA," and infrastructure team is able to achieve that based on what SLA says then. And also introducing audit versus gatekeeper. Security team, they were part of the team.

They were part of the dev team, not really blocking the deployment, more about as the audit, more about helping the developers what needs to be done.

The fourth phase, now operationalize the DevSecOps, like codify the CI and CD and make sure that have a continuous authorization is in place. So if you look at really closely to the ATO process, ATO process is RMF process looking for those security controls. If you look at really detailed security controls are, and 80% of the controllers is the baseline infrastructure component

dependencies. If you're able to get 80% done as part of your manifests or the Chef recipes, whatever deployment script that you're using it, then you're doing only maybe 10 or 15% of your ATO based on the code specific. That organization build up that components. Now, build up environments like immutable environment, automated environments. Now, build up infrastructure as code, like dev team is able to build up dev

environments as same as in staging environment. And sharing the same artifacts with other team members. Artifacts are, if I'm going to use my environments, an environment can be the same as other team members. So that concept changed a lot as infrastructure piece was covering up all the environment dependencies in it.

And introducing a new governance as based on DevSecOps. New governance are putting security, putting RMF team members into the process, they can keep check any deployment quickly in iterative phase.

The next step was the fifth phase, and building up a mature DevSecOps, which basically go live. Now, as soon as when we have changes, so we have an ATO processes maybe in an hour, then we can push the changes direct to the production environment. That was the last stage. And most important here, they were able to build up the lesson learned, put into a documentation, put into the pipeline itself.

Because typically in DoD, we have a two years or sometimes three years as a person works on the topic, then move somewhere else. To retain the information requiring you get a lot of lesson learned from the things they were building, put in the pipeline. How you collect that information is build up a platform that you will get the notes from the wiki pages or SharePoint. Any information goes in the pipeline, record it somewhere else.

It's kind of building up a continuous learning for the program itself. And also building up a metrics and deploy monitoring like, how can we collect application performance monitoring that goes into the pipeline? So, so far we kind of measure the fifth phases, and this is the basically overall level of the building from waterfall to DevSecOps that goes parallel. So we kind of summarize everything else. It's a parallel effort. How we do development activities, application

architecture that eventually lead into the DevOps component, having microservices in it and supporting containerization, you have a cloud-agnostic environments. Right.

So let's dive into the more DevSecOps initiatives in DoD. That's what Nicolas Chaillan and DoD CIO office is pushing from higher up level and to every program in DoD. So what is the DevSecOps initiatives? So, it's a joint program under OSD and DoD CIO office and US Air Force, and these are military services.

And setting up a pipeline is setting up a framework. That framework and pipeline will enable that initiatives from higher up level, from DoD level. And it's based on the container technology and using a Docker or Kubernetes having a side container in it, and designed for specific taking leverage of reusabilities. So if you remember the ATO process, ATO process is specifically based on

approval, right? If I am able to approve some containers in a higher level, the other CIO level, that same container can be used many other components in the other programs, which is going to cut down a lot of ATO process. And also at the same time working with other team members in DoD like DAU and build up DevSecOps curriculum. So actively part of that group too. So what is the real value so far? It's helping us. It's enabling DoD programs across the other services deploy

any application easily, in a timely, and having rapid deployment and developments. Having common environments as a common tool stack and approved component of each tool stacks, it's going to help us any other program and take a leverage of it. Right.

And another value stuff that comes from the DoD, make sure that if you are able to fix one of the images or any components in the pipeline that we had been utilized. So imagine that if you have a CI pipeline running your program, if you're able to pull up any container from a central repositories, it has been already patched, it will give you as verified version, then you can use your pipeline.

Let's say you're using an Apache or using a Mongo and Django stack. Right? If it is approved in a higher level in a container base, you can use that component in your pipeline. It's ready, it's good to go. That's going to enable you to deploy quickly, which basically authorizing it once, and you can use as many times as you can. Authorization is going to come from higher level component, basically from DoD CIO level or Air Force CSO level. If it is approved, then

you can use the container over and over and again.

So what is technologies behind it? And technologies using a DevSecOps pipeline, which is not just DevOps, and building hardened containers. So I'm sure some of you guys already look at it, the hardened containers available from DSOP GitHub environment, GitLab environment, basically. You can download it. And based on the container, then you can use over and over and again. And every time we're adding more container into the

pipeline, so it's keep adding more. Anybody check that repositories? Anybody aware of it? So it's keep increasing actually every time, and you can see better and better, more container approach than you can use eventually. And also creating a microservices concept of any components. In other things in DoD, we have been seeing a lot like rewriting the code, and we would like to have more common repositories.

We can use the same code basis for any module that has been developed. If you're able to do that, we save a lot of time and money. That again goes back to the ATO process. Right? We can leverage any components as Kubernetes and utilizing the defined metrics as part of our components. So I'm just going to quickly go over the tool stacks. I'm sure you already saw that tool stacks.

It doesn't really matter what type of tools you're using. The goal is have this in place, how you plan it and how you develop, how you build, how you test, and then how you monitor, how you secure it. So regardless, this stuff basically is going to be more container-based in left and right side, but the concept is building up a continuous integration using that tool stack. As long as if you follow up all these steps, like how you plan,

do the right testings, make sure to have an artifact catalog in place. And much closer look at architecture perspective. So this is a typical architecture we are offering as a DoD level, and we are expecting any program, they can pull up the code repositories from microservices. That's kind of part of DoD programs. Then we are expecting every DevSecOps CI and CD pipeline have container base.

Let's say you have your pipeline ready, and then you can build up and get your container from these app environments. Then eventually when we build up any artifact catalog, we can pull it and put in your repository. You can use over and over again. And also part of that, we would like to have every containers has the data mechanism. We can collect the data and using Elasticsearch, other

component, we can reuse over and over again for data perspective. So remember, if you don't know your application performance metrics, you cannot really build up right monitoring on top of it. If you don't have a traceability in your data collection, then how can you go back and analyze anything that is covering up your pipeline? Let's say you're going to build up some AI systems, or you have some AI system you build up, then how are you going to make sure that

your model is working properly if you don't have a proper modeling component in it?

The work is not completed yet. As of today, where we are, so I'm sure you heard about the Defense Innovation Board. It specifically came out, software is never done. So now we have the enterprise architect framework. So DIB study is more acquisition perspective. There's a four line of effort. That four line of effort will tailoring acquisition process and building up a digital infrastructure, which is kind of like

software factory concept, and also building the workforce, because it's requiring a lot of skill set changes. Remember that that six years journey, actually, most of them is train the people. Train the people using Agile, train the people how to use the tools, train the people how they build up a microservices concept, train the people how they can understand basic engineering principles.

Another parallel effort has been happening right now with the DAU. So I'm actively participate at DAU DevSecOps Academy program. We delivered so far two pilots. We would like to expand it more, and idea is to cover up more than 200,000 people to train the engineering principles of DevSecOps. When I say engineering principles, like how can we convert monolith application to the microservices? What are the patterns?

And how can we train the project manager, PM offices? How can we train the architects? How can we train the engineers to using this component process? How can we do acquisition component to DevSecOps environment? How do you build up environment? If you're in a position that which tool question you should ask to pull out and pick the right tools, right components. And also the DevSecOps Enterprise Framework, and it's available, you can download it,

and you can Google search it. It's already signed available. It's going to tell you what are the principles and components are of DevSecOps framework, and you can use it for your establishing your pipeline. So we have a couple offerings from Air Force, and you may have heard about it, which is Cloud One LevelUP, and these are the capabilities. You can use a platform and taking advantage of common infrastructure already available.

That's kind of like Cloud One is supporting the AWS or Azure government, which is kind of pay per usage components, and you can use it. And same as the LevelUP as well, and this is another software factory team, and which is taking leverage of the DoD hardened containers, and you can use it for your own purposes. Are we done? No, actually, we're not done yet. So these are the problems we have been

looking for help. It is not the solution yet. How can we build up embedded DevOps and run real-time embedded system? So far, if you look at all the DevSecOps journey in DoD, it's all web applications so far. Right? Kessel Run, Kobayashi Maru, all of them is webby. Now, there is a possibilities we can do the DevOps principles on embedded systems. It is requiring all your contribution to make that

happen. Again, goal is not really deploy quickly every time, but we would like take advantage of DevOps pieces traceability. Right? Requirement traceability. We heard so much traceability, we can do more architecture work. We cannot really fail and learn in the safety-critical systems. It is not practical. You cannot do the production life testing in DoD.

So what are the idea and concept of building up modularities, so we can do the more simulated environment testing, more simulation-based testing we can do, so we have a better quality of the work than we don't want to test anything in the production environment. And so we're looking for more trained professionals, and even the DAU is not enough to train a lot of people. It is requiring everybody's contribution.

It's nice to see people are traveling and understanding concept. The one thing as a community, especially in DoD, we're not really sharing how we did it in DoD. That program, nobody really aware of that program's exist in DoD. And I remember the first time when they were told DevSecOps and DevOps, it was five, six years ago. Nobody was believing its possibilities. Really, if you are in a member of DoD community, we should share our experience.

Everybody's dealing with the same problem. ATO is the same for everybody. And same as for the architecture, same as for everybody's. Acquisition process is the same for everybody. But when it comes to the sharing in DoD, we're not sharing how our journey is. We have to build up a sharing mechanisms. And also other things like how we can build up the DevSecOps pipeline that will support AI systems eventually.

It's another journey. It's another important factor. Build up your pipeline, make sure eventually your pipeline will utilize any AI components. Are you ready or not?

That said, I'm going to finish up here. This is quick SEI DevOps page. We kept publishing any repositories for any kind of lesson learned, and some video channels we're publishing as well. And I'm going to go back to another blog post. So if you follow the SEI repositories, you can get more information from SEI websites. And I'm going to go back one more time.

If you're going to take the pictures, this is the repositories, and you can download it. So we kind of rebuild up the AWS version of the pipeline as a service model, and you can see the building up pipeline is not that really difficult anymore. It's much easier. One click. The big problem is, how can we use effective those environments? Should ask that question yourself. How can you use effectively the pipeline we built so it really matches our organization and architecture components? That said, I took a lot of time. Thank you. Thanks for listening.