00:00:14

Hey everybody. Thank you for coming to our talk today about rebuilding security culture with security champions. I'm Anne Marie Fred, a senior principal software engineer at red hat,

00:00:27

And I'm par senior vice president consulting for nap group.

00:00:33

Little bit about ourselves. I have more than 20 years of software development experience of which more than 10 years were in the DevOps area. I was a DevOps stays Raleigh conference co organizer for three years, and I'm also active in the Linux foundation and CD foundation opensource projects. I was an HR manager for three years, and for the last four years, I've been acting as a security focal. Most of my career was spent at IBM, but the last nine months I've been at red hat.

00:01:06

I am part of the DevOps central excellence practice for the Naus group. Uh, what I'm doing in Naus group is driving chaos, engineering, security champions, and most importantly, cultural transformation track for the organizations. I'm also, uh, part of the governing board for OTs and opensource project who authored book onsite, reliability engineering, and, uh, published white paper on digital skills. I'm also on board of expert panel for cloud credential council global ambassador for DevOps Institute on the influencer panel for DASA, which is dev agile skill association and company manager for Nabu group and a regional chapter

00:01:53

A quick disclaimer. These are our personal experiences and not the official position of IBM red hat or Nabu group. According to former FBI director, Robert Mueller, there are only two types of companies, those that have been hacked. And those that will be in fact, in the 2020 Sonatype dev community survey, 20% of the organizations they surveyed either suspected or had verified a security breach within the past 12 months. And most of those breaches target data that the company owns the cost of a data breach can be astronomical, GDPR and other privacy regulations hold companies liable for security breaches, and ignorance of the problem is not a defense from the liability. This can cause major damage to a company's reputation. If you think about famous breaches at places like Facebook, Equifax, LinkedIn Anthem, and so on. In fact, according to IBM's 2021 cost of a data breach report last year, the average cost of a data breach was 4.2, 4 million. And according to Derek weeks, the time between a vulnerability being announced and the exploits appearing in the wild used to be 45 days, that is compressed by 93% over the last decade to three days. So how do we keep off with this? Well, it's a race and the only way we can be fast enough is with continuous security practices.

00:03:36

Unfortunately, according to Toby Irvine's research for every 100 developers across the industry, on average, there's only one security professional. And because of these scarce resources, security is often neglected in transformations. Security is treated as a last minute gate, say a month or two before shipping a product, the teams will do a threat model and a manual penetration test fix as many vulnerabilities as they can, but because there's a fixed end date, eventually they just ship it.

00:04:12

Security experts, uh, report about 18,000 vulnerabilities, uh, caught so far. Uh, moreover in 2021 was marked with a significant increase in reported zero dates. So at least, uh, 66, 0 days issues have been publicly revealed this year, doubling the total number of those identified in 2020. Uh, so the question is how can we reduce our cycle time, right from identifying vulnerabilities or to stop them from, uh, having being exploited within a duration of three days, uh, let's hear

00:04:51

In order to move faster, while being more secure, we have to up our game and to do this, our technology teams need to become more self-sufficient with respect to cyber security, we need to scale up the number of application security subject matter experts from that 1% to more like 10% of our technical population. And these would be the people who have the background information that they need to be able to dig into the details of security tool, findings, penetration tests, and hacking reports.

00:05:23

So specifically what motivated us to make a change here at IBM digital, we had done some internal DevOps surveys in 2019 and it identified security work as one of the top three pain points of our developers at red hat. A lot of this is become is coming along because we are restructuring our entire product security programs. And that's largely in response to the May, 2021 executive order on improving the nation's cybersecurity red hat software is an important part of the supply chain of many other companies. And so we need to make sure that our security practices are state of the art. And I know at nawe group, this came about largely because of responding to the log for Jay and long for shell vulnerability in 2021.

00:06:10

Yeah. So, um, interesting story from the Naus group and literally critical in the context of this theme of security champions. So a bit of background, uh, NA was group like any other financial institutions and technology organizations used different versions of Lockford and, uh, Java libraries. So the first challenge, uh, was, uh, we were using it very actively directly, but were also being consumed indirectly through the tools through the coding frameworks, which were in turn calling back lock for J and in the worst part, it was not even known whether we are being using the lock for J or not. So that compounded the issue across the industry and financial, the second challenge or the global financial industry like us is the way we are built. So what we do is, uh, the way we build our tool set and the way we implement our technology, uh, into them is sort of in a dependency mode, our core capabilities used by the other middleware, which is then used by the other applications.

00:07:16

So it might happen that you're not using lock four J but some of the services you consume are in turn consuming there. So there is a multi version multilayer and multi facade issue, and problem that exists. And couple with this, uh, consider our bank is close to now 400 years old banking crew. So we have a legacy and complexity to it. Complexity is, uh, what I, you know, spoken about, um, in the, um, years viewpoints and legacy as we end of life tool set, or there are certain languages, uh, which are at the end of their life. So as a result, there are upgrades which becomes quite difficult. And the more you get into the detail, the more you start getting drawn. So the story of lock four J and its relationship with the developer dev op security champion piece of thing is very much around the base level and core understanding about what the job is.

00:08:19

The job is to develop applications and to do so is in the safe and security and you can't be safe and secure until unless, you know, the tool set, the library is the underpinning things. You are importing into your application. So if we are able to take away all those challenges and, uh, then the life becomes much easier in addressing not only lock for J but other vulnerabilities as well. So what it is all about, it's all about building a culture of dev sec, uh, next slide. So why the DevOps culture brought a lot of innovation to software development, uh, security couldn't always keep up with the new speed at which code was developed and reuse. So DevSecOps is an attempt to address this by fully integrating security testing into your C I C D pipelines, as well as developing the knowledge and skills requiring the development team so that the testing results and fixes may be done intermediately. It has always been to integrate security as an ignorant component of the entire software life cycle, whether you name it, DevOps dev SecOps, or sec dev op dev SecOps is about, uh, security that is built in rather than security. That act a perimeter around programs and data. So if security is left until the end of the development pipeline develops adopting organizations, companies may find themselves back in the producted development cycles, where they were hoping to avoid in the very first place. So the question comes up, why do we need security challenges?

00:10:11

Well, as we said before, one cybersecurity professional for every hundred developers leave them spread too thin. And furthermore, the people who are closest to the code are the ones who are best equipped to keep it secure if they have some basic security training. So in a little more detail, what are security champions? What we're talking about is one management, designated person per small team or squad. That's at least one 10th of the development population, including developers, architects, testers, SREs, and so on. And this person will have about 10 to 15 hours of extra application security training. That's enough for them to be a local subject matter expert for their team.

00:10:55

Now, if I'm gonna compare security, focals, and security champions to similar terms, you might hear a security. Focal is somebody who has a role, more like mine, where it might be a 50 or even a hundred percent time commitment, advising teams, maybe five to 15 teams on security and security focals work closely with the other focals across a division or business unit. They're the primary point of contact with the corporate cyber security experts. And they're responsible to ensure that security champions are reacting quickly and reporting back status on the urgent and important security work. On the other hand, security champions is more like a 25% time commitment. So in some weeks, like when you're doing threat modeling or penetration testing, it might take up a hundred percent of their time. But in other weeks it might only be an hour or two of work. A security champion only needs to advise one small team or squad, and they work closely with the other security champions and with their security focal across a, a unit or a single product team, security champions, monitor communications from their security focal.

00:12:00

They make sure the work gets done within their own teams. And the report back, there's a playbook that we found that works for rolling out a program like this across your organization. First, you need a small working group for the program, just a handful of people's enough. Then you need to create and publish a program description that describes what you're doing and why, and explains all the details that people need to know using that documentation. You can share it with your management and executive team to get their buy-in. So they understand what you're asking them to sign up for your team also needs to choose a good training plan, make sure they've paid for it and set a due date for the training to be completed. Then all the HR managers will appoint a designated security champion from each team and squad. Finally, our working group will make sure that we've set up the communications channels that we need, and we'll kick off the program I have in the backup, some more detailed information about what I've put in program descriptions before, but a few highlights I wanted to call out of, of course, you're gonna have to have introductory information.

00:13:09

You also need to have the expectations of your security champions in terms of time training communications and what the responsibilities are. I like to call out that they're highly valued subject matter experts. This is something that's really good on your resume. Also security champions don't have to be the technical lead or the most senior person, but they should be motivated and want to learn more about security. And we wanna make sure that everybody understands that these are not the only team members who are responsible for security work. Some of this security work can be honestly tedious, especially patching and fixing bugs. And we need to make sure that that's the responsibility of the entire development team.

00:13:50

When choosing a training plan, there's a few layers to this, and we're gonna put some examples in slack if you're, uh, here listening to our talk. So everyone in the company is hopefully getting some training on cybersecurity basics. That's an hour or two on things like how to have good passwords and recognize Phish attempts. Hopefully all developers are getting some basic security and privacy for developers, 1 0 1 information, you know, things like, um, how to make sure you're encrypting the correct data and handling personal data correctly and not checking in secrets into your source code and things like that. Now specifically for the security champions, we're gonna add a little bit more. One is security and privacy by design. And this includes things like, um, all of our corporate sec cybersecurity standards and how to meet them and, um, how to make sure that your applications are secured by default, how to run good code reviews, checking for possible security problems.

00:14:48

And so on. We also give them maybe four or five hours of training in the O OS top 10 ans top 25 common vulnerabilities and software so that they know what those vulnerabilities look like. So they can recognize them when they see them. And they also know how to mitigate the problems or fix them in their own code. And finally, we want them to have an hour or two of training in the basics of threat modeling. This isn't enough for them probably to go out and run threat models all by themselves, but it's enough for them to be very helpful to a qualified security architect in running threat models with the team. And it's probably enough for them to keep threat models up to date over time as well.

00:15:29

The good news is according to Sonotype developers who receive training on how to code securely are actually five times more likely to enjoy their work in this bonus. Another thing that's really helpful to us is a secure engineering Guild, uh, at this meeting, which is usually weekly or biweekly, weekly, we're expecting security, focals, and security champions to attend. But I also like to leave it open to anybody else who expresses an interest in application security. We wanna make sure that it's either recorded or at least it has a really good agenda in MI medics so that people who miss the meeting are still getting the important communications that happened there. And a few examples of topics that we've covered in the past include status updates, corporate initiative, security alerts, security tools, and how our teams are adopting them, like very specific information on how to make it work within our own environment, um, pen test or remediation and how that's going. Interesting security news. And if the meeting's long enough, we'll also put in five to 10 minutes of timely security education.

00:16:33

Uh, we don't, uh, recommending any tools out here. It all depends upon your company's culture and the technology it's currently utilizing or approving or has available for collaboration. So whatever it is, go for it, however, depending upon the technologies you choose to utilize, and it may be even a standard emailing mailing list, what you need to ensure is that the tools are easy and straightforward. Um, the next is you'll almost certainly need to capture the interest and participation of the teams. And the next is, uh, the security champion should have efficient communication champions between themselves and with the rest of the security teams. And lastly about the cabins make our time, um, place for champions to meet on a regular basis. And these meetings are useful for discussing any challenges that the security champions are having or, uh, adjusting the target as needed. Uh, you may start with, uh, let's say Aly, uh, biweekly sessions, uh, which could be, uh, initially beneficial.

00:17:33

So now what we'll be talking about is the challenges that we have faced or facing in bringing this security culture in the organizations. So let me start with the first challenge. So people are busy, uh, things are really complex and prioritization is always hard. So there are <laugh> I know there are no special challenges for security. It all depends on our colleagues or developers roles in day to day lives and how it reflects on security. So the only special thing is that it's fast moving and complex subject matter, and that's where and why security champion has important role to play. So what is a recommendation from based on my experiences, um, is four things, education, awareness, involvement, and responsibility let's start backwards, um, which is the responsibility of everyone within the organization for the security of the organization. And that covers all aspects of security.

00:18:34

For example, even closing the door, uh, behind you, to the extent of developing your code, uh, safe and secure for your application. So it comes down to involvement, do the people understand it or feel it and know that it's serious so that where the tone from the top comes in. So for example, our CEO, CIO, and CEOs, our comes and talks about this part of our goals and objectives to be safe and secure. And it aligns with the NA group purpose, which is to serve our communities and customers. So setting the tone from the top is vital. The next is education and awareness. So everybody needs to be aware of the undertaking of their security responsibility within the group. And certain groups needs to be educated further. And that's where developers and security champions comes into the picture. The next challenge is about lack of awareness and it could be about security principles, or it could be, um, just knowing what is truly susceptible or how exploited sector.

00:19:45

So taking an example from NA group and maybe some of your organizations, uh, would be doing something similar is, uh, what we do, what our security teams do is they sent across our colleagues, some purposefully phishing emails, let's say, and when you click on them knowingly or unknowingly, you're asked to take up some pre-defined training programs. The benefit of this is this leads to beneficial adjustment in behavior, uh, which lowers the risk are to the business and possibility of refiling. Um, the third risk is about the buy-in and being a bank. It is heavily guided by our compliance risk governance and extra, extra layers of, uh, regulations making security or paramount importance. So what are we doing is rather than, you know, going to every person and team and asking their views and opinions, we have formed a working group, um, which is trying to build a security champions model.

00:20:48

So in the words of sleep jobs, if I, you know, people don't know what they want until you show it to them, or me words of Henry full. If I ask customers what they wanted, they would have told me a faster house. So the summary is, uh, build a security champion model, uh, with the other likeminded diverse SME groups, and then put it tutor with the rest of the audience or the, um, there's an interesting paper, which was recently published, which talks about the human element as an important factor in the it security. And as for the findings, the lack of interpersonal interactions and shared language is a key impediment to the functioning of connections between the various groups. And in order to develop empathy, trust, and cooperation, it was suggested of appointing security champions to function as a channel, explaining security to your coworkers, assisting them in mastering new behaviors and reporting security that isn't working. So in all, um, the challenges for our organizations is IBM red and <inaudible> group all have globally distributed development teams working across time zones. So you, you neither either to find a time zone with S for all, or, uh, let's say a two different timezone meetings that could be possible. The other recommendation is anchor and support synchronous communications with instant messaging, shared docs, uh, media recordings, email, and so on are more only challenges let's hear out from

00:22:21

One unexpected challenge I ran into at red hat was getting the training program set up the training program. We chose cost about $250 per user, but it took us some time to get the budget approval for that. And by the time we got the budget approval, there were only three months left on the contract for the training plan and they weren't allowing any more new users in. So we had to wait for the security team to negotiate a new contract with the training company. What I learned from this is that you need to choose that training plan and request the funding as early as possible, because it could take a while. Also, if you already have a good training plan available to you start with that. For example, at IBM, I was lucky that we already had several good training modules in our internal learning system. And so our working group just had to evaluate the modules, choose our favorite ones and chain them together into a training plan without worrying about funding.

00:23:19

The biggest problem I probably ran into at IBM was compliance fatigue. Our developers were already spending up to 25% of their time on things like data, subject requests, privacy, and security reviews, internal audits, patchings through software with fixes and to them, this just felt like piling yet more security work on. So, uh, what we did is when we formed our secure engineering Guild, we set one of our primary goals to be automating as much of the manual toil as possible. We shared new automation with each other in the Guild meetings, and we pair a program with each other to implement changes. We also partner with our CISO organization to pilot new tools that they were considering to make sure that they worked well for us.

00:24:08

So what are some wins we got out of this? Well, the oldest program or the three is the one from IBM digital. And when I looked back 12 months after starting it, we found that indeed about 10% of our developers had become subject matter experts. They completed their training programs as asked when handling critical alerts, which we called CISO overrides. Those could also often be handled within one to three days. So that is from the time that the, uh, cybersecurity team would notify the security focus of a problem to when we told the security champions about it, to when they scrubbed all their applications to see if it applied, uh, when they pulled in the cybersecurity team, if needed for further investigation or fixed the problem themselves, and then got the change pushed out to production. All that could very often be handled within one to three days.

00:24:59

And importantly, for me as a focal, cuz I was responsible for more than 100 applications. I knew that no teams were falling through the cracks by spreading the communication out to my 10 or 15 security champions and having them report back to me, I knew that we had touched all the applications. We also saw that far fewer security reports from things like pen testing and tools and hackers were coming back with, uh, false, false, positive, uh, marks or could not reproduce. And usually we didn't get a false positive or could not reproduce that that's actually code for the developer saying, I don't know what this means and I can't seem to reproduce it myself. Right. Um, when we give them the education they need, then they were able to actually fix the problems. Instead people also uncover new vulnerabilities in their own career views. We saw broader adoption of threat modeling and our teams were thinking about security every week. So key takeaways, the security champions program is straightforward and repeatable, but you need management support a few people a few months to get started. You need funding for online training, one security champion per team, uh, one or two security engineer guilds, secure engineering Guild leads per organization. And that's usually an architect or security focal in terms of help that we need. Um, we'd love to hear if you tried something similar yourself and what challenges you ran into and how did you overcome them?

00:26:24

Uh, I have, um, you know, certain help that I would definitely require is, um, I would love to hear out what is the either number of security S per developers that you have, what sort of education and time are you providing these security champions to enable them to perform their role nicely? Uh, what sort of contacts, uh, information and security resources does the security champions in your organizing need to discharge their account? And lastly, what tooling or less an extra tooling required by the security champions in order to build security champions in the life of the developer.

00:27:02

Thank you for coming to our talk.

00:27:04

Yep. Thank you.

00:27:06

Bye now.