00:00:13

If you had told me 10 years ago that the next speaker would be sharing publicly her experience report of elevating developer productivity in her organization, I would have laughed at you because the business of this organization is quite literally secrecy and security. So for Virginia Lorenzano has been a technical director at the national security agency for five years, heading up the dev X initiative, which she created in 2018. She is a crypto mathematician by training, but gravitated towards solving the problems that play developers in their daily work, preventing them from achieving their mission. She will tell the amazing story of how they're changing the way software is created and delivered throughout the agency. It turns out that Virginia and I were sitting in the same room at the 2017 DevOps enterprise summit marveling at the work of Kurt. Hockaday at the joint warfare analysis center who works very closely with the intelligence community. So I am so delighted that Virginia will be sharing her amazing achievements and that she was recently detailed to us. Cyber command, Mar force cyber or Marine Corps, cyber force. This is part of a rotation program that candidates for advancements go through. So here's Virginia,

00:01:27

I'm Virginia Lawrence on out, and I'll be talking to you about improving the developer experience at the national security agency. Also known as decks. I so wish I could be with you in person. I imagine most of you don't envision someone like me, small nerdy, feminine. When you imagine an NSA employee, hopefully we'll be together next year to set the stage for who dev X serves. I'd like to share some basics about NSA. Yes, security is our middle name. And while there are a lot of details about our work, I cannot share. There are plenty I can. NSA is responsible for providing both singles, intelligence or sand. She United States, policy makers and military forces. We are also charged with protecting the networks and sensitive information on national security system, known as cybersecurity behind these two missions. Our developers systems and network administrators and other technical folks founded in 1952. NSA is both a member of the U S intelligence committee community. I see. And the department of defense with over 40,000 affiliates where the largest member of the IC and based on our budget and size, if we could give you all the numbers, we would be in the top 10% of the fortune 500 companies. And we're the largest employer of mathematicians in the United States like myself.

00:02:54

So who am I? Well personally, I try to his advice and be the change I wish to see in the world. And I'm never satisfied with the answer. We've always done it this way. Professionally. I'm a career civil servant with over 20 years at the NSA, given the size and scope of our agency, it's been fairly easy for me to move around and try different jobs. I started out in NSA's oldest training program, the cryptologic mathematician program, choosing to stay in crypt analysis. After my three-year rotational period was up focusing on the automation of algorithms. I might've stayed focused on automation, but a beloved mentor boss pushed me into technical leadership saying, you can do things that others can. You can explain technical concepts to leadership. We have enough programmers who do that. And I did as a crypt analysis technical team lead, I revolutionized how 150 employees perform their daily work in support of NSA sick admissions.

00:03:58

Then I joined the analysis fellows program. One of our mid-career rotational programs to learn what life was like for an NSA intelligence analyst. After graduating, I set out to become a technical director throughout my early career. I found myself and others desperately wanting to use modern software tools and stuck in said with make, I'm putting the versions into the names of the programs. It was bad. Five years ago, I became the technical director for the processing solutions and services division. Um, and that group was tasked to make the dream of modern software development tools, reality for some NSA analyst developers. Currently I'm detailed to us cyber command Marine Corps, cyber force as my last professional requirement before seeking a defense intelligence, senior leadership position. What follows is about my five years as a technical director, founding and scaling defects, and some of the bureaucracy hacks, we leveraged along the way.

00:05:03

All right, I've got my dream job, but I don't know anything about agile practices or containers. How am I supposed to technically advise teams working on this stuff? I mean, I'm a math work whose skills include translating conversations from manager to intelligence analysts to crypto mathematician. So I dove in, I became versed in agile practices and I confirmed my teams were agile and it wasn't enough. We were still failing to deliver meaningful solutions to our customers. Okay. So I started looking for more. My boss suggested reading the Phoenix project and I haven't looked back dev ops practitioners taught me the language to describe the problems I was seeing in my teams and had me wanting to have that at NSA, others were coming to this conclusion to NSA formed an internal large government working group to admire the problem.

00:06:04

Now, during this time I also found myself asking don't they know this needs to be fixed. Don't they know this isn't working it slowly realizing there might not be a day while no actions came of it. I did get to know other champions in the software development space. And at the same time, I brought my new dev ops admiration to work and started a technical exchange meeting, inviting the, you know, over 150 practitioners in my office to come to the table and to talk one outspoken contractor took me at my word when I said, if I suggest something that you think is crazy, please tell me. And he did for Virginia. You want to adopt these practices, but we do not have the tools. All right, I'm a technical director. I can be a detective. Was it really that bad? Or was he exaggerating? He was right.

00:06:58

It wasn't good development teams were rolling their own software development, ecosystems spending considerable time and money on stuff that wasn't their mission product, what enterprise accessible tools that there were lacked, both the resources, people, money, hardware to support their user base and the teams that were supporting either just the tool for their team or for the enterprise or making custom solutions. In the case of boss tools, they were changing the source code, and then they weren't pushing those changes into any internal repository or to the outside, which meant with every new version the changes had to be made. Again, the teams that were supporting cop's tools were making custom one-off integrators to make the cops tool, do whatever it is we have to do at NSA, our back portion marking. And again, they weren't sharing it with other people and they weren't talking to the cops companies to say, Hey, could you make some changes? So yeah, this was not sustainable.

00:08:09

Okay. The world is messy and I cannot fix NSA is complicated, disconnected from the internet system, but maybe I can help my developers. Yeah. I can help a few more developers using those DevOps technical exchange meetings, my growing network of software development, advocates and interviews with developers. I learned some of the things developers want and also need, what are those things? Well, they want modern development tools so they can focus on sinking problems or cybersecurity problems. They want reliable collaboration, work management tools. They want a centralized place to share information with each other on best practices. I mean, that all seems reasonable. Why is it so hard? Well, again, NSA, we mainly work disconnected from the internet and that makes things complicated. Classified development needs proper marking and access controls. We actually write software that is protected by, you know, certain caveats. And then there's cultural barriers. Many teams have this attitude of, well, I can't trust another team to support a tool for me.

00:09:22

Alright, enter a bureaucracy. Heck we're big. We have funds. We have a mandate to work with industry. Let's use this bureaucracy to our advantage. We worked with an NSA channels to form and foster a public private partnership, bringing those classification, marking abilities or project labels. Role-based access controls are back an attribute based access controls feedback into the core feature set of a cots product had to remind ourselves, can't lose hope. These things take time. We're talking government contracts, we're talking adoption of big software changes. It took three years to go from good idea to people banging on the keyboard could actually use it, but it was worth it. Even today. My teammates and I still get, thank you notes, both on the computer and hand written ones for making this happen. And those thank you notes. Drive our desire for change

00:10:28

Bureaucracy hack number two, TEDx members. Weren't standing still while we were waiting for these software changes. Not that we existed yet, but we weren't standing still. So thanks to my dev ops learnings. I recognize that my organization needed to change for failure, reverse culture to make good use of the tools that were coming down. But how well as a civil servant with job security, I encourage folks to model failure and getting back up again. I personally did it publicly trying to stay on postmortems within my larger organization, sharing my reading list, feeling hating heated strategy, discussion meetings, more than once I earned negative attention. You don't understand lectures in difference to recommended books and the admiration of junior colleagues who still to this day, talk about some of the books that people share later with you on our internal social media platforms. What I kept learning was that failure and getting up again, the perception of that activity is in the eye of the Boulder.

00:11:40

Our next year, accuracy hack. By this time my network or software development advocates had connected me with this guy named Jacob and together we pitched and founded Devin, our dev X for variations on learning from mistakes and reading the room. The initial pitch focused on duplicative efforts and was unintentionally shared with the leaders of those efforts. And it was seen as a condemnation of them and their teams. It did not go over well. And instead of giving up, we went back to the drawing board. We landed on capturing key metrics from thing to show your senior leaders, how comparably large organizations save money and exponentially benefit from centralized support, a foundational software development tools, calculated productivity, accelerations possible via centralized tools. We should have our part of NSA, including the teams named in the initial pitch could benefit from centralized tools that worked. We had laid the groundwork for Debbie.

00:12:46

Now, for those of you wondering, because I haven't given any timelines, debits went from idea to money to people in less than 18 months, like government standards. It's nearly instant. And Jacob had coined our motto that I alluded to earlier. There is no day we are there. So what exactly were we trying to do with dev ex unofficially, make it suck less to develop NSA officially, we set out to improve the developer experience by delivering tools and services to increase productivity. We set lofty goals like Devic speaks for the developers. We elevate their needs. Um, term dev X wants to help senior leadership understand that NSA is a software agency focused on those things in cybersecurity. How did we start off that left decline? How we look to industry leaders like Amazon, Microsoft target, and we attended conferences like this one. We also read or listened to books, podcasts, and talks.

00:13:54

We immersed ourselves like we were starting a startup. His big quote here is from one of our systems engineers. And I'd like to read it to you. I have never, before worked with civilians, who did the type of external research dev X members did plugging into the latest technology information, reaching out and working with external companies, attending and actively participating in conferences, reading technology, and business books and bringing that all back into inside, making changes happen, not accepting the status quo or the fact that a policy is in the way of Angela evangelizing and making it part of the culture of your teams, all building into the larger fabric of the enterprise. What we were doing at the start of dev X was exciting, but it didn't feel that crazy. It looks revolutionary. Now that I captured that note and just like, thank you notes for that cots product.

00:14:47

A few slides ago, we keep quotes like this around to remind ourselves as a program and civil servants, how far we've come, all right on to the work of being Debbie. We have people, we have money, okay, ours and division. Many of those brown field enterprise accessible formerly under resource tools. I mentioned earlier. So what next? You set an early goal to cultivate relationships with users so we can effectively speak for them. And we identified a particular relationship in need of repair. We made the connections and we've set up a ride along for a peak into the day in the life. Wasn't meant to be day of the meeting IDEXX tool, critical to those users daily work degraded.

00:15:40

So instead of the ride along, we got to repair relationships during a period of our secret talk, make human human connections, both with your teams and your customers share laughs, maybe a donut talk, some more coordinate messaging, present a unified front. Instead of standing back, dev X managers sat alongside the support teams, fielding questions leading the engineer's time to try tests for fine, try again. Get X leaders wrote and refined communications plans, practice talking to customers, coordinating messages, telling the truth. Clearly for various levels of leadership, we modeled the communications patterns that our teams would adopt. FedEx engineers use this time to build our mom trained muscles so that we could see and react to failure signals earlier next time, maybe so early that no one else would know about it. Now, as you see from the title of this slide, we were rewarded NSA's four-star general praise their communication skills. During this event, we earned challenge points, which in the DOD are a way of rewarding collaborators and marking participation in events onto our next hack.

00:17:05

Don't wait for permission when there was no policy law or governance for bidding, an action, civil service necessitates action, ask yourself, how can we change this existing process to suit our mandate? As an example, I'm gonna say has an evaluation process for some software purchases that X added rigor for the tools in our purview requiring defensible evidence. We did not ask if this change was acceptable. Now the expectation that people coming into this process have is I want the tool. I have money. So you're going to say yes. In reality, when we asked for proof of need, it was difficult to accept. And some of our senior leaders came to us and said nicely, but basically who gave you the right to make teams justify their acquisition requests? And we pointed to the Devic sounding pitch. NSA literally gave us money and resources to reduce duplication of effort in this space. Scrutinize the requests is one of the ways where would you think potential duplication? It got to the point where one particular team went to senior leadership, demanding dev X fall in line and approve their request. Instead our CIO approved of Devex its processes, and that team quietly adopted the centralized tools for that one team. The savings included license costs, compute costs and the salaries for up to four engineers. All because we asked, what is it you need to do or need to protect that our tools don't do or don't protect.

00:18:49

So just exactly how did that X enable different software teams requiring controlled access to their software projects to share the same space I supporting our back HVAC and product labels across our suite of CIPD source code and binary, repositories collaboration and workflow management tools. How did we make this process sustainable? Well, first dev X teams intentionally do not change local copies of FOS products. Instead we either write modules that sit outside of the products that we use, or we love leverage public private partnerships to have companies support those features in commercial offerings. For us. In other words, we don't change the inner workings of fossils anymore, and we make modular changes to cost tools that we can share with others. As a result Delphix is able to provide cots and FOS tools and ensure that only those who should act source access and artifact can access that artifact and teams are able to store classify, need to know artifacts in shared tools, leaving teams, able to focus on their mission.

00:20:11

A few weeks ago, Jean asked if I could share the kinds of projects that teams are making, using our tools. And I can't, he also asked if I could share any numbers and I can basically though dev X is small. We are mighty. We have under a hundred people supporting a whole suite of software development ecosystem tools, and we've enabled 250 builds per day, about 3000 active chops users per day, 4,000 and growing active CICB users per day, 20,000 active workflow collaboration users per day, over a hundred thousand software projects. And more than 5 million software builds to all of my team members out there. I am very proud of what we've been able to accomplish.

00:21:05

Now. No dose talk would be complete without a discussion of challenges for dev X and larger NSA. Blameless, postmortems are both a challenge and a success. We hear a lot of reasons why we probably shouldn't do them at NSA internally. We've done them for our products to great praise from users. Some of the challenges we've encountered are comments. Like everything I do is close hold. So we asked that those people focus on the why and the how, but not the luck. Everything my team does is unique. You're not the only person using that technology. Talk about your technology stack. We have a culture of hiding and perfections. I personally remind participants at every event. It's okay, we're all human. We're not supposed to be perfect. I've seen participants lay blame during and after meetings. We continue to model blameless and show how word choice can greatly affect the perception of what you're saying.

00:22:04

And unfortunately right now it's a personality driven practice. So one of my career goals is to normalize blameless postmortems at the NSA. Some of our recent successes include identifying and correcting a bug in one of my division's most used tools, getting teams outside of dev X to participate in large cross-organizational postmortems, reviewing communications surrounding events, not just on the technical components, because how you share information with your technical workforce can be even more important than what you had to change. And finally, identifying on advertised policies, impacting all software developers, NSA, as a result, informed developers are empowered to contribute to the next generation of solutions to policy.

00:22:53

Another challenge where we could use your help. Many middle managers still view Cotsen FOS tool support as a sustainment activity. We haven't yet helped them understand the need for creativity and experimentation. So we're interested to hear what metrics you've captured for your various levels of leadership to help paint that picture. How did you help your management understand things like the tools their employees need to do their job, that software powers their mission and that an evolving software development ecosystem is critical to their overall success. As I mentioned earlier, DFX team members have read a lot and I wanted to share with you our reading lists, having professionally and personally benefited from seeing the resources that others used. We wanted to share ours with you, to all the authors and contributors of these books. Thank you for sharing your insights. You are helping dev X members grow as leaders and helping us improve the developer experience at NSA. And finally DFX members will be here on this channel for questions and can be reached after the talk through media relations@nsa.gov. Thank you for coming.