Las Vegas 2018

Transforming a 150-Year-Old Non-Profit

We didn't know we were a technology company. We didn't know we were a data company. We were a command and control driven company with our board changing annually. This talk is about how we are transforming into a company of federated decision making and continuous delivery of customer value.


Dan spent 12 years in the military as a fighter jet mechanic before transitioning to a career in technology as a Software/DevOps Engineer/Manager. He's now the Chief Architect at the National Association of Insurance Commissioners. He's leading the technical and cultural transformation for the NAIC, a non-profit focused on consumer protection in the insurance industry. Dan is also an organizer of DevOpsKC and the DevOpsDays KC conference. He is an active contributor to the CloudEvents project and opensource.com where he'll soon be publishing a book about monitoring.

DB

Dan Barker

Chief Architect, National Association of Insurance Commissioners

Transcript

00:00:06

So the N-A-I-C-N-I-P-R, uh, is the National Association of Insurance Commissioners and National Insurance Producer Registry. So they're two separate companies. Um, the NIPR is a wholly owned subsidiary of the NAIC. Um, I know that when I say and, and I'm talking about insurance, everyone gets very excited. Um, hold your applause for it. Uh, so we exist to protect insurance company comp consumers by supporting and, uh, enabling state insurance regulators. Um, so the NIPR also will, uh, try to make sure that producers are licensed and are, have the, the correct amount of education, aren't committing felonies in one state and then moving to another and continuing to try to sell insurance. Uh, that used to be a problem. And that's one of the reasons that the NIPR was formed. Uh, our annual revenue is around a hundred million dollars. We have a lot of different customers, so we have 9 million direct.

00:01:06

A lot of those come from NIPR as producers. We also have all of the regulator staff. We have the 56 members of our association, which are the chief regulators in the States, the 50 states, uh, Washington, DC, and the, uh, five territories. We also have insurance consumers. So we have a, an online website called Insure You, and that is some, a place where we curate information so that you can learn more about insurance. Um, the insurance industry, uh, I didn't know about it until I joined, so we'll work on advertising that better. Um, and we also have the insurance industry who will buy data from us, um, and, and other, uh, facilities that we, we provide them. We have about 600 employees across the two companies. That is 500 in the NAIC about and about a hundred in, in IPR of that staff. About 250 are technology related. Um, so that's includes engineer, software development engineers, uh, quality and operations. And we have about 20 top level applications. Applications is a difficult word to quantify. Um, so we have several hundred actual small apps that make up these 20 larger apps. Um, not really microservices unfortunately, um, but we're moving that way.

00:02:33

So where do I fit in? So I'm the chief architect. I report to the CTO. The CTO reports to the Chief Operating Officer and the Chief operating Officer reports directly to the board. And the CEO also, uh, reports directly to the board. Um, the board is four members of our 56, uh, member association. Um, it's a unusual, um, pairing of having both the COO and the CEO basically running the company together. But it's worked out really, really well with these two individuals. Um, and the CTO owns all application development and operations. Um, when we started this, uh, the CTO did not own the operations side of the house.

00:03:17

My responsibility, responsibilities are all technical aspects of the cloud and data transformations. Uh, and I'm also helping to lead the cultural transformation. Um, we've brought in, I, I help organize the local DevOps, Kansas City meetup and the DevOps Days, Kansas City conference. Uh, and I was able to bring in another organizer to help lead this cultural transformation. So we'll talk about the N i's transformation journey. There's only 30 minutes, so it's, uh, we can't talk about everything. I will be at the speaker's corner, um, I think is what it's called, uh, to answer any questions, um, fee. Feel free to reach out to me on LinkedIn and I can connect you to other people in the company that may be able to answer your questions more effectively. So let's start off by defining what our problem was. If we didn't have a problem, then we probably weren't gonna do this. So it's important to, to recognize that you have a problem and then you can probably start measuring to make sure that you're solving for that problem. So we have silos. So we had not only the development and operation silos, but also within development. We have, uh, many different silos for the different business areas.

00:04:28

And each one of those has a different culture, different processes. All of their Jira projects look different from everybody else's, um, makes it very challenging to move people around from area to area. 'cause they're basically joining a brand new company and everyone is using different technologies, um, in different areas of the company. We also were not being as efficient as possible. So I said we had 500 members in the NA or 500, uh, associates in the NAIC. And every new person is highly scrutinized, so we can't really add more people. So we need to make the most effective use of our people as we can. And also we have, we're a nonprofit, so we don't, uh, generate a lot of revenue. Um, we're trying to make sure that we don't generate too much revenue, um, and we have to reinvest that back into the company.

00:05:23

So there's a real balancing act of not, um, spending too much. And we have, you know, the industry also doesn't want us to spend too much. Um, and we get our money indirectly from all of you. So thank you. Um, and I wanna make, and I joined to make sure that we spend that as efficiently as possible and we, uh, give back as much as we can. So we also want to improve our technology. So we are using a lot of older tools. Um, it's been very much, uh, like was talked about in one of the keynotes that we were just keeping the lights on do with less. Um, we don't have enough money for that. You know, we don't have enough money to, to fix, uh, or to write tests, um, 'cause that's overhead. So, you know, trying to move through that has been challenging. Um, and we have a plan called State Ahead. It's available for, you know, free access to anybody on n ic.org. And that will tell you our three year plan. Um, we're at the beginning or we're, you know, in the first year of that plan. So we have two more years. Um, but all of that's public also. All the fiscals I write, um, for the budgets that we request are all public, and you can all comment on them. So please don't, um, <laugh>.

00:06:39

Uh, so all of your comments are taken into account, and then we decide if we move forward. So as we started this journey, we decided that, uh, we needed to make sure we held culture as the most important part. 'cause of my experience and the CTO's experience is that if the culture isn't right, everything tends to revert back to, to the way the culture is. So it's a kind of a similar thing to Conway's Law, um, where the organization, uh, influences the technology. But I think the, the decisions we make about the technology are highly influenced by our culture. So we kept that as one of our top priorities.

00:07:19

So we put people first. Um, and this is evidence of our focus on culture and trying to create a more learning culture. So we did lunch and learns where, uh, I think it's about twice a month we bring someone in or we have someone internal to the company give a presentation. It doesn't have to be even on a technology thing, really. It's just trying to teach others and trying to make our culture more of a teaching and learning culture where we're always trying to contribute knowledge back in and get knowledge back out of the system. We also encourage training and outside opinions. Um, so we brought in people, um, we've brought in, uh, multiple companies to help train us. Um, we also have gotten consultants to come in and give us their opinions, um, so that we can learn from them. We alar aligned our organizational structure.

00:08:13

So the operations team used to exist in a different area of the company, and we moved them in very quickly after I started. And now the operations and development teams are under one leadership. We still have silos there that we're, we're actively meeting regularly to figure out how we're going to integrate all of those teams, um, and, and make, make the operations side feel like they're not outsiders still. Um, because, you know, we brought them in under the same leadership, but that doesn't mean that they don't feel like they're outsiders, just like they did before.

00:08:48

So we also, uh, identified influencers to try to give them more authority and more freedom to make decisions and to, uh, engage their areas. Um, we particularly did this in operations so that individuals over there that are passionate about this stuff, we can give them the resources they need to move forward without, you know, us trying to hold them back or other leadership trying to hold them back. And that helped enable small wins with the staff. So they were able to go and create, you know, some Terraform deployment script, um, for a database or something that's internal, um, or some of the networking stuff, um, that we're doing, you know, the internal operations staff. Instead of the cloud staff, which I'll talk about, um, the internal operations staff that are still working in the regular data center. Were able to, um, create these, these new pieces of infrastructure and start to migrate on their own. We also use an award system. So this was the coolest thing, um, joining this company. So, you know, when I go from to a new company, I, you know, they, they always tell me about this award system they have, and then I never actually witness it happening. Um, it's, you know, like, uh, like, don't do that. You'll get fired. Um, <laugh>.

00:10:08

So within, I think it was less than a month, I think it was about a month, maybe less than a month, one of the people who reports to me, uh, got one of these for a project that ended the week before. And I was shocked that there was that fast of feedback, uh, in the system. So I used these to target specific behaviors that I want to encourage in the culture that we're building. Uh, and I think that is probably the most effective tool, uh, I've had at this company. So we also needed to understand our context. Uh, so the CTO and I and multiple others, uh, have been interviewing peers, managers, our customers. We send out surveys, uh, and we need to, we, we needed that initial feedback, but we also need to get continuous feedback, uh, from all these individuals to make sure that we're going the right direction.

00:10:58

So we send out regular small surveys, um, that are very targeted. Um, we do some, uh, cloud panels where they get to just ask us questions, um, which I'll talk about here in a second. Uh, we also have to, to encourage more of this understanding of the context that, that we're existing in is, uh, we introduced Slack, which blew up. I mean, like everybody ended up on it. Um, and there's, uh, uh, Etsy, uh, created something called Mixer. Um, and there's a Slack app called Donut, which I renamed a mixer for nostalgic reasons. Um, and so we have a mixer channel where people can join and they get matched up every two weeks and they go have coffee. And we have had some of the greatest interactions and acceleration of projects because two people met and now they're off building something new, uh, to help the company. We also did some application analysis. So we created a Excel spreadsheet that listed out lots of very valuable information about our applications. It's very time consuming, but if you don't know what your application's dependencies are, you're gonna break a lot of stuff. And that's really time consuming. So it's better to do some of the research up front, uh, and you won't figure out everything, but at least you'll have some baseline and some documents started that you can add to.

00:12:18

Uh, we also focused on business value. So we introduced, uh, uh, scheduler. 'cause we Hadron jobs just everywhere, and it was a disaster and it was hard to troubleshoot. So we focused on just aggregating those into one common scheduling system. Um, that was a great quick win. When we were doing this, we also made sure that we didn't recommend anything unless we did substantial research. So we wouldn't have recommended that solution if we didn't think that it was gonna be a winner because it was very early. And we didn't wanna risk, uh, essentially collapsing the transformation program because of a, a, a giant fail at the very beginning. We also introduced, uh, Dynatrace, um, an a PM tool, which has proven out over and over again, um, to be helpful. Even the most rudimentary of, of these tools is valuable. Um, the ability to troubleshoot a customer problem in real time with data, uh, is immeasurable. So we also had some grassroots projects that we tried to support. So if we identified those, we would try to support those. Um, we had, I talked about earlier that we, we really needed to tell people that we can spend the money. Like we can upgrade and we can afford the technology, especially if it's providing business value, if it's one of our differentiators. Um, and we want to commoditize the things that aren't differentiators for us.

00:13:46

So our first project was a cloud project. We explored serverless. Um, so it's all done on Lambda. We learned a lot of, a lot of things from that, uh, process. So it's, uh, a redesigned application, and we wanted to start pretty small and gained some fans. So we wanted to get a, a, a good solid win. We wanted the COO to be bought off on this and to help drive the, the, the change and champion it to the board. Um, so this was a, a great example of this. We had operations heavily involved, um, and it went really, really well. But it was small, it wasn't, a lot of people say, you know, take something that's like really critical to the business and do that. Um, I like to build up a little bit more. Um, so doing smaller things that are less critical and, and kind of getting a cadence of wins, um, has proven out pretty well for us. So how did we sell this to the top?

00:14:41

So we bought books like, uh, leading the transformation in AWS way. Uh, and there have been several others since then, like the Phoenix Project. Um, we visited Microsoft and AWS when we were trying to decide which cloud vendor we were gonna go with. We also visited peer companies. So there are other non-profits like us or other companies that are similar size or have similar sized infrastructure. And we went and visited with them and learned about what they were doing. We also shared case studies. And when we were doing all this, we included the naysayers, um, because the best solution is just to bribe them all of the knowledge you have. Uh, and then they'll hopefully make the decision based off of your knowledge. Um, so this was really valuable because it created a shared, shared experience that we could all look back on. So we could all say, Hey, at at AWS this person said this, or this was the story at, um, a IAS the initial thing is hard for me, um, <laugh>. So I think it's a, a IAS, uh, is another nonprofit and, you know, we can reference back, uh, hey, they, they were doing this with their data platform or something. Um, that was really helpful. And, and I've seen this happen in multiple companies now, where having this shared experience is really valuable for talking about it for the next, you know, year or two you.

00:16:07

So we also made broad involvement. So we enlisted almost every department in the company. Um, it's actually kind of funny, the, the mixer I was talking about, this isn't necessarily, uh, selling to the top, but we had someone from the library research department join Slack when it was still kind of in a beta phase. Um, and she just started joining channels and ordering books that were being talked about <laugh>. I was like, this is amazing. <laugh>. So I gave her an award, um, <laugh>. So I mean, that was, that's the kind of stuff we want in our culture is to, to just keep facilitating that kind of interaction. And we also did some Cloud 1 0 1 and AWS training with leadership so that they would understand more about, you know, what is involved in this and where we're gonna go and, and the level of effort that it's gonna take. And we didn't just talk about the upsides. Um, we were completely open that there were a lot of downsides that could occur. There were a lot of risks. Um, and we wanted to make sure that we were all on the same level playing field of understanding that this wasn't just something we could do overnight.

00:17:16

And we also talked about it publicly, like we're doing now. Uh, so we, we've done meetups. We've done AWS events. Um, we try to, to share it as often as possible. We have, you know, the documentation is out online, um, for free to just take a look at. And we, we request feedback and we also use Gartner Research. 'cause you know, execs get sold on Gartner Research. Um, choose wisely.

00:17:45

So how do we sell this to our staff? So we needed to make sure that we chose the right teams and the right projects, so we didn't want to, so, so in these types of transformations, you have, uh, teams that are, you know, they, they don't have enough knowledge to be successful yet in, in one of these events. Um, so you need to pick the people who are the most passionate and then give time and space for the people, you know, the other teams that are, that are less skilled to come up and be skilled in these new technologies or these new concepts. Um, this isn't most of the stuff that we're doing when we're thinking about AWS or Kubernetes containers, microservices. You stop thinking about, I'm gonna log in and press some buttons on this system. And you start thinking about, I have this overarching service and these are my SLAs and SLOs, and this is what it needs to, to interact. This is how it needs to interact with the customer. And we also wanted to make sure that we didn't choose a project that was too large, um, or so insignificant that it wouldn't, wouldn't matter.

00:18:55

So we also discussed, um, our transformation goals and how we were gonna measure that, um, which has changed over time. Um, but we wanted to make sure everyone was on the same page with where we were headed and how, what we thought success looked like. And the most important thing I've learned about this sharing often is you just need to keep sharing over and over and over again. You'll get very bored of saying the story <laugh>. Um, but you may see suddenly, you know, in a meeting, a twinkle in someone's eye, you know, like the third or fourth time you've said it. And, you know, they finally were like, oh, hey, the way you said it this time, finally that worked. Um, so just keep changing it and keep going over and over again. And also hit the hard questions up front. So the, at the very beginning when we did our first cloud panel, we were asked, um, basically, uh, you know, am I gonna get fired?

00:19:55

Um, and we were very clear that we were not out to, like, we don't, we're not, the goal of this is not to fire anyone. We have 500 positions and we have, uh, one of the questions I've, I've learned to ask is, well, how long is your backlog? Like, most people complain about not having enough time to do anything. So if, if I'm giving you, you know, three extra people because of automation, then you're probably gonna be able to execute on your backlog and, and maybe finish off that seven years of work that you have sitting there.

00:20:26

And then we also wanted to make sure that everyone learned so that they felt included. So those, those teams that weren't ready to make the the leap yet, we wanted to make sure they had enough knowledge and enough opportunity for that knowledge, um, as possible. So we send people to conferences like crazy, I mean, insane, the COO's constantly. Like, can we, we spend more, can we send more people to conferences? And a lot of people are like, we're kind of tired. <laugh> conferences are hard. Um, so we also do the Lunch and learns, like I discussed. Um, and we do self, uh, paced learning. So a lot of those online type courses. Um, the what, uh, something Academy, Linux Academy, I don't remember academy. There. There's a whole bunch of different kinds. Um, I don't remember all of them, but we have like every one of them. And basically it's, you know, whichever one you want to use, go use it and learn. Um, we've also brought in experts from GitLab, Heptio, uh, new context and second watch to help teach us. Um, so these are people who are experts in these fields, and we want to learn, um, from them. And so we'll bring them in and have them teach us classes.

00:21:35

So next, um, which is actually not next because I removed project three or project two. So project three was, project two was kind of an iteration on the, the data capture. It was just kind of similar to the, the first project. Um, but the third project is something that we've been working on and is all around business intelligence. And so we create a lot of reports. Um, a lot of what we give back to commissioners, our reports with some algorithms that have been run, um, so they can have this data about different companies. And we found that a lot of what we have for applications now can be solved with, uh, Tableau. Um, we can make it a, a Tableau dashboard and we might eliminate three applications. Um, and not running applications is the, the best thing to do. <laugh> as an app developer, um, <laugh>.

00:22:26

So we, we've really focused on that. And the person, uh, running it, Rupert Klein, he's been amazing. And we actually have a, a a hon I think is what they call their hackathons, um, coming up. And it's just huge. And people from all over the city are coming in. Um, and that's really that culture of DevOps applied to data, uh, is what we've seen there. And we're trying to make sure that, that, that we have, we definitely have the culture, and we're trying to make sure that the tools can help facilitate more of that and more of the federation. So we have kind of a centralized core that is knowledgeable about this space. Um, and then some, some, um, evangelists on the edges who are helping teach their areas, and then they come back to the central, uh, group if they need some access issues resolved or if they have some particular questions to ask.

00:23:18

So selling change, uh, across the, the company is, uh, much easier now that we have a couple of wins. And we have an example particularly of this, uh, Tableau culture that we've developed that it's being, um, so widely adopted and it's solving problems really fast. The commissioners have really enjoyed it. Um, we have a, an auto data piece, um, that is, uh, basically all the commissioners want it now. We did it for like two states and now we have to do it for everybody, um, which is a bit overwhelming, but it's, it's great that we have the problem of everybody wants the tools we're developing with. So for learning, um, we need to understand and translate what technology and culture change means to the business. Um, so everybody needs to be on the same page, uh, across the business to include, you know, the developers, operations, business, uh, the leadership. Um, we also need to share, uh, no matter if it's a big win or a small win, just share it, share it over and over again, shared in the hallway, send emails. Um, if, if a team has done really well, make sure that you recognize that and you praise them publicly. Um, slack has been great for this. Um, we're able to, you know, um, congratulate people in real time very quickly, publicly so that everybody can see it.

00:24:46

And also we need to continue to grow. Um, so this is all more than tech. Um, and we need to explain to everyone in the company why this, uh, is so important and what this will mean for our future. Uh, so we're focused on data moving forward, and we need to get our, kind of our bedrock, uh, ready so that when we start to accelerate our data program, we have all of our infrastructure, um, very solidly placed.

00:25:21

So now we're in that acceleration phase. So we've gotten a lot of the groundwork laid. Uh, we've done a lot of the cultural transformation pieces. Um, we've created a platform team. So we have a centralized team, which some people have claimed is creating another silo, <laugh>. This is all a timing issue of perception. Um, so <laugh>. So that, that's been a challenge. Um, and we've tried to, to stave some of that off with the training and everything, but we don't have enough people who know enough about the newer tools and the newer culture to bring everyone in at once. Um, we would just drown, uh, in requests. So we're trying to use a model where we have this centralized team. Um, they're creating this migration platform, and all of this will be self-service. Um, so we're using like GitLab and, and Kubernetes and, and Terraform and filling in the blanks, uh, you know, where, where our infrastructure and idioms of the company are a little bit different than what's provided.

00:26:19

Um, so we're, we're trying to provide that and then provide that as a completely self-service system where developers just come in and they, they take on the burden themselves, but we also offer some internal consulting of if you have any questions, you know, feel free to ask us and we'll try to partner up with you. And if you need a month of our time, then we'll schedule some time and bring you through. Um, but there are some teams that are like, just give us the stuff and we'll, we'll go and do it, and like, leave us alone <laugh>. We can, we can solve our problem if you give us the right tools. And so we're trying to bring in more people now, um, trying to make sure that they have education. So we've, whenever we do education, we actually include, uh, operations people, even like the desktop support people, um, because we think that, that some of those jobs may be different in the future. So we wanna make sure that they are trained and we bring in security because I mean, they should be trained on the things that we're doing <laugh>. Um, and bringing them in early, as early as just learning about it has been really valuable. Um, because when you go and talk to 'em about it, they don't have to, they don't have that burden of not understanding the technology.

00:27:30

So, thank you. Uh, I have another slide that I'm supposed to leave up. I think I also have, when this gets published, I'll have some resources on here. Um, but things I'm looking for are, right now we're still working through some of the organizational architecture decisions that we want so we can kind of mold the organization, uh, to the way we want our applications to look, um, kind of using some Conway's law, but we don't, we don't have a, a definite, um, decision on where we want to take our organization. Uh, there's a DevOps topologies website, which is pretty good that we've been using. Uh, and we kind of have an idea, but case studies and examples, as was mentioned in the, uh, one of the keynotes are really helpful to convince other people. Uh, how have you gone about migrating hundreds of applications? So we're, the next two years are, uh, are gonna be us just migrating all of our applications.

00:28:27

Some will be redesigned, some will just be moving JBoss into, um, into Kubernetes. Um, we're, we're moving off of Subversion onto GitLab, uh, and if you want to share, so I am more than happy to come and, and talk at other companies or to help, you know, meet with leadership teams. Um, and we want people to come and meet with our leadership teams and present at our company. Um, we're working on, um, funding people to actually come in. Uh, if you come in, uh, I would also have you speak at the DevOps KC meetup. Um, <laugh>, I'm sure that can be arranged. Um, so we want more sharing and we're trying to do that actively within the KC community, but we're more than, uh, happy to do it outside of that, as I was talking about, having those different, uh, partner companies, uh, have you done BYOD?

00:29:19

So this is a big issue right now. We have a bunch of VDI, uh, and we don't want that anymore. Um, it's creating lots of limitations. He's, he, he works at NIPR, he's the director of IT there, so <laugh>. So he's very happy about this and I've heard some good stories and some bad stories about BYOD. So if you have any tips, please let me know. And then if you've open sourced any tools, we're working on that now. I basically stole the stuff from Google's, um, open source docs online. Um, but if you've actually done it and, and have some lawyers that we can talk to our lawyers, that'd be awesome. Thank you.