Las Vegas 2019

How BMC IT Learned to Reinvent vs Just Run the Business

Hear about our digital transformation journey, the challenges, lessons learned and successes. Agile, DevOps and SecOps enabled us to transform an IT culture that valued control & governance over agility, contracts over value, standards over flexibility and safety over failure. By prioritizing efficiency, business requirements and user experience, we now build in automation with business logic and ensure the entire deliverable travels through a holistic build and test process.


Vinod measures and models the current application development, deployment and support processes to determine if the processes meet the organization?s mission and strategies, and if and how well the current IT systems enable and support the processes. As a result, spends a great deal of time working with different PMO, BSA, application teams, infrastructure teams, and operations on how best to adopt DevOps and derive value from current and future technology investments.


Vinod believes that, an agile enterprise is one that reflects agile and lean values and principles. Agile development practices have been so successful at connecting the once isolated developer to their business user. DevOps extends the gains made with meeting business users' needs to operations' needs. The ultimate goal in this role is to provide perfect value to the customer through a perfect value creation process (Lean) that has zero waste.


David Riggan is an accomplished IT Executive with 20+ years of global experience leveraging information technology to rapidly deliver business value. He has experience with a broad range of disciplines including Business Architecture, Portfolio Management, Analytic and Data Strategies, general strategy development, investment strategy, Lean/Agile, offshore development, vendor management and process re-engineering.


David is always asking how to improve "Time to Value" and then continuously executing, measuring and adjusting. David asks, how can we make the whole greater than the sum of the parts? Where is the non-value add work and how can we remove it?

VC

Vinod Cheriyan

Sr. Manager IS&T Data Architecture, BMC Software

DR

David Riggan

VP IS&T, BMC Software

Transcript

00:00:02

So today we will talk about our digital transformation journey at BMC. It, so if you know, BMC, we are also a software company, but at the same time I work in the it department. Right? So we'll be talking about more about how did we finance form our it organization and how did we get eight times faster time to value? How did we save about 15 million us dollars in savings? And while keeping the head count fairly flat, all right. Get this protocol. So six years ago in Oregon, we were an organization that was tech in the 1990s. You know what that is really means. We had business requirement document that formed part of a contract. How many of you are still lawyers? Let me ask this question very specifically. How many of you are still lawyers writing up your business requirements document? I don't see any hands.

00:01:01

I actually only one which means this section of it will go fairly fast. Right? So let me fast forward. So we were in a waterfall development model. We use, we had very little automation, right? So if at all, a new automation, it was, it was more along the lines of this. We have a DBA, he does wonders in his automation. We can refresh databases. We can refresh applications, servers. That's what the obligations teams still need to submit a ticket to the DBA to use the automation, right? That's the real world, right? So there was automation, but in effect, there was very little automation. We are tools for everything. So we had sales team who sales application, ADR team using Excel spreadsheets. We then got JIRA Riley, whole bunch of tools just to manage the requirement pipeline, right? So we had tools for everything, but in effect, really not any useful, right?

00:01:56

It is also quality was someone else's job. So here is how it goes, right? If you develop an application, QA completes his QA effort, goes into production, finds a bug. Now who is at fault, the devil will say, it's a QA because he didn't test enough. The Chios say it is a dev because you introduced the bug in the first place. So that's how it went. Right. We also had the silo mentality. Remember when I talk about the DBA teams and the apps teams, what do they actually do? It's all my thing. I'm not going to share that with everybody else, anybody else in the organization. Right? So that was a situation that we are dealing with. Now, what happened as at assoc, we had very long delivery cycles at best. We could deliver one in every three months. In some cases it took about a year.

00:02:48

We have missing our commitments. What do I really mean by commitment? Right? So it's what the business is really expecting it to. Do. You differentiate the commitment right? In, in the waterfall model, our commitment is to deliver what we wrote up in a document here we are talking about what did the business really get? What he really wanted to get. We also had a hero culture. So now if I can do things on my own, right? And we are rewarded based on the projects that you deliver, then we kind of incorporate the hero culture into our organization, right? So the hero culture actually led to poor morale within the organization, right? Somebody who delivers a project successfully, he's just the hero, right? So, but if you look at our overall transformation, right, is that really the culture that we want to adopt at the end of the day, we had frustrated customers.

00:03:42

So this was just a disaster, right? So we'll talk about some, uh, our it portfolio. So we are a typical it organization. If you look at it, this is all the tools that we use internally, right? So we got SAS applications, actually a whole lot of them with the cloud that you're seeing, we have applications, digitally, native containers, Dockers, so on and so forth. We also have applications on prem, right? And we also have a application that's been built 10 years ago. And so what really happens in an organization is for you to deliver a product. We'll talk about what do I mean by product layer today, for you to deliver a product, you really need to have all these systems talking to each other, right? So you cannot just say, Hey, look, my I'm all containers and on microservices. But if you are back at back office system, or maybe your BI system is old in or turning on an old technology, then your enrollment cannot move fast enough.

00:04:43

Right? Your business value cannot be delivered fast enough. So that was a situation that we were now, we dug one layer down and really looked at what was the real problem that we were seeing. Right? The crux of the problem was we were spending most of our time trying to run, we spend very little time in re-inventing. So what did we do about, so when do I talk about run and reinvent, right? And reinvent is about maximizing the efficiency of existing operations without impacting the customers. But at the same time, you are finding some value somewhere else, time, somewhere else to reinvent your team. There's when I remember when I talked about flat head count, that's what I really mean. Right. So how did we really approach it? We developed a multi-faceted strategy and we said, remember, somebody talked about today morning. I think it was Jim.

00:05:39

Talk the talking about it. When you get up in the morning, do you have one thing that you're going to do today? All right. So that one thing is what matters, right? So we established a core mission statement that is all about how do I maximize the time to value. Then we put a set of strategic teams around that to say, here are the things you do to strategically execute your time to value goal. So keep in mind, always that simple message of one thing that you are going to do for the day really matters. Right? So that's what things in the people's minds. Now let's look at each of these right automation, where if you really look at everything that we do, right? So you, you automate, you find more time that you can spend somewhere else, right? So we all know that now what is really different about it?

00:06:31

So if you really look at agile DevOps, all of these things together, what are we trying to do? We are going to go faster. We are going to improve the quality, which by the way, include security, right? Then we want to make sure that the cost is optimized, right? So we cannot increase the cost. We want to move faster. But at the same time, we cannot compromise quality. Is that even possible? That's what we really need to look at. How do we automate things to increase that speed? Look at self-service capabilities around. But the example that I talked about, the DBS and the other teams will have their own automation, but it's not really useful. So bring the capabilities to the people who actually needs it. So in our example, our customer support organization, right? They get tickets and they should be a router someplace. So these routing rules changes very often. So what did we actually do with it? Right? So we said, look, uh, in a project or in an initiative, you're going to take about three, three weeks out. We go on to build a rules engine that is going to help you build your own rules. That's how we look at self-service right? So now when they no longer need to come to it, it's all going back to the business data. Now you got the flexibility to do what you need to do.

00:07:56

One of the other things are, if any of you dealing with your shadow, it situations where it is, the businesses developing their own stuff. How do we change that paradigm around? If you're going to tell her you're not going to, Hey, buys use software. It's just not going to work. He enabled a platform where they can develop things on their own. Now it needs to be low code, right? It needs to be high-performing only then, and it should support rapid application development, right? Only then they are really going to bolt on to buy onto it. Now that's what we did with our citizen Donovan. We train people. We told them how to use the, the rapid application development platform so they can do the development themselves. So we enabled all bunch of innovation models, right? Hackathon, ideation, time, exploration time, make sure this is part of your organization DNA, right? So enable people and provide people the framework to do innovation.

00:08:57

You also look at to see how do we develop people? What is the best way for you to learn? Right? So innovation are all, are all, all of those things are good, but at the same time, you need to provide people a platform to learn. So what we did is we used, eh, uh, a training tool that they can go consume on their own. Right? Whether it be your Udacity, you Demi Coursera, doesn't really matter. Give them a tool to learn. Right? And some of these things that you'll say, Hey, they are going to learn, but then nothing really comes out of it. And that's okay. But they also provide them an opportunity to come and showcase what they actually did. So by doing that, you are moving the people up, their skill set up at the same time, you're moving the work down. How can they learn?

00:09:44

If they'd all automated, their repeater, things that they do day to day, they'll not be able to advance further in their career. Right? Also remember technical debt is one of the key things. So you have to identify where your problems are in your organization, right? Whether it be the bad code, maybe you're not doing security scans, part of your CICB pipeline. That's that was our situation before, right? We didn't have any security we will do at the tail end. Guess what? There was not reaction time. So we incorporated security, indoor CICB pipeline. So things like that look at to see where is your technical debt? Really look at the broader umbrella of technical debt. Now, how did we really address this problems are great. But what major steps did we take? We reorganize all of our it teams into practice areas, right? So when we talk about product, we are talking about a product that offers a sales capability, customer support capability.

00:10:39

If you keep going down further, you'll be too much into the weeds, right? So you need to look at what is your product that you are going to deliver. Remember the project versus product conversations. I'll not be able to speak to that within the limited time, but look at how do you organize your teams into the product focus areas, create enablement function. There are going to be some things that you want, the people are not going to go do on their own sometimes, right? So you want to have some enablement functions in your organization that will help you to do, uh, the, the looking, always looking for the best in the organization to further advance it. So what we did is for agile DevOps, information management and data science, we create a horizontal. They always stay ahead of time looking at what is next, what is next?

00:11:26

Right? Then they go out to the teams, whether you call it agile coaches, scrum masters embedded into the team doesn't matter. But the fundamental is provide that horizontal platform. Our earlier horizontal platforms are used to be what a database server change that nomenclature. I changed that things around and say, what matters to me in the organization? Also look at how are you going to do a change and release manager? And look, if you're going to provide all the tools for people to learn, do you think that they are just going to learn by themselves, especially our senior resources? No, ma'am maybe not. Right. So tie some of the incentive backs to their learning. So what we did is we provided them a platform and said, you can go learn on your own, but at the same time, you need to meet at least 80 hours a year in terms of your learning objectives.

00:12:15

So think about some of these things. That's what we did that doesn't necessarily mean that's an approach that's applicable to your organization. By the end of the day, provided them a platform to learn and make sure that they are learning through that platform. We aligned incentive to our desired outcome. We had just talked about what our desired outcomes are, right? The strategic things. So you earlier, our incentives were given to the people who are delivering projects. We changed that around. We said no longer you are getting any incentive. If you are not doing one of those strategic initiatives, right. That was one of the game changes. You also look at nurturing champions, how do you nurture champions within the organization? Right? So you want to have people with skill, but they also need to be training other people, right? So giving some incentives to them, one of the lessons learned for us, we thought, you know what?

00:13:07

To make some of these changes, our best partners are going to be the most experienced people, the architects, the managers, but guess what it turned out to be that some of the newer junior folks actually got onto the bandwagon and they are driving the change. So, but provide them an opportunity. Now, what do I really mean by operationally? Right? One of our interns, we're presenting at our CEO SLT last week. That's an opportunity, right? So it's not only that culture that you create provides an opportunity, whether it be, you know, wall of fame, whether it be like presenting at a conference, right. Paying for them to go present at a conference that didn't submit call for papers. Somebody talked about, you know, the CTO types and the opportunities right today morning. But how do you really do that? Look at what are the delivering and then based on the value that they deliver end of the day to the business, not the requirement that they deliver anymore, right?

00:14:02

The value that they're delivered to the business, you have an opportunity to go talk about it, small wins and market, your successes, right? It's very, very key, right? So if you're anybody here in it, who is not doing marketing, not doing marketing, that's good. All of you are doing marketing on your own, right? That's great. So make sure you deck your cards. What are we really doing as an organization market, those values to the rest of the team now make also make failure is cool. That doesn't mean we are asking people to just go fail, right? We are saying, look, if you try to innovate, you fail. That's cool. And that's very, very hard for a company that's been in, in the, in the, in the industry for 30 years, right? With all the historical in place. Now let's look at what do I really mean by run?

00:14:55

And Raymond, your security team comes to you and say, look, I got a critical security issue identified. That's going to be impacting 50 of your applications between sales and customer support and marketing. Now, what are you going to say? Your business comes to you and say, look, I need this product function. That is again, across multiple products that needs to be delivered right away. What are you going to say? You're going to be, are going to build a, we are building a digital initiative obligation, right? And let's wait for a year and a little, all be cool. It's not going to work, right? So you need to look at how do we really do that? So here is one of the examples that we did. I think somebody rumble the today morning, somebody talked about the data center with open. I think it was going to be where with the place cards.

00:15:47

That's exactly. When I was looking at the numbers are going, this is exactly the numbers that we go to. 2014, we had 36 data centers. We had 64,000 square feet space. And we were spending about $7 million. Today. We have four data centers, 7,000 square feet. And we are only spending about 2 million in our expenses, right? So these are the ways to look at how do we arraignment now? It's not only that we had all of that space on the data center, but the, when the dev asked for, Hey, can you give you a man and Maryland refresh, like in production, the infrastructure guys will go, alright, talk to me in a month. That didn't work. So what did we do? We today they have a self-service capability. They push a button, they get an environment in about 15 to 30 minutes. Right? So look at the next thing is, how do you now the infraction is all done.

00:16:41

How do you really go about getting your pipelines streamlined? Right? Once your dev is done, you should be going through the pipeline fairly fast. If it's, if you're not doing continuous deployment, right? You can still go faster to at least get to a stage where a UAT is being performed, at least get to that stage of continuous delivery. Now, part of the continuous livery incorporate things that is really needed in your organization. You'll find that some of you will need a security scan or, or maybe, you know, unit test or regression test, right? If you have, whether it be your invertebrate pyramid and, um, in regression test or your starting UI going up or vice versa, you can still do this irrespective of which level that you are in. Right? So as soon as one of these things fails, make sure that you proceed. You get that feedback immediately back to the customer, immediately back to the developer, right?

00:17:34

We also look at optimizing our business flow, right? So let me talk about an example to get our BI environment refresh. We got a 10 step process to go pull from here, pull from here, get to the next step. Now this is what my, this process usually takes about 15 minutes, but sometimes it takes two hours. So my next process is only going to start off with two hours because this sometimes take two hours, right? So why didn't you do that? Optimize your flow. You say, as soon as this step finished, as I know that job successfully completed, proceed to our next step, whether it be near CICT, whether it be optimizing your business flow, you want to get your Tableau report out to the business users, even there, right? So look at how do you optimize your business flow shift things left security, your quality, right?

00:18:20

And optimize your business flow. Now we did this all on the foundations of innovation, automation, and self-service right now let's look at one of the key things that we learned in our experiences. We need tools, tools that can do things that more enterprise wide, right? So we looked at it, uh, the capability doesn't answer, what do I really need? Whether it be the CIC or a business law. I needed the ability to look at a predecessor and a successor, right? What is going to be next? Right? We look at how do you optimize that flow? We, in some cases, you'd deal with your legacy application. You may want to just kill and restart a service. The tool should have the ability to do that. Sometimes you need to orchestrate your microservices on containers. Sometimes you have to do API calling into different applications. Sometimes you have to make HDS a actually a fear first commands to execute on a command line, right?

00:19:20

Sometimes you have to just go self-service right. So an mobile Annabelle, right? So if, what if, if your application developer for where to push the code and say, Hey, I want to get a feedback. When they come in tomorrow, he's at a restaurant with his family. He can still push the code in, but through the self service, those, these are capabilities that you need to enable and also make sure that you provide the, the, the, the team and the ability to innovate fast and specifically around the automation. Right? So when they develop the code, construct your automation at the same time, we call it as job as code, right? So that allows you to create, build the automation at the same time as developing initial, let's, all the development gets completed. Okay. Now let me automate, right? Let's go back and automate at the same time that they start developing and make sure that you have tools that label, right?

00:20:10

We have used tools. We'll schedule a job. Guess what it didn't run. Now, the next process has already kicked off and we just lost million-dollar right. That doesn't make sense. Make sure that you select something that will be sustainable and that you can run. Let's look at an example, because we got only about 30 minutes, right? So I learned to be able to talk through hundreds of examples that we have, what did we do with our Salesforce platform, right on our Salesforce platform. Guess what you need developers and environment to develop from, right? So for those of you, anybody using Salesforce platform today, I got a couple. So if we have to look for a full service handbooks, depending on your environment and complexity, sometimes there's $200,000, $300,000. Right now you need 20 of them. You do the math. Now, if you get a sandbox with only the things that you need, right, then you are much more agile in terms of things that they can deliver instead of sticking to three or four full service boxes, look for the sandbox and then push things in that civil warranty.

00:21:12

It's not only that when really things happen in production on the right hand side, right? You want to know what actually happened all the way, right? Your SOQ inquiries exceeding a hundred per transaction. You have a shared page limit, exceeding 170 MB. How do you really know I am in the accelerator model development? Right? So when he really gets to production, that's when I, I get to know that's what happened now, what happens without something in place? Right? So everybody's scrambling, trying to figure out where is the real problem, right? We don't know. So what we did is we use Salesforce API to extract the, the law, uh, use control and to extract the log from Salesforce. We loaded into our TrueSight tools, which is doing analytics, which will then look for patterns and trends. For example, the hundred and 70 MB page limit. Right? Do look for the pattern sentence. When it identifies something, it will create an incident and get us into the robot, actually worked on it. Okay. So that makes it a full loop, right? So when these things happen, the blame game actually goes away a whole lot, right? So it's no longer the business reporting no longer the QBR reporting no longer the manager is on he's on the issue. The person actually knows what to actually work on. Even if it happens in production. Now that doesn't mean that they slip things into production, right?

00:22:35

Let's look at another example, see all these different boxes here. Do you think an army of people coming in is going to really going to help you? Not really. Right. So what is the solution around us? All these boxes here, right? All these boxes with people look at how did we deliver value in this type of situations? Right? Take one issue at a time. A lot of times that you will see that you have, you have automations already in place, but it is not connected to good. So you some type of a workflow orchestration tool to connect all these different pieces together. For example, we had a surveys to start the application we had to start. We have to start the database. We have a survey surveys to push the data in. Now the issue was, these are all disconnected pieces. All we did, we put all these things together and orchestrated and ran it reliably. So that then the PO, when we no longer need these people in between is the people actually who really need to do their job through a self service. They're able to do that, right? That's how you increase your agility in the organization. Avoid those white spaces, right? If the ticket's going back and forth or emails or a hundreds of emails out of those white spaces.

00:23:50

So in effect, what did we actually do? Right? So these are the key elements that led to our faster time to value. So if you look at it the faster time to value, be improved security, we improved quality and increase knowledge. You use some type of a measurement dashboard to tell people and what your team is actually doing. Remember what I talked to talked about? How many of you are marketers? You fall, if your market is, this is what you'll be doing is create a dashboard, tell people what type of efficiencies that you are getting and how are you improving over a period of time, right? So we didn't get from 2014 to today with eight times faster, we started with one time faster, two times faster, right? So if you look at some of these things in terms of how did we actually get time to work on it?

00:24:38

If you look at this specific example, 80% of our repetitive tickets are resolved automatically. So you know what? Some of our repetitive tickets sometimes have SIS service or bitches or payment gateway goes unresponsive. You look at your typical monitoring stuff. You will see that the gateway is still up and running, but it is not processing transactions. So what did we do? We use canola to inject transactions into it, to see if it actually working. If it's not then create an incident, that incident is then worked by somebody else. And in that was our first step, right? The next step that we took is now we know fairly confidently, this is working in terms of the process, right? We said, you know what? Let's automate that process. Why do we need this person to be in the middle to be looking for it? Right? So we automated those processes.

00:25:29

Now let's look at a couple of things in terms of the lessons learned, what was our journey? And the lessons learned in our journey. Culture is crucial, but it is not a prescription. You all know that right now. Also keep in mind, keep the system thinking in mind, part of your training, you want to let people learn from the other systems. So for example, if a change is happening in our Salesforce and mermen, and there is something happening in the back office and the Oracle environment, and then something happening on the front office and the Java application, you need to know how all these things are connected together. So you train people through your different training methods and make sure that the people really know how the systems are really working. Now that's one way to address does this, right? So there is technical ways to what does the store, right?

00:26:18

APA management, for example, right? So you don't have to be knowing all the dependencies. All you need to know is where is my gateway? Where is my API? I pass this, what am I going to get back? Right. So there is different ways to approach it. But in any case, you want to dial up that understanding of the system, thinking what I mentioned is a design principle and it is not really a tool, right? So remember the example that I talked about when customer support agent was trying to route, we could have just automated that, but what did we just do? Which is automated a process that was not really useful. That's not the real problem solver right below. So incorporate your automation and into your design patterns. Don't automate copy processes, right? So in, in the same example, if you just automate a hundred different ways to create your routing rules, then that you are basically not really effectively impacting the user community, right.

00:27:13

Communicate the intent. And don't dictate how one of the key things remember our, uh, our mission statement is that here is what you are coming to work to do today. That's the level that you want to come in and get actually right. Then they know what they need to do after that. So don't dictate how don't get into the weeds of, Hey, you know, do the SQL statement and do the Java application, keep your managers away from it. Let the team do what is right to get the value delivered. What is in it for me is very, very important. So remember the waterfall model where people will write document, you know, why did they do that? Because they could avoid the blame game and they could say, not me, but what are you changing it to now to say, look, here is your incentive. If I don't see the value, I'm not going to do it.

00:28:04

So how do you market your successes and make sure that there is something in it for each individual that is working on it, get your execute the level of commitment. And in fact, uh, David, who was the VP, who was our VP, was going to be planning to be here today, but he cannot, he could not make it, but he looked, he is also a safe, certified, and he also teaches safe classes right now, whether you like safe or not different ballgame, but that's the type of commitment you want from an execute and execute to going teaching people how to do these things, right? That type of commitment is what you need in your organization. Be prepared to learn and accept a beep pair to fail and accepted. We kind of talked about it, right? So keep in mind. It is not a sprint. It is a marathon for us to get here.

00:28:55

It took six years from here to the next step. I'll probably go on to take several years. Also. Re-invent is not going to have, is going to get finished by a certain timeframe. Make sure that your execute is buy into that. Right? The first thing that I, when I see these values, I was talking to going back and talking to our CEO CEO. Didn't tell me, okay, now, where are you giving me my money back? Can I hit cut your head count? That's not how you kind of want to approach it, right? That's where you execute. You kind of support is important. The question should be, where are you? Re-investing now? How are you getting to the next level? Right? So change that paradigm around in your organization with that. Thank you. I know we didn't get a chance to kind of do the Q and a, hopefully this was helpful. I'll be at the BMC booth. If you need to find me. And I'll also be at this speaker sessions, if you have detailed questions. And also one other thing, I know if you have a card with you and scotch of that card, there is a whole bunch of gifts waiting for you. Um, so thanks for marketing team, uh, getting that arrange and thank you all for coming today. Really appreciate it.