Las Vegas 2019
Blue, Bold, Brave—Bank of New Zealand DevOps Journey
In this presentation, BMK will share his experience with BNZ's Transformation journey, realizing fast is good, confess with some of our mistakes, how did we track, measure our progress and some of the learnings.
BL
BMK Lakshminarayanan
Solutions Architect, Bank of New Zealand
Transcript
Hello and good afternoon. I sincerely appreciate every one of you here in the room, because I know there is a Disney magic happening there and thank you very much for coming along. My, um, my talk is on bank of New Zealand DevOps journey. If you wonder, where does bank of New Zealand and where is New Zealand? And there you go. Okay. We are very small, the very bottom, and that's where we are about bank of New Zealand. We are 150 years of all bank, and we have 180 branches throughout New Zealand. And we have 5,000 plus staff members, 25, 2 0.5 billion in revenue and our purposes enabling high achieving New Zealand. And our mission is to help customers be good with money. We have full set of products ranging from personal banking, business banking, institutional and private banking. Our purpose, helping the new Zealanders be good with money. We make by enabling digital banking for our customers. We help our communities. We run skim savvy. We run community finance. We run other local initiatives and help the Newsland small businesses and also help New Zealand businesses to grow
Banking in New Zealand. I would like to give you a context because we need to understand everybody's talking about banking. Everybody's talking about DevOps and DevOps and banking is actually for a long time, you know, in this DevOps enterprise summit, but the market that we are operating on. And what is the compensation for us in the bank in New Zealand in terms of our context, New Zealand is very pretty small. Actually we are 4.5 million in terms of our population and there are totally 26 registered banks and there are 21 non-bank institutions. And out of the registered banks, four of the major banks are owned by Australia, including us bank of New Zealand is owned by national Australian bank headquarters in Melbourne about me. My name is BMK. I'm following this DevOps community since long time. And I would like to give you a bit of a quick story about how I was attracted.
I didn't come to the conference. I did not read about DevOps, but I was involved in one of the production rollout for lending applications to do the back of drop zones. I was a lead developer, an architect for the program and should do lever production rollout on Friday evening seven. O'clock thinking that I'm going okay. My developer, I've done a number of production. Roll-outs I have ops team six offers on board. We can finish all this, maybe in a couple of hours and go to bed early Friday evening. It started seven o'clock and we finished our rollout on Monday morning, three o'clock and the Monday morning, three clerks, very critical. The reason because Monday at nine o'clock, the application went online. It means the business users started using it. And that turned off base of marathon effort to get our application to production opened my eyes.
That was even before I came to know about DevOps term, I went back Monday morning to my work. I started writing down all this information and then started investigating more. What could we do do better? That's how I started my journey. Then I introduced actually a deployment tool I did through the evaluation of process, will we could automate a lot of things. We used to write word documents and wondering like, can we do something different 2014, June? This is I'm talking about. And I went to the operations team and I told them, Hey guys, I would like to do an automated deployment. The infra guy says, Hey, Mick, who does that? After five years now, the developer goes to ops guy and says, Hey, my application is fully automated deployed, but I have to configure manually a couple of things. The same ops guy says, Hey mate, who does that?
So that's the success for me in terms of changing the culture, changing the team, changing that option, changing the people mindset about this and my community. I'm involved in DevOps days, New Zealand. I'm one of the core organizer. I co-chair the cloud-native summit Wellington, and I'm one of the CNCF ambassador. I run cloud native meetups. I went to meet ups in New Zealand, Wellington based one is the cloud native. And another one is future of ICT. Future of ICT is to help students, women, people who are looking for career change to come into it and then help anybody who's returning to work after a break. So the whole purpose of the meetup is to help community. My mission make your life better, make my life better, make others life better, make the world better. And I'm a community, man. I would like to draw a lot of inspiration for the community, right?
History about a bank of New Zealand, 150 years means it's not easy because we inherited a lot of systems, application, culture, and whatnot. Still. Some of our systems are 40 years code that has not been changed 150 years means we have done through the year. Changes in the farm of programs are smaller projects to improve certain areas, certain values, not looking at the holistic view of the entire experience of the customer. So we've improved in one area. What the goal book talks about as local optimization. So we did really fantastic. We build actually a lending drought on system better, but we failed to make the lending request system, not that great. So we had those kinds of scenarios, other business challenges, as usual, most of you have heard since last three days, we have problems in terms of business looking after, or we have a waterfall process.
Business starts. Their thinking is that it systems are slow to change because business thing, they think the speed of light and it systems are not moving as fast as they wanted competing priorities and timelines because we are bombarded by technology teams. Remember by the initiatives from new initiatives, for launching onboarding experience, lending experience, digital experience, mobile apps, and the other area. We have compliance regulatory requirements to meet and other areas we have to look after the BAU projects because the team that I used to be used to support 70 odd applications, and we are maybe a dozen developers in the team and supported by few contractors. But that's what the case is. And if one of my friend told me very jokingly, you ability to keep up with the regulatory changes is the biggest challenge and forget about introducing new features because that's what happens in.
If you're in a highly regulated financial industry, you need to meet a deadline and delivery line to be compliant, not to lose your banking license, the bankers who are helping the customers and the frontline, they have their own struggle because they have to switch between systems to help and service the customers. One of the classic example we had in the lending journey is that we had 21 systems. The bank has fringe line people. They have to help switch between their systems to service one lending request for one customer. That's really frustrating for bankers, but some of them carried on some of them with the bank for 10 years, 15 years in 30 years. Now we have now because a small country and we have a lot of veterans in our bank and these processes are time consuming. And you know, these systems though, I talked about 21 systems.
If you think they are all interconnected in a modern engineering and modern architectural practice, they are not these systems. If you have to copy and paste, open another one window, copy the data, open another system based there. So this is what actually know the bank has the afternoon. And the challenge is that they are always busy in copy pasting. They're always busy in context, switching between multiple systems and the challenges spending quality time with customers, quality engagement with customers. They struggle with that. So what does it mean for the bankers? Multiple processes, multiple systems, multiple people. It is insane complexity and it's not easy. And there is no way that bank has, can give the feedback directly to the technology teams. And there are no avenues. They have to come through the business. And the business has to come through the portfolio teams and the portfolio teams, our project managers and the project managers have to engage the technology teams.
So this is how the whole cyclist, and probably it might resonate with a lot of, you know, if you're coming from that background, technology challenge, challenges or other different sites, what the Tekken ops. I mean, I'm coming from an, I come from a developer background. So the challenges for developers are like this. We have to deal with legacy systems. Now we have mainframes because we are a bank. We have databases, which are 20 years to, to use. We have some of the Jane monologue products and databases and systems. We have to get the data out of it. We have to move the data between multiple system using batch foil, CSPs, overnight process, whatnot, deal with heavyweight processes and documentation and release calendar. Probably again, this is something that most of you heard since last three days, or maybe hearing again and again, centralized data modeling, operations and security.
I have to create a word document hand out to ops team. The ops team will then go and deploy it. And the word document becomes actually the Bible for them to follow. And even if we want to go and change something, no, you have prepared the document. That's what we are following manual testing and deployments dependency on other deems times for upskilling learning and grow is an another challenge. I think the bottom one is very, very critical. If you want to be innovative, transforming the organization, the technical service owners who own the system from technology, these business systems, they have their own challenges, the complexity gold model systems, because the system is getting grown every time when somebody is coming, we are keep adding a band band-aid or no pasting saying plaster saying, oh, let's do it at a new feature. Or you need to add a new broker capability.
You to add a new lending capability unit as document upload capability. So we keep adding all those plasters and the system, it become a gigantic, you know, it becomes, it's a nature. The monolith, when we say we don't understand the complexity, but that, that reality is that your monolith systems, they are not easy. And everybody is now in journey monitor microservices as we do. But that is one thing. Juggling priorities and timelines. Yes, because we have a small developer team. We have to always switch because if I am busy in delivering a new feature for a new system, somebody else has to come. If they say that I have a requirement for compliance, I drop what I'm doing. I start looking after the compliance project, because that takes the priority. And I, because I have to meet a deadline, then people have to switch between multiple systems and becomes projects to BA you forget about what we are talking now because that's modern products to product, but this is actually projects to BAU the technology teams.
Every time somebody comes form a project team that develop a system, they throw it to the BAU and they disappear. And what happens the team with 10 system started maintaining 20, 30, 40. And now, I mean the team when I left, actually I'm in a different team now, but 70 applications, the technical service owner under the operations team challenges also same because the ops team is sharing resources and we have rippling incident effect. It means if something goes wrong in one server due to one application, and we take down other few set of applications and priorities of patching projects, BAU always challenging. And then manual operations, what have we changed? And this is one of our, it's a really a busy slide, but I would like to focus only on the middle portion of it. This is the industrialization. Okay. It's similar to what Jean showed this morning.
And what we are doing is the other to achieve our one BNZ that plan. We need to make our system simple and we need right skills. And we need to fund for industrialization because you cannot throw the legacy systems and start introducing new systems. And you had to keep supporting them. You need to keep fueling them. You need to keep watering them. I know some of the systems are the trees to grow. At the same time. You also introduce new capabilities, build new systems as you migrate on. And that's our tech strategy and business and technology. We came together. We said that the best way to introduce new technologies, best way to introduce new applications is to introduce new ways of working. We have to work together to improve the customer experience. What did we change? We talked about a lot of teams. The talked about here, the Spotify model.
We talked about projects to product. We also have been through the same journey three years before we started this and we farmed different squats, different tribes. And we had dedicated product owner. The product owner is not from technology. The product owners from the business, more collaboration. Now business is sitting with technology. They are able to understand the challenges, what kind of problems we have in technology, why they can't make certain changes as we are requesting them to do. And they see and appreciate the progress. And it's, they're not seeing 80% of project report complete at 90% project complete. This is the burndown of our finance budget, timeline, nothing like that. They are sitting with a team. They're trying to understand that, oh, this is the complexity because you guys have to deal with multiple system, getting the customer data from mainframe, getting your load and security data from a different system.
So the challenges are in real. So they started appreciating that in terms of enabling the technology, the technology team changed, how did we change our delivery teams? We no longer in one team where the pool of resources that we go out and we start working on different initiatives. And we come back, we started forming delivery squats, and these squats are full stack product and stream teams. They are co located. They have a single goal and they are domain aligned. And this is similar to the projects to product approach the domain. When I say, I'm not talking about domain of anything, this is about pure domain-driven design concepts, concepts. And we have bounded context. We have to agree on what a silver boundary and what services we are building within them. And this also resonates with the talk from team typologies, where they said, reducing the cognitive load, because I am not worrying about what I'm doing with the lending securities, what I'm doing with unsecured lending, because the teams are only focused on that particular problem.
Platform teams becomes enabler and upskilling. We run experimentation training programs and we have Guild and certification assistance. Now these are some of the tools overview, and now I don't have a pipeline view or anything like that because most of us, we are in the CIC journey. And we know that how the pipeline looks like, but this is how we started. We are one of the, the first in New Zealand to start with the container journey in terms of our bank, we started with the data, had open shift and we lead and we spend a lot of time and the team have struggled to, you know, the steep learning curve to understand the product, how it works, how to containerize, how to help the developer productivity, developer experience. These all matters sharing of our stories. Now, if you're doing a small team, okay, the team knows that, oh, this is what you're doing.
Look at you're using Jenkins. Your applications are getting deployed fine and fantastic. Now, how do you make an impact in the rest of the other organization? The one way that we did was we used to run internal billions of DevOps states. And I started the moment so many 2016, and I used to invite people who are, they come over for a conference from states to their New Zealand. And some of the industry, people from government organizations who are going through the same journey, I invite them. I bring them as a guest speaker. I asked them to share the story. So this is one way that we call it out like international conferences, right? Like now there is a book on that as well, but this is, this is I started doing 2016 and we also started running architecture and design from countries delivery because your DevOps journey and delivering business value is not just about Jenkins and containers.
You, our system need to be designed architected, including your database, your communication between your systems, your API contracts, your evens, these all has to be know. We have to think about it, put a lot of thoughts around it and then make the system because there is no value in delivering your monolith crap, using a very fast pipeline and you need to deliver them in a smaller batch places. And again, connecting to the cognitive load, internal forums, chapter meetings, Brownback session, and one of the things that we have done, and I would like to also, um, caught on this NAB national Australian bank is the second number of highest AWS certified people work for a bank in the entire world. Second to AWS themselves. It means they are launched a program called cloud Guild and this Tata training, the people on the cloud. The interesting approach on this is actually when everybody's working like that, they all start understanding ubiquitously, the same language.
They understand the same aspects of it. I'm not talking about glacier and somebody is talking about a tape drive. No, we have same kind of an understanding what technology is talking about, what the operations team is talking about or what the businesses they want. And we haven't run it customer feedback. I don't want to read the points, but I'd like to give you an example, think about the banking experience like this mum with a six month baby in the hand goes to a supermarket. She buys formula for the baby, walks to the checkout, wants to pay for the money for the formula, using a card or whatever the system that you know, they have past mission. We call it pass there. Let's say that when you tap your card, if the banking system doesn't work, if that says, sorry, that we are, we have, we are experienced some technical trouble.
We can help you think about that situation. The baby goes to sleep empty stomach. You don't want that experience for your customers. And everybody needs to think about that. Think about ourself. I bank only with one being that myself. And if you're a primary bank and you have a car, you are hungry, you are taking your friend for lunch. You're standing in the queue. And when it comes to pay and you couldn't be able to pay using your mobile device or your card, then you situation is really bad. It's not about you going hungry, but the situation that you're in and this all matters to improve the customer experience. This all matters for technology and I am a technical developer, but I see value. When you put smile on the customer face, that matters not how many processes you have, how many documents you write, how many lines of code you do, how many containers you're running.
Those doesn't matter. What matters end of the day is the smile achievements. We have built useful, useful functionality and usable functionality to users. We have done that. And one of the small business customer, it's a real story. One small business customer, three o'clock. He woke up, he needs to pay wages for the employees. He logged onto our system. He applied for our overdraft in 15 minutes, the fund in his account. And that tells the story itself, how much the team has struggled to bring that value to the business. Because three o'clock the business, small business owner in a pressure because he needs to have a fund to pay wages and you are helping that individual and that time in that moment. And that matters a lot DevOps challenges. Okay. This is interesting thing for me. I want to talk a couple of minutes about this.
This can security stay in compliance audits and our back, our back, most of you might be knowing role-based access security. This is the biggest area for us, right? And I will probably, when I'm summarizing, I'll summarize with one slide, which will help you. And the database challenges, changing the data management team to adapt to modern engineering practices, to unlock the data. It's not easy. We are not successful. We are working with them. And the data management team is helping us in achieving that. After 40 years of same naming standards and consistent conventions, they used, we were the first, my team were the first to break that and we introduced, we could make more meaningful names. We had six characters, eight characters, 10 characters, kind of naming conventions because we came from an old, the DB two and the mainframe systems. But we changed that. And not only that in 150 years of history last year, we were successfully able to deploy database changes, automated way.
It's not just application, but also actually the database and external dependence is our biggest thing. Because you know, if your vendor is not ready and if your other teams are not cooperating, if other teams, they say like, oh, you have to create a ticket for us. I call funnily. It's not TDD. The DV stands for test-driven development. It's not DVD. It is TBD ticket based development. You know, if you want somebody to do something, you have to raise a ticket. If you have that culture, probably that is not going to help you. So you need to work with them. Ma facilitate, make them understand how we can collaborate and how we can be successful. Our learnings, somebody was talking about establishing a common standards, language that is very, very critical. If I'm talking a and if you're talking B, it's not going to help us, we need to understand is for a ubiquitous problems are not unique.
If you think that, oh, in the entire world, we have a unique challenge. That's not the case. Reach out. The lot of people who have gone through the same experience and the, one of the, the important thing that I keep coming back to this community, DevOps, community and DevOps enterprise summit, is that it amplifies your learning. You hear the stories, those are your inspirations spread and share because if you have success to shout out, do that, because that will increase the team and don't give up, keep challenging to go faster. You need to discipline. And it's not. If, if a formula one driver is on the track, if you need to go fast, he needs a better control on his system. Okay? It doesn't mean he has a control is going slow. This is a quote from Adrian Cockcroft who's with Amazon at the moment, I needed to have a sense of architecture as well, because don't think just having Jenkins, as I said, and containers and cloud is going to make you go faster.
No, these are enablers, but your system needs to be designed for doing that other confessions. This is another important thing. If you are embarking on more extensive technology changes at the same time, you will have to pay the price for it because it will slow you down because you're changing. You're introducing a new APA gateway. You are changing the, your enterprise database system. You're introducing a new, um, integration platform. And you are also introducing a new streaming platform. All those changes. Don't start to do that. Now you think about it. I don't have a solution for it, but think about it. Solving puzzles, not problems. This is important. You might ask. What is the difference? Do you want your team member to go and set and fiddle with the Jenkins scripts are with the Kubernetes Yammel file and do all the glory, add a lot of bells and whistles for that, or do you want them to set and solve the problems?
What the businesses have to think about it. And please make use of the platform teams make use of the different teams who are capable of doing it. Jean was talking about this particular point on this Scott improvement of daily work and one of the ideals. So the focus on enablers, this is very, very critical for us because if you don't have, this is again, the court of Satya Nadella this morning that Jean's showed him. The, the PAC developer productivity is very important for you. Okay? Just introducing containers, doesn't help what you have to do. You have to give them self-service capabilities. Do they have to use those system? Those are very critical, understand, 15,000 people working for a developer productivity in a big enterprise. And you are size of 5,000 staff members, 700 people in technology. And you need to have quality investment in helping those developers to deliver the business value, help that I'm after decentralized governance.
How do you guys do it? And if you have answer, I would like to talk more about it applying product mindset to platforms, because okay, now I have a domain teams and stream teams, how we are developing capabilities and introducing new features. How do you do that? With the platform teams, sourcing and procurement sourcing and procurement is another area probably I'm expecting in another forum paper, maybe in a year or two from it revolution books, because sourcing and procurement in a highly regulated industry takes months. If you want some capability. And if you evaluate, and if you want to buy that today, sorry, forget about it. It takes ages to get there because we have to meet a lot of obligations, no compliance, waterfall funding, agile funding, and ask discussion. Are your vendors that you're working with, the products in your suit? Are they DevOps ready?
Do they will connect with you? Do they have API is to handshake? How do you do it? If you're introducing new vendor tools in disconnected, it's not going to help my sincere, sincere advice. I'm not that experienced, but keep learning, keep challenging and keep fighting. One day, you will make the change. And I'm still working with teams since last two, three years in this journey. But we started in 2014, some of the areas, and don't give up and one day you will make the change and you will share this, the benefits, closing thoughts. There's a Maori proverb.
The cushion goes like this. What is the most important thing in the world? The answer is . It's about the people. If you're doing a transformation, take the people with you in the journey, don't forget about them. And the important thing that you need to be successful is your own people and help them out. And this is a quote from a Sanskrit, Rick Veda. It says Sangha chatroom, somebo that thumb some woman armsy Chinatown. It means you all move in harmony, speak in one voice. Let our minds be in agreement with this. Thank you very much for the opportunity. And as I started, sincerely appreciate every one of you in this room for, for going the Disney magic and being in my session. Thank you very much for that.