Las Vegas 2019
Rethinking Operations to Deliver Business Value Faster at DBS Bank
Join Shaun Norris, Pivotal CIO for APJ, and Dapeng Liu, of DBS Bank as they share how a team at DBS Bank has scaled deployments from 10 per year, to 4 per day. We will look at the operations side of this transformation, and how by building a solid foundation of automation, deployments have increased from 10 per year, to 4 per day.
We will start at a high level view of why automating operations can help development teams increase velocity, and then dig into the details of typical ops activities such as HA restarts, VM lifecycle, network provisioning, patching of polyglot microservices, sharing security credentials, setting up centralized logging and more.
We will also cover how DBS moved key applications from monoliths to microservices, how using an automated platform has automated a number of key operations activities and how the mindset around production change control has shifted in the organization.
DL
Dapeng Liu
VP, Development, DBS Bank
SN
Shaun Norris
Field CIO Asia-Pacific, Pivitol
Transcript
I'm Sean Norris. I'm the field CIO for pivotal covering Asia Pacific. I'm based in Singapore. This is my fifth DevOps enterprise summit. I was at the first one way back in, in San Francisco in 2014. And it's the fifth one, the fourth one here in the U S and it's a real privilege to be here and introduce my friend and, uh, you know, kind of working partner that Pang Lou from DBS stepping. Do you want to, uh, introduce yourself quickly, please? Yeah.
Uh, hi everyone. My name is Stefan. I am from, uh, DBS bank Singapore. So I came all the way from Singapore to here just to share some interesting stories that we think is worth sharing. Uh, I lead the groups, uh, credit risk development team and, uh, uh, uh, maybe will actually talk about, more about, uh, what's going on. Right? So,
So we're just waiting for our slides to come up. Um, one of the, uh, this'll be really fun if we do 30 minutes with no slides, we probably can cause we've done a dry run, but, uh, you know, the interesting thing when, when that Peggy and I got together and had coffee in Singapore six or seven months ago now, and started this idea of doing this talk together, it was really based around the idea of the accelerate book. So the work that Dr. Nicole Forsgren and team have done, if you saw her talk yesterday, it was great. Um, the, the work they've done was fantastic, and that book has been something that as I go around and talk to customers and in my roles, working for large banks in the past, that was, um, really informative and especially kind of these big four metrics. So when I got talking to dapping from DBS and he started telling me the transformation story that they've been on over the last really two and a half, three years, I was like, this sounds to me like one of those elite performers that we read about in the state of dev ops reports and in the, uh, and in the accelerate book.
So, you know, let's, let's start at the end of the story in terms of business outcome. This is really a technology talk, but you know, in July of this year, um, first of all, how many people have heard of DBS?
Wow.
That's, that's good. Um, so DBS is development bank of Singapore. It's, it's one of the, you know, I'll let that Pang explain a little more, but maybe let's start with talking about the award, the DBS one in July this year.
Uh, can we have the slide please?
It looks like we may actually have some slides. Oh, they went,
Okay. So, uh, this year we have, uh, achieved the global recognition, um, being recognized as the best bank in the world. Uh, thank you. So, uh,
So, so that was the, uh, it was a magazine in the European union called Euro money. And in July of this year, they voted DBS bank, the best bank in the world. So you may not have heard of them, but hopefully you'll go look that up and reference it. And, you know, I think we're going to use that as a kind of placeholder. We're not going to kind of go into balance sheet reading and prove to you that the kinda the, the dev ops outcomes talked about an accelerate are, are definitely in place, but let's just use that as a placeholder and agree that DBS is one of the trendsetters. They're one of the elites in the banking world. And so, you know, as we go back and kind of remind ourselves of what accelerate toxis talks to us about, I was hoping to have the big four metrics up here.
But, um, since we don't, you know, remember what they are, I've talked about them enough that they should come straight to me, there's really one on throughput and one on stability. So, so you got two metrics on throughput. This is how long does it take you to get a change into production? And then how often do you go to production? You know, in, in a lot of the organizations I've been part of the speed that you went to production was maybe every month. And we're going to hear from DBS, kind of, that was the position they were in, you know, a couple of years ago. And yet you also, it's not good enough just to have lots of throughput and be going really fast. You also need stability. And so the other two of the, kind of these four key metrics really talk about, um, how fast you recover, if something breaks and how often do things break when you go to production.
So what's your, what's your change failure rate and how fast you recover. So those four metrics, uh, I, I'm hearing a lot more about them as I go around and talk to customers across Asia. And, you know, if we go in then and look at DBS, if, if we put this, I want to use this accelerate report, or the state of dev ops report is kind of a backdrop for what we're going to kind of dive into and talk about what DBS have been up to. So, um, it seems we're still a slide lists and timeless, so we may go on all afternoon. Um, you can leave whenever you're bored, but, uh, you know, in the, in the accelerate book, there's, the punchline is really this, that if you adopt dev ops practices and principles, you are going to improve your delivery capability of software and operations. And that in turn is going to improve your business outcomes, that your however, you measure that whether it's profit revenue, you know, turnover, et cetera. So, you know, we, we've already kind of gone to the end of that story and talked about how DBS was voted best bank in the world this year, but maybe let's talk about what are some of the specific principles and practices you, your team picked up debt paying over the past couple of years.
Yeah. So, uh, since I'm not from the marketing departments, I would just leave the so-called sales talk to my marketing department by no means I want to steal their job. So, uh, uh, giving you an example, ISO, uh, what we have been and what we are now, uh, when I joined the bank about three years ago. So, uh, back then, uh, our in-house developed so-called legacy, uh, cardiac profile application. It was about a million lines of code, uh, donate Don framework, right? asp.net, and don't judge Ashley. You know, he was really, really painful. Uh, then we cannot afford to have more than one release every month, uh, into production. So, uh, we used half this, uh, tradition saying, then every time we go into production, we need to have a party because it just like way too much effort that to put in the, uh, we need to give ourselves a pat on the back.
So now, uh, we are talking about like, after, after the three years of, uh, development, we are talking about almost 100 times into production every month. So if you do the math, uh, and, uh, since we only do like a week, day deployment, we don't actually do a Saturday, Sunday. And, uh, we actually talking about like, um, about four times a day into production. So, uh, that's, uh, where the half the, uh, the title from, uh, 10 times a year into four times a day. But then, uh, it, these two, now this looks very close to each other highway. Doesn't need to be. So if it's just like change the unit. So if you actually convert that four times a day into a year now, that's roughly about 1000 times a year.
W we have some slides up now. So, you know, a really quick, really quick rundown, we actually don't have a monitor here. So we're going to do the awkward thing and look behind us. But, um, you know, really, if you think about it in, in, in the accelerate backdrop, like lead time, went from 32 hours down to three and release effort went from 12 to two and infrastructure provisioning now, instead of taking a week happens the same day. And, and so w when we talk about those things, we talked about these already, and you've seen these slides as well. This is some fantastic material. If you've not seen these, go download them from the door report, read them, take them back, share copies in your organization. Um, we talked about the punchline and why it matters to every business. So then, you know, we, we we're, we're kind of gotten to this point in our slides where DBS has been doing things like using cloud native architecture, using a PAs using private cloud and automated delivery pipelines, and self-service of infrastructure provisioning, all the stuff that you kind of read in the state of dev ops reports, and they celebrate books saying these are good things, and they're correlated with high performance.
We now have, you know, a real life example of a team in a highly regulated, complex financial services industry that are doing these things. They're achieving these sorts of improvements in their technical outcomes. And, you know, like we talked about they've voted best bank in the world. So as we, as we kind of thought about how to structure this and only having well less than 30 minutes now, um, you know, there are seven key areas that the accelerate book talks about in terms of practice areas that you want to think about adopting. If you're going to get on the path to be wanting one of these elite performers, and we wanted to pick two, and we also wanted to talk about both dev and ops, but shade it a little more towards ops, because it's maybe the forgotten part sometimes of the DevOps story in terms of like what our platform teams doing, what our infrastructure teams doing to help achieve these improvements in delivery.
And so we want to zone in on two specific things. One is how DBS adopted a platform as a service. And the second is what sort of things are they doing around infrastructure as code? So let's talk about infrastructure as code. First infrastructure is code, you know, really the, the accelerate book tells us that if your teams are manipulating their infrastructure as code, they're almost twice as likely to be an elite performer than if they don't. And so, especially having declarative version controlled environments, if you manage to see my colleague Cornelia Davis, his talk yesterday, fantastic on using Kubernetes and having a declarative functional model for managing your infrastructure. So, you know, that's the theory, but let's hear from dapping and DBS on what sort of things have they actually done around infrastructure is code.
Yep. So, uh, from the self permission implant view, we don't have a cloud Foundry to help us to, um, kind of like the probation, uh, uh, compute Ram and the disc. Uh, however, actually, uh, we think about the, uh, the, the, uh, the whole, uh, software development as a whole. And then suddenly you sent that in a sense that, uh, infantryman shouldn't necessarily only be, uh, classified as only a mile, uh, the hardware or the, uh, or the soccer, the computers, but also, uh, the stuff nationally, uh, relevant to the, uh, code, but it's not part of the code. So therefore actually we think about that pipeline. Also, the automation part of it should be also considered as a, uh, as the, uh, part of the infrastructure. And, uh, uh, this days, this days we spend enough effort to make it real in a sense that you can, you can automate, uh, not only Lombardi environment provisioning, but also the rest of the stuff.
Think about like I, uh, provisioning of the repository provisioning of the, uh, pipeline, which does the bill, does the test, does the scan, does the, uh, does the deployment, right? And, uh, until now we have about 260 live, uh, repositories live in, uh, in, in our kind of like a live code base. And then that's another, that's another 130, uh, core repositories have already been decommissioned. So if you do the math, uh, putting into perspective, that's about 400 repository in, in three years time. So, uh, let's, let's do the, let's assume people working on about 270 days a year, that's about 800 days. And then therefore it's like every other day we have putting on the new repository. So, um, uh, this kind of like activity start becoming a norm. And, uh, for the, uh, pipeline perspective, we have the one-to-one mapping from the, uh, core repository towards the pipeline. So every, every one of them will be actually backed up by exactly one pipeline. And, uh, every time there's a push into, uh, uh, a bit the repository, uh, the pipeline will be automatically trigger. So, uh, in last month alone, we have managed to push into our testing environment no less than 200 times. So, uh, this has actually a fundamental kind of differences than the, than the old legacy application that we are doing. And, uh, next slide, please. Very cool.
Um, so let's talk a little bit more about this pipeline automation. This is pretty cool. You were sharing with me how before, when it took a long time to get a pipeline, you ended up with really kind of giant microservices. I think this would be a great kind of anecdote to share with, with the community,
Uh, the story hasn't been always this romantic. So, uh, you all start up, we all know we all start over with something. So, uh, one salon, we had, uh, some, uh, so-called microservices, but by no means they are micro. So on. The bigger ones are, has a hundreds of, uh, controllers in a tons of people working in the same, uh, repository. And then people are competing with, uh, the, uh, much conflicts I saw so many punches were going on. So we said, uh, uh, despite the, actually the constant emphasis about putting up the right architecture, uh, doing microservices, right. And then, uh, we don't actually see people actually spin up new repositories. And in a sense, in a sense we are, we were right into the portrait directory of being a certified, the model is right. So, uh, so then now we kind of like, uh, sat down and then talk about this problem.
Why do we think, uh, there are some good things that we need to do, but the people choose not to do so. So then, uh, then, uh, we had this conversation, I think this is one of the most important conversation I ever had for the whole development, in a sense, uh, we say, what is the reason why people wouldn't want to spin up in the Newman repository since, uh, given the fact that a code basis has already grown a little bit too big. And, uh, the, the feedback was, Hey guys, what? I only need to spend about a hundred lines to get my, uh, uh, ticket out of the automat hand when, uh, by the setting of the pipeline took, took me about three days. So, uh, in Tufts, these are risks award, and it doesn't really kind of like gel in. Right. So then suddenly, uh, that kind of like realization kicked in, in a sense that, um, just because this something's right, just because doing some things right.
Does not mean that people will do so if it's not easy to do so, then we say, okay, let's stand on. And then kind of like a, uh, streamline the pipeline pipeline set here. And then, then, uh, more in the more kind of microservices start to pop in actually these days. And if you set up a new pipeline, it takes you no more than five seconds. Uh, then, uh, if you actually kick in the pipeline, they run it in the, in a kind of like the whole, uh, whole, uh, testing, uh, environment takes you about 10, 10 something minutes from end to end. The very beginning of when you're having a, uh, a application running locally, uh, on the port 80, 80, given a JDBC URL. So that's all you need to have, uh, a pipeline highway. That's another problem kicked in, right? So, uh, as I'm and more pipeline, and we are putting into the, uh, into the, uh, environment, we figure, uh, every, uh, microservices has their own, a little kind of like pipelines, uh, specification or definition of power.
And, uh, apart about 50 to 70, we feel that it is becoming a little bit of a maintainable because every now and then there'll be a little bit of a tweak of the bespoke requirement for the need of the requirement, uh, for the need of the pipeline. So then now we say, what can we do? All right. So, uh, uh, then I figure out actually, there's a way to inherit all of the pipelines, uh, then for each and every one of them, you don't have to put on the full blown definition. You, you probably can only, uh, I mean, it is sufficient for you to only provide the necessary, a key piece of information. What are the, where are the pipeline? Where do you want to, sorry, where's your code? Or where do we want to deploy that? Right. So, uh, it's actually a very iterative approach. I now actually anticipated to land in such a situation. Now, today we have all of the, our kind of like a, uh, 200 something, uh, uh, repositories. We have 270 pipelines. They all inherit from a single kind of like master definition, the rest of them, you don't really have to do. And so, uh, uh, as, as I said, right, it's actually a kind of very iterative approach, as well as off we go, we figure out more problems and then we try to find solution to software.
Very cool. So the other kind of key part of the story is, you know, the use of platform as a service. And if we kind of refer back to this backdrop of the accelerate book and the state of dev ops reports, you know, teams using platforms as a service in last year's report, we're one and a half times more likely to be elite performers. And, you know, our theory is that this is likely because developers are spending more time above that value line and less time wrangling infrastructure and doing kind of non-value-added heavy lifting. Um, I, I think you're going to be enjoying hearing about, you know, how DBS has, has been on this journey as well. So, so definitely share with us a little bit about kind of how the platform has enabled your team. Yep.
So, uh, as a developer manager, I actually put myself in a sense in a, in a position where, uh, my job is there to optimize our developer's time. So, uh, I really want them to focus on one thing, especially to basically dealing with the business logic other than, uh, anything else, uh, in this case, I'm talking about back to the, uh, context of the past in this case, I'm specifically is a cloud Foundry and, uh, he, and the stuff that actually we are enjoying in a, in a daily basis, uh, first about the kind of like, uh, uh, abstraction of a, uh, of the environment. You don't have to, uh, kind of like the call, anybody to spin up the environment who ran your application. All you need to do is specify how many, uh, uh, ran, uh, what is, uh, what is the, uh, disc requirement and then boom, uh, they go, you have your environment running, and then, uh, we don't have a way in this, in this case, it's more like a stutter was a shortcoming where we don't have the processor storage.
So, uh, uh, it turns out to be a very good kind of like a constraint in a sense, a economist of right design of the application to be one like, uh, towards the stimulus spectrum of it. So we can easily kind of spend one a month. So this, this scaling and another very, uh, key point, uh, for all of the, uh, achievements so far is about a control of the DNS layer, where we do have the capability to manipulate the route. And in this case, the, the, uh, the load balancing algorithm, and therefore this give us the capability to do blue-green deployment. So with all the 100 deployment that happens every month, uh, they all happen at like 2:00 PM during office hour. So we don't have, uh, to request a special kind of downtime. So, uh, coming back to our kind of a interesting tradition, right, we used to have, these are traditional, let's go for a party after each deployment. And, uh, and, uh, I don't think, uh, as a bank, we don't have that kind of money to, uh, host that many, uh, kind of like a parties after each and every deployment. And even, even if we do, uh, there's a physical limitation of, uh, people's belly rise, how much okay. You stuck stuffed in. So
This is literally like deployment was so rare and so special that used to throw a party every time. This was like two years ago, and it used to be at 1130 on a Saturday night, you know, green window. And now how many times a day on average are you going?
So, uh, in a day it's actually about four to five and sometimes actually catch with up to 20 something. So yeah, just think about 20 parties in a day.
And so, so this is around like 140 applications in production and 600 containers. So it's a sizeable chunk of, uh, of business,
Actually a special shutout for the concept of a bill packer. For those of you who are not very familiar with cloud Foundry, uh, he has a very interesting characteristic where the content us are built inside the platform instead of outside the platform. So, uh, as a developer, I don't really need to do anything, but to just Spotify, uh, here's my quote. And I run that on a platform for me. I don't have to specify the, uh, basic age. I don't have to specify the user I, you use ID. Uh, we, uh, and I don't spend, I don't need to specify the order. Let's say a GVM, uh, flags and, and, and, and the switches. And I don't even need to specify like which points, right. Exports, all of this data has been taken care of by the platform. So it's a huge time-saver for all of them.
Speaking of time, savers, I'm going to let you in on the, kind of behind the scenes speaking gig here, you know, Jean talked about, you know, you, sometimes you want to step over to the other side of the velvet rope. Well, our clock just came back on and we have 10 minutes left, so let's, let's crack on and go through. There's a few more things we want to talk quickly about things like incident management and legacy processes, like change management as well, because, you know, this is a bank, it's a traditional organization that's been around for a long time. And as you can imagine, kind of ITIL traditional processes are still kind of the norm through, through lots of technology. So let's talk about incident management. Let's, let's kind of quickly go through and kind of explain how incident management has improved as you've been on this journey.
No, I have another interesting story. I want to share with all of you. Uh, we used to have these reporting module, uh, under the various, uh, Santos circumstances and a one of the query war file out. Uh, and I will take way too long to kind of run the query and then it will eat up way too much memory. And, uh, as a result, uh, and the, the, this application actually crashed in production environment, uh, while in the, uh, in the old days, legacy, uh, world, uh, a crashing, running out of memory is going to be, uh, sometimes going to be triggering the whole world, right? So important in the whole ocean already. And, uh, uh, fortunately we are running multiple instances of this report, uh, uh, module. And, uh, no one was actually raising a incident report to us, and it's all is more life.
Our, uh, kind of platform operation teams start to look in us and saying that, Hey, guess what? There is one of your microservices start crashing now. Then we started like, uh, go in and take a look or what happened. So now we do have the option to say that, okay. So in case of Liza, uh, like, uh, catastrophic failures out of memory, stuff like this, the platform is there to help us to automatically revive the failure service. So then we don't have to kind of like immediately, uh, putting everybody into the war room and the figuring out what happened next. And we do have the capability to kind of say that, okay. So in this case, let's understand the problem first. And then, uh, we have the option to take, uh, actions whether immediately, or actually maybe can, can do that later. And another interesting thing, uh, uh, I observe, uh, is more like, uh, these days we, uh, we are talking about rolling back less than a less often. We are talking more about room for what's right? Anyway, we score, we are going to roll. So, uh, why, uh, roll backwards, let's roll forward, understand the problem and fix it, and then kind of like move on. Uh, so from the traditional kind of incident management point of view, it used to be a very reactive process. Now we tried, we try not that, not completely there yet. We try to be a little bit more proactive, preemptive to figure out as off the application running what's happening there.
And change management is another kind of topic that I think a lot of people in the room are probably still grappling with to some degree. And so kind of talk about some of the tactical things you've done around improving change management, even though you haven't quite got to where you want to yet.
So, uh, after all, as I said, we, I still at the bank, we are my team. At least my team is not here to break all the classes. Right. So kind of like, uh, accept a lot of, uh, risk control people. So we still follow them exactly the, uh, the process that had been set up the bank and, uh, uh, one of the key pinpoint for the team to raise the form, right? So, uh, going in production, uh, we have to raise actually several forms. And, uh, over the many years, there are many, many different fields being put up into the thumb. And, uh, just because you have the right information does not necessarily mean that you will know how to fill out the form. So, um, people actually like joking that it takes some serious art to, uh, kind of fill out the form in a sense that people won't want to reject it.
So the most of the error happens in a sense that you always happen in us, uh, in a way where people fill up the right information into the wrong field. So when we say, okay, so what can we do to, uh, uh, ease the pen? Right? So I ended up like, uh, all of the information that's needed to fill up the form is already there in our core repository. So then, uh, okay, that's easy. Let me just write that generator or some little automation script to automatically fill up the form in a sense that the right information with Philippine to the right field, and then the right sequence of events will happen exactly the way it needs to be. Then, uh, help us a lot to kind of like this, uh, by submitting the, the, uh, detention requests, uh, the chances of getting that change request to be rich projector is actually very, very low now. And, uh, for now preparing a change request to take on a moment, let go, um, the upfront, uh, and take takes about five minutes or 10 minutes to put out a single change request. And then we don't really have any rejection these days, animal. Yup.
So along with the kind of specific technical kind of principle or the process changes you've done in, in the ways of working, I found it really interesting when we talked about this before the, the, the culture changes that have happened and, and sort of three themes came out around your team typology or structure. And then what's happened to the team around things like sustainable pace and how engaged employees are. We've heard a lot already at this conference about this. And I, I think this is going to be an interesting area for you to share as well, duping.
So, uh, the team used to look like a more or less layers upon layers. We have the front end developer, we have backend developer, we have the so-called cross cut developer reach on debt, taken care off the framework, building part of it. And then, uh, uh, and a very interesting story. So when I first joined this program manager always came to me every day after a standup and asking me a single question, uh, for this feature, is this supposed to be a financial backend or the cross cut? So, uh, that kind of confusion, and it's actually a very unfortunate, then we say that, why do we still kind of like, uh, giving people titles in terms of what do they do? And then, uh, we look at not only about development teams, you look a little bit further, right? So then there'll be the self-correction between QA and developers and also the BAS.
So we, we, we think, I, at least actually right now, we are still having that conversation now, in a sense that why do we still give people a title? Um, maybe we should only give one title to all of the people that are relevant for the, uh, for building a software. We should call them the software maker, right? So in the making of the software, there are different activities that need to carry out. And then, uh, just because you are doing something does not mean that you have exclusive access to do only one activity, if you can do more than one, why not? So, uh, and then, uh, the other one about the, uh, automation again, right? So, uh, there used to be this kind of like invisible war. My job is done and throw over, uh, the one and then it's ops problem. And, uh, um, we have, these are so-called locker Gator setting up all up the log, actually piling up into a single, uh, lock mining system and lock system. Anytime there's a 500 error happening in the production, no matter what kind of life, 500 arrows, we could be Pfizer, or one of Pfizer or to anything that will be actually automatically feed back to the development team. So the operation will not actually need to know what's there and the exception happened in the production, but instead, uh, they are my should looking the operational right. Looking at a capacity planning, stuff like this.
How about sustainable pace? Like you you've shared with me when we talked about this, how, you know, if you're having developers work at almost every weekend and we working weekends and deploying on weekends as normal, the engagement is often it's often not a very enjoyable job. So what changes have you seen in that area? Yeah,
So, uh, when I first joined the bank, it used to be an accepted norm where people can shoot work out over time and sh is okay for you to come back Saturday, Sunday. So, um, we don't, we no longer do that in a sense that by throwing more time into the project, the ceiling, I mean, you can only scale the team out so much. The ceiling is actually very low. So, uh, instead of we changed the, uh, team building philosophy, uh, slightly in a sense can be look after the people and then transitively let the people look after application that's running lecture. Right? So, uh, in fact, um, most of the companies are actually very good. Uh, putting a lot of effort emphasis on the stocking at hand off the, of the day, we put a lot more actually emphasis on the ending of the day, right? So six 30, this is the time to cut off your work, and then you should just go home. You should go home. And every time if you figure someone, uh, actually, uh, I need to stay behind after six 30, that's a good opportunity for conversation. Why do people need to stay back? Uh, is there lack of support, lack of expertise or lack of our planning, that has to be a reason. So, uh, in terms of that, uh, I think people are getting actually much happier, which shows on the next slide.
And I really liked this policy that if you do have to work a weekend, that you get twice as much time off during a weekday at your choosing kind of when it's convenient after it, I think that's a real, you know, sort of positive step towards engagement, but let's kind of finish off here in the last minute or so, and talk about the improvements you've seen in kind of measurable engagement. Yeah.
So, uh, my first year a team engagement score and the survey came back to be only 66, or I, I taught myself actually not that bad. I can win the election by this. actually, it's not good enough at all. Right. So you look at our department, uh, Irish, it's about 80. So, uh, okay. So I say now mine, uh, we know our problem. Let's start now problem out. And then the next year came out to be 89 research, which was pretty good. And then two weeks ago, two weeks ago, I have my this year score. We should actually guess what it's actually about. It's actually exactly a hundred percent. So, so thank you.
well, I don't think I don't, I don't think a hundred percent means we are done now. There's a lot more than we can do. Right. So, but that does mean something where you can see the, uh, the progression from 66 to 89 all the way to a hundred. And, yup. That's another actually very interested in talking about culture. I do see a lot of culture in this conference. I do. I love it. And then, uh, um, not a lot, a lot of organization, they all say that we are learning organization, but where's the rest of test for it, right? So we run this launch on the learn series every Tuesday, a lunchtime, we get everybody together and then we buy the food for them. And then we, hopefully someone will put up with that topic and discuss, and then this topic can be very broad.
Uh, uh, we have topics about credit risk, uh, approving. And then we had talk about the latest and greatest technology. We either had a topic about black holes. And then, um, so, so the, the idea is, um, well, when we first started this series, it was really a pain to kind of, again, the speakers, because people just generally speaking 10 or to kind of speak in front of her audience to share something. And then, uh, these days we had to set out a lot, like a job scheduling stuff to make sure that people don't fight too much on desks, lot of speaking. Right. So that kind of sharing mindset is already there. So I believe a personal belief, like when you do have a learning architecture and you do have a learning culture, and then the sharing part is the automated test for whether you have passed out or not. Obviously again, I don't think I don't see that we have already condemned, so there's still a lot more to be done. So, uh, but yes, we, we think we are in the right trajectory now.
I think that's a perfect spot to, to wrap things up here. I just want to thank dot bang and let's give him a round of applause for coming and sharing. .