Las Vegas 2018
How Experian is Accelerating Testing in a High-Risk Enterprise Environment
This talk by Tatiana Salazar, IT Manager at Experian, details how Experian transformed their testing process to uphold their stringent quality standards while accelerating delivery with DevOps.
The talk will cover what changes were required to prevent testing from throttling DevOps speed, how they started phasing in these changes across the complex global organization, and what’s next for expanding the scale and scope of their DevOps quality transformation.
Tatiana Salazar is IT manager at Experian leading their QA and Testing Center of Excellence (CoE).
TS
Tatiana Salazar
IT Manager, Experian
Transcript
I'm going to present to you, uh, about how we are accelerating testing in a high risk environment. Um, well, before we get into what's Experian, I wanna give you a little bit of an overview of myself. Um, I am the manager of the Testing Center of Excellence, and I am based out of the Costa Rica office in Experian. Um, this office has been, uh, it opened 10 years ago, and I've been working with Experian for this senior. So basically I'm one of the founder employees. Um, so fun fact, when we started at Experian Costa Rica, we were barely 80 employees today. We just got an email that was telling us that we should celebrate the thousand employee. So it has been quite a journey. Um, now let's, let's talk a little bit about experience. So if you are familiar with the company that I work for, probably this is, this format is what you are likely to understand, and it is a, a credit report. I know the, the, it looks like a credit report, although the information is not a credit report. Um, so yeah, Experian is, uh, one of the three main credit bureaus in the us. Uh, we have headquarters in the uk. But, uh, my point with all of this is that, uh, although you all related to credit reports, we are much more than that.
What else are we, we are an army of data scientists that work with a lot of data that come through to make your lives easier in terms of making decisions. So for example, if you wanna buy a used car, our automotive business unit will help you make that wisely decision because we will have all the information of what has, what, what has that car has gone through, uh, since the very beginning. We also help in, uh, insurance companies, um, to provide you health. We also, uh, have a marketing services, um, business unit, which, uh, helps companies adverse advertise targeted to your needs. And also when you go and you Google something and you see a small adver at the right of your, of your screen, and probably it's something that you bought a few weeks ago and you say like, why in this world would this know?
Yeah, it is probably experience <laugh> <laugh>. Um, so we operate in over 35 countries around the world. Um, so there's a lot of countries where maybe there's not big operations, but there is small sales offices. Um, now, uh, a little bit about the background of the, of the business unit that I work for. I, I'm part of the internal IT services. I am not part of all of those business units that I showed you. So I am not customer facing, but we work with all of those business units. And, uh, here in that picture is Barry Levison. He is my CIO. And a few years ago, he, uh, he announced that he had a very defined vision and mission for us, and it was to align ourselves into the DevOps journey, and that we were starting a DevOps transformation, and he was planning to turn all the waterfall into Agile.
Okay. A little bit of an overview of my team. So we are mostly all of us made based out of San Jose Costa Rica, which by the way, if you haven't been to Costa Rica, you should. It's Paradise <laugh>. And we have two team members in the Kuala Lumpur, uh, office at Malaysia. This is, this is my team. We are a team of 10 in Costa Rica. Um, we are a merge of two quality assurance teams, which gave services to different completely different products. So one of the teams tested, uh, corporate type of, uh, Veryan products such as Oracle and Salesforce.
And, um, the other team, uh, tested more internal type of, uh, web based tools and web services. So we, we did a merge of both, and we created the center of excellence from scratch. This, this transformation begun in, um, in let's say August last year. But, um, it, it was, it was a rough start. Um, because when you get two teams that work totally different, and then you tell them, we have to be agile, we have to move into DevOps, and probably half of the team doesn't even understand what we're talking about, it's a struggle. And they all wanna keep working only in their silos. So first thing was to make them feel as a whole team. And then we started the transformation. What do we do? What, what was the first thing that we did once we were already settled as a team, we looked for the better tool for testing automation that will be, uh, that, that will support our DevOps, DevOps transformation. And that tool was, uh, trace and Tuska. Um, so we, we've been very committed into this transformation and to build a very solid center of excellence.
Uh,
We've created different roles and, um, to this, we, we started our Tosca journey in February this year. And as of today, uh, I have the two first Renes, um, certified testing architects in my team. And I also have, and I'm very proud to say two TRE testing heroes winners in two different categories. So that's how committed we are, uh, to this transformation. Now, I'm going to tell you about the team that's based out of Bogota, Colombia. This team is not exactly my team, but it is the first team that partnered with me from a testing center of excellence perspective. So this guys came to me and told me, I need your help because we're working on a huge project and we need, we need to do fast testing, um, a little bit of the, of the team. But, um, important, uh, Experian bought the credit bureau in Columbia, and it is now in the middle of transforming it into experience standards. So it is a big project. It's a huge project. These guys work and save. So they do project increments every three months. They get together in the office in Bogota, Colombia, and all the different small teams scout teams, how we call them, get together. They talk about what they did, what they're going to be planning for the next three months, and then they, everybody goes back home and work on their normal agile teams, uh, within their, their scout teams.
So based out of this experience and this partnering that I did with this team is that I have a few points that, um, are kind of lessons learned about a DevOps journey. It's, this has been very interesting. It's our first experience and we've learned a lot from them as they've learned a lot from us too. So first, first point, define success. What defines success in a DevOps transformation? Um, I would say that the fact that our CIO is so committed to bringing this to reality, that we are all estimated as he's, and we are all walking hand in hand with him. We leave it by heart. We truly believe that this is what is best for us, and we all work every single day to make to, to get into DevOps.
It, it takes an enterprise. So what do I mean by this? Right now, we are, we are doing, uh, an approach that is from the inside to the outside, as I mentioned before. I work for the it, uh, business unit that gives, um, that gives all the services to the rest of the business units within experience. So this transformation that I'm talking to you about is from the inside, is we are the core and we wanna give the example to the rest of the Experian business units, the customer facing ones, but we need to do it with the best practices and the best standards. So once we're ready, we will start preaching out. We do already, but
<laugh>,
So, okay. Building up. Um, I would say here the most important thing is that we all need to understand what is exactly our, our, our place in the DevOps journey. In my case, um, I, I am in charge of all the testing automation and quality assurance. So I know exactly where I have to sit and who do, who do I need to collaborate with.
Um,
How did we start this, uh, this journey specifically within my testing team? Um, so we were stuck with a lot of waterfall projects, a hundred percent. So what did we do? We just grabbed everything that we had pending and all our BAU activities, and we created a backlog, which then we waited. Once it was waited, we started tackling the less weight, uh, type of activities. And we, we planned to use 80% of the capacity of the team so that we can leave some time left for, um, getting architect training or, you know, or other sort of emergencies. We also have an on-call rotation because we still do a little bit of operations. So we, we had to leave time for that. So that's how we started tackling. And then, and we started transforming our water projects into Agile, which, which I can tell you this is a good success story because right now we have our daily crumbs that nobody misses and everybody enjoys.
And
Though it was a difficult transition,
<laugh>,
Okay, second step that we did our definition of done, how, how can we begin to partner with different teams and tell them you have to follow our best practices if we don't even know whatever definition of done is. So before we started giving advice to any team, we decided that this is our definition of done. So the automated test is defined. The code has been reviewed by other peers. We've done at least three successful runs, uh, with distributed execution. I'm sorry for the typo there. Um, it's documented in Confluence so that if any other team member from the development team wants to use it for, um, for, for daily builds, they know what, what type of tests we have in our inventory. It's decommissioned in UFT, which is our old tool. And it's also integrated with j with Jenkins. 'cause we want continuous testing. So next key, I think I would say a, a success factor in the DevOps journey is having the right roles and having the right tools. So first of all, I'll give you the bit big picture of how our DevOps infrastructure looks like. That's what we have, that's what we use. I think it's pretty standard. It's what mostly all the companies use. Um,
In, like I said before, in terms of testing automation. Um, the chosen tool was, and you can ask me all the questions that you want from a testing perspective. If you have other questions about the rest, I can refer you to my colleagues though.
Bank of experts, what do I mean by bank of experts? And this, I don't wanna confuse you because in a DevOps, in a DevOps transformation, you have to think about the teams not being siloed and that they have to be collaborative, collaborative and like they should not be, you know, differentiated. But what we've learned is that although you may have your scout teams, as we call them, that are the mixed teams with the different roles within the same team, you need experts outside those teams that will consult and help them, um, with their expertise. So I I, I wanna quote one of the dev managers who is always telling us like, it is impossible. And it is definitely, um, it's not sustainable to think that in a, in a scout team, you will have all the expertise and all the knowledge you need experts that will help you. So that is exactly why the center of excellence, uh, was built. So we have the testing center of excellence, we have a DevOps center of excellence, which are all the experts with the tools that I showed you. And we also have released, released, uh, center of excellence. And it'll keep, they, this center of excellence will keep building. Um,
It is very important because from the feedback that I've gotten, uh, from scout teams is that normally when the testing center of excellence didn't exist, then you will have to fight for, to get a tool, which wasn't standard. So test this or te this, this group used this, the other group used this other tool, and now we have a recommendation and we have a set of experts that will guide you through the process of onboarding with the tools and onboarding with the best practices. Um, also very important is that when somebody needs support, it's faster rather than what we had before that nobody, like, it was nobody's land.
So finally, um, what, what, what, what do we give, uh, from a, from a center of excellence perspective is a double impact. 'cause we build standards. We, we build, uh, we have a tool, we have all the infrastructure, um, in order to support this transformation. And what people are getting is they are going through a smooth transformation, and they are not being as, they're not feeling it as painful as they were feeling it at the very beginning. Um, now very important to say, since this was announced, it hasn't been this great as a story as I'm telling. We have had a lot of stones that we've, uh, walked through. Um, so I think, I think that's it. And now, yeah, now we have questions.
So this is high risk.
You have some regulatory challenges in experience. Could you talk about that?
Okay, so for example, we went through these, uh, the, the European, um, regulation, um, I forgot, I just forget the GDPR. Yeah. So we had to accommodate like in three seconds how we were going to handle test data, for example, um, thankfully trace and is what was, is a great tool for that. Uh, task is a great tool for that. You have different approaches than you can use. So you can build your, um, your set of, of, um, synthetic data. You, you can also, so synthetic data is, um, it, it'll take your real data and, and it'll mask it up. Or you can start building your own set of, you know, in the database to try to emulate the database and, and create data that's dummy. But it's, it works with the tool that you're trying to test. Okay, the pencil there, there's different, there's several different, uh, and tools that, that we test. So in terms of a production environment, we had to mask that out completely. Uh, for example, for our EVS tools, that's completely max masked out. Nobody has access to production data, but us from a testing perspective, and well, we have to go like through signing contracts and things like that. Um, no, sorry. In production? No, just in our test environment. Production production data is totally covered. Nobody can see it.
No, I'm
Not holding, I'm just trying to see if you're doing it.
No, at least from an EBS perspective, no, there is other tools that don't have such, uh, um, uh, this type of, uh, difficult data so we can do more tests with, uh, within, within the production environment. And
Sorry, um, I was, you mentioned that you had to define, you had to come up with your definition of done Mm-Hmm, <affirmative> before you could go out and sort of teach this digital, this transformation among other areas of experience. Did the teams that you're teaching your methods to, do they get to define their own definition of done? Or does that become the definition of done?
My definition of done is what we call the standard. Um, so, so they should, they, my, my speech would be they have to follow. Now that doesn't mean that it has to fit everybody or that, um, because we're barely starting, there's something that we cannot improve. So it's welcome to discussion, but, um, it's pretty close. Like we really thought those, those terms, we really thought them through before we released it and presented in a place like this. Um, so yeah, those are, those are the, the definitions of standard. Like if you wanna board with the testing team and use our tools and follow our standards, and you wanna get our full support, you have to be on board with what we tell you. It's the best thing to do. If we permitted, uh, non standardization, then support will be a headache.
Two questions. Sorry, <laugh>. Um, so one of them is, how many tests do you currently have automated within Tosca? Mm-Hmm, <affirmative>. And the second one is, what were the key reasons why you decided to move away from UFT and go to Tosca?
Okay, I'm gonna, I'm gonna answer them on the other way round. I would say that the first, the first main reason why, and I, I actually have a slide for that.
<laugh>,
I knew this, this question was coming <laugh>. So first, first, uh, of all UFT, it's definitely not a good tool for supporting a DevOps transformation. It's not very agile and it is a pain to integrate it when with Jenkins, for example, whereas Tosca is super fast and with the new release, it's gonna be even faster.
Um,
Then, uh, people required extensive programming knowledge. So I'm not saying that TOSCA is, uh, like you can bring your little sister and she will be able to automate with tosca, but I can tell you is way less time consuming. Scripting is way too time consuming in terms of developing from scratch and maintaining. So that, that, that is good. That said, we, that said, we still need like the heads that will be super technical. We still need them, but we need less of those. And they are focused on infrastructure, architecture, you know, uh, customizations, higher profile things. Um, also, UFT tend to be very, uh, very slow. It will bring the machines down. So you are basically, you need a VM in order to be able to do stuff while you are running tests. And that is that, that's, you know, it's more expensive. And again, a little bit with, uh, the, the programming knowledge is, uh, the learning curve. The learning curve is higher. You have to know visual basic scripting in order to use UFT. Um, what was your other question? I'm sorry. <laugh>. Oh, okay. Okay. So right now in our Tosca repository, we have, we have two repositories, one for the Columbia team, one for my projects. And in my repository we have about 250 tests.
And in the Columbia team, we have about 170, because we are moving them out of UFT into tosca. And also we are building new tests. That's what we have right now.
Question.
Hey, could, could you touch on non-functional requirement testing, like performance and security? Have you guys integrated that to the pipeline yet?
Excellent question. And I forgot to mention that. What, during my presentation, so right now we are just using tosca, but in our roadmap, we, we will include flood, which is performance testing. Uh, we're still working on that with them. Um, security testing is something that still, uh, they have their own people and we don't, we don't, we don't mess with them. Um, <laugh>, so in our, in our pipeline, we have flood for performance, BI for BI testing, for big data. And we also are going to move out of A-L-M-H-B-A-L-M into Q test.
Uh, I have question, what kind of infrastructure your automation is running on? Uh, are you using OnPrem or use cloud services
For that? No, on-premises. Uh, right now, so we are in the middle of moving everything to cloud, but Experian, since we handle so sensitive data, we, we cannot just say today we're gonna use the cloud. We have to go through a, a very big security process. So right now they're on premises, but we should be moving into cloud eventually.
Yeah. I have a question. Uh, what percentage of your testers are actually still doing manual testing versus using tuska for automation?
That is the best part of my story. I have zero manual testing, believe it or not. How many, how many, uh, re in terms of resources that the, all, all the team, all the team is doing automation. Maybe. I would say that the two folks in KL would be the ones that are still not automate automating as much because, um, the, the, the Oracle EBS team consumes a lot of their time on gathering requirements and things like that, that are not test testing per se. I can't hear
They're not doing in sprint automation. Right? They're, it's a lag. Right.
Uh, so I've, this is you, you just touched my pain point. <laugh> <laugh>, we, we, we try, because there is a big time zone difference. So I, I have different scrum meetings with the Costa Rica team and then with the KL team, so I've been trying to onboard them, but it is quite a pain point. And the other problem is that, um, when, so the experts are in Costa Rica and it's difficult to support them, although we do breakout sessions very late at night, but it's still a pain point. Any more questions? Any last question? There's a question here. And, okay.
Do you factor in or handle data quality, data integrity in your testing? And if so, how do you
Yes. How do you deal with that? Okay. Um, specifically when we are moving, um, when we are transitioning from one thing, one big data handler to another. For example, from mainframe to Hadoop, we do a lot of data integrity.
Thank you.
Uh, anything database specifically related testing, keep database changes, uh, going together with application changes? Do you test directly database changes?
No.
Uh, application change?
Yeah. Yeah, yeah, yeah. Like we don't test at a database level. Okay. But if there are database changes, we would test at an application level.
Okay. Thank you.
At the end.
But let's just take this last question. Maybe, uh, quick question. You said, uh, in your current roadmap, you have a LM and you're trying to move to Q tests. Mm-Hmm, <affirmative>, what's your, uh, which tool does your, the agile team, the project team uses to track stories and other things?
For storing, uh, for managing test cases?
No, the stories.
Oh, no, the stories is Jira.
Jira, yeah. So why, why not use Jira for test cases
Also? Um, because Jira is not meant to do, like if you have manual tests, my team does not have a lot of manual tests, but other teams that, that you partner with will have, and JIRA's not exactly made to handle a manual test because it will not keep, uh, it will not keep like the, the registry of how many times it has passed per round, things like that. There's a lot of statistics that you actually need that will be in a test management tool. We good? Thank you. Thank you.