Building the World's Most Trusted Digital Gambling Brand Through Adherence to DevOps Principles

William Hill has a long history of adapting and innovating to provide an exciting and safe betting and gaming experience. The bonds of Brotherhood are at the heart of William Hill's brand campaign, and this resonates with the collective approach being taken to drive our technology forward.


In this talk, we will provide an insight into the challenges faced and how, through adherence to DevOp principles, tech will support the organisation's ambition to build the world’s most trusted digital gambling brand and a business with greater scale, more geographic diversity and higher profit margin.


This session presented by GitLab.

GS

Gareth Sephton

Infrastructure and Cloud Tooling Product Owner, William Hill

Transcript

00:00:08

Hello and welcome everybody. Hope you're all doing well. Staying safe, living your best lockdown life. Thanks for joining us today. I'm going to tell you a little bit about William Hill, uh, how we're using devil principles to take the company forward from good to great. My name is Gulf Sefton and on the infrastructure and cloud tooling products. I want to out William Hill. So let me start by telling you that little bit about who William Hill are a little bit about the history of the company and some of the major milestones and the last eight years. So William Hill is one of the world's leading betting and gaming companies employing over 16,000 people worldwide. Now started back in 1934. William Hill started taking started the company by taking bets of a phone and by mail. So right from the start, he was taking an innovative approach to reach and excite our customers. In 19 66, 5 years after legalization betting shops in UK, William Hill started to expand the company by acquiring existing outlets, and that acquisition would have become a major driver for the company's growth over the following years from then I continues to be part of our growth strategy to date

00:01:26

In 1998. So the launch of the online sports book super quickly followed in 2000 by the online casino. So regionally established in the UK, this was moved to Gibraltar in 2009, not move included the migration of service from our UK data centers to the data centers in Gibraltar, which is really interesting projects, really quick timescales. But I think that nobody at William Hill would want to repeat anytime soon. In 2012, we established William Hill us with a focus on retail and mobile gambling in Nevada. This gave William Hill a foot in the door for when in 2018, the Supreme court of the United States declared the professional and amateur sports protection act prosper on constitution. So this allowed states to start legalizing gambling and regulate sports betting. So William Hill was one of the first companies to be able to capitalize on the opportunity with access secure to 24 states already and expecting multiple and each state's comes with its own interpretation of the rules, which keeps us on our toes. 2016, we acquired the better than game of digital solutions company, grand parade in Krakow. So this allowed us to rapidly ramp up the scale of our development teams, bringing in that existing expertise gram per hate, hard, particularly around UX UI design. So it really enhanced our creative capabilities. Then in recently in February, 2019, we completed the acquisition Mr. Green, and with that's an expanded European footprint in a fast-growing online betting and gaming markets.

00:03:06

So whilst this just scratches the surface of what William, who was long established history, w what he does show is that even from day one, we've used innovation acquisition and making bold decisions to take the company forward. And also we are not an organization that was born in the cloud. In fact, we were born in the back streets of London. So this history provides valuable lessons that we can apply to date set company forward, but also creates a few interesting challenges that we need to address. So one of the things I didn't show in the previous slide, but possibly an important, most awful William Hill as well was I joined the company in 1999. So fresh from university to university, driven by a desire and a motivation to remove that minus sign from the start from the start of my bank

00:03:58

Now, without really knowing it at the time, and probably not being a conscious decision, as I reflect back on my role as I've had, I can mutton against that. The areas that dev ops is looking to address, and he shows up high in a hug this front row seat to some of the challenges that have grown up over the years and that we're trying to fix. So I started in tech. My tech gen is an application is on the list, which was a fancy word for a developer back in the 2000. For some reason, every single one of our tech jobs seem to have the word analyst in this. And we're Back in that role. I took an interest in our version control systems and the delivery life cycle we're using. I mean, for the version control systems, it was, it was an act of kindness. You know, these were not so much service. It was a desktop hiding in the corner, somewhere gathering dust. The only maintenance that was ever performed in it was when it actually got lit up accidentally got turned off either to flip the Hoover or someone to charge the phone.

00:04:57

We recognizing the importance of treating these tools where care was key. And this led me on my journey through their vault role show here into my current position, a tooling product owner. Now, along the way, my roles have involved. Most of those have involved working closely with development and with operations before, Mostly in nice and collaborative, we've got things done, but it be fair to say that along the way, the husband, the odd bit of frustration and conflict, if I have a lot of sympathy for anybody that does an environment monitor role, if it was anything like that, when I did, When did the observations that I've managed to make along this path and the kind of hope this isn't too controversial, but whether you work in development or operations, we all have a lot in common.

00:05:44

Everyone I know wants to do the best job they can, but they get frustrated when they have no control over completing a piece of work or that an issue they're working on has been caused by factors outside their control for many more roles, bridging that gap between development and operations teams was key to my success, but that came for just ensuring we're providing the right information, just opt in that interface between the two. So when I started to hear about dev ops and start to read up on dev ops, I was hooked from day one that I remember reading in the DevOps handbook. There's a paragraph early on. It says, imagine a world where product owners development, QA it operations, and emphasize work together not only to help each other, but also to ensure that the overall organization succeeds those words expressed better than I ever ever occurred.

00:06:38

What I wanted to achieve in my career and what I do get a buzz I was trying to fix. Now, I confess that Jerry, my career, I was probably never the most technical person in my team, but built up a reputation of being someone that we could go to, to get things fixed. Now that was mainly trying to get the best out of others, but fixing things myself. It's my fried in that right information, connecting those teams together. And for me, that's working together is how I see William Hill tuck and the challenges we face today. In fact, one of the things, if I'm ever asked to give a quick summary of devils, I'd like to summarize it is it's to stop asking people to do something for you, but ask them to show you how to do it. And if that's too difficult to show someone how to do it, make it easier, simplify it, automate it. So let me just delve a little bit further into some of the challenges we face it, William Hill, which I'm pretty certain will be not unique to us.

00:07:39

So, as I touched on before William Hill has been around for while now that time has built up a fair bit of technical architectural debt. If at times, to understand the dependencies between some of our older services, it's incredibly time-consuming. I thought some of the older products you realize that the one guy that might have known about it as retired, left the company, moved to the coast completely off the grid. So we pretty much hope and pray that that service will never break until it's replaced. Previous slide. I mentioned the positive side of regulation changes with what happened in the U S allowing the company to grow well. That's probably one of the exceptions. Most legal compliance changes tend to require significant work for us to remain compliant and to maintain our revenue income. So, for example, in the UK, following the outcome to try a new review in April, 2019, new regulations came into force limited machine stakes to a maximum two pong bets, so that significant parts of our revenue, but we don't sit there and moan about it. We just look at how we can innovate reshape the retail side of the business and move on, learn and adapt.

00:08:46

So with a large tech department, which is based in the UK crackle Balter, Molter under us, we need to look out for teams and working in silos. In fact, it's probably not just the geography here, teams in the same country, in the same office, on the same floor, even sat next to each other, cause sometimes end up working in that silo. And the silo teams may at times give the appearance of autonomy, but we also need to duplicate an effort and a coach that builds up the almost tribal in them mentality. And at times even unknown impact in another team's work With the tools we use in tuck tech, the siloed nature started to lead to a few issues as well. She had a lack of clarity around ownership and responsibilities of the tools and services we used. We had a fragmented to chain, which was impossible, if not very hard to support. I mean, there was bits of goodness, not chain together. We didn't have a strategic direction for the two were no, no feature roadmap. So when someone had a problem, it didn't know what to go to. That would end up fixing the issue locally or within their team. And that diverted their attention from the product delivery side.

00:10:02

We had no consistent patterns for integration between these different teams either. And the two providers centrally would probably not exactly what the user wanted. We built what we thought people needed, not what they asked for. It's really the challenges we've all faced recently. Coronavirus. It's definitely imparted to everyone, even the summit today being done virtually. So for William Hill and the industry wherein relies heavily on betting events for us to able to battle this. Obviously worldwide pandemic has caused a few headaches, but on a positive side, it's also how to unify an effect. So our teams have been almost to progress. We've redirected that effort into our gaming, into the areas of the business, where we'd still get that return on investment. And also with everyone working remotely, we've had to get better at creating those virtual things. So blood geographical split becoming less of an issue. Well, the biggest challenge that I've listed here, my view is keeping teams motivated One of these. If we get this right and have motivated people, they'll take care of the rest. Say my early days in line management roles, I was probably guilty fall into the trap of assuming that financial reward was a key motivator. So look into where I could get the guys in my team, the next pay rise or a bonus or some other financial reward.

00:11:24

But over the years, it's obviously it's become clear to me that that's not the biggest motivator and it, um, pink eloquently X highlights in his book drive it's Ptolemy, mastery and purpose are much more effective motivators.

00:11:43

So how will William Hill tried to give that purpose and help us address some of these challenges? Well, we have a clear company strategy in tech. We talked about DevOps, a lot, an army and a lot. It's a very opinionated topic. One of the push bucks or more, I get negative comments, we'll get all those conversations particularly around. Would it be, yeah, that sounds great. Really good stuff, but our business name will never buck with it. Well, I think that you can see from the company strategy, it's clear that whilst it might not say in big bold letters do devils, the company is behind this. This is not just a tech thing. This is an organizational thing. So we are going to too much of these JLL. We can see from the strategy where it will support the devil's journey. We talk about having collaborative and agile teams. There's operational efficiency through automation, scaling by selected core platform components And continuous innovation at the heart of this, or focused on giving our customers the best competitive offering.

00:12:52

Now let's company strategy is great. You would still get the murmurs at, down at the tech level. I thought, yeah, but what does that mean for me? And we listened to that, but tech operations and want it to break it down and make it very clear where our people suck within this. We wanted to make sure the purpose was really clear to them. So show here, there are six areas to the strategy, but purpose today, I'm going to mainly focus on public cloud people to change. Let me just give a quick nod to the service side of it. So from an operations team at William Hill came from the 24 7 365 days a year operation going. We consider ourselves to be pretty good at keeping the lights on for our revenue generating systems, always that our customers use. However, is part of the strategy we realized we needed to look into and make sure we're providing the same level of service to our colleagues internally at William Hill, The public cloud. So William Hill, our and a migration path from our physical data centers into the cloud. And at the moment specifically, AWS, Now that would take a whole nother talk. And I had a lot of time to go for all the benefits of the move to public cloud is intended to bring for us, but I'm just going to keep it short here.

00:14:10

I think one of the key things there is it's acting as a catalyst for change for the over as we need to address. It's given us that fresh playing field so away from our on-prem data centers and the challenges and the tech that we build up there, including the tools and the products we use to manage those on-prem physical data centers. So it means we can take that step back and look at solutions without constraints, but making sure that we still understand the lessons that we've learnt from running those tools and from running those physical data centers, the technology might change, but the knowledge and experience we have remains Within two chain. We acknowledged that to win, to be market leader. We have to get our ideas into the hands of the customers before our competition. So here, this is tech ops looking at fit for purpose to achieve that lets our product teams focus on delighting. Our customers Thought process. There took his down, uh, looking at Gartner's pace, Larry model, and essentially looking at the two chain as a system of record, which does say whilst it's incredibly important for William Hill. It's not what our customers want. It's not differentiating for our customers and that's the bit we want our product teams to be able to focus on.

00:15:28

And it heart of our strategy of people not to be corny, not being cliche. They seriously recognizing that by having talented, motivated, and engaged people they'll contribute to a high-performing organization, but to move forward, we have to provide training and development plans so they can provide the mastery of their craft. But alongside that, there's no point training if we're not given the opportunity to put those skills to use. So in fact, we do monthly say this and in our monthly surveys, we're seeing more and more feedback that people want meaningful work. If, if we're given the training and no opportunity that just something to substitute bench forever. So we've talked a little bit about some of the challenges and our strategy. It's probably good to understand a little bit. How are we structured within tech? So William Hill is divided into divisions, subdivisions Org. I'm showing the subdivisions, same places. They can split it further. It's involved focus, product teams. And these are spread within the subdivision across the UK to volt a crackle or leave a geographical tackles. We have Now not all teams have the exact same makeup, but essentially they are a mix of developers, testers, engineers, project managers, scrum masters, product owners, things you would expect in a delivery development team. Now operations provide support and service to all these areas, which to some extent, puts us in a very privileged position.

00:17:02

We can take lessons learned from supporting, wanting to provide the help and guidance to another setting in the middle. Again, historically probably been guilty, guilty of doing this based on what we thought was best for teams pushing these standards out to everyone. And in my experience, actually, you just start talking standard. You're just going to find yourself in argument discussion of this discussion, new standard after new standard, It's a tricky word and I've never really seen someone get a successful standard, pushed out. So with people at the heart of our strategy, we thought it best to go out and talk to the people. So we started the conversations with our heads of tech, for the divisions and we discussed what was to be above and below the line. Now that wasn't a new bet opportunity, new fancy gay mountain sports book. It was about what it made sense for us to look after centrally provided that service and what we wanted our delivery teams to be focused on what was really going to be a differentiator for them to excite our customers. Now, the overwhelming response from everyone was pretty much to put anything over an application development below the line or in our key cases on the line. So on the line highlighting where it was important that we were collaborative. It wasn't what we thought. It wasn't just what the dev teams thought. It was a collaborative effort

00:18:37

Following on from that, we then took a lethal design thinking approach. Again, design thinking is probably a whole different presentation, but a key to that design thinking approach. And the first stage is to gain an empathetic view and understanding our colleagues and their problems. So this was a crucial step for us to set aside our own assumptions and gain real insight into the users and their needs. We had to get to know our colleagues. What did they like? What did they dislike? What did they want? Or what did they feel would make their life better and easier at work? Now this took over 30 hours of interviews and the majority were held face-to-face, but it wasn't just because we want it to have a jolly out to crack out. We saw it as an important that we, we went and met the people.

00:19:23

We didn't question why from our side, we didn't have any agenda. If we didn't go out there with we like this, therefore we're going to push it. We focused on their individual likes and dislikes. In fact, before we went out, we even talked about avoiding the question why, and we've got a much better response. If we asked, what is it you like about something? What is it you dislike about something rather than asking why they liked it or why they disliked it. That's going to why almost puts it like defensive. It feels like you're asking them or challenging their point of view. So as interviews allowed us to build up a list of common themes, common challenges that helps us look at where we needed to folk our initial focus, our initial efforts. And she said, this was The common response is coming back. As they wanted things to be centrally managed, but stable and reliable tooling. We wanted a low barrier to entry with better training and documentation, potentially one it control over their pipelines, but not necessarily the responsibility for the underlying infrastructure. We wanted fewer tools to consolidate all the tools we had available.

00:20:38

They won't a Toni, a self service level. So again, not having to understand everything that happens under the hood, but even to do their piece of the puzzle And then we'll get better visibility. We wanted to see what was happening when our releases, where things were. And also we wanted to have that clear ownership. It wasn't to look for someone to blame. It was look to someone. We could help that we can move things on with where we can put our requests, where we can work together to make the best tooling for William Hill

00:21:13

When the tech ops and the to shame piece that gives us our goal, which is to create a centralized platform, tooling service that any development team can use to become more productive from getting production like environments, central lab deployment pipelines, bringing the monitoring in, or even the, the security tools that our award-winning InfoSec team have to offer, bringing it all together by dev team. That's not just our development teams and the channels. That's everyone, our network teams, monitoring infrastructure. We're all pushing to that as code approach. Therefore this platform has got to be right forever. The key for our success here, we see built in part as a platform, not individual products or tools. It's a platform where we bring the best of all of William Hill's tech teams have to offer Again, that I guess some of the benefits of that two chain platform, we can provide capabilities that we'd agreed earlier, made Vermont value being provided centrally, but we can also tap into the skills and experience across all of our tech teams. We don't need to have one team that knows how to do everything. We integrate services via API APIs, or even getting people to contribute to the code base,

00:22:32

The best practice and safeguards. We can build that into the tooling and where we can need to enforce it. We can do so. So that not only contributes to improving our delivery timescales, but it also gives teams confidence that we can trust the products we're delivering to our customers. Now, through those interviews, we discovered that of all the services we were using, all the tooling we were using internally at William Hill that were being used around the CIC area, particularly get luck was the one that everyone liked Say light, no one had any real negative opinions, which fair for our developers and engineers is high praise indeed. So we'd been using GitLab as ourselves control repository for over five, six years. And the license was up for renewal in 2019. This is part of that renewal. We need to start looking at how we can maximize the return on investment we had with that product. Should we start to look at, we'll get lab to see up starts to give us that single platform so we could use that to power everything we're doing and eventually essentially avoided any unnecessary complex compatibility with over tooling.

00:23:48

Get lab has the extensive documentation along with a strong community, she starts to address the onboarding aspect, the low barrier to entry. It provided teams with ownership of that pipeline without the support overheads and even to the SaaS version to get love.com even takes away the operational overhead from the team on lighting that was looking after the current self hosted version mentioned it was wider light within the organization. So having that buy-in from the start is incredibly valuable. They started to reduce our tooling footprint, so where we're using Jenkins Concourse or our own bespoke, then we could start to consolidate that into one comes with reporting and dashboarding things that we'd been having to piece together from our own external services And keeping it all of a key thing for us. It's part are varied use cases. So we are moving to public cloud, but we're going to be in a hybrid world quite well. In fact, there's a fair chance. We will never be able to be fully cloud of regulations where bats need to be stuck. I mean, we will always have some on-premise physical data center services. This is safe to elicit.

00:25:09

If you looked at that continuous integration delivery and deployment are the holy grail of the devil sushi. Being able to take safely, take that code commit through the various quality gates are put into the customer's hands with no humans reduction is the dream. So we're looking to do this by taking input capabilities, many areas and changing this together. We recognize we're not going to be at a deliverable on day one. It's going to be an incremental approach And we're going to really have to work with the expertise within William Hill. Again, this is where the platform approach API mentality was microservice contract based testing starts to pull all this together and it gives us that flexibility and control. We need to be able to adapt Again. You're looking to get the most of our relationship with get lab is probably why we've decided and agreed across the whole of tech to get love. Runners of get love itself. We'll provide the platform or the engine behind the platform.

00:26:09

Similarly, there is, we help you in looking at so we realized that we've got big infrastructure operations team with lots of experience, move to AWS. We've got development teams who know how to the code. So we want to deliver the infrastructure for then into their channel accounts from the sensor and creating a pipeline to do so. So ensuring that he's kept in all the goodness, we get out to CIC D. So it includes a layer that enables deployment of varying configuration. So we can still give the development teams control. They need before we let CA approach, right from the start, we're starting to run test culture security checks and deploying that into our own testing central tech ops text environment versus where we can do functional tests based on the contract of what we'd agreed. We are delivering to the channels. This is using get loved on, is built in AWS. The code is all stored in get lab.com. So we're really embracing that cloud first mentality, Free desk we're working closely, we'd get lab and AWS processing. It's an only look into the internal experience, but where we can maximize the relationships with our vendors and third parties. Part of the aspiration here is to provide a library or shopping list of proven infrastructure builds, taking that open source approach. So anyone can contribute to it and encouraging that reuse of best practice rather than dictating standards.

00:27:38

So let's summarize a few things here. So there are different paths along the devil's journey. Thankfully for me, William Hill attacking the approach, but providing central tooling to enable devils it's a tyranny products on it. It was probably a good job choice they made for me. It's not to say it's the right way. It's not to say it's the way we'd even recommend to others. But for us, it's an approach that capitalizes on the strengths we have internally, But we do accept like change the new norm. If it doesn't work, we'll adapt. We'll change. We'll move on. But we incrementally cut those changes now and we'll listen to the feedback We want. Autonomy mastery and purpose clearly are excellent motivators, but our experience says, we need to use this with caution too much. Autonomy contributed to silos. The master is great, but I expected everyone to be tech intact, to be massive. Everything is either going to be an impossible challenge or very expensive. And again, check out that purpose is understood. If it's not, it's not going to have the desired effect. You may have all these fancy graphics, fancy slides, but if the people don't understand it, the people don't see where they fit into it. It doesn't give that purpose.

00:28:51

I think everything is use your lifelines. Talk to people. Don't feel you have to go it alone, Accepting that we don't have all the answers and leveraging internal or external expertise will take us from good to great That as mentioned this tech, this isn't just internally. So with the third parties that we work with looking to establish working relationships, not just buying products off the shelf, I think as a final word, if we are to build the world's most trusted digital gambling brand, and we need the people that work at William Hill to believe in it and to do that, we need to nurture a culture of trust within our day-to-day work. For me, that is going to be key to our successful DevOps adoption. Thank you very much for listening.