Las Vegas 2019

Experimentation and Evolution with Wardley Maps

This session explains how Cat has used Wardley Maps to inform her always evolving strategy for the modernization of Ticketmaster's core ticketing platform not only in terms of technical capabilities and architecture but also process maturity, organizational design, and more. Attendees will leave ready to start Wardley Mapping right away!


Cat Swetel

Engineering manager, Ticketmaster



Hello? I'm Kat. Well, thank you, Dominica. Um, yeah, everyone be sure to squeeze into the middle of, so that there's room. Okay. I am wearing sunglasses it's because I have a light sensitivity, and it's not because I think I am cool in any manner. Uh, this is my talk, so who's heard of Wardley maps before. Wow, everybody. Okay, cool. I mean, not everybody, but pretty much right on. So John Willis, I don't know, a few months ago, he's like, oh, you should come to a DevOps enterprise summit and talk about worldly maps. And uh, I said, no, thanks.


This should be interesting moving right along. I'm just going to tell you my story. Basically, I worked for a Ticketmaster. I manage the, um, core ticketing engine. So basically the transaction engine at Ticketmaster. And this is how I came to work at Ticketmaster. This guy that I used to work with in financial services also highly transactional. Of course, he called me up and he said, uh, do you want to work at Ticketmaster? And I was like, Ugh, I don't know about that, mark. And then he said the magic words, emulated VAX. And I said, where do I sign? I was extremely interested in that. Of course, being a millennial. We like all things vintage. Uh, but I think it has to do a lot with my personality, right? Uh, the bottom line, you don't need to worry about reading this Sol, explain it to you.


But, uh, basically Simon Wardley has this idea of pioneers, settlers, and town planners. And that's kind of how we like categorize ourselves. And the settlers is the people most likely to take a 3d printer man out of Legos and turn it into something that people actually want to buy. And that's totally me. And in the emulated VAX, I found my 3d printer made out of Legos. So, uh, yeah, you don't see people taking pictures. There's the link. You can check it all out there, Simon Wardley, if you don't know him, he's a strange old man that lives in a swamp and he believes that you should give away all of your intellectual property. There's no such thing as ownership. So there it is. Anyway, back to my sweet baby vacs, who, who knows sort of access, is there anyone who doesn't know? I went to this conference a few weeks ago and I was supposed to be there recruiting people.


And I don't know how many people I ran into who never heard of a fax before. So an old computer. That's what it is. So deck vacs anyway, moving right along. The first thing, when I say that I run an emulated backs and my platform is chock full of macro 32. The first question that people have for me is, is that VMs VMs is my favorite operating system ever. I loved it so much. No, it's not BMS. So I'm very sorry to disappoint you. I don't know VMs, but I appreciate what you have to say. I am sure it was the best operating system ever. I love what the people at Ticketmaster, uh, cause there's people that have been there since the very beginning, the guy that sits behind me has been at Ticketmaster's, uh, for 34 years. So the platform that I manage is only 43 years old.


So he's basically been there only 43 years old, whatever. Uh, he's basically been there since the beginning and they call it VMs, VAX made slow and uh, think it's so ridiculous and funny, right? Something that's that close to the middle would be slow, but it was for this specific use case, uh, it was slow. So in ticketing you need to have a bunch of transactions all at the same time. So it is fax made slow. And that's kind of, um, the first thing here is that they, in order to meet the needs of all of these tickets going on sale at once and you have to sell out a whole arena, there needed to be a custom operating system in order to do that in 1976. So it's not BMS, it's a custom operating system. It's in the code literally called ticket master operating system.


And now you may be asking yourself, where does she get parts for hardware backs. Right. That was the question you, um, yeah. Uh, I don't, I do not get parts for our hardware backs. It runs in custom emulation. So a custom VAX operating system running in custom emulation because that's how we roll, right? Yeah. There's one guy it's like, ah, this is hilarious. I have my own backstories. Uh, but that's basically it, right in order to meet the need of selling tickets in a computerized fashion, which is something completely outlandish in 1976, it required a custom operating system. So that was a decision that the founders and the original programmers made, right? We're not going to build an application that will run on BMS. We need to have a custom operating system in order to sell tickets and meet the needs of our customers. So it sounds ludicrous now, right?


To have a custom VAX operating system running and custom emulation. But at the time accustomed VAX operating system was the only choice, right? The only that made any sense at all, right. Is everyone familiar with this evolution access? Certainly some of you are because you said, you know, Wardley in the swamp. So it goes from Genesis, which this like a brand new thing. Holy smokes. What is that custom? Which I think I've already used the word custom, like a thousand times product and then commodity. And so that's like electricity or something like that, but we'll go into this more later. So you don't have to worry if you didn't catch all that right now, I have some news for you. It's an even start on a VAX. It started on a borrowed PDP 11, which is ridiculous. Am I right HUD? But I think that's a really important point in this story because of this.


So in 1976, first is everyone familiar with Mel Conway? Conway's law? I think I've heard Conway's law like a thousand times since I've been here. Anyone not familiar with Conway's law or Mel Conway. Uh, so basically Conway came up with this law that people love that says basically, uh, that our systems will reflect the communication patterns within our organization. Yes. Makes sense. Cool. Right on that was a long time ago and Conway kind of came up with that lot and he has actually continued to think post Conway's law. And this is something that I love from him and you can look it up. Uh, he's written a lot about it, but basically about the history of our very young industry and uh, when the Ticketmaster operating system was built, it was built at this very interesting time between the pre software economy and the software economy.


So it used to be, uh, that you would buy hardware and you would employ programmers to program, right? But there wasn't an idea of buying software. And when, when the original, when the founders were creating this operating system, it was right at the shift into the time when you would buy hardware and you would also buy software to run on that hardware. So it makes perfect sense that at that time they would think we should build an operating system, right? Yes. Software was just starting to become an idea. And so the way that they chose the first where they would install the first implementation of this operating system is that they found an ex mathematician who ran a record store. So he knew how to operate a computer because so many businesses didn't even have a computer there. So this operating system basically had to be everything for the business, right?


Because this was, you need a computer and you also need this operating system in order to sell tickets. Does that make sense to everyone? Does that not make sense to anyone? I can't see you anyway. So I don't even know why I'm asking that question. It's completely irrelevant, but don't, you all feel heard right now. Terrific. So this is what we end up with, like the custom operating system and the product, which is the hardware, right? You can go out and buy this PDP 11 EV anywhere. The important thing that I think when we map things on that evolution access, we can ask ourselves this question, how are we treating this component versus how is the rest of the industry treating this component?


So Pete and the other founders made this very conscious decision to say, yes, an operating system, BMS is a product that people will buy, right. But we realized that that will not be able to meet the needs of this market. And so we're making the choice to create a custom operating system. Yes. So part of plotting things along that evolution access is to make that, uh, apparent when you are treating things in a way that the rest of the industry is not treating them. So compute now, right? If you have each beautiful baby server cradled in your arms and you give it a name from your favorite, sci-fi, uh, you are treating compute in a way that is fundamentally different than the way the rest of the industry is treating compute. Yes. Yeah. Some people are like, oh, I feel too seeing right now. That's fun.


Uh, so the other way that I like to think about this access is in terms of being conspicuous, I think evolution is fine, but then you get people that think what's the difference between evolution and ubiquity. Yeah. Like does evolution mean that, uh, it's evolved? So everyone has it. And so for me, a useful frame is how conspicuous is this? So things that are in Genesis, people are just filled with awe and wonder, and don't really know how to handle the thing. Right. They're more just in shock that the thing exists and they're not even like, oh, how do I use that? How do I interact with it? Commodity? Do you notice electricity? No. You only notice it when it's broken. It only becomes conspicuous when it's not behaving in the way that you think it should be. Right. Does that make sense to everyone?


So that's the frame that I'm going to use throughout the rest of this. So it basically goes from no one even knows what the heck that thing is. Of course everyone notices it it's completely foreign to, uh, this thing I never even noticed until it's broken and it's basically never broken it. Acceptably, living in California. Nevermind. Moving right along. Okay. So this is a really high level map of the ecosystem here. So the customer need is I need to sell tickets right on. So you need content. You can't sell tickets to just do nothing, stare at an empty stage, right? You need actual tickets in order to have tickets. You need this operating system. You need a printer. Printers, printers are amazing in the, uh, um, when in 1976, a printer would be like, be most expensive thing that this business owned. So there's all sorts of interesting things in the code to optimize how the printers, someone's extremely excited about printers over there.


All right, me too. Uh, so basically you, you need these other things. You need reports. Like if we sell this many tickets, uh, what's the take for the venue or the client, what's the take for a Ticketmaster, all of these things, right? And they've made the choice to have this custom operating system, the printers and the compute, our product. Terrific. So what are we doing right now? Why did they call me up out of the blue and say, cat, you come work at Ticketmaster. It's because just like a lot of other businesses, Ticketmaster is currently going through this shift where it's shifting from a purely transactional model. Right. We sell tickets to a relationship based model. And lots of you are probably involved in this cause of keep hearing, hearing about it in the halls, right. Service pivot, anyone familiar with that language, right? Yeah.


People in blue suits love to sell us on that. Yeah. So basically now there's a shift from, uh, distinct transactions to maintaining a relationship with customers. Now our clients want to know, not does someone have a valid ticket to enter this event, but do, who is that person? Can I send them an offer afterwards to get a commemorative t-shirt are they going to want to go to the next tour? All of these things. So it's this persistent relationship across touch points and transactions. So what does that mean? If you're going to have relationships with people, people are unique. Yes, sure. They are. I promise you. Uh, so what we need to be able to do here is absorb the variety of the market, all of these different people, with the different ways that they'll interact with all of these different clients, all of these things.


Uh, so we need to be able to absorb all of that variety. And also as tastes change and new things happen, we need to absorb those perturbations in the market. Does anyone disagree with that statement that we should just treat people all the same tea repick so that's basically what they invited me in to do. Right. And now it's not, they it's us wonderful. So if we want to absorb all of this Friday, we have just one pot of money, right? One pot of money, regardless of what type of variety it is, money, time, energy, all of these things.


What if my platform is causing variety, right. If I have a handcrafted artismal deployment, that would be a source of variety. Yes. And I still just have one pot of money to absorb all of the variety from the market and also all so that's value added variety. And then all of the variety that I am, uh, enacting on myself. Yes. So I still just have one bucket of money. I, I don't get, it's not like if I cause myself more variety, I get extra money to take care of that. And I think this is just a thing, right? Why do organisms die?


Because the cost of maintenance to the organism outpaces, the metabolic rate of the organism. Yeah. What did we do with agile? We'd said we're going to learn really quickly. Right. And we're going to deploy frequently. We're going to get information. We're going to get feedback and we're going to incorporate that really quickly. How do we as technologists, metabolize information through code, do we not? Yeah. That's typically how we metabolize information. And after you deploy that code, what does it become? A maintenance costs. Yes. Yeah. Some of you, I guess, need time to come to terms with that. So we had the agile manifesto and we say, we're going to vastly increase. The metabolic rate of these organizations are going to metabolize information more quickly. And that's exactly what chick master did. And then what, we don't have the ability to balance that with the cost of maintenance. So then what do we see the rise of DevOps? Yes. And I think what dev ops in essence is a movement that says we have to balance the cost of metabolizing new information with the cost of, uh, maintenance within the organization. Right? Yeah. I think that's useful frame and you're not on stage, so it doesn't matter what you think.


All right. So we want to absorb this variety from the markets, but we have just our one bag of money and time and energy and all of this stuff to absorb all variety, any big cybernetics fans in the room, this is extremely niche, but I really appreciate you being here anyway. So we have this one pot to absorb all of that variety, whether it's value added variety, right? Increasing our metabolic rates, we're able to metabolize information more quickly, or it's this kind of maintenance variety where just weird things happen in the course of conducting business. So we have to keep that in mind, as we move from transactions to relationships. So we're going through also a shift in the way that we conduct business and we still have to deal with that paradigm and we have to not die, which I highly recommend for all you business minded folks.


So if we want to do that, if we want to decrease our investment in maintenance, and we want to increase our investment in metabolism, which is basically what that is, right. If we want to maintain relationships, then we need to increase our ability to metabolize new information. Then what we need to do to enable that variety in the higher order systems is we need to evolve and perhaps standardize the transactions, right? So that we can build relationships on top of the transactions. So basically we're faced with taking these things, which are all globbed up together, right. And pushing them all into product. And there are tons of customizations made for each installation of this platform and all of these things. So it sounds quite simple, but unfortunately it is not. So this is like, oh, high level, very cute math. This unfortunately is just a small zoom in of a, of a real map.


And if there are any folks from Miro, which they create this canvas, I think you only need like a free membership or something just saying, I'm out here spreading the good word. Uh, but there's this template it's very useful because it takes you through the creating of the maps. Step-by-step yeah. And it has all these tips, like actually embedded in the template. So this is a, for real Zs map and you see, uh, the operating system in the emulation there and the arrow that we need to move it into products. Yes. So that we can enable the higher order innovation, but all of that absorbing that variety on top of the transactions, very simple. Right. We just move, remove, whatever customizations are present in the code. And we're good to go. Yes. All right. So I guess I finished seven minutes early. No, unfortunately not the code turned out to be very simple.


It turns out to be very simple, to somewhat standardize that. Right? Yeah. So it's just code. We can do whatever we want with it. The things that were much more difficult are all these things in the bottom, which are the actual practices. And yeah, they're like kind of blurry because I didn't want to put anyone on blast, but, um, all of these practices on the bottom, which looks like a hot mess of spaghetti, they need to be evolved along with the technology. So if we evolve the technology to now, it's more of a standard offer, offering a product, but we are treating it in our practices still as custom, we have a bunch of custom practices for our standardized technology. Are we getting the benefit of having the standardized technology new? Unfortunately not so kind of my like unofficial rule is I kind of take whatever the furthest left thing is for the component.


And I treat them all as that. Right. So we would not be able to get those product benefits, the standardization, right. Enabling the higher order, uh, innovation and absorbing of the variety until we drove out the variety in the actual practices. So this is the whole big thing. Like I didn't just make this shit up. Wardley actually has a whole whatever table about it, which again, you can go look up, but we have to take those practices and take them from being this kind of like emerging, beautiful artismal thing and make them more, what are a set of good practices for doing this? It doesn't have to be best practices. We don't have to have driven out all of the variability, but we need to set up good practices for handling this. So the first thing is basically this bottom arrow and then the big, like crossed out part, which is basically everything needs to be in source control, no brainer.


Right? Yeah. But when you're dealing with a platform that's 43 years old, it's a bit more of a brainer than you might expect. Uh, so everything needs to be in source control. We don't copy anything from like one prod box to another or any of that kind of stuff. Uh, so also source control should just be an off the shelf thing whenever possible. Uh, so that's kind of the first thing there and you see the heads, that's obviously manual intervention. So we're trying to get rid of some of that. And then we also need testing. I don't know if you know this, but there's no macro 32 unit testing framework just who would've thunk it. Right. So we had to come up with this way of testing the macro 32, and there's also Pascal in there. And then, um, there's all the emulation and all that jazz.


I'm only 31 years old every day. I'm like, whoa. Anyway. So we came up with this testing framework so that we can have confidence when we deploy something to prod. Right. And we don't have to hand it over to testers to like, just dream up whatever testing hub. So you see, right. We're building a CGI pipeline here bit by bit, literally bit by bit it's faxed assembly. And then we move on to the deployment. Yes. So we build a CGI pipeline and that involved actually a bunch of things like, uh, we containerized our emulated VAX, which is the definitely the most ridiculous thing you'll hear all day.


I like every time I say the words out loud, it just feels ridiculous, but that's what we did. We containerized our emulated back so that we have a proper CIO pipeline. And then, um, moving on to the deployment, which originally made kind of home grow that, um, automation, but then, you know, moving on to tools. So at, through this process of mapping and remapping and Madmen, all of this stuff, we came up with these principles and the first one is show respect for history. So I was actually hired by Ticketmaster, right after a bunch of failed rewrites of the platform. If you have something that has had 43 years to get really good at its job and its original intent was, I want to be so customed for this thing, that I'm the very best thing. You're not going to be able to just send some people off in a realm and rewrite that.


Right. So, uh, first thing always show respect for history, go and find out why is this implemented the way it is. And do we now need to treat this in the same way? Do we still gain an advantage from this? Or is this source of variability that we could drive out? And then all of these other things, that's obviously the most important buy when possible this is just a thing, right? Like there, when you're out so far ahead of the rest of the technology industry, like they were in 1976, you have to build all of your own tools because tools don't exist. That's not a thing they exist. Now we have a whole exhibitor hall to, uh, show that. So by when possible visibility is a priority, so we want to make all of the things visible. Dominica could probably talk about that much more better.


Uh, so we have a mano repo which contains motto repo, I don't know, but, uh, it contains all of the code that is associated with this platform in any way. And then it also contains the code for the gateways to this platform, uh, skills duplication. We always favor that over speed. So if there's an opportunity to duplicate skill somewhere, we, we would do that rather than just try to get things out the door and then standardize then automate. So we're, we are conscious of the fact that, uh, premature automation just kind of like OSS buys things in a way that could be potentially harmful. So we want to make sure that we've actually standardized things, even if it takes a little longer and then automate it all because we want to increase the variety that we're able to absorb by driving out non-value-added sources of variety.


So what we're doing here is creating disfluency right? So not the way that you've always done things, but examining is that the right way to do things now. And I think that's really important and I want to get to this because, uh, our industry is really young. How do you measure the age of a baby in years? Probably not right. Weeks or months. Has someone ever said to you, this is five years old, I'm done with it. We are babies. So we have to learn that respect for history, our industry, and we, our babies, our industry is so young. So this, this developing this ability to create disfluency and developing this ability to show empathy for the past and respect for history, but also looking forward. That is the critical thing that we have to develop now. Not a super awesome, uh, ability to deploy everything in Kubernetes. And that's it.