Las Vegas 2018

Crowdsourcing Technology Governance

In the past, architecture choices, ensuring quality, and security and compliance technologies were the ways to enforce governance through tollgates, centralized approvals, and silos. Three years ago we started recommend_tech, which uses crowdsourcing to make technology choices at Target.

Dan Cundiff (@pmotch) is a Principal Engineer at Target Corporation. For the last 5 years he worked on Today he's building Target's IoT platform. Dan is an advocate for the use of and contribution to open source software at Target and was one of the founding members of Target's open source office. Previously Dan has spoken at GitHub Universe, Linux Foundation Open Source Leadership Summit, API Strategy Conference, Cassandra Summit, Jenkins User Conference, RSA Conference, Gartner AADI, and more.

Levi Geinert (@levi_online) is a Director of Engineering in the Target Dojo but recently transitioned from Principal Engineer and is responsible for supporting, and refining the agile and DevOps software engineering practices and patterns. He advocates for Developers, and also helps to operate the Target Dojo which has helped drive DevOps adoption across Target. Previously Levi has spoken at ChefConf, #That Conference, multiple smaller venues and internal Target conferences.

Lucas Rettig (@LukeRetigg) is a Principal Product Owner at Target Corporation. He currently owns Target's Item-Location service, the Item and Location Grouping service, the GraphQL platform, and Autobahn, a data movement and caching platform. Prior, he was the Product Director of Supply Chain at Infor Retail and before that spent several years in engineering at Target. Luke has a deep-rooted passion for Product Management and is an advocate for engineering enablement.


Dan Cundiff

Principal Engineer, Target


Levi Geinert

Director of Engineering, Target


Lucas Rettig

Principal Product Owner, Target



Good morning. How's everyone doing? Good. I'm very excited to be here today to talk about something that's very near and dear to me. Um, a little bit about me. Uh, I'm a director of engineering in the Target Dojo, where we help teams with guidance on engineering choices as well as changing the way they work. I'm gonna talk a little bit about how we move from a governance model to a guidance model, and how that's helped support the empowerment of our engineers and our business to make, um, smart decisions, uh, take smart risks and be empowered. We're gonna talk about a real example of how we've implemented this by crowdsourcing some technology guidance. And then we're gonna talk about how that helps us create awesome products.


A little bit about Target. We're big. We have a lot of stores we give back to our community in many ways. The interesting thing to note here is that we have a lot of team members, over 300 product teams, as well as 4,500 people plus in technology. This is why guidance is important to us. A little bit about our evolution. We started our journey with DevOps by rebuilding our engineering culture. Uh, Ross Clanton and Heather Mickman presented on this in 2015. Feel free to check that out at some point. But it was grounded in empowerment of our engineers, DevOps, uh, agile and so forth. This had a very big impact on our engineering capabilities at Target.


DevOps and automation were key components of that. Again, um, after the DevOps movement, if you will, we got into changing the way we work. The Dojo has supported this. Uh, many of our business today is from business teams that want to change the way they work. They're seeing how our engineering teams are empowered and taking risks, and they're being supported in doing so, and they want, they have interest in that. They also want to understand how to interact better with their engineering teams that are operating with Agile Scrum or Kanban. Today, many of our challenges are focused on business challenges and not just DevOps. And finally, we be, we realized that to have great products, we needed to have our customer at the table. Why are we talking about this at a DevOps conference? I'll show you a tweet. That's one of my favorites. Caitlyn mentioned that it sucks when you have an idea for a product, but it's much worse when you build the wrong product, right? And so aligning on our customers, internal or external, was very important for us, again, to build those awesome products.


All right, so we talked about governance and why that was something that we wanted to get out of the way. Um, we don't want to require teams on how, how they should work, how they should operate, what tools they should use, because when we empower them and we make them accountable, then they choose the right tools. We know this because we have research to support it nowadays. Thank you, Dr. Forsgren. Um, and I would extend this to not just tools, but languages, frameworks, and all the choices that teams need to make to accomplish and build those awesome products.


Now, it's great to remove governance, but what happens when you don't provide guidance? We know that it can get very noisy in a tech landscape. If you do any searching of images around tool choice, it can be overwhelming. Teams that are given no guidance have to consider every possible option. So guidance allows us to streamline and reduce the friction required to make those choices. Now, what does, what does gui, excuse me, what does guidance look like? Um, there's a few things we'll show you on that, but I'll talk first about some of the principles. We think that guidance should be accessible, available to anyone to contribute. Not a specific leader, but anyone at any level, we think it should be transparent. We hope that anyone can view it. It really, it's not a great situation if the engineers aren't able to find out what are the approved patterns or what are the, the best tools that they should be using.


It needs to be able to change. We know that the stuff changes very quickly in our industry. Tools, languages, frameworks, et cetera. And finally, cultural. Just like DevOps, our guidance model needs to be cultural. The people that are providing the guidance need to also support and promote other leaders to step up and help guide. In many cases, our Dojo is not the expert, but we maybe have an awareness of what people are doing, what patterns are being used. We'll bring in those teams to actually help the teams that are considering a, a specific technology or framework.


I'll also mention the Dojo. You may have heard about it. It's a big part of our culture. It helps us, again, empower teams, support them to make smart or take smart risks. I wanna announce that we've recently had over 350 engagements in our Dojo. 250 of those are 30 day challenges. We've had over 180 tours of our dojo from many of your companies, as well as internal teams. Again, trying to help guide people towards positive change through empowerment. I'm gonna pass the, the baton to Dan, where he's gonna give you an example of how we crowdsource our technology guidance. Thank you.


Thanks, Levi. Um, so what I want to, um, so I'm Dan Kund. I'm a principal engineer at, uh, target Corporation. What I, today I wanna show you is, um, is actually how we, um, take what, uh, Levi talked about this, uh, idea of empowering engineers. And, uh, the idea of going from governance guidance. I wanna show you a real world example, uh, of that in action. Um, but first I want to talk about, um, to set the stage for what we created. I have to tell you how we did things before, and maybe this is actually a familiar process at your company, so you may be able to empathize with this. Uh, before we had a centralized group that would choose technologies for us, we called it an architectural review board, an arb. And what's difficult about these centralized groups is that they have to make decisions for, uh, many different product teams.


And the larger the company you have, the more likelihood that they're probably not gonna be, be able to pick the technology that works, uh, for everyone or be able to take into account all the different product teams unique things they need. And so, um, you know, that, that becomes an issue. The other one is, is that if you are, uh, someone who wants to be able to influence that ARB or that centralized group that's doing that governance, it's probably gonna be pretty hard to do. Um, because, you know, you have to go through their process. They probably meet on a regular basis. They have some form you gotta fill out and things like that. So these things are, are problems. What we did is, um, and this is about four years ago, whenever engineers were starting to feel empowered, um, that we said, there was a couple of us, we got together and we're like, Hey, this can be done better.


Let's try and disrupt this. Can we do this in a different way? And what we did is we just created actually a, a repo in GitHub that was a simple list of technology choices. And we wanted to use that as a stage to be able to make a couple of new disruptive choices and see if this thing could take off. So we did. We just created this repo, and I, I actually have to show it to you right away here. So, um, this is, this is the repo, and it used to be just a single file in here. It's been broken up into different categories. But what you see here in this list is a different set of technology categories in here. Uh, collaboration tools, uh, application frameworks, caching, data stores, et cetera. Uh, and you can go into any one of these files here.


Let's take a look at, um, the data stores ones. There's typically some front matter on the file, just like an introduction, some motivations for why the people who maintain this file, uh, why they make the choices that they do. Uh, but if you're a developer and you're coming here saying, Hey, what choices can I make? Or how can I influence this document? Um, they usually just scroll down a little bit further and you'll see, uh, different sections. And again, this is in the data stores one, uh, they, they'll actually see what the technologies are and, uh, what their status is. So let's say you're looking for a relational database store and you're wanting to see what your options are. Um, Postgres is a choice here. And the other fields to the, to the right, the, the use case and the other stuff that's just more like metadata.


But the real importance one is the status. So there's recommended, there's, uh, limited use and do not use. And you're gonna be tempted if you try to do this yourself. You're gonna wanna have all kinds of other statuses, but keep it simple. Um, recommended means you can use it. If it's limited use, there's probably some reason why it doesn't. It, it has some, maybe there's some, uh, licensing constraints to it cost, maybe it doesn't implement well in certain environments inside of Target. And then the, the last ones do not use, don't use this. You shouldn't use that here at Target. In fact, we would put things on this list that maybe we've never used at Target, um, and we just wanna put in here and make sure someone doesn't think they can go choose it. And we'll explain why. We'll say why it's not a good fit.


So let's talk about the motivations for, um, why we did this and this. There's a theme here. I think whenever using GitHub or you know, Bitbucket or GitLab, you want your developers already day-to-Day in, uh, GitHub. They're writing code, they're doing social coding. This is where they're out already. So if you want to do something like this and you want it, you've empowered your engineers, put it somewhere where they're already working, put it in version control. There's a lot of benefits to that. So people that are familiar with the pull request workflow, um, this is, I have a change and I'm requesting that you pull in my work. Uh, anybody at Target, uh, you can be a junior engineer, a more senior person can open up a pull request to any of these technology categories and suggest a change. We've had junior engineers do this who, uh, they just maybe have learned something new and they wanna bring it forward and say, Hey, can we start using this?


I think this is a gap in our technology set. Uh, some people have come forward and had to make very large changes, like, we need to change an operating system or a much, uh, bigger, uh, you know, technology choice. That requires a pretty big discussion. So you can do this through pool request, and it's a perfect workflow to this for this. Everyone can comment and, uh, have a discussion on it. And when it gets merged, then that's it. The, the technology choice is final. And I'll show an example of that here in just a minute. Another nice benefit is if you're putting it in Git, um, you can go to any one of these markdown files, uh, see, uh, what, uh, and run, get blame on it and see why did it change. And if you, for this example up here, this is for Kafka Streams, I can see the actual commit that was done.


I can go back and see the pull requests and any associated issues with it. And for posterity, I have everything captured. Understand, say a year later, why did we make this choice? So it's all just right there in history and get, right. So here's an example. Pull request. Uh, this is a person, his name's Dan Moss. He is actually, uh, suggesting a new database technology. You can see his justification up there. He's made a case for it, a sort of pitch. And it started a discussion. And this was only a couple days old and had already had a lot of comments on it. So this is exactly how it takes place. Also, take note too, there will be people who, and there's a whole spectrum of people, right? Who will chime in. There are some who will be very motivated to make comments. Uh, there are also some people who just want to listen, but they'll react with emojis. So you can see that there's three thumbs up. There are some comments in some of these pull requests where a lot of people will add all kinds of different input, whether it's comments or different emoji to kind of show their support or which way they're leaning. And take note of the label up there, the CIO form inputs. I'll talk about that in here in a minute. It'll be some important here in a second.


So because it's in gi then we can do all kinds of other things too, right? Just like you are, uh, working on a project and you're putting code in there and it can trigger workflows and stuff like that. It's nice to point, it's good to point out that there are other things that you can do, um, uh, is triggering having an interface into a process like this. You can have the alerts sent to Slack. You can have, uh, uh, some sort of job kickoff, which maybe renders these in PDFs, which maybe is more digestible for certain people. Um, the good thing is that you can stitch all kinds of, uh, alerts and nice things to workflows.


This important thing I I wanna bring up is having seen this exist for four years now, um, resist, uh, complexity. There is gonna be a real temptation, uh, to say, Hey, we could add more status categories. Um, we could, uh, do this process to this. We could really require all these additional steps and things like that. Don't do that because at the end of the day, uh, it is developers who are coming here and, and recommending these changes. And if you make it too difficult to do, they're not gonna come here and do this. So, absolutely resistant. I mean, I think the only changes we did is we used to have that single large file. It was nice to just do control F on, but it got pretty cumbersome to vertically scroll so much. So we did split it up. And then there's been some other things that, helpful things that people have added, like decorating some of these pages with motivation and, uh, uh, the reasons why we've made these choices and just more clarity on things. But other than that, it's always stayed pretty simple. It's just to get repo. You can open up a pull request, you make your pitch, and if it get merged, it goes.


In the earliest days, it was volunteer led. It was the, uh, you know, there's a set of influencers probably in, in all companies, and there certainly was here when we started this. Um, we, it's volunteer led, uh, but make sure that, um, you know, over time you don't, uh, let, I would actually keep it fresh, keep new people coming into it. You don't want someone to have a real dominant voice. Uh, if someone's gonna make a suggest a change to this, you don't wanna feel like they're coming into a conversation with bias or anything like that. So always keep, uh, uh, fresh new people coming into this and helping groom and stuff like that. Those people will probably emerge and come forward and they'll start doing it anyways. And you'll just kind of embrace 'em and be like, Hey, thanks for helping us maintain the vitality of this community and this, this process we have.


Okay, so there are some changes, um, that have a very, uh, steep cost associated with them. Perhaps maybe the switching cost of moving from one database to another is high. Uh, maybe there's just a big disagreement. It can't be resolved in the poll request commentary. Um, there is another part of this process where we can actually take it all the way up to the CIO. So our CIO has a near quarterly setting where if you've made a pull request on something and it requires the CIO level input, um, you can actually take your pitch. If you're a junior engineer, whoever you are, you can take it to this setting. And the CIO, his drs and principal engineers and other influencers will be in this room, and you can make your pitch for that change. And in, in most cases, they're just, you're kind of rehashing probably existing conversation.


There's a set of, you know, unbiased, um, you know, criteria that that is typically followed in that setting. Um, and you're just making your pitch. Just like if you were doing a startup or something like that. You're making your pitch for why we should make this change. And, um, the nice thing is, is that, um, not only can people come to that setting and get that type of attention if it's an important thing, it's nice to know also that the trust goes the other way. That our leaders at the top entrust our empowered engineers to let this guidance based process actually work. They trust this process, and they already take into consideration all the input that was already had with the, the pull request conversation beforehand. So there's a lot of trust built into what's, what dialogue has happened so far. Um, and you know, the, they're taking this engineer's input into this, this CIO level setting.


These next two slides here are, uh, just, um, you know, this is on the repo. You can see how many watchers there are getting alerts for things starred and forked. Um, this is also just another nice thing that GitHub shows you, uh, to give you an understanding of, you know, is the community help, uh, healthy, how many people are actually making changes and things like that. What, what I really wanna underscore here though is, um, that just because it's a, there, it's not centralized with governance, that it's guidance. That doesn't mean you shouldn't measure it. You should measure to make sure everything is healthy and that things are on track, uh, and that there's some good vitality in the community. So you should still do that. So, uh, next, uh, Luke is gonna talk about, um, everything from a product owner's view.




Alright. I'm Luke Greig. I'm a principal product owner. I own a couple products, um, at Target. So I have our, our enterprise GraphQL platform that I own. Um, one of our enterprise caching platforms that, that I own. And then I have, um, our core item location data, data service that we provide out to the enterprise. So I'm gonna talk to you guys today about just a view from my lens on why DevOps, and particularly this, this theme of engineering empowerment and then also product team empowerment. Some of the things Levi touched on really hit home and really drive awesome products. So before we get started on that, I'm gonna do a little survey here. So how many people in the room, show of hands, have been on a team or are currently on a team where there is a separate technology and product slash business roadmap?


Okay, um, how many people in here have built something that wasn't used or adopted? <laugh>? I, yeah, I have done that a lot. Um, so I think we all know this, like building the wrong thing. And that quote that Levi had up there actually hits home too. I hadn't really read the entire thing until I was sitting here. Um, but building the wrong thing is a nightmare. So it's, it's a nightmare from the, you lose time, you have unhappy users, you waste a ton of money in engineering cycles doing this, and then more times than not, you end up rebuilding it. So I think that building the wrong thing is actually stemmed from two different things. So this is an extract of code from my brain. Um, but I think that if your engineers or your product development team is constrained, you're dead in the water.


And if you aren't, you actually have a chance. So I call this like my square peg, round hole example. So I've worked on so many products where it was either a package implementation where I had a feature, particular experience that we had to, we couldn't deliver it how we wanted to deliver it, and we ended up having to do a process work around, or it's 10 clicks to get through this workflow or whatever it looks like. And so I think that having a development team that is empowered to make their choices about their technology is gonna ultimately get you to, to a better result, at least give you a fighting chance. So I think that the other symptom here is actually kind of back to some of the roots of, of the DevOps movement to begin with. But I think that, um, I've, I've worked on many teams that have this kind of siloed approach.


So if you have separate technology and product roadmaps or objective key results or whatever framework that you guys use, it's gonna drive this difference in silos. Your engineering team's gonna want to do something that their organizational leadership's driving on them, your product team's doing something else, your design team's doing something else. I think another characteristics there is if you use language like the business or not our business or our users, like, those are just some key, key telling that you have some silos. And then I think the other one that um, is near and dear to a lot of people is, is my product owner only prioritizes features and functions. They don't care about debt or defects or, or, or any of those sorts of things. So, um, I think ultimately like those lead to bad things. And I think really where we need to go is to get to awesome products is you have to have shared goals, you have to have an empowered team, you have to be on the same page, you have to break down some of those silos.


So I'm gonna talk about what I think the role of a product manager is, and then I'm gonna talk about what I think the role of engineering is in getting, getting to this kind of siloed breakdown. So I think that product, product managers or product people really have two things that they, they are accountable for. So one of the aspects of those is kind of your traditional, like textbook stuff of you're the voice of your consumer. You do a lot of product marketing. You're, you know, some of the quotes are you're the CEO of your product. I think that is absolutely important. You have to prioritize the backlog. You have to make sure you're, you're, you're meeting the business objectives that your product actually is entails. But I think the other side of this that goes kind of under the radar a lot is, is you have to spend just as much time ensuring that your development team knows what you're building, why you're building it, who you're building it for, so that they can come and build the right solutions.


So I think this, this becomes even harder when, um, you aren't necessarily building for your core customer. So at Target it's, I don't know what our official thing is. I always think of it as like suburban mom with a family. Like my products GraphQL and caching and data services don't service suburban mom with a family. They service engineers that are using, using my data or using my platform. So I think it, it becomes really hard to do that, but I think it's so important to get grounded in those, those core product things and say, here's who our customers are. Here's who our competition is, here's what they care about, regardless of whether who, regardless of who those people are. And I think really important is product. People need to let your development team and engineers do what they're really good at, problem solving creativity. Like let them own and drive how you're going to achieve these objectives. Don't get so caught up in on the detailed features and functions and what, what really good looks like, like let some of that go. Get step up, get higher level, here's what we're building, here's why we're building it. Here's how I'm sequencing my things, why I am sequencing it this way. And have that healthy dialogue and trust and empowerment to let your development teams build what they need to build.


So I, I come, I'm not a engineer, but I, I'd say I would give this advice to, to engineers that have, you know, skeptical product counterparts like I do. Whatever you can do to get grounded in that. What and why and who, so ask your product person or your, your end users, why are we doing this? Really get grounded, really understand that. Um, and then I think for the, the debt stability technical stories, I hate, I cringe when I hear all those terms 'cause I just view this stuff as this is things we need to do for our product. But, um, I I really came outta my shell when, when I had a lead engineer that would come and say, Hey, here's what your objectives are. Here's why we need to do this particular thing. Pay down a debt switch, whatever, because it's sucking up our velocity and it's gonna stop us from being able to deliver what we need to next to next.


That aligns to what your vision is. So if you have creative ways to articulate that, why doing something aligns to what the overall goals and your end users and benefit are, the better off you you'll be. So what does awesome product look like? I've kind of, I've built this up and Dan and, and Levi did this too, but it really looks like you have an empowered team. So Target went on this engineering empowerment journey and talked about some of that. Levi alluded to the Dojo and a lot of the other tools at Target are really getting into around how do we center around business value and product value. I think that's, that's the key here to get to awesome product. But really some of the detailed things is the product team drives your backlog. It's not your product owner writing every story in your backlog and tagging it in feature versus debt.


It's your team all contributes, all says, here's what we're trying to do, this iteration or why this is important right now because of what our, our users have. And they're articulating what that is. Um, and I actually think that like things not being labeled as debt or stability or feature, you might wanna do that for reporting and measurement purposes, but it's just your backlogs just to list the stuff you guys need to do to, to enable the value that you're driving. So I think the obstacles, this, this is kind of my reflection on this, the obstacles that still remain for us. So, um, when we started at Target on our DevOps journey and breaking down silos within our, our IT organization and driving to empowerment of engineers, I actually think those, those same challenges are still right here with us. It's just more of that bigger, broader community. It's, it's your product team and your business teams doing those things, breaking down the SI silos, empowering those teams to deliver the right things. And I would say that you're never really done, software is never done. It's only ready to be shipped. So, um, that's, that's what we had. So thank you all for listening to us. Thank you.