Las Vegas 2020

Teaching Old Dogs New Tricks; Infrastructure as a Product

The transformative benefits gained in evolving from a project to product mind-set should not only be for the devs. In today’s digital age, it is more important than ever for the infrastructure itself to be fast, fully automated, self-healing, and scalable to meet the ever-changing business demands. The move to the cloud provides these capabilities, but the real transformation is in the people. Infrastructure teams and their traditional silos across the enterprise must transform to measure their people in terms of their collective contributions to a shared goal rather than by their efforts. The hero culture and its rewards just won't scale. This value can only be realized by changing from a project to a product mind-set.


In this talk, we'll take you through the journey that started with automating everything in Discover’s traditional infrastructure through the capabilities of the cloud to a more valuable shift in treating the infrastructure as a product. The shift to a product-centric organization has allowed Discover to move from a less valuable traditional functional model to one that is agile and focused on delivering business outcomes and capabilities that provide real value that can help anticipate customer needs.


We'll talk about our journey from a plan, build run organization to a culture of empowerment and autonomy, focusing on the tenants of 'You build it. You Run it' . Creating delightful consumer experiences through the delivery of products and services that are supported and run by the engineers that designed them.

We'll walk through our transformation and the operating model that was invoked, with distinct focus on both 'what we deliver' as well as 'how we deliver' to drive efficiencies and enable faster outcomes across our value streams to meet the rapidly changing business landscape. We'll share how we were able to take 'traditional' infrastructure engineers and evolve them into more valuable e-shaped ones. How moving from traditional silos into cross functional teams allowed us to deliver in an optimal way and invoke shared responsibilities that brought the right people to the problems while promoting self service and empowerment across all our teams. Faster feedback loops and rapidly delivering capabilities through product characteristics allowed the company to move faster and made its platforms and services more resilient and relevant. Most importantly, now being able to focus people on value-added work, rather than toil, allowed for increased business agility to give Discover an advantage over its competitors.


Lastly, we'll share how we measured our success by utilizing Objectives and Key Results (OKRs) to drive and map team alignment to broader organizational goals; How we standardized, scored and measured consumer satisfaction across our products and services to increase value and established 360 Leadership forums with our Cyber Security and Enterprise Architecture partners to broaden alignment, ensure prioritization and allocate just in time resources to rapidly respond to the evolving and changing needs of our business.

While there has been much success, the road has had its fair share of bumps and, at times, potholes that almost swallowed the department up. Discover is a company of very tenured technologists that have at times been extremely resistant to change. As they say, the technology is easy; it’s the people that are hard, and we have the bruises to prove it. We'll share some lessons learned in the hope you won’t make the same mistakes and that you can accelerate your own journey. We’ll share our perspective and guidance to get started, quickly mature, get unstuck, and find yourself delivering infrastructure with more value and intention than ever before.

HM

Heather Martin

Director, Value Stream Engineering, Discover Financial Services

ER

Edward Russell

Infrastructure Tooling & Automation Product Management, Discover Financial Services

Transcript

00:00:12

Hi, I met Russell, a director within our infrastructure product management department here at discover. Um, I've spent the last 18 years within technology here, uh, with roles primarily in operations and application development. Uh, when I'm not working, you'll most likely find me on a golf course or at home spending time with my family.

00:00:31

Hi, I'm Heather Martin and I've worked for discover for about 15 years and currently a director in our business technology department. I'm a mom of three littles shown here also now known as my coworkers who provide me great joy and sometimes great pain all in the same day. I'm most proud that I was able to finish up my master's this past August, despite all that was going on in the world, which only took about three years, but truly an invaluable experience outside of work. I'm a huge runner fitness junkie, and I help lead our women in technology community at discover I'm a huge advocate for the human trafficking cause volunteering, educating, and raising awareness.

00:01:12

Uh, so as we mentioned, we worked for discover, um, I'm sure you're all aware of our award-winning credit card and personal bank offerings. What you may be surprised to learn is that while we are a financial services company, we are very much a leader in the technology space. Uh, we lead with technology first to effectively compete in play, to win across our industry, providing our customers the value in personal experiences. They desire. We also strive to make a difference in our community, through financial education, providing our employees with many benefits and resources and by making positive impact in the community through giving and volunteerism. Uh, this photo was taken last year, right before our entire technology organization spent the day in our community, restoring two elementary schools that have no resources to do so on their own volunteerism runs deep throughout our culture. And it's one of the many things I love and value about working here. So as a reminder, I want to be sure that everyone understands the views expressed in this presentation or ours individually and not necessarily those of our employer.

00:02:14

So just like any good problem. The first step is recognizing and accepting that you have a problem that needs to be solved. As we prepared for 2020, we reached the stage where we recognize that the way we were working and delivering capabilities to our partners and application development, wasn't going to continue to meet their needs and allow us and them to continue driving value for the business and our card members. We knew that business needs for technology would continue to grow and that we would continue to face pressure to deliver faster and response to competition from fintechs and other financial services companies embracing the cloud. We also knew that as our internal consumers became more accustomed to cloud offerings, that their expectations for our more traditional on-prem offerings would have to be optimized to meet the needs of our consumers while they continue to modernize their applications and offerings to take advantage of the newer cloud solutions.

00:03:07

We also recognize that we needed to improve our feedback loops and provide quicker innovation to our consumers instead of waiting for new technologies and demand to surprise us, our options were to continue on the way we had been in a traditional plan, build, run organizational model, which we really knew wouldn't scale. Or we had to identify ways to do things differently across the whole organization, rather than in pockets as we had seen happen previously, the first step in our journey was to get a better understanding of our inefficiencies and opportunities. For years, we had been working under the premise that our consumers didn't need to know anything about the way that infrastructure was delivered. This caused us to become disconnected from them and their needs. And it really enabled us to believe that we were doing a much better job than we really were, uh, to begin improving our performance. We had many conversations with our consumers asking them for feedback on our service, where they felt we can improve and what ultimately mattered most to them.

00:04:08

Our consumers were more than eager to share feedback with us. In fact, they relish the opportunity to provide real meaningful feedback on our performance as a partner in a lot of cases, really for the first time, uh, for those of you that have operated in a similar plan build run model, I'm sure none of the following feedback will surprise again. Uh, we weren't consumer-focused uh, they told us that it was often a challenge to figure out how to engage us properly, to get work done. What they really wanted was that what they received with many of their vendor relationships, which was a single point of contact to help guide them to the correct solutions. They wanted reusable patterns based on their needs and as simple way to request and receive them in reality, what they wanted was to be able to focus on solving business problems and not on figuring out how to get environments so they could write test and deliver their code.

00:04:59

Uh, they also told us that we delivered to slowly, uh, seeing is that we were acquaint siloed in our delivery. We didn't recognize how long it took for us to stand up a fully functioning environment in our silos. We thought we were doing a great job. Uh, sure. We could deliver a new VM in 15 minutes using automation, but it wasn't usable to our consumers. It didn't have the necessary firewall rules. It didn't have the software and configurations that they needed and a whole slew of other work that we were passing back to application development, uh, when they would try to consumer offerings. So while we were counting quick delivery of a server, in reality, it would often take weeks for that newly created server to be useful. Our consumers also told us that we were too quick to hide behind tickets and processes. Uh, what was successful for them one time would be unsuccessful in next, our solutions weren't outcome-based.

00:05:49

And we expected our consumers to know the secret formula of service. Now requests to submit an order to get working infrastructure. Nothing was more frustrating for them than to spend an hour submitting 15 requests only to find out that they submitted the incorrect form or didn't provide the right information. Plus very few of the processes were connected and orchestrated. So our consumers are left with very little idea on how long the work would actually take for them to get the equipment that they needed. The last major piece of feedback was that we were slow to embrace new patterns in technology, especially as our consumers became more accustomed to cloud offerings, they wanted us to adapt our internal offerings to anticipate where they would have needs in the future versus playing catch-up to their new needs. As we had in the past, in essence, they wanted us to be more cloud-like in our solutions and delivery.

00:06:42

So we took all this feedback and started to think about how we could do things different. We engage some external partners seeking wisdom from those who were further ahead of us in this journey with hopes that we could learn from their successes and failures. One of the first steps was assessing how we were organized and identifying our opportunities there and how it led to some of the feedback from our consumers. As part of our analysis, we identified four significant areas of opportunity. First, we quickly realized that our plan build run model had caused a large disconnect between the operation of our products and services and the teams that were actually designing them. Our engineers weren't involved in the day-to-day delivery and support of the solutions that we were providing. And as a result had very little idea that our performance was less than ideal. Uh, we also recognize that our technology silos were leading to poor delivery instead of focusing on optimized end to end solutions. Our teams are only focused on their portion of the outcome that was being delivered. This played a major role in our slow delivery and a form for everything model that we had worked on her for so long.

00:07:49

We also identified that our people managers for spread very thin and frequently had to make trade-offs. They either excelled at technical delivery or people leadership, but rarely both due to the scope of work assigned to them. They frequently had to choose between one or the other many times, focusing on getting the work done and setting people leadership off to the side only to be addressed at review time. This also carried down to our engineers, just like our managers. They wanted to do a good job and we're dedicated to providing solutions to our consumers. So they would often put off or disregard training opportunities and would also miss opportunities to build and improve the automation capabilities needed to more efficiently deliver solutions. Instead, we we'd hear that it was quicker to just do it manually so they can move on to the next request. As I'm sure happens in many organizations, we also relied way too heavily on heroes. We had way too many single points of expertise on teams. Refried currently recorded heroics on major incident and change calls and people considered working crazy hours. A badge of honor.

00:08:53

So at this point we were kind of like this, but hopefully I haven't painted too bleak of a picture. There were things that we were doing very well and we had made some great strides and part of our organization to deliver quicker and more efficiently. A few years prior, we stood up new teams for our cloud solutions efforts. As we started up these teams, we went into them with the directive that we would focus on improving our delivery by actually being agile and focusing on consumer feedback all while building a strong employee and engineering culture, we focused on end to end automation, making everything self-service and pipeline driven in teams built around a, you build it, you run it support model. We had a lot of success with this experiment and we proved that by focusing on the right behaviors and guiding principles that we could drastically improve our performance and improve employee engagement. In fact, others started to see what was happening with these teams and they began to adopt some of the principles. We created new teams focused on automation and self-service enablement in both our network services and operations areas. These teams did great work, simplifying the delivery and support of many of our systems.

00:10:05

We had learnings that we could take advantage of what if we took what we did with our cloud solutions area, made some changes based on feedback and experience and got everyone to buy into the plan. That's what Heather I'll walk you through next.

00:10:18

Thanks said. So how could we drive adoption of this model that ed talked about at scale? How could we get all these traditional siloed infrastructure teams to adopt a product mindset for us, it started with some guiding principles that served as a guidepost to everything we did not so much about what we do, but how we all want to work here is what we focused on. Be consumer obsessed, but customer focused, establish regular feedback loops with your consumers whenever possible, look to eat your own dog food consuming our solutions first and focusing on the user experience creates simple frictionless experiences, reduce, eliminate, or create soft handoffs, deliver value, focus on the things that add value and possess end-to-end accountability for the capabilities we deliver work on those things that drive value and optimize those that don't, it's okay to say no. When things are not aligned to business goals or our desired outcomes, be a trusted partner in all things be transparent.

00:11:42

Your consumer is going to understand that you may miss a deadline and you may miss an SLA, but be transparent about that and take accountability, present and share new ideas. Often don't be afraid to fail, learn and get better. If we know anything from what we did in cloud solutions, it was important that we looked to engrain a culture to experiment. Often. We didn't want our engineers to be afraid to throw up shots and fail. We wanted them to actually expect some of them to miss be comfortable being uncomfortable. So I think all of us here could probably say we've been extremely uncomfortable over the last nine months. So learning how to embrace, embrace that discomfort is key. Teach your engineers that they need to try new things, learn what others are doing. Try not to always reinvent the wheel and apply, explore outside your own domain.

00:12:51

And then finally be flexible and frequently iterate. Sometimes you're going to need to pivot and focus on efforts that result in the greatest value delivered the highest consumer valued parts of your product first. So we benefit from productizing the infrastructure as an offering, no matter where it runs, whether that's in our public cloud, our private cloud or within our data centers as traditional infrastructure patterns. So some of the big lessons and principles we took away from our experiment and cloud solutions were things like the end to those long running big bang releases. We were traditionally operating in a waterfall model with a series of linear sequential phases that resulted in long running project work that brought in the consumers at the end of the release cycle. We needed to end those long releases. Those never ending technology, migrations and upgrades, and instead release more frequently.

00:13:53

And, and instead of enabling features and capabilities that maybe we only cared about focus on the customer and value, not technologies evolving to product allowed us to make it better. Without even being asked. We brought in the work to the engineers versus bringing the engineers to the work so that they owned it and continue to make it better. There is no done in product delivery. Objective is to bring in our consumers early so that we can focus on the things that drive business value, limiting our own agendas, where we only focus on the technologies that we may want to advance established feedback loops that improve value you should be bringing in your customers early and often focus should be on improving value through regular input. Not at the end when it might not matter and worse. Once you've implemented something that the customer doesn't even want or need iterate quickly and frequently, there's no defined end in the state of continuous improvement. Focus on breaking down the work to stand up minimally viable products, gain some feedback and improve, remove the desire to make it perfect or focus on all the things you still want them to do or need to do before you can release.

00:15:22

And as we shared adoption of our guiding principles that we applied to everything we did was key. This made it easy to build things that promote itself. Service made our products easy to consume and focus us on the user experiment experience in everything we delivered next, it was about changing our operating model, moving from an implementation partner. One that was accustomed to just taking orders to an agile service provider. We focused on improvement across four key pillars. The efficiency moving from a siloed organization that threw things over the wall to one that was focused on outcomes, delivering value, and when needed cross-functionally. This allowed us to focus our engineers on the business outcome versus working to complete their tasks or close the consumer requests as quickly as possible time to market. As you heard, ed mentioned, our service delivery was slow and often failed to meet the needs and expectations of our consumers.

00:16:34

We needed to improve our time to market focusing on and allocating time to drive for business initiatives as defined by our value stream partners, not just those infrastructure solutions that were needed to run and keep our applications current or compliant, improved service delivery, where the products and capabilities we are focusing on ones that actually drove the most value to our consumers. Did they enable and support our key business initiatives. We needed to not only improve what we delivered, but how we deliver those capabilities. Our solutions should be easy to consume frictionless whenever possible, self serviceable finally, and maybe most importantly, leaning in to customer satisfaction. We had traditionally been unfocused on how to satisfy our customers with the products and solutions we delivered. So we needed to start asking more questions about the user experience. What was that like? What were we doing? Great. And maybe where could we improve?

00:17:53

So to operate at scale, we first had to change how we are organized, moving away from our traditional siloed model of plan build run. We first organized our engineering teams around products and product, family capabilities. Those engineers were focused on the what building, managing, maintaining, and supporting the life cycle of their products and services. We stood up a new chapter model that focused on the how with chapter leaders acting as player coach for the engineers across their domain, helping to drive good product practices, principles, and hygiene. They became responsible for coaching and developing our engineers through skilling upskilling, and sometimes rescaling helping to anticipate the resource needs across the domains. So we could deliver value when it was needed. Not months later layered over our product organization was our value stream engineering and services organization, which included our it process teams like incident management, change management, continuity management, any management you can think of, et cetera, as well as a newly created value stream engineering organization, which invoked a cross-functional engineering model to focus on accelerating the needs of the value stream, prioritizing the delivery of the needed infrastructure solutions and patterns, as well as automating and enhancing our enterprise solutions and capabilities.

00:19:26

Basically taking our enterprise delivery that last mile previously, the intake of work was decentralized and often invisible across many tools, whether that was service now, tickets or JIRA stories or conversation in Microsoft teams, emails, phone calls, hallway conversations, et cetera, to effectively scale and focus the organization around the highest value work we had to ensure we had a common system for intake that intake framework, how to ensure the work was visible, being reviewed and prioritized across the product or value stream engineering teams. We needed to ensure that the engineer stopped asking us, which number one priority do you want me to work on today?

00:20:18

So we had a good model in place and we were moving forward. And then the wheels came off a little bit. We had a plan until we didn't. I love change, but what I'm about to share with even fatiguing for me first, a pandemic in the middle of the transformation. This hurt our plans to fill so many new roles. As every organization slowed down to understand the effects this would have on their business models and plants as if a pandemic wasn't enough to deal with. We were in the middle of some major leadership changes from a new CIO, a transition of our champion sponsor VP into another area of the organization and an onboarding of our new product VP whose first day just happened to be the day that every domestic employee began to work from home full-time. So that's a heck of a first day. Of course, leadership also had some new ideas around our it transformation, but lucky for us, the direction was still to productize our engineering teams, but maybe with a slightly different path, the runway became our new focus, our new it wide transformation initiative that was being adopted across our entire business technology organization, a series of company back north stars. So we jumped on board and leaned into the work streams that would allow us to scale this transformation further.

00:21:57

What we learned and continue to learn is that change cultural change is still critical to drive the right sets of behaviors and expand adoption. Those that had already made the shift to this new way of working happily and enthusiastically accepted the changes and jumped on board, viewing this as just another model to get to the same outcome. So what can you learn from us? Some mistakes definitely aren't work worth making. So let me leave you with some of these closing lessons. Either have a craft beer habit or wear skinny jeans, but not both. You can't be everything to everyone. And it's better to execute on a small number of things with excellence, then many things that are half gate. So say no so that you can say yes to deliver things of the highest value in the right way, in the way of transformation. It's going to get hard and Rocky and changes, maybe thrown at you like a pandemic and the ones bright lights that lit the demo lit lit. The lighthouse seemed, Jim, decide on what your objectives and key results look like and use them as your guiding light.

00:23:21

Don't make your transformation journey. A side hustle. Transformation is hard, but cultural change is even harder. Make sure you have leadership. Buy-in crystal clear clarity and dedicated resources and time to transform it. Shouldn't be a side hustle or you will never succeed. And you are very likely to burn out, stop boiling the ocean and trying to fix everything at once. It's definitely easy to look at all the things that are broken and get stuck in quicksand, but break down your transformational changes just like you would any problem to ensure clarity measure and iterate, rinse, and repeat. But you have to start somewhere.

00:24:11

Stop focusing only on the delivery of capabilities because tech debt really does matter. Take time to do it right upfront. If you have a mountain of tech debt already, you have to start chipping away at it. No matter how overwhelming that may seem and whenever possible, make decisions grounded in data, especially when decided to allocate your precious engineering resources. You don't want to be in a position where you're saying, I feel, or I think this is what we should work on back those decisions with data. So this product thing, this transformation thing, and this cultural thing may be hard, but it's worth it.

00:25:01

So what are some of the key learnings or takeaways of things that you can do become a community and do it together. So at discover, we've had much success through bringing people together through various communities of practices, allowing those that understand the way to show and help everyone get better from communities that are focused around tools and technologies, to roles like product owners, scrum, masters, or SRS, we found the best way to drive your adoption is through the people that champion those ideas, slow your roll and push for the upfront investment. It's better to be valuable than fast, rather than not in faster focus on the right things and make time to ensure how you deliver is just as important as what you deliver. Don't cut corners else that will come back to bite you. As you start to scale out your solutions, tech debt is real and it will slow you down.

00:26:01

Share the responsibility. Both the applications and infrastructure need change with the blurring of lines and the focus on efficiency in DevOps. It's more important than ever to share responsibility and knowledge, the better and more reliable applications measure and validate your thinking. This is something that we had traditionally struggled with across our infrastructure services teams. And it's extremely important. So adopt a way to measure consumer satisfaction and seek input regularly. Have your product teams make this a regular practice? Yes. Even for infrastructure establish metrics, both qualitative and quantitative. What are the teams working on? Is it the right balance? Are you measuring your engineers, their skills and their competencies? So you know where to continue to grow and hold true, your OKR or organizational objectives. Review those often score them, share, and use them to create clarity across your organization. And finally take time to take care of yourselves. It's not all about building shiny new capabilities. I'll refund. That might be what you need to ensure your engineers are keeping their sword sharp and they make continuous learning part of your team or your department norms.

00:27:31

So once you get the right people embracing the idea of productizing your infrastructure, it's important to cultivate a culture and an environment to optimize and sustain that productivity. I deeply advocate doing this through social interactions, outside the day-to-day delivery of the product sheet of play, which I know can be extremely challenging during a pandemic, but look, to create an environment that's psychologically safe. One that fosters openness failure and vulnerability for our teams that was finding ways to connect socially to open that up. Why is it important to find a way to connect socially? Well, that's how you create trust, respect, and improve collaboration. Maybe that's virtual happy hours for you or other team building activities, but find a way to connect with your team outside of the day-to-day delivery. Get to know your people learn what motivates them, what they are passionate about and how they like to be rewarded. Thank you very much for your time. We hope you found this useful. You can find ed or I on either LinkedIn or we've shared our email above and of course we're hiring. So please check out our career site to learn more about some amazing opportunities in technology and engineering. Thanks.