Las Vegas 2020
Advancing the Lifeline with DevOps
Motorola Solutions' mission critical two-way radio business was established over 90 years ago and has been the dominant leader in the market with over 85% market share for decades. Configuring, upgrading, and managing the radios has evolved from simple, stand-alone applications installed on a PC to enterprise grade, on-premise, distributed solutions.
In 2018, Motorola Solutions began the transition to move device management solutions to the cloud in conjunction with the launch of the newest radio, APX Next. The RadioCentral cloud platform has transformed not just how our customers manage their devices, but also kickstarted the organization on their DevOps journey.
JA
Jonathan Akers
Product Owner - RadioCentral, Motorola Solutions
RD
Ryan Dobson
Director of Engineering, Motorola Solutions
Transcript
Hi, welcome to advancing the lifeline with dev ops. Um, my name is Ryan Dobson. I'm the director of engineering with Motorola solutions, working with the device cloud engineering team. I've been with Motorola for 20 years, worked in all different areas of software development. Um, and now the pleasure to work with this team. Uh, Jonathan acres is with me today. Jonathan, you want to go ahead and introduce yourself?
Yeah. So thank you, Ryan. I'm John acres. Uh, I've been at Motorola for 17 years and worked in a lot of different areas and it's been a pleasure to work in, uh, some of the new technology that we're going to be showing.
Great. Uh, just a little backstory on Jonathan and I, and our experience here. Uh, last year, uh, we attended the DevOps enterprise summit in Vegas and that we, we really got a lot out of the experience. Um, both the people we met as well as the sessions that we attended. So it's really our privilege today to be able to share with you, um, what our story has been. Uh, we started on this transition, um, to dev ops in 2018. And, uh, the story we're sharing with you today is a about our journey there. So first off is this iconic symbol. Um, it, it means quite a bit, uh, the different folks, right? When you first see it, uh, the internally inside the Motorola, we refer to these as the bat wings. Um, and, and they've been around for quite some time and it may, uh, kind of stir up different images when you first see this, uh, for a lot of folks, it can, uh, you immediately start thinking about cellular devices.
Um, for other people it might be radios. Uh, there are quite a few things that have had this, um, symbol on them over the years. Uh, and Motorola started, uh, back in 1928 and when they started, they actually were doing radios. Uh, two years after that, 1930 is when they released their first two-way radios. And since then they've been a leader in the public safety, two way radio communication business. And over the years, they extended into quite a few different areas of technology, right? Leading the way in cellular, um, semiconductors, uh, lots of different projects in 2011, the company split into two. So Motorola mobility and, uh, Motorola solutions and the Motorola solution side retained the two-way radio business as the core business that they had the mobility side, so that the cellular. So Jonathan and I are with the Motorola solutions. We worked in the two-way radio business. Uh, and that's what we're here to talk with you about today.
So like I mentioned, Motorola solutions took the two-way radio business with them in 2011. Since that time they've expanded through acquisitions and some organic growth. And now the core of the business is around this mission critical ecosystem. There are four main parts that create this platform, uh, the first being mission critical communications. So this is based off of the core that I mentioned before that Motorola solutions has been in the business for a long time, but it's expanded beyond just voice communications into data, um, and other adjacencies video security and analytics. So this is relatively new to the Motorola solutions portfolio, but they've, we've aggressively been going after completing this entire ecosystem and video security, as well as the back end, analytics plays a key role in that. And the third is command center software. Uh, this is the back end systems that provide incident management, um, as well as, uh, um, dispatch nine 11 services, um, those types of things. And then the last is our managed in support services, which wraps them all together and provides an end to end solution for our customers.
So double-clicking into each of these areas. It gives you some idea the scale, uh, at which Motorola solutions is operating. So on the mission critical communication side, we have over a hundred thousand customers, uh, all around the world. Um, this includes both on the enterprise and the public safety side, the command center software handling millions of incidents, um, and, uh, focus on our nine 11 dispatchers and call takers, video analytics, you know, uh, analyzing, um, hundreds of thousands of alerts a day, notifying first responders, uh, of those events, and then providing the back end analytics that support it and that our managed service and support where we're monitoring customer networks and proactively, uh, providing support for them, uh, as a total package.
So, as I mentioned earlier, we do support both the government, public safety business and enterprises. On the government side, we work with, uh, federal customers, uh, department of defense, uh, lots of three letter agencies in the us as well as foreign governments and militaries on the enterprise side. Our products are used by, uh, theme parks, uh, data centers, as well as airlines, mining, operations, uh, all kinds of things. So when we're talking today about the services that we provide in an easy way to refer, to think of NDAs, any two-way radio device that you see that has those bat wings on it, those are the devices that we're talking about today. So as you go to the airports or you go to different, uh, locations, you can look at what devices they're using. If you see those bat wings, that's why we support. So I have a short video here. That's been a introduce, uh, the ecosystem a little bit more and show you how it's all integrated together to provide these solutions for our customers.
Now on a device cloud engineering. Uh, so the purpose of this organization is to provide cloud services and capabilities that enable the mission critical work group communication solutions that I referred to earlier. And really that is in these three care key areas, device management, device analytics, and telemetry and feature and surveys offer enablement. So first in device management, uh, and the device management functions, I'm referring to our firmware upgrades for our devices, uh, configuration of our devices, um, new feature enablement. So as, uh, we're selling features and controlling the sellable elements of those features, uh, all of that is wrapped together. Um, the second is in device analytics and telemetry. So our devices, the newer devices that we have, um, do have an IOT aspect to them that allows us to monitor, gather the data back into our cloud services and present that data for consumption both to customers and to internal stakeholders, so that we can make decisions about, uh, what features we should be doing, how successful the rollouts are, things of that nature.
And then the last is on the feature and service offer enablement. We are bundling a lot of the services and capabilities that we're starting to offer together and through the, the tools and mechanisms that we provide from a device cloud engineering team, we're able to control those elements, that turn things on and off as well as expire, uh, those services that we're offering. So that's really the core of what we're doing in device bout engineering. Uh, the team has, uh, this is something that's been around for quite some time. And just recently in this last two years, we've transitioned to really be focused on cloud. So prior to that, we were doing quite a bit with on-prem software as our solutions, but we were really transitioning.
Now, I'm going to walk through what that evolution has been, uh, when we started, uh, like I mentioned, Motorola's being in the two-way radio business for quite some time, uh, back in, uh, when we started, right. Any customer configuration that was needed. So if you needed a new channels in your device, or you had new devices, you were rolling out really the only option you had was to come to the manufacturer, right? You worked with Motorola, and we would put in those configurations for you as technology progressed, um, trying to enable customers to do this on their own was the primary focus. So we created several different solutions, uh, over the years that were first dos and then windows based, um, commercial off the shelf software packages that would allow you to program one device at a time. Uh, and as, as those solutions were being used, they obviously there were a lot more demands as, uh, fleets grew.
And again, technology tends to mature. So in 2010, we introduced a series of solutions that provided fleet capabilities and enterprise grade solutions for our customers to manage fleets. Now, typical fleets for, uh, our public safety customers can range from a couple of hundred devices for a police department up to 40 to 50,000 devices. If we're talking about, um, a state or a country-wide system. Uh, so the solutions that we had to provide had to be fairly, uh, scalable and mature as well as reliable for on-prem. And as our customers were using those right investing in our on-prem solution servers, uh, equipment, the, the demand for new features and capabilities, uh, that are very well aligned with the cloud started to come up and, uh, particularly around, uh, defining role-based access for their users and, uh, agency partitioning on the backend. So you can imagine if you have 50,000 devices, you're not just trying to provide access to those, to a very small number of people.
You could have hundreds of different people that need to be able to configure, manage, upgrade these devices. So those solutions, um, we're starting to reach kind of their end of life for on-prem. So in 2018, um, Motorola solutions started work on, uh, their next generation two way radio device, which is called apex. Then, uh, when the, we were first having discussions about this device, we knew that there were several new things and capabilities that were being provided with it that would provide us a great opportunity to mature the device management, uh, solutions that are used in order to, to manage it. So, first of all, this is a mission critical device. Uh, this is a two-way radio used by public safety. So no matter what, uh, it has to work, right, um, this is, this is the definition of mission critical. Uh, it it's different than most of our other devices in, in really two critical ways.
One, it has an LTE pipe and that LTE broadband data pipe allows us to provide services that we don't typically provide to these, uh, two-way radios. So we can now do things like messaging and mapping and apps, uh, and, and we can run an IOT framework on the device as well, that would enable us to be able to manage it as more of an IOT device. Uh, there's also a new UI. Um, so that picture had a nice big screen on it that big screen enables us to do a whole lot more as well, but the focus for our organization and our services was around this streamlined ownership experience. So, and this goes from one customers are, uh, we've, re-imagined what it was like for customers to go from ordering a device, looking for a device to when we ship it to them. They open it out of the box, a lot of focus on the autobox experience, and then they they're able to configure and manage it through its life cycle, as so, as part of that, we introduced this new platform, radio central, and this is, uh, the Motorola solutions cloud device management platform of the future.
Um, and it it's really, really exciting. We've got, uh, not just a way for us to be able to manage devices from anywhere and to do it over the air using the broadband pipe. Um, but it also allows users to connect, um, from wherever they're at, in order to do this management. It's, uh, it's really exciting to see this and we can provide these newer features that I was referring to. So role-based access, you can see partitioning all these great things that come with the cloud, including the scalability and the security and the reliability. So it was a really, really great fit for our overall market demand. I have a short video here that I want to share with you as well, that summarizes the impact that radio central has had on the apex next device and the ownership experience.
Next, I'll hand it off to Jonathan who will walk us through the radio central platform and our dev ops journey.
Thank you, Ryan. So as Ryan and I were putting together the materials for this conference, it was really cool to see how far we've come in, such a short amount of time. When you look at this timeframe of 2018 to 2020, we're really talking about the end of 2018 and the October timeframe to a lot of this being realized even as early as 2019. So things like our release cadence, we came from a three plus month off ramp on not off the main line to where we are today, which is on our main line shipping from our main, um, and promoting those changes, uh, to production at the end of the sprint or auto promoting all the way to production. We came from a mix of manual and automated to a hundred percent automated. There there's zero discussions of any manual tests and the things that we do today.
Um, and this is kind of built into the culture of the team. And we get into meetings now, and we have passionate discussions about which kind of test, um, strategy to employ. Uh, but it's never a discussion of whether we have time to add tests or not key metrics we use to track things, uh, purely like just the number of defects, the team velocity, um, or the number of successful nightly builds to where we are today, which is focused on the dev ops key metrics onboarding, uh, back in 2018. This was not a primary focus of the team, but it's something that we, um, we made a focus for our team. We wanted to bring in developers and have them be able to make their first commit to production within their first week. And when you set this as a goal, um, it touches everything from your CIC to, um, the ease of setting up the developer environment to documentation and moving it closer to the source code and things like reading these ownership.
We came from a place where we had shared ownership to where we are today, where the team asks for component ownership, either scrum team or individuals, uh, they wanted to hold each other accountable. Um, and that's something that came up in retrospect is from the team tech debt. Um, this used to be taken kind of at an ad hoc in nature, and it was always hard to prioritize, um, to where we are today, which is, um, every sprint and sprint planning. We're talking about, which pieces of tech that, uh, should be taken by which of the scrum teams. And, uh, as everyone knows, whenever you're building a product, even if you're starting from scratch, you're having to make a decision, every sprint of how much tech debt you're biting off, um, to keep that low, um, you know, and balancing that with hitting your, your product dates, that when you have to ship incident response, um, we, we came from a place of having a no incidents to response to, uh, where we are today, which is fully integrated with Runscope and PagerDuty.
And I did miss our team initiative. So, uh, we used to, uh, you know, developers would have a chance to do some of their own initiatives, but now it's kind of built into the culture where the team is enabled and empowered to go off and, and build things that are going to make their jobs easier. And a lot of the ideas that we've had in what we're going to show you came from those retrospectives of the team, uh, asking for time to go build things, that'll help them get their jobs done. Uh, so things like our dev ops team portal, which we'll show you later, um, our self-service portal, which is an, uh, uh, front end to our, uh, backend API APIs, um, and things like chatbox integration, et cetera. So this is your classic, uh, text deck slide. Um, and this isn't just a show off the number of integrations or what we've done, but each of these have solved a problem for us. And what's really cool is how far we've come. Uh, like I was saying in such a short amount of time, uh, none of this was in play in 2018, but now we are utilizing all of this, um, in 2020.
So what we really want to show is our dev ops team portal. So again, this is something that came from the team and from, uh, from things that would make their jobs easier. And for the team, it shows them really buying into the dev ops concept, but they own everything about the product. So it is our one-stop shop for automated dev ops metrics. We use it to track our production costs over time. Uh, that's something I won't be able to share today, but it is something we take very seriously. Our team owns our production costs, and we sometimes even rearchitect in order to drive costs down. Uh, but we are watching it constantly. Uh, we could track things like the deployment health, uh, component ownership, and we'll show that later our delivery performance metrics and our CICD pipeline health.
So I think everyone, uh, in this conference is probably very familiar with this, uh, with the DOR report and this chart. Uh, but we wanted to show, um, we, we are, we use this as a metric for us to find out if we're how we're doing in the DevOps world. And, uh, this really was a great insight for the team, uh, to see, to understand, and then actually automate and we'll show that later, but doing a little self-reflection. Uh, so our team, uh, for our deployment frequency is rated high. Um, we came from medium. So we've improved, um, in that category for lead time for changes, uh, we're medium today, Time to restore service, uh, were high and change failure rate were also high, But the point was is to figure out how we can get to a leak, um, and slowly keep moving the chain to see if we can get, uh, we can hit those elite levels. So we set an organizational goal to hit the elite level on 2020. Um, and we're going through an architecture change. We're overhauling our CACD and everything in order to align with this goal with our, for our team.
So as part of that elite initiative, some of this got built into that dev ops team portal. Uh, so we host all of this data for, uh, what we're going to show you here in that portal for all the developers to see in check. So, as you could see from below, this is our deployment frequency. And so this chart is showing you where we came from in 2018 to where we are today, which were big bang, uh, promotions of all of our microservices at one time, uh, back in 20 18, 2 smaller, more frequent deployments that are happening, uh, more frequently. So for our evolution through this, we started with manual promotions to all of our environments. At some point, we turned on auto promotion all the way through our staging environment, and that caused a lot of problems. Uh, we learned a lot of lessons when we did that. We actually had to back out that change, uh, because we broke, uh, teams that were dependent on us at the time, but we gathered all the data and all the things that we'd have to go fix, and we tackled them over the next couple of sprints. And once we turned on that auto promotion back to stage, we'd never had to turn it back off. We also had a goal, uh, in a mandate that all new services are auto promoted from auto promoted to production from the start.
So for the lead time metrics, this is also hosted in our custom dev ops team portal. So as you can see here, we have this graph is showing you all of our microservices and their average times, uh, for deploying work items to production and the lead time. Uh, so we have an average of 19 days, uh, but you can see here that we're actually highlighting that we have a minimum of 3.4 minutes. So there are some microservices that are auto promoting all the way to production, and those can reach in 3.4 minutes. And the other ones that are not auto the auto to point to production, uh, have to wait until we have a promotion.
So in our time to restore metric. So this is hosted in PagerDuty, uh, currently, uh, and we do review this, uh, weekly with our team and what we noticed from the, the grasp below you could see that, um, where we came from was we had a group of, uh, five people on PagerDuty on-call. And over time we realized that the number of incidents that were slipping through that team without an acknowledgement or without a resolution we're getting escalated, um, much more frequently. And so what we did was we actually switched to having one person on call. Um, and once we did that, you can see the, uh, the green bar, which is the meantime to resolve a higher, um, the, uh, the incident at the time dropped to close to zero, um, the red bar, which is the meantime to acknowledge the incident, also dropped close to zero, and then our number of incidents, we actually were working on getting better over time. So the change failure rate a metric, we are still working on automating this, if there's anyone at the conference that has some, uh, ideas or, uh, around this metric, uh, let us know, but we're, this is something that we're currently working on.
And then one of the other things we wanted to highlight was this component ownership piece. So in our dev ops team portal, we actually have cards for every single microservice and within those cards or showing one here in the top left, you could see this as the firmware operations service, we have a primary owner assigned a backup owner, and then we call up the things that are shown in our static code analysis, things like the bugs, the code coverage, the code smells, the complexity. And then one of the most important pieces for us as the deployment status, we built this due to the fact that the team was having a hard time keeping track of what versions of what microservices are deployed, where, and as soon as you start spreading out and having more environments and more regions, uh, this was really hard to keep track of. So in this example, it's actually highlighting in yellow, that in the development environment, there's a different version that's deployed. Then the latest version that ran, and this has been invaluable, um, in our day-to-day operations. So this team in this example is actually getting a daily email telling them, Hey, there's a discrepancy. You need to go look at it.
Great, thanks, Jonathan. Uh, so now I want to close things out a little bit and you know, that all the great things that we've done with radio central and the stuff that Jonathan walked through really doesn't matter, unless it impacts our costumers and makes their lives better. So here's, uh, two examples of some feedback that we got as we started to roll this out, you know, and the things that we're talking about IOT and dev ops within the consumer space, these are fairly well known, but to the mission, critical public safety customers, these are completely new, right. Uh, in this industry, um, we're really pushing the envelope. Um, and in order to do that, uh, it, it, it really takes a lot of trust from the senior leaders and from our customers that the solutions that we're going to deliver are not only going to meet their needs, but to exceed them.
And with that, uh, I'm just going to wrap things up, um, a couple of things, uh, one, you know, advancing the lifeline with dev ops. This has been a journey we're on. We still have quite a ways to go. Um, I wanted to first acknowledge, uh, the incredible team that, uh, we have that Jonathan and I work with. So, uh, we get the privilege of being here and to present and share the story, but we have a team of rock stars that, uh, created and deliver and continue to surprise us with new ways that they're coming up with in order to push the envelope. And we love it. Right. So, uh, I'll speak for both of us on that. And, um, and also to, uh, a thank you to gene and the, uh, the DevOps enterprise summit committee for selecting us, you know, being able to share our story. This is, it means a lot to us. Um, we hope that some of this that we shared with you today, you can take back challenge your team with ask questions. Um, we're certainly available both Jonathan and I reach out to us. Uh, we'd love to talk, learn from you more, uh, as well as, uh, share what we know and with that, uh, thank you and have a great day.
Thank you. Bye.