Developer Productivity Engineering – The Next Big Thing in Software Development

By 2022, IDC has predicted that 65% of the global GDP will be digitally transformed. Two-thirds of the products and services that you pay for will be driven by software. There has never been a more important time to foster developer productivity, but many of our methods have not evolved.

In this keynote style talk you will learn why DPE is the most important development in the software engineering world since the introduction of Agile and DevOps concepts and tools. DPE is a new software development practice that uses acceleration technologies to speed up the software build and test process and data analytics to to improve developer efficiencies by as much as 10x. The ultimate aim is to achieve faster feedback cycles, more reliable and actionable data, and a highly satisfying developer experience.

Justin Reock, Field CTO and Chief Evangelist at Gradle is pioneering DPE as a practice and set of technologies and is one of the world’s leading advocates. Specifically, Justin will provide an overview of the key concepts and tools, business impact on key business objectives like time-to-market, cost and quality, the business case for DPE, and the role of AI/ML in DPE moving forward.


Justin Reock

Field CTO and Chief Evangelist, Gradle, Inc.





Hi, everyone. Thanks for joining today. I'm Justin Rak, the Chief evangelist and field CTO at Gradle. You may have heard some of my other talks before around open source and developer productivity in general. My background is predominantly in software development, moving into enterprise architecture, but now I really focus more on, uh, evangelizing, uh, developer productivity engineering, which I'm gonna talk to you about today. So let's kind of jump right into it. I'd like to start this talk with the following quote from Eric Pearson, who was the CIO at Intercontinental Hotels Group. And he says, it's no longer the big beating, the small, but the fast beating the slow, okay? And I think that this quote adequately sums up the state of the art and the state of the industry right now. Okay? It's no longer the big giant software companies that are just dominating, right? It's the disruptive, nimble companies that are able to respond to customer and market feedback quickly and bring their features to market, uh, fast, right?


And that's, that's, you know, obviously what we hear talking about today with DevOps, right? That's the whole purpose, is to, uh, get around various constraints and to unstick bottlenecks and to convert our code into throughput faster. So we're gonna talk today about a very specific problem, uh, around feedback cycles for developers and how that impacts developer productivity. Uh, because the practice of developer productivity engineering is very practical and very pragmatic and very straightforward. It's about identifying bottlenecks and friction in the software development process, and then using technology to mitigate those bottlenecks. Okay? So, so here's, you know, your average developer, uh, doing their thing, they're in their local environment, but this could also apply to remote environments or even building out in CI environments, and they're waiting on feedback, right? They've written some code, they're gonna run a build, and they want the build system to tell them something.


Well, sometimes the feedback just takes too long, right? Sometimes the build takes minutes, in extreme cases, even hours, right? Or sometimes the test cycle takes too long. Maybe there's thousands of tests that have to be run, right? Maybe, uh, it, the, the feedback is okay, but, but, but it's a failure, right? There's some type of failure that has to be investigated, right? Maybe it takes too long to fix that failure or to try to collaborate on the fix with that failure with, um, other engineers. And maybe like the worst case, it's, it's a, it's a, it's a pitfall or a problem or a bottleneck that could have been avoided altogether, right? Maybe a flaky test, a test with like a non-deterministic, uh, outcome or a failure that is actually impacting lots of developers in the organization, but nobody's talking about it because nobody's tracking it, right?


So think about these pain points, these friction, uh, these bits of friction and these bottlenecks, and multiply that by, you know, hundreds of calendar days per year, and then hundreds of developers times that we end up sinking productivity and increasing cost, which is antithetical to constraints based work and productivity theory. So, backing it up just a second, okay? Software development is still a creative process, all right? Even, even as practitioners, even as folks who do this for a living, right? This is still creative work. Now, it's not purely creative work, it's also scientific work. I think it's fair to say that it's a bit of both, right? We experiment with our code, we form a hypothesis in our mind, we run the code, and then we get the results, uh, right? And we want to, uh, understand whether the work we did, um, accomplish the task that we wanted to accomplish.


And I think for a lot of us, when we started, uh, in this industry, and we started down this path of like learning how to code, the experience felt like this very often for us, right? These were learning moments, right? We were achieving, we were, we were running a hello world program for the first time in a new language, and we would run it and we would get the output, and that feedback would happen so quickly, right? Because they were very simple, simple programs, right? Maybe we're, uh, doing a, a standard hello world, or maybe we're just changing the background color on a shape or something like that. But the point is that we would get feedback fast, and that fast feedback is part of what led to the delight of the experience. But as a lot of us have moved into more professional roles with developing software, uh, especially in the enterprise, um, the success of software projects ironically creates a new set of challenges for us, right?


Um, as the projects become more successful in an enterprise, as more users start to use the project, uh, as the project grows in size and the number of developers working on it, or the number of repositories that are necessary to support the development, uh, just the number of dependencies that are being used, and just the overall diversity of the tech stack. You know, again, as enterprise projects get larger and larger, the tool chain efficiency, uh, starts to degrade if we're not managing it, right? So the, the build time gets longer and longer and longer, the test cycles get longer and longer and longer. And, and ultimately, the delight of the process and the happiness involved in the, in, in, in the craft begins to diminish and begins getting replaced with some frustrations, right? So I think now, you know, if you look at the day of the life of an average enterprise developer, their calendar might look a little bit like this for organizations that haven't, uh, fully invested in productivity engineering as a practice, right?


Uh, the developers might come in, they're ready to go, um, and they start coding, great. Uh, and then we're gonna wait for our local build to complete. We've made a bunch of changes, um, and something failed in the local build. So we're gonna spend some time debugging that build failure, maybe we fix it and then go to lunch. Great. We're gonna come back, um, and we're gonna code for a little while longer. We're gonna wait for our local build to complete. Now, we, we debugged the build failure that we had last time. So this time it's successful, but, uh, now we pushed that build to ci and there's a flaky test, right? There's a test that passed in our local environment. It did not pass in the CI environment. Now we're spending time investigating that. All right? So how much of this was spent doing what developers love to do, right?


Coding, uh, not a lot, right? A lot of this is actually just spent, uh, with various frustrations supporting the build, troubleshooting the build, debugging the build. So an outcome of developer productivity engineering then is to give developers literally hours back in their week to be able to work on valuable solutions. And I think that we are at a point where we can make a global appeal for this now, okay? Um, there's a study that's been, obviously it was published in October, uh, of, of last year. And some of you may have seen it, you may have even been quoting it in some of your presentations today. It's a very powerful statement, and it was from IDC, and it's a prediction that by 2022, right? Coming up, by the end of of this year, 65% of the global GDP will be digitally transformed. 65% of the goods and services that we pay for will be software services.


Okay? So this is now, you know, still this relatively small, you know, group of craftspeople developers who are quite literally lifting all boats in the harbor. And I think as much as possible, we owe it to this part of the workforce to have a happy and productive work experience. And that's very much, uh, the purpose of developer productivity engineering is to increase productivity by increasing developer happiness. Because right now, the current state teams do work far from their true potential. And the productivity of developers absolutely affects their happiness, right? Again, going back to when you first learned about coding, uh, it's, it's neuroscience, right? Those little rewards that you get for running the experiment and seeing the results that you want, it's a dopamine hit, right? Right? It's, it's, it's a small reward that leads to overall joy and happiness. And if you can't be productive, if your tool chain is blocking you from getting the feedback that you want, and from building code at the rate at which you want to build it, you're gonna be more frustrated than you are happy.


And so, as a result, low dev, low developer productivity is blocking business innovations. It's not that DevOps hasn't done amazing things in terms of increasing our ability to get, get code to market faster. Of course it has, but what we are talking about is a new set of bottlenecks that are actually further left in the process now that are actually part of the developer experience in writing code. So this is not like, you know, a replacement for DevOps or anything by any means, right? This is more of a continuation, right? It's a constraints based theory, uh, just like, uh, just in time manufacturing business process, re-engineering, ultimately moving into things like change management and then evolving into practices like Agile and DevOps. I mean, we, we know these processes deeply. We understand, uh, uh, gold rat theory and theory of constraints. And if we fast forward now to what developer productivity engineering is trying to do, it really is just taking a look at the same, uh, types of bottlenecks that have been identified by other practices like this in the past, and then taking pragmatic solutions to do something about it.


Okay? So let's talk about what those solutions are. That's a lot of theory, but the solutions are actually very straightforward and pragmatic. Um, so when we say that DPE is the next thing, like this continuation of DevOps, though, it's quite literal, right? We, we really say, okay, well, this is the next set of bottlenecks and friction and pain points that need to be addressed. So we use acceleration technologies and, uh, data accumulation and analysis technologies, all right? Um, we use acceleration technologies to speed up the build itself to, uh, decrease the amount of time it takes for a developer to get feedback about the build. And that applies to local builds, remote builds, and CI builds. We also apply, uh, technologies to the testing process to allow more tests to take place in parallel. And we're working on a, a, a, a machine learning gradient boosting based technique called predictive test selection, uh, which was actually, um, pioneered at Facebook to avoid running certain tests that probably won't produce any valuable feedback for us to begin with, right?


So, so really, you know, we just take these acceleration technologies, we apply them to the build and the test to speed up the feedback cycles, and then just as important, we monitor, uh, metrics like build times and test cycle times and failures and test results so that we can determine flaky tests. And we use all this to paint a picture of overall build performance so that it can be monitored over time by a group of production engineers. Okay? So that is fundamentally different than some other approaches to productivity that we've seen in the past. Uh, and I think we can, you know, kind of talk about two categories of productivity work. We have developer productivity management, and we have developer productivity engineering, right? And I think it's really important to separate the two, right? We're talking about developer productivity engineering in this talk, which has a different focus, whereas productivity management might focus on the people.


So how many lines of code are being produced? How many story points are, are being generated by, or, uh, how many, how many story points does a team's capacity have? What's that team's velocity, right? DPE focuses on the process and the technology, right? It says, can we make the build times faster? Not are they fast enough, but how fast could they possibly be given the right types of acceleration technologies, right? And so our metrics then are based on outcomes, right? I mean, symptomatically, a lot of them are the same as you'll get from a productivity management solution. Like this will absolutely increase a team's overall velocity. It will allow them to work through more story points per sprint or however you, you, you measure your productivity. Um, but it's gonna look at those raw concrete outcomes. The SDLC focus right now for productivity engineering really is just sort of in the, um, the, excuse me, the build and test parts of the SDLC right now, um, because that's where the majority of the developer experience happens, right?


Um, thi this landscape is changing a little bit, certainly, right? Um, you know, I think, uh, as we start seeing more infrastructure as code, and we start seeing more developers actually getting feedback from production, um, for very fast releases, so pushing things out to service mesh where they can wait, uh, network traffic and do things like blue-green testing or canary releases, or very easily as part of the release process, then we may start see seeing DPE sort of creep into more of the, uh, uh, CD and deployment side. But right now it's really focused on test feedback cycles, build feedback cycle times, and tracking that over time. And as a result, the ROI from taking DPE initiatives is very straightforward and easy to calculate. It's very hard and proven. So just to give an overview of the overall, uh, solution and the five, um, pains that we try to address with developer productivity engineering as a practice.


Um, we already talked about idle and wait time as a, as a pain, and as a result, we want faster feedback cycles. And so build caching and test distribution are the two technologies that we utilize there. We're gonna do a quick super quick demo of a build cache so that you can get a better idea of what that is, and I'll show you where you can link to a video on test distribution. Unfortunately, we don't have enough time today to do a demo on that one. Then the next thing, inefficient troubleshooting. So we mentioned that, you know, the, the, the build time is just one part of it. The build time and waiting on, uh, feedback is just one part of it. What if that feedback is negative feedback? What if it's a failure? How can we improve a developer's ability to debug that failure, collaborate on that failure with other engineers so that they can get to a root cause?


With that, we utilize something called a build scan. And you may be familiar with this if you've built, um, with Gradle in the past, if, if you haven't run a build scan, but you build with Gradle, you can do it right now, it's part of the open source tool. You can just do dash dash scan at the end of a gradle build and it will run a scan. You'll kind of get a, a chance to look at that. We'll, we'll see one at the very end. And these are also available for Maven, the Maven, uh, free Gradle enterprise plugin, kinda a freemium feature for Maven. Um, but you can also run, build scans against the Maven build as well. So what the BUILD scan does is just collect a whole bunch of forensic data about the build itself and context around the build, uh, and puts it in a shareable form and a URL that can be passed around the business.


Failure analytics allow us to detect proactively things like flaky and non-deterministic tests, and then other types of avoidable failures that, you know, failures that may be occurring, again from multiple developers, but there's no visibility, no one's actually tracking that. And so because no one's tracking it, um, there's, uh, that, that failure effectively never leaves that developer's workstation. Um, there's no metrics or KPI observability, no one's actually paying attention to how well builds are performing, um, in a lot of organizations. And so that's another pain point. No one is really paying attention to how much time developers are, are sitting, waiting for builds to complete or test, uh, test feedback to come back from, from, from, uh, testing frameworks. And so part of this is making sure that you're rolling up aggregating all that data centrally and being able to visualize it. And then a side effect of this, which really doesn't have anything to do with productivity, but since we're, you know, things like caching, uh, are allowing us to, and, and, and things like failure analytics and flaky test detection are, are literally, um, making the build systems do less work.


A side effect of this is that we can utilize our CI resources more efficiently. A side effect then is that if we're building out on cloud and we're worried about CI costs on cloud, which we all know is a creeping cost, um, by literally asking the build system to do less, we can save on those resources. So it's just a side effect of DPE. It's really not part of productivity, but it's worth mentioning. All right, so those are really the five pillars of DPE. And if you are aware and conscious of these pains, and if you're using technology and process to address those pains, then you're going along the path of developer productivity engineering. So let's look at what kind of impact this can have, because very fast, fast feedback cycles are really important. Let's take a look at two separate developer teams and just do a little thought experiment here.


We have 11 developers on one team with a four minute build time with you. Ask any Java Enterprise developer if a four minute build times killing them, they're gonna say no. They're gonna say it's fine. Um, but compare that to a team of six with a one minute build time and look how much more often they're able to build. Look how many more local builds they can run in the same unit of time, right? This second, team will in all likelihood be able to ship more better features because they're able to build more frequently. They're able to, um, uh, have a smaller change set per build. They're able to avoid merge conflicts more often because they're able to build smaller change sets more frequently, and they're able to experiment on the code base more frequently. When we start looking at, um, the savings per year at very large teams, when we take the same principle, look at a hundred developers doing 12,000 local builds per week with a nine minute build time, reducing that with our acceleration technologies like caching to a five minute build time can translate to 5,200 days a year in engineer savings.


Okay? So we talked about this build cache, and it's just a tool for fast feedback cycles. Uh, the, the, it was introduced to the Java world by Gradle in 2017. It's not the only build cache technology out there, um, but it is, um, available for Maven and Gradle. And it's important to understand that this is very different than a dependency cache, right? A dependency cache like, um, an Artifactory or a SONOTYPE Nexus, those hold your binary dependencies, fully compiled binary dependencies that need to be downloaded, uh, to various projects. And they're useful, right? They're, they're, they're, they're complimentary to a build cache, which actually caches, um, outputs from various tasks or goals within the build. So, you know, as a Gradle task completes, uh, the inputs from either the Gradle task or the Maven goal are effectively just cryptographically hashed. Uh, and a key is generated.


And if code hasn't changed, uh, or if tests haven't changed, that would affect the output. Uh, then when we actually go to run the build, we just generate the key look in the cash first to see if we had literally the exact same output that would've been generated based on the changes. And if we do, we just pull it from cash and it's usually a lot faster. Okay? Um, several open source projects, uh, more than just are this list are using this technology now, and they're seeing, I think, very dramatic results. The, uh, commons IO Java library has, um, is using caching and brought their bill from a minute 23 down to four seconds Spring Boot from 21 minutes down to six. And this is even better now, actually, we can go and we we're gonna look at this dashboard, uh, as the very last demo. We've got about five minutes left, um, at the, at Tommy e project, which builds lots of stuff. This is Tomcat, but then also things like active MQ for, for JMS, uh, from an hour, 27 minutes down to 20 minutes with, uh, caching. So we're gonna take a look at a super quick demo really fast.


Okay? So this is just a very simple Maven project. Uh, it's actually the, uh, camel Spring Boot router, uh, archetype that you can get from Maven Central Archetype repository. And I've modified it to add the, um, greater enterprise Maven extensions open source, uh, well freely REDISTRIBUTABLE extension that you can just add to your project. So we're gonna do a Maven Clean Verify, and this should just be like a normal run. Um, I think generally this project takes 20 to 30 seconds to build and test on a normal run. All right? About 10 seconds probably. 'cause I'm, I built it pretty recently. Um, but let's, let's run this again now with caching turned on and look how much faster. Okay? We pulled, um, several of these tasks from cash and the build time only took three seconds. And we can take a quick look at a Maven, excuse me, a gradle build scan here on a Maven build and take a look at our performance and we can see that we avoided, um, uh, 79% of almost 80% of our overall build time using the cache.


Okay, let's jump back into the presentation since, uh, we are getting near the end here. So build caching is one, acceleration, uh, technology. Uh, test distribution is another. Uh, test distribution is the ability to distribute test workloads across multiple agents in sort of an elastically scaling way. We don't have time to get into it, but we do have a really good video on how we've applied this to the Apache Cassandra project, and you can view it, uh, here. Okay? Now the other part of this we said was observability and data. Uh, what gets measured gets improved, right? We all know that, um, performance regressions are very easily introduced, um, into any type of build infrastructure, right? Changing office locations, refactoring the code, changing the way that we manage, uh, binary dependencies and things like that. All of this can have an impact on build performance.


And so it's really important that we maintain vi vi uh, vigilance over, uh, over that part of the build and make sure that those metrics are really well understood. And so that's the other two parts of, um, the other two practical parts of this, uh, practice is doing failure analytics and really making sure that we can detect things like, uh, flaky tests and then, um, making sure that we're watching build performance over time. So we're gonna take a look at another very quick demo, and then we're gonna wrap up. Um, this is the spring framework, uh, Gradle Enterprise dashboard. So several open source projects, a number of open source projects we just give this technology to for free. Gradle Enterprise is enabling technology for the practice of developer productivity engineering, and Spring has chosen to use it, uh, to augment their build process. So I want to point out two things here.


First, let's take a look at failures. Um, these are failures that have been aggregated across the build process for all spring developers. And look at this one right here. So we have these two types of failures, uh, non verification and, and verification failures. The non verification failure is like an infrastructure failure, right? It's something from maybe like a network timeout, whereas a verification failure is like a, um, an assertion that wasn't met properly, right? So like a programmatic failure. But we can find out, like for instance, right now that 26 builds have failed with this particular failure, we can see and drill down to all the various builds where it's happening. These are different local builds taking place on, uh, uh, for different users. And we can drill down right into the failure and really try to understand what's happened. And we can even take this link and we can, you know, link to really any part of this BUILD scan, but we can take this URL, copy it, we could paste it and give it to somebody else, and it'll take them right into the failure.


All right? So this is one way that spring is using it. Um, and then another way I want to point out is, um, the trends dashboard. And this is what allows us to take like a, um, you know, a a relative amount of time, maybe the last 28 days, working weeks for working weeks, and take a look at how our build performance has been. Um, how many builds have taken place, what's the cumulative build time. If we scroll down a little bit, we can see how much, uh, savings have been taken place by our cash. So this is the way that, um, productivity teams can remain vigilant over the overall productivity of the development group. Okay, let's close this out. Wrap it out. I think we're coming right up on time. So just as some next steps, we do have an ebook on this subject that you're welcome to download for free.


Uh, you can, uh, go and hit this, uh, link or just look up the developer productivity engineering ebook. You'll find it, take a look at kind of, uh, it was written by Hans doctor, the inventor of Gradle, uh, and, um, and the person who sort of coined the developer productivity engineering practice and term. So that's a good start. If you wanna learn more, try a free, a free Maven or Gradle build scan. Um, just if you, uh, have gradle, just do dash gas scan or just try the Maven extension, check out our documentation, read more about build scans, and of course, feel free to reach out to me, Justin ak, that's j ak if you have any questions. And we'll take some live q and a now.