US 2021
DevOps for Salesforce
Application development is increasingly happening on low-code environments like Salesforce. These SaaS platforms offer simplicity and make it easy for business specialists to build applications themselves. But managing their end-to-end development process is often far more complicated than the development itself. Andrew Davis, author of the book Mastering Salesforce DevOps, will explain how Salesforce has grown from a tool for salespeople into a $30 billion company driving the creation of millions of jobs. We'll look at the unique challenges of DevOps for SaaS solutions and the options for addressing them.
AD
Andrew Davis
Sr. Director of Research and Innovation, Copado
Transcript
Hello, everybody. I'm happy to have a chance to share with you today on DevOps for Salesforce and the full title of this talk is why the business has been bypassing it for decades, but is now in a way over their heads and how you can help without going insane. My name is Andrew Davis. I'm the senior director of research and innovation at Copano. I'm also the author of the book, mastering Salesforce, DevOps. So I want to level set. I want to make sure that, um, everybody here knows what Salesforce is, how it works, basically why it exists, so, and why you should care. So I want to go over upfront Salesforce, the business, the technology, the acquisitions, the community, the parties, the mascots, and the real estate. So I want to start, first of all, with sales, why sales, Salesforce, you're not in sales. Why do you care?
So why do you start with sales? Why did they start with sales? And why does it matter? First of all, profit, let's start with profit to get profit. You need revenue to get revenue. You need sales to get sales. You need customer data and to get customer data, you need a Rolodex. And if you want something that's a little bit easier to update than a Rolodex, you can use Microsoft Excel. So this is the origins and the roots of Salesforce. So, but prior to Salesforce, people had realized that maybe there were better ways because there were issues with the data attrition, salespeople would leave and they would take the Rolodexes with them. And they would entropy people, you know, particularly Rolodexes. They get wet in the rain and so forth. And there's management management wants to see the data. They want to see the customer information, and more importantly, they want to see your forecast.
So they want to get all that data in one system. And that system is, came to be called a customer relationship management system CRM, and the famous players prior to Salesforce was Oracle and Siebel. And then Oracle bought Siebel. So customer relationship management, I want to explain to you how to get started with Oracle or Siebel in less than three years and under $5 million. So first of all, you need to start with budget, CapEx, capital capital expenditure. You need to get all the money committed upfront, and then you need to get time and attention from it. This is an actual diagram with an actual it org right here. And then you need to install the CRM, which involves a lot of CD rom disks. And then you need to host your on-premise CRM, which involves a lot of servers, but Salesforce comes along and they think we've got a clever, disruptive idea.
What we're going to do is we're going to set this up. So you only have to pay monthly or yearly with a monthly, uh, pricing point. Um, that's operational expenditures, not cap ex, and even better. You can bypass it, which as we all know, is the root of all trouble until lays is the it team and there's software. There's not going to be any software and there's not going to be any service. So who wouldn't love it. So Salesforce was pushing the cloud before the cloud was cool. This is 1999. These people are in the cloud way early. And so summarizing the benefits of Salesforce, they host all the infrastructure. There's no AWS, there's no GCP, whatever you have to deal with major upgrades, three times a year, they provide the support, the security training, all that stuff, application itself built in out of the box addresses most of the common needs, and it is highly customizable.
Just got a lot more customizable over the years. So I started with the question why start with sales, but then the question becomes why stop with sales things have worked so well so far, there is one negative byproduct of sales, and that is that it generates customers. And the problem with customers as everybody knows, is that they complain. And so you need customer support. So what Salesforce did is they thought we'll help with the customer support as well. We'll provide a service cloud. Then they got an even better idea. We can't, it's very hard to provide customer support. Let's let the customer support each other. Let's have a community supported by the Salesforce experience cloud that where the customers can help each other and get all the information they need. And since it's going so well, let's get earlier in the whole process and let's get some prospects with the Salesforce marketing cloud.
So as you can tell, Salesforce has really spread out to address a whole lot of the business needs. And Salesforce has done a really good job with this. They've received a lot of industry accolades and are leaders in many categories, highly regarded company, very successful financially and high customer loyalty. Not only was Salesforce pushing the cloud from way back in the day, they were pushing low code. Nowadays, everybody's talking about low code, but Salesforce has been in this business for a good long while. So understanding low code, first of all, think about traditional application development. The way this works is you want to build your awesome app. You need to start with some custom code and infrastructure that costs some money and you need software developers. And they cost a lot of money, right? And they need a computer science degree from a reputable university.
And that cost a bunch of money as well. But Salesforce had the idea, Hey, let's do let's. We bypassed it once before let's bypass them again. Let's do the Salesforce development, the low code way clicks, maybe a little bit of code. Um, and for that purpose, you can use a Salesforce admin. Now you do need to pay them money as well, but it's less than you would typically pay a full software developer. And partly because they come from non traditional backgrounds, it's a good compromise. They can get paid more to be a Salesforce admin, you know, than in their previous job, but less than a full software developer. And there's no university degree necessarily require you can get trained on Salesforce, own training platform, Trailhead for free. Now there's one thing that I should mention that you do also want to pay Salesforce a bunch of that money.
So if you were wondering what to do with that excess cash, Salesforce has a suggestion. So custom application is I believe the fun part of it all, and this is the way it works in Salesforce. You've got a customizable database. You can build business process automation, that's graphical tools to build the business processes, there's code methods to do it, um, customizable UI again with graphical methods or code-based methods, but also there's a huge application ecosystem. It's a business app store and that's a quite cutting edge back in the early two thousands. When they started this program, the applications can have all of these aspects just built in naturally. And if that's not enough for you, you still need something more clever. Then you've got a robust version API that you can use to plug in your external systems. I mentioned these Salesforce admins. So I wanted to talk a little bit about the community and the culture and the world that Salesforce has built up around them.
And I really mean a world. It is huge. Dreamforce is the largest single vendor tech conference in the world, about 180,000 people back when you used to do that kind of thing. Um, they've got a lot of cute fuzzy mascots appearing in lots of different forms and a lot of, uh, focus on personas, especially people coming from non traditional backgrounds coming into the Salesforce platform, becoming awesome admins and so forth. And the Salesforce community is, shall we say, dedicated, highly, highly committed community of people building on the Salesforce platform. So that was just getting you up to speed on what Salesforce is now for the fun part. Let's talk about dev ops for Salesforce. What does this look like? How is it different if at all, which it is a lot. So I want to disambiguate two things. First of all, Salesforce and the Salesforce, people get these two confused, understandable.
And then I'm going to ask to ask you to check all of your dev ops tools at the door, because they're probably not going to work in the Salesforce platform. I'm going to talk to you about Salesforce metadata. I'm going to talk to you about XML hell sale, an initiative called Salesforce DX and as well commercial tools. And there are commercial tools that are available and do work for Salesforce. So in terms of building on the Salesforce platform, I mentioned low code. This is a picture on the left of building the schema on the right it's building the business process. He is one of many, many ways to make it easy to build on the Salesforce platform. So easy is great, right? But some conditions apply. The problem with easy is that easy means fast and fast means more and more means complex actually. So the idea with Salesforce is it's like building with Legos, you get all these individual nice building blocks, and the vision is you can assemble all of those building blocks into an amazing structure.
That is your Salesforce org, which looks basically like the Salesforce tower made out of Legos. But oftentimes what teams end up with is something a little bit more like this Jenga tower, where there is a significant amount of risk in making changes to that system because it has become so incredibly complex. And so Salesforce has gone from a world of easy to a world where we have so much easy. We're really struggling and it's not quite as easy and more. So the development life cycle in the Salesforce platform is where things start to get kind of complicated. Now think about our, of our awesome Salesforce admins. Now there are a lot of Salesforce specialists who do have, you know, CS degrees and, um, a lot of it architecture background and so forth, but it's a broad mix, right? So you have to think about the blend here.
You've got all these people who are learning to drive with Trailhead getting started, but you've got large app, large organizations where an increasing amount of their business processes are running on the Salesforce platform. I mean, some organizations going all in, you know, massive portions of their applications are low code on the Salesforce platform. And this is much more like traffic engineering as opposed to learning to drive. So a very different scale. Now, the challenges that you face include things like keeping your different orgs in sync with one another, including your development testing and production orgs tracking changes that you've made propagating changes systematically the risk of interactions and side effects, accumulating technical debt is probably sounds familiar to all of you from traditional systems. And so what you get is often something that looks like this. So this is often the way that the Salesforce orgs end up being a very, very tricky to manage.
And so when we talk about Salesforce, this is their marketecture, right? It looks like this. You've got all of these applications that do all these different things, but only a subset of this is what I think of as the Salesforce platform, because my skills are on what you call Salesforce core. And the underlying technology on Salesforce core is covering part of this, but not all of the different capabilities. So when we're talking about Salesforce here, just understand it's not all of the applications that they've acquired. So I want to take you back to a topic that you know, and love the difference between on-premise infrastructure as a service platform, as a service and software as a service. So infrastructure as a service has know, you know, they take care of some underlying aspects like the virtualization and the underlying networking and so forth, but you have to build on top of that platform as a service, you are, they take care of more of that stack that the hosting provider takes care of more of that stack, but you still need to provide the code and database and so forth and software as a service.
Basically they take care of everything. And all you have to do is configure the application and supply a bit of data or a lot of data. Now, Salesforce is interesting because it functions like a platform as a service, but the way you configure it is basically like a software as a service in the sense that everything that you do on Salesforce is configuration what Salesforce calls, metadata, um, all of the, even the code and soap that you load on to the Salesforce platform to cover the UI and the business processes and so forth. They're all treated like configuration. It's very different from just pushing files up to a service. So understand that difference. Um, and what that means in practice. When I ask you to check all your dev ops tools at the door, this is the periodic table of DevOps elements, DevOps tools, thanks to digital, uh, AI, uh, that we know in love, huge Pantheon of different tools that serve all different aspects of the DevOps process.
The problem is that almost none of these are relevant to Salesforce. There's only a very limited set that actually will work on Salesforce version control, a little bit of build scripting, maybe your CGI tools and so forth, but it's a limited subset. So we wanna think about, um, I often get involved in conversations where we have two communities that are meeting each other for the first time, the dev ops community, you folks, and the Salesforce community. Again, the Salesforce community is amazing. Dev ops community is amazing. And so I'm delighted to be in the middle of these conversations where these two communities are meeting, but you have to understand there's a knowledge gap. And there's a difference in the expectation. So the dev ops teams that the Salesforce folks are often meeting for the first time when they're starting to get into things like version control and deployment automation and so forth.
The dev ops teams make a bunch of assumptions about how applications work and these assumptions are simply not true in the Salesforce platforms. First of all, these dev ops teams kind of assume that everybody can just use kit, but on the Salesforce platform, that's not a comfortable experience, not a familiar experience for these low code click-based admins. The dev ops community tends to assume that an application is just a bunch of files. You can push them around, but Salesforce, remember it software as a service, it's not files it's configuration. And you tend to assume that if you need to merge the files, you can do so easily. I'll explain why it's not nearly so easy in the Salesforce platform to just merge files. You tend to assume that you can just, you have comprehensive build scripts. You build something in Java. There's tons of options for build scripts, but that's not true in the Salesforce platform.
Many more limited options, um, tend to assume that you can build test, deploy things very easily and automatically also not true. They, the deployment jobs and test jobs can run for a long time. You assume that you can build one and test on any computer, but you can only run these applications on Salesforce itself. And that you assume that you can define an app, an environment to run the application, but again, only Salesforce provides that environment. So in short, your dev ops, mind tricks or work on Salesforce. So I want to understand, I want to help you to understand how things do work with Salesforce. We start, uh, with environments, the operating environment. So Salesforce controls the underlying environment. The production orgs are hosted by Salesforce. The development and testing org called sandbox is they're hosted by Salesforce. There's a new kind of short-lived environment.
Scratch works. It can be used in some circumstances, but again, they're hosted by Salesforce. So every Salesforce instance is a big monolithic database. Um, this is Salesforce core and the Salesforce platform lives on top of this. Originally they were Oracle databases and grew. Salesforce is evolving that a bit Salesforce provides. These maintains them, um, does all of the database admin stuff, all that kind of stuff. And they provide a lot of standard apps. And these standard apps, they just work out of the box. You want a customer relationship management system. There it is service cloud. They just work out of the box. And there's also these third-party apps that you can install from these, um, the app exchange. Salesforce has app store, and you can build your own custom apps. And all of these things are integrated naturally because they're sitting on the same platform.
They have access to the same data. They can inter operate with one another. They can, um, you know, you can trigger, uh, one, uh, process and a third-party app from the inside of your custom apps and so forth. So nicely integrated takes away all those integration problems. That's why people want to build a lot on this one platform now where things start to get a little bit tricky. And again, I should say it's quite easy, really, very easy, relatively to build on the Salesforce platform. What gets trickier is when you want to begin to move things between different environments, where you have a development and a testing environment, you want to move your changes from that environment over into your production environment. So each Salesforce org has one doorway through which you can get changes out or in, and that's called the metadata API. So from the metadata API, you can pull a set of related metadata.
So you've got a little bit of user interface, a little bit of data layer. You've got a little bit of business processes, maybe some code config, whatever your related metadata and on your production, or you've got the metadata API. So your vision is that you just take it out of one, origami, send it to the other Oregon that basically works, but it's very easy to have a lot of deployment errors. And the deployment errors are things like you're missing dependencies missed a dependent field. Um, you're missing test coverage on part of that metadata. You've gotten malformed XML, especially if you're processing it by hand and trying to use version control. We'll come back to that and you can get these unknown errors. So I personally have addressed tens of thousands of Salesforce deployment errors. I'm pretty proud to say, it's not exactly that fun though.
And the errors can go on and on and on. So it's quite a pain now over the last few years, really just the last five years, mostly people have been beginning to try to track this stuff in version control for obvious reasons. And once you've got an inversion control, it's not a big step to begin to do deployment automation. That's where I kind of got involved in this whole activity of just trying to make the whole deployment process easier. But this is the big picture. Now Salesforce provides one built-in method for doing this in a declarative way in an admin friendly, low code way. And that's called change sets. Chain sets were introduced 13 years ago, and they basically haven't evolved much from then. I've got a really, really rough user interface, not what you want to be doing your work in. So, um, a few years ago, Salesforce launched finally because of this popular demand, uh, an initiative called Salesforce DX, which stands for development experience.
Um, so it's kind of a moonshot project really, really ambitious. And they thought way outside the box, they thought, okay, in an ideal world, how would or could Salesforce work and came up with brilliant ideas? And they launched it in 2017. One, one of the, one of their brilliant ideas was we need more funding for our development tooling teams. So those developer tooling teams increased significantly. And so that allowed them to get a lot more work done. They began to improve the metadata API and a lot of the other API APIs. They adopted visual studio code as the IDE of choice, moving away from eclipse. They have a new command line interface where you can do some scripting. They created this new kind of org scratch org. They created a way to packaging for enterprises, um, and an easier way of tracking your changes, the changes that you made in one word, just imagine you're a low code admin.
You don't know anything about the metadata API you're making changes. How did the changes that you make in the nice Salesforce user interface translate into metadata? It's a very tricky challenge. Um, people struggle with that now in 20 16, 20 17, when Salesforce DX came out, that basically created a Renaissance in terms of, um, open source activity on the Salesforce platform, where they began to really, um, grow the activity, the enthusiasm for things like version control. So the dotted line here indicates the percentage of GitHub repos that are, uh, related to Salesforce. And as you can see, when Salesforce DX was announced and in the years, following things really started to tick up. So a larger and larger proportion of activity on GitHub is done by Salesforce community. And the red area down here are repos that mentioned SFDX, which is an indication that they're using the Salesforce DX scripting lines.
As you can see, this is growing actively these days. So how Salesforce scripts work. Um, I mentioned, there's this metadata API. There's just a couple of ways to work with this. The traditional way is using amps. Now some of you may know ant from 2001, not the world's easiest tool to script in and so forth. And the Salesforce DX initiative launched this new command line interface SFDX, but both of these fundamentally are dealing with a large amount of XML. All this metadata is stored in an XML format. There is no other way to move changes out of a Salesforce org or into a Salesforce org. It doesn't matter what kind of, kind of tools you might throw at the problem. Commercial tools build your own and forth. They're all going through the metadata API. This is a little picture of Salesforce metadata. This is inside visual studio code.
Um, but imagine some of these files are literally hundreds of thousands of lines, long of XML, um, highly complex. Um, now the problem with dealing with XML, I mentioned about XML, hell that if you really want to manage XML properly, you have to parse it. So imagine I have these two snippets of XML here up in the upper left. How hard could it be? The answer is you don't want to know because when you try to merge them, you come up with something like how you hard don't could want it to be no, and that was not your intention in merging them. So get doesn't naturally do a great job of merging Salesforce, XML, which leads to a lot of pain. So I'm going to show you an actual merge of metadata in Salesforce. These are two XML files being merged in Salesforce. So a little rough.
Now that's just for two files. When you have a lot of files, this is what it looks like. And so this is an actual project that I worked on, huge number of XML files being merged simultaneously. And as you can tell, it's pretty rough. So, um, if you really want to get Salesforce dev ops working, you want to have a dev ops process version control the works on Salesforce. What you need is the intersection of these three things. You need somebody who knows Salesforce, you need somebody who kind of knows DevOpsy stuff, whatever that means broad term, and you need somebody who has time to write scripts. So as you can tell us a somewhat narrow intersection of those three categories. Um, so, uh, I had the good fortune with, uh, co Pardot, my employer to run a couple of, um, research reports. We had the very creative, clever name state of Salesforce dev ops report, not ripping off anybody else's ideas.
Totally our idea. Um, the technical practices of dev ops, we found are less mature than in the broader it industry. The adoption of version control much lower and so forth, but the software delivery performance is actually comparable because Salesforce is an inherently stable and fast platform. You can go a long way, just building in a haphazard way before you really encounter problems, but teams really struggle as they get bigger. And a lot of the companies that we work with it for part of, they have 100, 200, 500 Salesforce developers and admins working together in the same org. And one of the conclusions we noticed in here is that as teams scale up their lead time really starts tank. And the lead time, of course, um, you know, that means time to value. That means a delay in delivering real business value on that Salesforce platform, which is the opposite of what Salesforce is supposed to do.
So one of the reasons why you have such a long lead time is because it's all one big model, that thick database. And so everything gets really tightly coupled and all of the components become highly interdependent. And so these, this is an actual graph I created of Salesforce components and how they're inter related with one another. And, um, you know, I tried to pull it apart with some colors, but as you can see is very, very complex. So Salesforce launched this initiative of unlocked packages, promising modularization. The idea is take that what they call the happy soup of Salesforce metadata. I refer to this as the treacherous spider web of doom and refactor that sort of rationalize it as these package structure, where while still complicated, it's not nearly as complicated, uh, uh, a structure when she, once you divide things into packages, at least you can be clear what the interdependencies are.
And so the idea is you take all of this mix of org metadata, and you gradually fork it off into packages, package one, package two, and you reduce the loose Ord metadata. And these packages basically operate the same way that any partner apps you might install, uh, from the Salesforce app exchange would work. Um, and you have less of this loose org metadata that's on, on, on, uh, contained. But the challenge is that the obstacles to packaging are Legion. So it requires refactoring much of the code base. Um, and with, as with any refactoring effort, it's, there's no immediate, there's no immediate and obvious value add to the business. It would either stop or slow down other business initiatives. So it's hard to get investment buy-in from the leadership teams, and it requires some sophisticated software engineering techniques. Now there's brilliant people in the Salesforce world who figured out how to do this using solid principles, object oriented, design, and so forth.
But those are not necessarily the majority of Salesforce developers, um, or the majority of Salesforce developers, not necessarily at that skill level. Um, and the development environments, just setting up these scratch orgs can be hard to provision. In some cases it's a, not an, not, not necessarily an easy lift for all teams. And then once you've got things into packages, you have, um, a lot of difficulties with package interdependencies, unlike something like node JS, where every package has its own dependencies in a separate folder, um, they have this diamond dependency problem, as it's known in the Salesforce world, where you can only have one version of the base package installed in any Salesforce org. And so you, um, anyway, so a little bit tricky to sort out those dependencies, and then even if you do have it figured out, then you have all these other problems with multi pipeline packaging builds that are a little bit tricky to manage.
And so it's for all of these reasons that you have a number of commercial dev ops tools for Salesforce. So it's quite a niche. It's an unusual platform, uh, to develop on traditional tools don't work. So there's a unique market for commercial dev ops tools that help to address some of these challenges. Um, one of the main challenges that, uh, is commonly addressed is providing a graphical interface for these low code Salesforce admins to track their work. So you can use version control, even if you're a low code Salesforce app, that's very, very powerful and helpful. They provide a gooey for managing the flow of changes for visualizing, which changes in which environment, um, they handle the Salesforce metadata, all of that XML in a very intelligent way. And it's important to understand that while Salesforce itself is sophisticated, there are a number of very sophisticated applications that are in turn built on the Salesforce platform.
So these applications built on the Salesforce platform, they have their own configuration, but that configuration is stored as data as complex relational data. So they have their own networks of this complex relational data. So if you want to have a development life cycle for these complex applications for CPQ configure price, quote, applications, or industry specific apps, you need a way to move that data in a smooth way. These commercial tools, they also tend to provide, uh, tools for reporting and team collaboration and some such as co Pardot. Um, my employer, this dev ops tool is actually a Salesforce app itself. I know what you're thinking. That's a little bit meta, right? So it manages the Salesforce development life cycle, but it lives inside of Salesforce. So it's a Salesforce app. So it is customizable in a low code way, the same as any other Salesforce app.
So I'll give you a little bit of a visual of a Cokato in this case, you may not recognize it, but the screen that you're looking at is Salesforce. This is Salesforce itself. Um, this is, you know, roughly their user interface, but the central diagram is your, uh, delivery pipeline, your, uh, development environments, your testing environments, your production environment. So visualized in here along with visualizing, uh, the flow of packets of changes, user story, Quanta of changes in the history of your deployments and so forth. This also is a Salesforce user interface, and this is a value stream management tool actually built on the Salesforce platform to track metrics on your development progress in Salesforce. So how long do these user stories spend in the development phase and testing, and how often are they sent backwards and so forth? So as you can see, these are concepts to the broader dev ops world, but they've been ported onto the Salesforce platform for the Salesforce platform.
So lots of layers there, but hopefully that's giving you a picture. So if you're curious if you liked this topic, if you want to learn more one great way to learn more about Salesforce is to go to Trailhead. Salesforce is free learning platform, trailhead.com, awesome learning platform, gamified fun, very accessible, um, really, really excellent tool in terms of learning. Um, and if you search for dev ops, you'll find these two modules, uh, that I wrote together with Copano, uh, to explain of ops for Salesforce in a little bit more detailed, but there's all kinds of great content out there. So now thank you so much, really grateful for your attention and time. Here's the help that I need. Um, if you're working on Salesforce, let me know. I'm really curious to know who here has been faced with this challenge of dev ops for Salesforce or managing Salesforce deployments. Also, if this is sounding similar to any of the other platforms you're working on SAP, Oracle, Siebel, um, uh, service now, any of these other low code platforms, let me know as well. I'm curious to see how you're handling those changes. So thanks so much for your attention, really a delight to be with you today.