US 2021
DevOps for Salesforce
Application development is increasingly happening on low-code environments like Salesforce. These SaaS platforms offer simplicity and make it easy for business specialists to build applications themselves. But managing their end-to-end development process is often far more complicated than the development itself. Andrew Davis, author of the book Mastering Salesforce DevOps, will explain how Salesforce has grown from a tool for salespeople into a $30 billion company driving the creation of millions of jobs. We'll look at the unique challenges of DevOps for SaaS solutions and the options for addressing them.
AD
Andrew Davis
Sr. Director of Research and Innovation, Copado
Transcript
<silence>
Hello everybody. So I'm happy to have a chance to share with you today on DevOps for Salesforce. And the full title of this talk is Why the Business has been Bypassing it for decades, but is now in way over their heads and how you can help without going insane. My name is Andrew Davis. I'm a Senior Director of Research and Innovation at copa. I'm also the author of the book Mastering Salesforce DevOps. So I wanna level set, I wanna make sure that, um, everybody here knows what Salesforce is, how it works, basically why it exists, so, and why you should care. So I want to go over upfront Salesforce, the business, the technology, the acquisitions, the community, the parties, the mascots, and the real estate. So I wanna start, first of all with sales. Why sales? Sales force? You're not in sales. Why do you care?
So why do you start with sales? Why did they start with sales? And why does it matter? First of all, profit. Let's start with profit. To get profit, you need revenue. To get revenue, you need sales. To get sales, you need customer data. And to get customer data, you need a Rolodex. And if you want something that's a little bit easier to update than a Rolodex, you can use Microsoft Excel. So this is the origins and the roots of Salesforce. So, but prior to Salesforce, people had realized that maybe there were better ways because there were issues with the data attrition. Salespeople would leave and they would take the Rolodexes with them, and they would entropy people, you know, particular Rolodexes, they get wet in the rain and so forth. And there's management. Management wants to see the data, they wanna see the customer information, and more importantly, they wanna see your forecast.
So they wanna get all that data in one system. And that system is, came to be called a customer relationship management system, CRM. And the famous players prior to Salesforce was Oracle and Siebel. And then Oracle bought Siebel. So customer relationship management. I want to explain to you how to get started with Oracle or Siebel in less than three years and under $5 million. So first of all, you need to start with budget, CapEx, capital, capital, expenditure. You need to get all the money committed up front, and then you need to get time and attention from it. This is an actual diagram with an actual IT org right here. And then you need to install the CRM, which involves a lot of cd-rom discs. And then you need to host your on-premise, CRM, which involves a lot of servers, but Salesforce comes along and they think we got a clever disruptive idea.
What we're gonna do is we're gonna set this up so you only have to pay monthly or yearly with a monthly, uh, pricing point. Um, that's operational expenditures, not CapEx. And even better you can bypass it, which as we all know is the root of all trouble and delays is the IT team. And there's no software, there's not gonna be any software, and there's not gonna be any service. So who wouldn't love it? So Salesforce was pushing the cloud before the cloud was cool. This is 1999. These people are in the cloud way early. And so summarizing the benefits of Salesforce, they host all the infrastructure. There's no AWS, there's no GCP whatever, that you have to deal with major upgrades three times a year. They provide the support, the security training, all that stuff. Application itself built in out of the box, addresses most of the common needs, and it is highly customizable.
Just got a lot more customizable over the years. So I started with the question, why start with sales? But then the question becomes, why stop with sales? Things have worked so well so far. There is one negative, the byproduct of sales. Uh, and that is that it generates customers. And the problem with customers, as everybody knows, is that they complain. And so you need customer support. So what Salesforce did is they thought, we'll help with the customer support as well. We'll provide a service cloud, but then they got an even better idea. We can't, it's very hard to provide customer support. Let's let the customer support each other. Let's have a community supported by the Salesforce experience cloud that where the customers can help each other and get all the information they need. And since it's going so well, let's get earlier in the whole process and let's get some prospects with the Salesforce marketing Cloud.
So as you can tell, Salesforce has really spread out to address a whole lot of the business needs. And Salesforce has done a really good job with this. They've received a lot of industry accolades and are leaders in many categories, a highly regarded company, very successful financially and high customer loyalty. Not only was Salesforce pushing the cloud from way back in the day they were pushing low code. Nowadays everybody's talking about low code, but Salesforce has been in this business for a good long while. So understanding low code, first of all, think about traditional application development. The way this works is you want to build your awesome app. You need to start with some custom code and infrastructure that costs some money. And you need software developers. And they cost a lot of money, right? And they need a computer science degree from a reputable university.
And that costs a bunch of money as well. But Salesforce had the idea, Hey, let's do, let's, we bypassed it once before, let's bypass them again. Let's do the Salesforce development. The low code way clicks, maybe a little bit of code. Um, and for that purpose, you can use a Salesforce admin. Now you do need to pay them money as well, but it's less than you would typically pay a full software developer. And partly because they come from non-traditional backgrounds. It's a good compromise. They can get paid more to be a Salesforce admin, you know, than in their previous job, but less than a full software developer. And there's no university degree necessarily required. You can get trained on Salesforce's own training platform, Trailhead for free. Now, there's one thing that I should mention is that you do also want to pay Salesforce a bunch of the money.
So if you were wondering what to do with that excess cash, Salesforce has a suggestion. So custom application is, I believe the fun part of it all. And this is the way it works in Salesforce. You've got a customizable database, you can build business process automation. There's graphical tools to build the business processes. There's code methods to do it. Um, customizable ui again with graphical methods or code-based methods. But also there's a huge application eco ecosystem. It's a business app store. And that's a quite cutting edge back in the early two thousands when they, they started this program. The applications can have all of these aspects just built in naturally. And if that's not enough for you, you still need something more clever, then you've got a robust versioned, API that you can use to plug in your external systems. I mentioned these Salesforce admins.
So I wanted to talk a little bit about the community and the culture and the world that Salesforce has built up around them. And I really mean a world. It is huge. Dreamforce is the largest single vendor tech conference in the world. About 180,000 people back when you used to do that kind of thing. Um, they've got a lot of cute fuzzy mascots appearing in lots of different forms and a lot of, uh, focus on personas, especially people coming from non-traditional backgrounds coming into the Salesforce platform and becoming awesome admins and so forth. And the Salesforce community is, shall we say, dedicated highly, highly committed community of people building on the Salesforce platform. So that was just getting you up to speed on what Salesforce is. Now for the fun part, let's talk about DevOps for Salesforce. What does this look like? How is it different, if at all?
Which it is a lot. So I wanna disambiguate two things. First of all, Salesforce and Salesforce people get these two confused, understandable. And then I'm gonna ask to ask you to check all of your DevOps tools at the door. 'cause they're probably not gonna work in the Salesforce platform. I'm gonna talk to you about Salesforce metadata. I'm gonna talk to you about XML. Hell sale an initiative called Salesforce, tx, and as well commercial tools and that are commercial tools that are available and do work for Salesforce. So in terms of building on the Salesforce platform, I mentioned low code. This is a picture on the left of building the schema on the right. It's building the business processes one of many, many ways to make it easy to build on the Salesforce platform. So easy is great, right? But some conditions apply. The problem with easy is that easy means fast.
And fast means more and more means complex actually. So the idea with Salesforce is it's like building with Legos. You get all these individual nice building blocks, and the vision is you can assemble all of those building blocks into an amazing structure that is your salesforce org, which looks basically like this Salesforce tower made out of Legos. But oftentimes what teams end up with is something a little, little bit more like this Jenga tower where there is a significant amount of risk in making changes to that system because it has become so incredibly complex. And so Salesforce has gone from a world of easy to a world where we have so much easy, we're really struggling, and it's not quite as easy anymore. So the development lifecycle and the Salesforce platform is where things start to get kind of complicated. Now, think about our, of our awesome Salesforce admins.
Now there are a lot of Salesforce specialists who do have, you know, CS degrees and um, a lot of it architecture background and so forth. But it's a broad mix, right? So you have to think about the, the blend here. You've got all these people who are learning to drive with Trailhead, getting started, but you've got large app, large organizations where an increasing amount of their business processes are running on the Salesforce platform. I mean, some organizations going all in, you know, massive portions of their applications are low code on the Salesforce platform. And this is much more like traffic engineering as opposed to learning to drive So very different scale. Now, the challenges that you face include things like keeping your different orgs in sync with one another, including your development testing and production orgs, tracking changes that you've made, propagating changes systematically the risk of interactions and side effects, accumulating technical debt.
This probably sounds familiar to all of you from traditional systems. And so what you get is often something that looks like this. So this is often the way that the Salesforce orgs end up being, uh, very, very tricky to manage. And so when we talk about Salesforce, this is their architecture, right? It looks like this. You've got all of these applications that do all these different things, but only a subset of this is what I think of as the Salesforce platform. 'cause my skills are on what you call Salesforce core. And the underlying technology on Salesforce core is covering part of this, but not all of the different capabilities. So when we're talking about Salesforce here, just understand it's not all of the applications that they've acquired. So I want to take you back to, to a topic that you know and love the difference between on-premise infrastructure as a service, platform as a service, and software as a service.
And so infrastructure as a service, as you know, you know, they take care of some underlying aspects like the virtualization and the underlying networking and so forth. But you have to build on top of that platform as a service. You, they take care of more of that stack. The the hosting provider takes care of more of that stack, but you still need to provide the code and database and so forth and software as a service, basically. They take care of everything. And all you have to do is configure the application and supply a bit of data or a lot of data. Now, Salesforce is interesting 'cause it functions like a platform as a service. But the way you configure it is basically like a software as a service in the sense that everything that you do on Salesforce is configuration, what Salesforce calls metadata, um, all of the, even the code and so forth that you load onto the Salesforce platform to cover the UI and the business processes and so forth.
They're all treated like configuration. It's very different from just pushing files up to a service. So understand that difference, um, and what that means in practice. When I ask you to check all your DevOps tools at the door, this is, um, the periodic table of, uh, DevOps elements, DevOps tools, thanks to digital, uh, ai, uh, that we know and love huge pantheon of different tools that serve all different aspects of the DevOps process. The problem is that almost none of these are relevant to Salesforce. There's only a very limited set that actually will work on Salesforce version control, little bit of build scripting, maybe your CI tools and and so forth. But it's a limited subset. So we want to think about, um, I often get involved in conversations where we have two communities that are meeting each other for the first time. The DevOps community, you folks, and the Salesforce community.
Again, the Salesforce community is amazing. DevOps community is amazing. And so I'm, you know, delighted to be in the middle of these conversations where these two communities are meeting. But you have to understand there's a knowledge gap and there's a difference in the expectation. So the DevOps teams that the Salesforce folks are often meeting for the first time when they're starting to get into things like version control and deployment automation and so forth, the DevOps teams make a bunch of assumptions about how applications work. And these assumptions are simply not true on the Salesforce platforms. First of all, these DevOps teams kind of assume that everybody can just use gi. But on the Salesforce platform, that's not a comfortable experience, not a familiar experience for these low code click based admins. The DevOps community tends to assume that an application's just a bunch of files, you can push them around, but Salesforce remember it's software as a service.
It's not files, it's configuration. And you tend to assume that if you need to merge the files, you can do so easily. And I'll explain why it's not nearly so easy in the Salesforce platform to just merge files. You tend to assume that you can just, you have comprehensive build scripts, you build something in job, there's tons of options for build scripts, but that's not true in the Salesforce platform. Many more limited options, um, tend to assume that you can build, test, deploy things very easily and automatically. Also not true. They, the deployment jobs and test jobs can run for a long time. You assume that you can build, run and test on any computer, but you can only run these applications on Salesforce itself, and that you assume that you can define an app, an environment to run the application. But again, only Salesforce provides that environment.
So in short, your DevOps mind tricks won't work on Salesforce. So I wanna understand, I I wanna help you to understand how things do work. And with Salesforce, we start, uh, with environments, the operating environment. So Salesforce controls the underlying environment. The production orgs are hosted by Salesforce, the development and testing orgs called Sandboxes. They're hosted by Salesforce. There's a new kind of short-lived environment scratch orgs. It can be used in some circumstances, but again, they're hosted by Salesforce. So every Salesforce instance is a big monolithic database. Um, this is Salesforce core. And the Salesforce platform lives on top of this. Originally they were Oracle databases, and Salesforce is evolving that a bit. Salesforce provides these, maintains them, um, does all of the database admin stuff, all that kind of stuff. And they provide a lot of standard apps. And these standard apps, they just work outta the box.
You want a customer relationship management system. There it is. Service Cloud, they just work out of the box. And there's also these third party apps that you can install from these, um, the app exchange, Salesforce's app store, and you can build your own custom apps. And all of these things are integrated naturally because they're sitting on the same platform. They have access to the same data. They can interoperate with one another. They can, um, you know, you can trigger, uh, one, uh, process in a third party app from inside your custom apps and so forth. So nicely integrated, takes away all those integration problems. That's why people wanna build a lot on this one platform. Now, where things start to get a little bit tricky, and again, I should say it's quite easy, really very easy, relatively, to build on the Salesforce platform. What gets trickier is when you wanna begin to move things between different environments where you have a development or a testing environment, you wanna move your changes from that environment over into your production environment.
So each salesforce org has one doorway through which you can get changes out or in, and that's called the metadata, API. So from the metadata, API, you can pull a set of related metadata. So you've got a little bit of user interface, a little bit of data layer. You've got a little bit of business processes, maybe some code config, whatever your related metadata and on your production, or you've got the metadata, API. So your vision is that you just take it out of one org and you send it to the other org. And that basically works, but it's very easy to have a lot of deployment errors. And the deployment errors are things like your missing dependencies. You missed a dependent field, um, you're missing test coverage on part of that metadata. You've got malformed, XML, especially if you're processing it by hand and trying to use version control.
We'll come back to that and you can get these unknown errors. So I personally have addressed tens of thousands of Salesforce deployment errors. I'm very proud to say it's not exactly that fun though, and the errors can go on and on and on. So it's quite a pain. Now, over the last few years, really just the last five years, mostly people have been beginning to try to track this stuff in version control for obvious reasons. And once you've got an inversion control, it's not a big step to begin to do deployment automation. That's where I kind of got involved in this whole activity, um, just trying to make the whole deployment process easier. But this is the big picture. Now, Salesforce provides one built-in method for doing this in a declarative way, in an admin friendly, low code way. And that's called change sets. Change sets were introduced 13 years ago, and they basically haven't devolved much from them.
They've got a really, really rough user interface, not what you want to be doing your work in. So, um, a few years ago, Salesforce launched finally because of this popular demand, uh, an initiative called Salesforce dx, which stands for development experience. Um, so it's kind of a moonshot project, really, really ambitious. And they thought way outside the box. They thought, okay, in an ideal world, how would or could Salesforce work? And came up with brilliant ideas. And they launched it in 2017. One, one of the, one of their brilliant ideas was, we need more funding for our development tooling teams. So those development tooling teams ex increased significantly. And so that allowed them to get a lot of more work done. They began to improve the metadata. API and a lot of the other APIs. They adopted Visual Studio Code as the IDE of Choice, moving away from Eclipse.
They have a new command line interface where you can do some scripting. They created this new kind of org scratch orgs. They created a way of packaging for enterprises, um, and an easier way of tracking your changes, the changes that you made in one org. Just imagine you're a low-code admin, you don't know anything about the metadata. API, you're making changes. How do the changes that you make in the nice Salesforce user interface translate into metadata? It's a very tricky challenge. Um, people struggle with that. Now, in 20 16, 20 17, when Salesforce DX came out, that basically created a renaissance in terms of, um, open source activity on the Salesforce platform where they began to really, um, grow the activity, the enthusiasm for things like version control. So the dotted line here indicates, um, the percentage of GitHub repos that are, uh, re related to Salesforce. And as you can see, when Salesforce DX was announced, and in the years following, things have really started to tick up.
So a larger and larger proportion of activity on GitHub is done by Salesforce community. And the red area down here are repos that mention SFDX, which is an indication that they're using the Salesforce dx uh, scripting lines. As you can see, this is growing, uh, actively these days. So how Salesforce scripts work, um, I mentioned there's this metadata, API, there's just a couple ways to work with this. The traditional way is using ant. Now some of you may know ANT from 2001, not the world's easiest tool to script in and so forth. And so the Salesforce DX initiative launched this new command line interface, SFDX, but both of these fundamentally are dealing with a large amount of XML. All this metadata is stored in an XML format. There is no other way to move changes out of a salesforce org or into a Salesforce, or it doesn't matter what kind of, kind of tools you might throw at the problem.
Commercial tools build your own and so forth. They're all going through the metadata. API, this is a little picture of Salesforce metadata. This is inside Visual Studio Code. Um, but imagine some of these files are literally hundreds of thousands of lines long of XML, um, highly complex. Um, now the problem with dealing with XML, I mentioned about XML hell, that if you really wanna manage XML properly, you have to parse it. So imagine I have these two snippets of XML here up on the upper left. How hard could it be? The answer is you don't want to know. 'cause when you try to merge them, you come up with something like how you hard don't, could want it to be. No. And that was not your intention in merging them. So GIT doesn't naturally do a great job of merging Salesforce X ml, which leads to a lot of pain.
So I'm gonna show you an actual merge of metadata in Salesforce. These are two XML files being merged in Salesforce. So a little rough. Now that's just for two files. When you have a lot of files, this is what it looks like. And so this is an actual project that I worked on, huge number of XML files being merged simultaneously. And as you can tell, it's pretty rough. So, um, if you really want to get Salesforce DevOps working, you wanna have a DevOps process version control that works on Salesforce. What you need is the intersection of these three things. You need somebody who knows Salesforce, you need somebody who kind of knows DevOpsy stuff, whatever that means, broad term. And you need somebody who has time to write scripts. So, as you can tell, is a somewhat narrow intersection of those three categories. Um, so, uh, I had the good fortune with, uh, coppa my employer to run a couple of, um, research reports.
We had the very creative, clever name state of Salesforce DevOps report, not ripping off anybody else's ideas. Totally our idea, um, the technical practices of DevOps we found are less mature than in the broader IT industry, the adoption of version control, much lower and so forth. But the software delivery performance is actually comparable because Salesforce is an inherently stable and fast platform. You can go a long way just building in a haphazard way before you really encounter problems. But teams really struggle as they get bigger. And a lot of the companies that we work with that cap, they have 100, 200, 500 Salesforce developers and admins working together in the same org. And one, one of the conclusions we noticed in here is that a, as teams scale up, their lead time really starts to tank. And the lead time of course, um, you know, that means time to value.
That means a delay in delivering real business value on that Salesforce platform, which is the opposite of what Salesforce is supposed to do. So one of the reasons why you have such a long lead time is because it's all one big monolithic database. And so everything gets really tightly coupled and all of the components become highly interdependent. And so these, this is an actual graph I created of Salesforce components and how they're interrelated with one another. And, um, you know, I tried to pull it apart with some colors, but as you can see, is very, very complex. So Salesforce launched this initiative of unlocked packages promising modularization. The idea is take that, what they call the happy soup of Salesforce metadata. I refer to this as the treacherous spider web of doom and refactor that sort of rationalize it as this package structure where while still complicated, it's not nearly as complicated, uh, uh, a structure when once you've divide things into packages, at least you can be clear what the interdependencies are.
And so the idea is you take all of this mix of org metadata and you gradually fork it off into packages, package one and package two, and you reduce the loose org metadata. And these packages basically operate the same way that any partner apps you might install, uh, from the Salesforce app exchange would work. Um, and you have less of this loose org metadata that's un un uncontained. But the challenge is that the obstacles to packaging are legion. So it requires refactoring much of the code base. Um, and with, as with any refactoring effort, it's, there's no immediate, there's no immediate and obvious value add to the business. It would either stop or slow down other business initiatives. So it's hard to get investment and buy-in from the leadership teams, and it requires some sophisticated software engineering techniques. Now, there's brilliant people in the Salesforce world who figured out how to do this using, you know, solid principles, object oriented design and so forth.
But those are not necessarily the majority of Salesforce developers, um, or the majority of Salesforce developers, not necessarily at that skill level. Um, and the development environments, just setting up these scratch orgs can be hard to provision in some cases. Uh, not an, not, not necessarily an easy lift for all teams. And then once you've got things into packages, you have, um, a lot of difficulties with package interdependencies. Unlike something like node js where every package has its own dependencies in a separate folder. Um, you have this diamond dependency problem as it's known in the Salesforce world where you can only have one version of the base package installed in any Salesforce work. And so you, um, anyway, it's a little bit tricky to sort out this, but dependencies. And then even if you do have it figured out, then you have all these other problems with multi pipeline packaging builds that are a little bit tricky to manage.
And so it's for all of these reasons that you have a number of commercial DevOps tools for Salesforce. So it's quite a niche, it's an unusual platform, uh, to develop on traditional tools don't work. So there's a, a unique market for commercial DevOps tools that help to address some of these challenges. Um, one of the main challenges that, uh, is commonly addressed is providing a graphical interface for these low code Salesforce admins to track their work. So you can use version control, even if you're a low code Salesforce admin, that's very, very powerful and helpful. They provide a GUI for managing the flow of changes for visualizing, which changes in which environment. Um, they handle the Salesforce metadata, all of that XML in a very intelligent way. And it's important to understand that while Salesforce itself is sophisticated, there are a number of very sophisticated applications that are in turn built on the Salesforce platform.
So these applications built on the Salesforce platform, they have their own configuration, but that configuration is stored as data, as complex relational data. So they have own their own networks of this complex relational data. So if you wanna have a development life cycle for these complex applications for CPQ, configure, price quote applications, or industry specific apps, you need a way to move that data in a smooth way. These commercial tools, they also tend to provide, uh, tools for reporting and team collaboration and some such as coppa. Um, my employer, this DevOps tool is actually a Salesforce app itself, and I know what you're thinking, that's a little bit meta, right? So it manages the Salesforce development lifecycle, but it lives inside of Salesforce. So it's a Salesforce app, so it is customizable in a low code way, the same as any other Salesforce app. And so I'll give you a little bit of a visual of, uh, COPPA in this case.
You may not recognize it, but the screen that you're looking at is Salesforce. This is Salesforce itself. Um, this is, you know, roughly their user interface. But the, the central diagram is your, uh, delivery pipeline, your, uh, development environments, your testing environments, your production environments, so visualized in here along with visualizing, uh, the flow of packets of changes, user story, quanta of changes, and the, the history of redeployments and so forth. This also is the Salesforce user interface, and this is a value stream management tool actually built on the Salesforce platform to track metrics on your development progress in Salesforce. So how long do these user stories spend in the development phase and testing, and how often are they sent backwards and so forth. So as you can see, these are familiar concepts to the broader DevOps world, but they've been ported onto the Salesforce platform for the Salesforce platform.
So lots of layers there, but hopefully that's giving you a picture. So if you're curious, if you like this topic, if you wanna learn more, one great way to learn more about Salesforce is to go to Trailhead, Salesforce's free learning platform, trailhead.com, awesome learning platform, gamified fun, very accessible, um, really, really excellent tool in terms of learning. Um, and if you search for DevOps, you'll find these two modules, uh, that I wrote together with kipato, uh, to explain DevOps for Salesforce in a little bit more detail. But there's all kind, all kinds of great content out there. So now thank you so much. Really grateful for your attention and time. Here's the help that I need. Um, if you're working on Salesforce, let me know. I'm really curious to know who here has been faced with this challenge of, uh, DevOps for Salesforce or managing Salesforce deployments. Also, if this is sounding similar to any of the other platforms you're working on, SAP, Oracle Siebel, um, uh, ServiceNow, any of these other low-code platforms, let me know as well. I'm curious to see how you're handling those changes. So thanks so much for your attention. Really a delight to be with you today. <silence>.