Supercharging PayPal’s Application Development

My name is Rama Kolli. I'm a product manager for developer experience in the Core Platform and Infrastructure team at PayPal, and I'm here to talk about our DevOps journey so far, with a particular focus on the developer experience we have delivered to PayPal developers.

This is a high-level view into the structure of this talk. I'll start off with an introduction to PayPal, its business scale, as well as the technology scale involved. I'll take us back a few years to what a developer's experience at PayPal was like originally, before the DevOps platform started offering self-service experiences for the software development life cycle. I'll share some of the lessons we've learned along the way and a look ahead to what we have planned next in the development of the platform, and then we'll have some time for questions and answers from you folks.

A brief history of PayPal. You may already know that PayPal originally was started in 1998, when Max Levchin and Peter Thiel started Confinity. It quickly grew to gain one million users and then merged with Elon Musk's X.com in 2000.

PayPal went public in 2002, and that was the year when eBay Inc. acquired PayPal as well. Fast-forward a decade to more recent history. By 2013, PayPal became available in over 200 markets worldwide. Many credit products were launched. PayPal separated from eBay Inc. in 2015 once again and relisted on the Nasdaq.

In 2016, we celebrated over ten years of support for mobile payments. And more recently, in Q3 of this year, PayPal has grown to support 218 million customer accounts, doing over $100 billion of total payment volume per quarter, with the mission of democratizing financial services. The goal is that PayPal will improve the financial health of both merchants and consumers worldwide.

To also get a sense for PayPal's business scale, over $13,000 in payments are processed by PayPal every second.

If we look at the technology scale that enables this level of business performance, what you see when you use PayPal is developed by over 4,500 developers spread throughout the world. These developers are working on a code base that contains more than 50 million lines of code. Our build infrastructure handles over one million builds a month of this code base and results in over 100 terabytes of application artifacts that get deployed into production.

There are over 2,600 applications in production of various kinds: web applications, REST services, message consumers, batch applications, et cetera. And these applications are deployed to a site that spans three regions and nine availability zones. Over 600,000 production cores support this workload, and this actually results in over 40,000 batch executions per day and over 230 billion hits per day across this infrastructure.

Now, going back a few years, if we go to 2013, this is what a developer's life looked like at the time. Let's say a developer decided that they are going to build an application to support some customer experience. The first thing they did is to file a ticket, and this ticket gave them access to the infrastructure that added support for this new application into all the set of tools that were a part of the SDLC process at PayPal.

Typically, it took a few days for this ticket to be handled and approved by an architect before a developer could proceed further.

After creating an application, let's say our developer wants to write their code, build it, and deploy it to a test server. To get a test instance was yet another ticket, and depending on capacity, it could take a while before our developer would get some test capacity to use for his work.

Let's say the developer wanted to connect to some other service to leverage some PayPal functionality. That's yet another ticket. And somebody had to look at it and approve that request, saying that, yes, this application A should legitimately connect to application B.

After developing their application, after certifying that code, when our developer is ready to deploy their code to production, that took yet another ticket to go create the production pool to deploy to.

To actually release the code, a developer had to sign up for a push window, which took another ticket. To increase capacity to the pool is another ticket. To do smoke testing of functionality before pushing code, that's another ticket.

Developers were drowning in tickets. In fact, the primary activity for developers was not to write code, but to actually file and follow up on tickets. Now, this is not an experience that any competent developer will put up with. So it was high time that we joined the DevOps revolution and offered a self-service experience for SDLC activities at PayPal.

In short, to keep track of these KPIs as we move to the self-service experience, before we started on our DevOps journey, it took a developer days to create a new app, weeks to be able to build and deploy their app to a test server, and it literally took months for a new application to go live.

We've since then moved from that world into an entirely self-service experience for all SDLC activities at PayPal. These SDLC activities are enabled through a custom platform-as-a-service implementation that PayPal has developed. And I'll walk through each of these phases and talk through what the developer experience is like as developers confront each SDLC phase.

First is the planning phase. Let's say our developer in the current world is trying to build an awesome customer experience. What they'll likely do is look through a web portal that lists all the canonical payments functionality that is available as REST APIs. It's up to them to then figure out what is the best experience they can build across the capabilities that they can integrate with. Or they might choose to actually go find and fill a gap in this portfolio because they are not able to do what they want to do.

The planning phase experience actually offers the ability for developers to consume other services, and if there are gaps, to actually offer to fill it with the help of guidance and consulting that the API governance teams can provide.

API governance is actually built into the platform as well, so that when developers look through APIs available on this portal and then later on integrate against them, they actually get what they expect to see.

Our developer now has a great experience that they want to build in mind. They know the services they are going to call, and now they want to start creating their new application.

The platform actually offers them a menu of language stacks that they can get started with. Developers have their preferences of the ecosystem that they like or one that they are very productive in, and we are mindful of that choice as we offer options to create applications on our platform.

Web app experiences at PayPal have been built for a number of years on the Node.js stack, on middleware that was built and open sourced by PayPal as KrakenJS. A lot of our Java workloads for building services and message consumers and batch jobs is actually based on a Spring-based stack that our team has curated and offered to PayPal developers. A set of highly scalable reactive applications are also built at PayPal on our own open sourced squbs stack as well.

Now, what goes into offering these choices to our developers? We are focused on solving for the most number of workloads that PayPal developers are usually creating, as well as the platform team takes up the challenge of implementing the security standards specific to PayPal into the language stack that is offered. A lot of our work also goes into instrumenting these language stacks so we can then monitor these applications once they get deployed across the different environments.

Now, this combination of these popular language stacks and integration into our DevOps SDLC lifecycle is such that over 1,000 applications have been built since this experience started being offered, and these applications actually are now serving over 85% of web traffic and also over 50% of internal traffic at PayPal.

Our developer has now created their new application, and a part of the application creation process is also the provisioning of the build environment for this app. There is actually a dedicated build instance created for this app that the developers have complete control over. They can choose to upgrade or enhance it with plugins that they are interested in using.

At the same time as their build instance is being provisioned, a test playground is also created for our developers. And a part of the application creation process is where a new code base for this app is created in GitHub. A build of that code base happens, and then the manifest that is created gets deployed to the test playground, and all of this now takes less than 20 minutes to do.

So our developer is able to make a "hello world" call to their brand-new application, not just running on their desktop, but actually into their own test playground.

Developers actually at this point are set up with a CI and CD pipeline as well, and all they do is then focus on building their application logic and adding to the test suites that are created for them.

Now, as application functionality is being built up, let's say our developers are going to integrate against any number of services that they have previously identified. And in the past, they've had to run all those dependencies themselves on their playground, and there was a lot of productivity lost from developers needing to keep all of these applications going at all times.

Instead of that, today, all 2,600 PayPal applications are deployed and run for developers to integrate against in a test cloud. And the scale of this deployment is such that there are four billion calls being made to this test cloud deployment of PayPal applications that support all developer activities.

So you can kind of see how developers' focus is now on building their application logic and being kind of the artist they are in writing their code instead of spending their time on the mundane activity of keeping up their test playground or their dependent services.

After an application has been built and once it is certified, now it is time to deploy this application to production. Where in the past, this step depended on an operations team to go create the production pool, today, this is an entirely self-service experience.

Implementation in the platform actually allows that all the site architecture principles that exist are supported by this self-service provisioning of resources. For example, there are rules around what type of app deploys to which availability zone. For example, web tier applications need to be deployed into the front tier. Batch workloads actually go deploy into the mid tier. These architectural concerns are implemented into the platform, and a developer simply chooses to provision resources, and provisioning happens where it needs to.

Whether resources need to be provisioned into DR colos or into sandbox colos, it's all handled by the platform.

We talked about releasing code to production in the past requiring a particular release window. Now, imagine you're a developer and your release window happens to be Friday afternoon, and you're dependent on release engineers to actually push your code and let you know whether the push was successful or not, and you're on standby to figure out if there is an issue with the push, if you need to resolve any issue. And I've known developers whose weekend plans were impacted because this push ended up going into early hours of Saturday morning. And this is not an experience that our developers enjoyed, just like anywhere.

Unlike that, deployment to production is a self-service activity for PayPal developers today. The timing of your deployment is entirely up to you, whether based on the availability of resources in your team or when it makes sense for your business, not based on release windows where there are release engineers available to handle your push.

Now, you can imagine deploying code to production can be a pretty risky event. And while you're in charge of how much risk is involved in a particular deployment, the platform supports you by offering various deployment strategies. For example, you can choose to deploy your code to one host in your production pool, observe its behavior for a while before choosing to further deploy across other hosts in that pool or across other colos.

The deployment system also is watching site health metrics at all times as you're rolling out your code and will actually pause and alert you to indicate if site instability is detected at any point. So you as a developer then have the capability to take your time, figure out what is going on, and identify if it actually makes sense for you to continue with your code deployment, or that if you should actually roll back and figure out if maybe you caused that site instability yourself.

So this platform has been available for a while, and actually, it's currently supporting over 4,000 deployments a day to 15,000 production instances.

Going back to the menu of language stacks we have available. If you recall, I shared how the platform team actually instruments these language stacks, so that when applications are created for them, they are able to be monitored and supported in the production site.

Now, the metrics that get published through these instrumented stacks actually result in a self-service experience for monitoring application health across all environments, and this is the experience that both our developers and site operators rely on.

Now, if you can look at this image, you can actually see that there are key metrics like transactions per minute, errors per minute, and latency for calls that's automatically calculated and displayed for our developers to look at. Also, availability across the different colos that the application is deployed to. So using these aggregated metrics, developers can figure out whether their service-level objectives are being met or not.

And apart from aggregated metrics, the monitoring platform also provides support for tracing individual requests across the multiple hops that happen within PayPal's data centers. So if a particular request fails, tracing is available to figure out why exactly that failed.

The monitoring platform actually supports over 440 terabytes a day, as over 350,000 metrics per second are generated from these applications and help us figure out whether the site is doing well or not.

Apart from the out-of-the-box capabilities, there are also custom dashboards and alerts that are supported on the monitoring platform in a self-service way. And our developers have created over 1,500 custom dashboards around particular application behaviors that are more meaningful for them, that help them support the health of their application in production.

PayPal being a payments company, a large part of payment settlement actually happens as a batch workflow. And these batch workflows are orchestrated by an enterprise scheduler provided by BMC called Control-M.

Now, while building these batch applications has been a self-service experience on our DevOps platform so far, our developers would like to also have a self-service experience around scheduling these batch jobs. We're working on a self-service experience for this using the automation APIs available from BMC's Control-M.

Integration of batch apps into the DevOps platform, as well as integration with Control-M, actually mean that Control-M agents are provisioned along with the applications and get propagated to all the different environments that batch apps are deployed to.

Over 5,000 executions per day happen on these batch applications that have been onboarded into our DevOps platform, and the self-service experience for scheduling actually is targeting those specific workloads.

Using the rich editor that developers have access to, they can define their batch schedules. They can actually execute ad hoc runs or dry runs of these schedules to make sure that the schedule has been authored correctly, and can then choose to publish that schedule for continuous ongoing execution.

The DevOps platform actually also has capabilities for self-healing. Signals are being read from over 140,000 production instances to identify if there are outliers of performance behavior. Now, if a particular host is showing distress through high CPU utilization, for example, the self-healing platform can automatically issue a restart.

If it determines that a particular host is unreachable, the platform can also replace that host and can deploy the same version of the application that's running on all the other hosts in that pool automatically onto that new host as well. To date, over 1.8 million remediations have been done by this self-service platform since it went live.

Now, let's compare what our developers are saying about this DevOps experience as compared to their experience in 2013. Of course, they love it.

Our developers are a lot more empowered and productive as they are using this DevOps platform. These testimonials actually show that they are able to create POCs and evaluate both functional and non-functional behaviors of their code as they're trying to build customer experiences.

They are thrilled by the ability to deploy their apps to production on their own at any time of their choosing. And you can compare the months that it took new applications to go live to a metric that our developer has shared with us, which says that in less than two weeks, they were able to create a new application, certify it, and deploy it all the way to production.

At this point, I think I can say that the rate at which change is deployed to production is not bound by limitations of the platform, but it's more a function of how fast developers are able to write their own code and certify and deploy it.

Let's take a look at a few lessons learned.

While our interest in building the DevOps platform was to increase agility by offering self-service capabilities, this should never come at the cost of instability to the site. And our site operations partners actually have built up an expertise over a long period of time. They have insights and also tools that they have developed that help them maintain the stability of the site.

It's incumbent on us, as we are building a DevOps platform, that we are mindful of all the tooling and restrictions that exist that help the stability of the site, and that they are built into the platform. It's a non-trivial thing to actually work with operations and understand their perspective and make sure that their concerns are actually built into the platform, and site stability is not actually sacrificed at the altar of agility.

Now, when you think of a DevOps platform, you're primarily thinking about application developers taking many self-service actions for their own application and its pools. But site operations is typically responsible for managing the stability of the entire site, and that includes a mixture of new applications on the DevOps platform, as well as legacy applications that are critical to the business.

So it's important that all of the operations tools continue to work, whether an app is legacy or has been built on the new platform. And also, the platform can offer a management interface for operations to take site-wide actions like, for example, bringing new colos in or out of traffic. So site operations is a key partner in the DevOps journey.

Now that a self-service experience is available for developers, what we've learned is that now the bar is higher for everything that we offer. Now, when it took days to create applications and months to push applications to live, perhaps there were enough pain points that certain other actions that developer took did not surface as huge bottlenecks. But it turns out that as many actions become self-service, now your target moves to other areas that developers are focused on that now become your problem areas.

And developers are now used to a much more rapid pace for being able to do their work, and they are not going to be as tolerant for delays in the life cycle anymore. So this immediately shows up other opportunities for automation that you can then invest into.

One example for this happening is essentially the self-service scheduling that I was talking about. Creating batch applications has been a self-service activity for over a month. But our ticket-based scheduling process, that is still very manual, currently comes with a two-week SLA, and that is not something our developers are happy with anymore. And that's really why we chose to invest in self-service scheduling of batch workloads as well.

Developers have a lot of freedom now. They can do what they want to do when they want to do it. But is that free? It really is not. It comes with a cost to the business in particular.

So the 4,500 developers that PayPal has, they've created over 12,000 test playgrounds for themselves. And as long as these test instances are helping developers be more agile and do their jobs, this is great. But if these 12,000 instances are not being used by developers efficiently, then that comes at the cost of business that I was talking about.

So it's necessary that the platform actually builds the right kind of behaviors in our developers so that they are mindful of the cost to business that they're imposing through the freedom and agility that is available for them.

Now, building these right kinds of behavior is possible by actually giving them visibility into the cost of the resources that they have provisioned for themselves. And that likely will help, but it could potentially be extended to charging back for the resources that developers have started consuming from the platform.

Now, where there is a self-service experience, there is the user experience that the developer has. And the more intuitive and user-friendly that experience is, the more developers are able to be self-sufficient and help themselves. Where actions are not intuitive, where things don't work as they expect, then that really quickly shows up in support costs for your platform.

And given developers are your customers here, even things like access to logs, more meaningful error messages, these are all a part of the experience of developers, and we need to focus on making sure that they're as user-friendly as possible.

Now, there may be a number of self-service experiences that are available, but if these are all built as independent tools that don't look and feel the same way, that still adds to an overhead for developers to know and figure out where to go find a particular tool and how exactly to use that tool. So preferably, a unified experience for the entire platform is what a developer would like to experience, and that way they are more self-sufficient.

Let's take a quick look at the timeline of development for our DevOps platform. 2014 is when capabilities on the DevOps platform became available, and we started off with support for web workloads early in Q1 of 2014. We added support for services later on that year, and we quickly found that over 400 applications had been built on top of the DevOps platform, with developers orchestrating all the self-service experiences for themselves.

2015 was when we had to separate from eBay Inc. once again, and we needed to invest into our platform so it worked independently after the separation happened. And following that, in 2016, we added support for batch workloads. We started our containerization journey, and the DevOps platform started supporting Docker containers in 2016 as well.

Earlier this year, we actually deployed our dev workloads into the public cloud, so our DevOps platform actually took on the challenge of supporting a hybrid cloud environment. And to date, as I mentioned, over 1,000 applications have been built on this DevOps platform.

And from here on out, we have a few more thoughts around where our investments are going to be.

This DevOps platform is actually powering the needs of 4,500 developers throughout the world, and immediately, the availability of the platform becomes directly proportional to developers' productivity. So it's really important that we track the success or failure of all the developer interactions that are happening on this platform.

And we also track exactly how long a certain action takes. Because not only should the platform be available for developers to do their work, but also the faster any of the action involved is, the more productive our developers can be.

So all developer actions result in metrics that we track to ensure that the site is not only healthy but is improving over time. And there's going to be numerous actions, such as creating applications, for example, that are not exercised that often. But we still want to have visibility into the health of the platform, so we run a lot of synthetic tests that we use to figure out how healthy the experiences that are offered to our developers are.

And what you'll see is kind of interesting is towards the right, the scale that's listed over there, which actually shows that anything less than 90% is considered a hard failure for the platform. And our goal is very, very high 90s, to be able to do any of these self-service activities at all times.

PayPal being a payments company, and also all the concerns around security that you hear all around you, there's huge opportunity for tighter security integration into the platform. As I mentioned, there are a lot of open source artifacts used for building applications at PayPal, and we are investing in scanning all the code that is written at PayPal and all the applications that are built at PayPal to figure out where any vulnerable dependencies are being used.

We're investing into the platform to be able to remediate these vulnerabilities in production at scale without actually needing to depend on application developers themselves to address these vulnerabilities.

All code changes are going to be gated through a security review process to make sure that before they end up in production, that there are no known vulnerabilities reported against them. We're articulating secure coding practices for all the different language stacks that developers are held accountable to be satisfying. And the platform will make sure that these security concerns are addressed before any particular code is deployed to production.

Containerization is a long-term effort, and there are significant investments around containerization into the platform. One of the key benefits of containerization is that by having a container image as a unified packaging and deployment format, our DevOps platform can then be made to understand this format, and thereby it standardizes the tool sets used across both the new applications built on the DevOps platform, as well as all the legacy applications that were created before.

And after a platform that understands containerized workloads, now there is an opportunity to not only restrict new application creation to those few language stacks that we previously supported, but also to allow onboarding custom applications that support specific non-standard workloads into the platform as well. It is also the means for us to sunset support for legacy applications over time, after onboarding them into the containerized platform.

Containerization as a service, which is a clustered deployment of containers, will help us gain a lot more fleet efficiency as well. And containerization is also a focus for driving a lot of developer productivity benefits for our developers as well.

So that, in short, is our journey so far on the DevOps platform. And I'm happy to take any questions.

Q: How did you manage complex relationships between different deployable assets? So one developer has a deployment, but what if he's dependent on six other services?

A: All application development actually happens isolated at PayPal. And all the interactions between applications are through these standardized interfaces. And if you recall the API governance portion that I mentioned earlier, there are restrictions around backwards compatibility that ensure that changes to the site are not happening in such a way that they are breaking other applications that depend on them.

And integration test environments actually ensure that these standards are met before any code is rolled to production. So it's very much supported by the platform and is also supported by the policy around how changes are accepted at PayPal.

Q: Thanks for the nice presentation, Rama. One question I have: looking at your presentation, looked like you had a very smooth ride. So I want to understand from you that so far, what are the challenges you faced in this journey? And in the past, what is your current challenges and how you are addressing them?

A: Sure. So I wouldn't necessarily say that it's been an entirely smooth journey, but a large part of it was also deliberate in terms of how we started down this journey.

So if I go back a few slides, and if you look at the timeline for development of the platform, you can see that we first started off with support for web workloads early in 2014. So focus on web workloads actually let us add support for stateless web experiences. That offered a simple deployment pattern, and it was a clear small target for us to build our platform originally on. And over a period of time, we've incrementally added capabilities and started supporting new workloads on the platform.

So it's actually been a deliberate effort to plan out the development of the platform in such a way that we start off with an MVP use case and then incrementally build on things from there.

Now, in terms of challenges, there are some challenges that we could have foreseen and there are some that we could not. Like, for example, the separation of PayPal and eBay Inc. was one of those unforeseen challenges we had. So as you can see, it actually impacted our development of the platform with significant portion of 2015 actually being spent on standing up a standalone DevOps platform for PayPal.

So the challenge actually turned out to be in terms of being able to react to things that were not foreseen and adjusting our schedules based on those events. Did that help address the question?

Q: Sure.

A: All right.

Q: I think you're good. Thank you. You considered writing in-house platform for the continuous delivery, the SDLC one. Have you had any consideration for vendor ones, and what was the criteria for choosing to invest into your own?

A: Sure. When we started on our DevOps platform journey in 2014, actually we were a part of eBay Inc. at the time, and a DevOps platform had been developed at eBay for use a couple of years previously. And because eBay and PayPal were sister concerns that operated together for over a decade, there were large influences in how site architecture was defined and what processes were developed for managing changes to the site.

And actually, it turns out that this ready-made platform that was already available helped us roll out a DevOps experience much faster than starting from scratch on a vendor-provided system. So that was a large part of how the platform was developed.

Of course, since 2015, we've been investing into the platform and developing it pretty independently. And if you look at support for the particular hybrid cloud deployment that we have tackled and the particular rollout of containerized workloads that we are handling, these are all being driven by PayPal's business decisions rather than anything else.

So I think by using our own platform, we both got good time to market as well as the ability to support the specific concerns for operations, for example, that PayPal has in a world where we both support new applications as well as legacy applications at the same time.

I think there was a question from the back over there.

Q: Yeah. Do you have a dedicated team to this? And do they have open source contribution from all the developers?

A: Absolutely. The bottommost layer of our cloud deployment is an infrastructure-as-a-service deployment that's built on OpenStack, which certainly is also open source. The platform-as-a-service experience that we've built on top is very much proprietary, but the containerization-as-a-service implementation that we have built is actually something we're looking to open source.

And as well as many of the language stacks that we support actually are a mix of open source technologies that we have curated for PayPal, or where we see gaps, they are open source technologies we have contributed back to the community as well.

So throughout the deployment, it's actually a mix of proprietary and open source technologies that offer the combination of experiences that we feel is right for PayPal.

Q: And sorry, just one question. Do you have a dedicated team that's assigned to this?

A: Yeah. We have the Core Platform and Infrastructure team that I'm a part of, is actually the dedicated team that supports this platform. And this organization actually has engineers that support all the physical assets from the data centers all the way to the application stacks that we provide on top of this platform.

One more question in the back, and then I can come back here.

Q: Yeah. The self-service tools that you mentioned, what kind of technologies you used to develop that?

A: So it depends on the SDLC phase that we are talking about, essentially. The platform-as-a-service implementation is proprietary and is not open source. The various language stacks that we have available are actually a combination of curated open source technologies or open source software that we have developed ourselves. And the infrastructure-as-a-service layer is based on OpenStack and so on and so forth.

So it's a mix of technologies with the right combination of behaviors that are appropriate for PayPal. It's not necessarily a clean one-size-fits-all type of model.

Q: Do you mind briefly touching on the branching strategy that you went with for all of your applications and artifact creation?

A: Sure. Branching strategy is perhaps there are many opinions and ideas on how that needs to be developed. In fact, it turns out for applications built at PayPal, we settled on a simple branching strategy, actually.

There's a master branch, a release branch, and a develop branch. The master branch actually represents the live version of code of applications running in production at all times. The release branch is where code is available for deployment into an integration test environment prior to it going live. And the develop branch is where active development happens.

So that, at a high level, is kind of a uniform branching strategy that applies to all applications at PayPal. And it, in our view, offers this right template that we can rely on as we try to manage the site at scale. And beyond those branches, developers are free to manage their branches on their own.

Q: I'm assuming all of this is a private cloud. Is that right? Or is it public?

A: Not necessarily. It's a hybrid cloud deployment.

Q: Okay.

A: Most of our production deployments are in our private cloud, but our dev workloads actually run in a public cloud today.

Q: Does the test environment, QA environment, run in a public cloud then?

A: It does.

Q: It does. Okay.

A: Yes.

Q: So why the hybrid? So you're saying all of the data's in the private, everything runs in a private cloud in production, but yet on the test side, it does in a public. Why do you deploy it that way?

A: Sure. Many reasons, actually: security, legal, et cetera, and also the site architecture as it stands today as well needs to change before we could actually run in an active-active mode.

So PayPal's payments functionality in production, it actually needs to undergo some change before production workloads can actually work in a public cloud. Whereas our dev workloads actually offered an isolated deployment that we could actually bring to the public cloud earlier, and we could take advantage of dynamic pricing and scaling that we typically need to rely on without investing in our own cloud deployment.

Q: And is that based on the sensitivity of the application? Is that kind of a decision that's made whether you're going to use the public environment for deploying a test app versus a private? Is it based on the sensitivity of the app?

A: No, not necessarily. All of our dev workloads of all applications actually happen to a VPC in the public cloud. But there's no actual data in the test environment that reflects anything in production. So there are particular conditions before we could move our dev workloads into the public cloud. But it's not based on an application-specific basis at all.

Q: Okay. Just one other thing. Do you have any measures on the usage of your dev workloads on public cloud? Just to have a student of some instance, you know, address that down, it's just going to keep on running. Have some kind of a guardian, something of that sort?

A: Right. So that's certainly an issue today, as I kind of alluded to a little earlier. One of the ways we are choosing to combat that is by doing cluster deployment of containers for our workloads, and that way we can actually shut down and initiate new workloads anytime we need to. But today it certainly is an issue until we move to a more clustered deployment of our workloads.

Q: Touching on Chris' question, you guys have to deal with PCI compliance, right?

A: Right.

Q: Were there some challenges you guys had with developing this DevOps solution for us to take?

A: Absolutely. Actually, just like different site operations concerns that I alluded to, all PCI requirements, all data handling requirements, all InfoSec requirements that always existed at PayPal was something that the platform was built to support.

So it wasn't necessarily the other way around. These restrictions were not necessarily negotiable, and actually the design and implementation of the platform had to account for it, and none of them, to my knowledge, were actually changed. And if you ask me, that's kind of the right way to go about it.

Q: Did you guys face challenges because of that?

A: Not that I know of. I'm sure that all the development actually, as it happened, was reviewed with all the concerned teams: legal, compliance, InfoSec, and so on. And in many cases, we actually had to account for those restrictions as we implemented various things. But I can't think of anything that comes up as a particular challenge that I bring up, actually.

Any other questions I can answer?

Okay, great. Thank you very much for your time, and I'll be around to chat with you all further.