Scaling Enterprise BizOps by Automating DevOps Practices

Hello everyone. Hope you're all doing good in these tough times.

We are excited to be here at DevOps Enterprise Summit 2021. Today, I will talk about how we scaled enterprise business operations by automating the majority of our DevOps practices while taking care of cloud and data security.

Along with me, I have Akash Mahajan. He's the CEO and founder of Kloudle, who was the lead cloud-native security architect; and Ramesh Karra, the VP of GTM and business at BYJU'S, who has helped us understand the business needs.

My name is Prashanth, but I go by the name Praz. It was my team who was tasked with scaling of business operations during the massive global danger that we are facing right now. It is my privilege and honor to share our learnings and approach with all of you today. I especially want to mention Anne and Gene, who have guided us with empathy for us to get here in front of you.

In the interest of time, I will lead the presentation and hand over to Ramesh and Akash wherever necessary.

Ramesh, over to you.

India is our largest market, right? And the reason is quite simple. It has the largest school-going population in the world. We have close to 260 million students in India. But while the quantity of learners is very large, we still rank very poorly as far as the quality of the education we deliver is. Our PISA scores are very low. And the reason is very simple: there's not enough high-quality teachers, high-quality content going around, right? Access is a big challenge.

Obviously, with the rise of internet penetration and smartphone penetration, there's a great opportunity to address this.

Let's go back a year, to April 2020. This was an unfortunate time, really, but it was also a very critical time and seminal moment for us. What happened really was India went into a nationwide lockdown, very similar to a lot of countries across the world, all in a bid to control the pandemic.

The lockdown was pretty intense in India. Apart from the fact that everybody was confined to home, obviously one of the key things that happened was schools were shut, students stuck at home. And remember, this is a time when schools are not really equipped to do any kind of digital learning. That's when BYJU'S decided that we had to step in. It was more a responsibility and it was taking our mission forward.

What we did was we did a number of things. One is, of course, we had an app which had access for a certain number of period, but we made it completely free. We opened it up for pretty much every learner. We also launched additional offerings. We launched a product called BYJU'S Classes, which was an after-school solution for students. And in this case, it pretty much became the primary solution, given that even online learning schools were pretty much shut down. The idea there is students get access to online classes from India's best teachers.

But remember, at the same time, the entire organization was confined. They were sitting at home, and pretty much this entire mission had to be done remotely. It was a challenge to all the stakeholders, but especially DevOps. So DevOps was pretty much the team, the backbone that had to ensure that this mission can get the best.

That's pretty much the context. I'm handing over to Praz to walk you through how we addressed it.

Thank you, Ramesh, for sharing the business perspective with us. Now that all of you have an idea of the enormity of making this happen, here's what we figured we will do.

We were clear from the beginning that these are our three pillars from our frame of reference. Everything that we do, we're bound within these three pillars. The pillar of data security meant that all aspects of DevSecOps, security alerts, preparing for an incident, in case an incident happens, and how do we react to it when bad stuff happens, and benchmarking with the global standards was encompassed within the data security pillar.

The pillar of continuous delivery meant automating all of the builds, deployments to servers without humans manually running commands on servers. And logs of all of these builds were available to all the developers. And Docker as our application containers was made available because of continuous delivery.

The pillar of continuous integration meant approval-based merge for production, semantic versioning for all the applications and artifacts, verifiable releases by following the entire build pipeline and integrity checks, and also monitoring the entire pipeline for its performance and build time. This was enabled through the continuous integration pillar.

Before we move further, I believe in show and tell, so here are the results. A simple customer-focused metric: the impact of our work meant that 12,000 students could simultaneously attend live teacher-led classes during the nationwide lockdown with all the schools shut. I'm not sure how big are the schools in Europe, but for us, this is equivalent to about ten full-fledged schools running.

When the weather was actually fine, we were preparing for rainfall. But what we did not expect was a massive thunderstorm called COVID that took over all of our lives in the last one year.

We were never planning to go fully remote as an engineering team, but we had to do that, and this was unexpected. We were used to using tools like Slack, Hangouts, Google Meet, Zoom and all that, but that was only when working across geographies or when we couldn't travel. We had to move our entire sales team of 12,000 people to a work-from-home model from a feet-on-street model. The entire business operations team had to use digital tools that we had to develop and deploy during the pandemic to this massive scale while dealing with a completely new way of working.

While the senior leaders have had the experience, our average age of engineers was 25 years, and this was the first time ever that they would be without mentors or someone who's onboarding them physically present with them. And this was a new challenge for us.

And the way we did this is by templatizing the pipeline. When anyone new joined, it was easy to get their code merged into the development branch, and once the code was merged by the tech lead, the developers could test their code. The development environment was maintained with data being auto-generated. We used custom scripts to push dummy data into the development databases so that no PII information existed there. The environment acted as a testing ground for new features.

Once the developers tested their code in this environment, now they could request it to be merged to the main branch. Code merged to the main branch could be done by multiple teams who had access to these repositories. Code merged to master branch triggered a build on staging environment, which was meant for automated testing by the QAs. Once a release tag was added to the main branch, the code deployment was triggered with a semantic versioning of the code deployed. Tags could also be added to park a release for later use. Alerts on Slack would ensure that the deployed code could be quickly tested on all the various environments, so we are not affecting the day-to-day operations. With this approach, we were able to achieve more than 500 deployments in a week when it was at peak.

If you remember, for us, the first pillar is about data security. So handing over to Akash to take us through this.

Thanks, Praz. We realized that if COVID goes on for some time, everything is going to get scaled even more. And the time to solve for security was at the beginning when we were just embarking on this. We came up with some principles or guardrails, if you will, on how we proceed.

Relying on globally accepted standards for application security like OWASP, infra and cloud security like the CIS benchmarks, we felt was the way to go. And by adding a step of approval to the pipelines, the team leads and managers felt reassured that they won't get nasty surprises, suddenly something will break in prod. So they had that.

We were already very big on automation, so creating a hard boundary between prod and the rest of the entire infrastructure was a no-brainer. This allows us for reducing the attack surface and having a blast radius if things do go horribly wrong.

We additionally deployed centralized log solution based on Elastic Cloud for developers with data masking and access control to make sure production environments don't require DevOps to remotely manage anything. Most of what we did, we documented in a markdown-based knowledge base, using continuous delivery, so that was fun.

And the important thing to remember here is that these are principles for us that we wanted to be guided by, but we weren't dogmatic to let them become a blocker, right?

So keeping in mind the principles and being pragmatic, we designed the pipeline to be a simple release-based model. And the primary interface for engineers was access control based on GitHub commits with the pull requests that team leads approved and merged. Once satisfied, anyone can make a release to staging by using a specific tag, and this automatically triggers a build, right? And this was done with Jenkins.

Once an app has been staged and the team lead is happy, they deploy to production, right? That's the approval step here. And we're deploying to AWS Fargate, right, after the QA was done.

The way we were doing it is by creating an artifact of an already-running application, writing it to, let's say, a secure S3. And this particular Jenkins only had access to write and no listing, nothing, with IAM permissions. Once the artifact was fully copied to S3, this triggers an event to start a job in production Jenkins, and no developer has access to this.

You will notice while the applications are running on managed service AWS Fargate, we chose to keep the Jenkins servers running on EC2. It works without hitches, and there is a ton of documentation we can refer to if required, and this is us being pragmatic here.

The important thing to highlight is that all the secrets required for the production were being added via pipeline during the job. So we would have source code, and the CI server would interact with the Secrets Manager service, which is a cloud-native thing, and figure out the dev and test secrets. Once you would have the artifact, and now the CI server is going to create a production secret, right? So they have access to that. They will bake it in the application, and the container will run in prod.

Thanks, Akash. Those were some great takeaways. Moving on.

For us, what was the real triumph of shared culture, automation, monitoring aspects of DevOps was that when the business was geared to scale massively, our team not only delivered and continued to deliver all the apps that were already working as before, but also was able to release and massively scale new applications without having to train them on a new way of development.

While you see, on the one hand, the business was trying to launch new features, build new applications, and also was trying to scale all the applications that existed. While on the other hand, our development and DevOps team ensured that the built-in efficiency that was in the system, and the continuous monitoring and the shared knowledge that was already there, ensured that all of these could happen without disrupting any of our existing applications.

This also meant that people were not stressed. In fact, by not having to commute to office, many of them experienced a more fulfilling workday. Imagine 100-plus engineers who are solving a higher purpose of education for all, well-rested, secure in their homes, enjoying building applications for the future, and not getting bogged down by the toil that is getting introduced as a security measure or half-done DevOps transformations.

I personally didn't lose any of my sleep because the transition was so smooth for us. I'll hand over the security aspects to Akash again.

The approach Praz and the business have towards making all their teams are taken care of is a great example to emulate. I love working with these folks for these reasons.

Now that we have passed the questions such as how do I transform my business digitally? What role would COVID play in that transformation? I think COVID basically accelerated digital transformation for everyone. We want to talk about security around that.

We have seen that going digital and leveraging the public cloud has become the baseline for everyone who is not limited by industry or regulatory compliances. But at the same time, COVID has stress-tested the security of the best of security companies. And at Kloudle, we ended up working with a bunch of customers who were compromised when their privileged users started working from home or remote. IP address-based security wasn't really a good idea anymore.

And this is where the maturity of the BYJU'S BizOps team has shone through because of the focus on data security, automation. Our level of maturity extends to something as commonplace as an SSH login to a production server. Sometimes it's required for the production troubleshooting. When someone logs in using SSH on a Linux server, it triggers an alert for the team, and the person who needed to SSH becomes part of a mandatory root cause analysis process. And when everyone is dealing with massive change, this allows us, the security team, to educate and make everyone aware of the responsibilities and the level of threat they may be facing without realizing it themselves.

So this is a great example of what we've managed to do as part of the cloud transformation. We envisioned the security team to always be an enabler instead of being a blocker. When employees are well-rested, they not only perform better, they're likely to adhere to the processes that keep data secure. This reduces their potential stress levels, and we can do all of that, and all of us can do with that in these times.

And this is what Praz had to say when we were able to achieve all this: that Kloudle monitored our entire cloud infra and provided a full view of security, enabled us with critical alerts that required our immediate attention. Reducing alert fatigue is a huge thing. Our engineering team grew 10x, but we still follow the straightforward GitHub release patterns while security magic happens natively.

That's a great testimonial for me to validate us being enablers instead of being a blocker. We always envision the security team to be an enabler instead of a blocker. We did this by building a continuous monitoring stream with automation to ensure that it didn't matter where the admins were sitting while doing the work, because COVID.

Based on the continuous stream of data, we reduced toil by issues getting added to the issue tracker automatically. If security issues were being reported, they were getting added to vulnerability management tracker automatically. This really helps with reducing the grunt work. And also updating Slack when a build got deployed. And by bringing a lot of these moving parts of operations, including issue tracking, to a single pane, we enabled discussions and conversations in the team Slack. While everyone may or may not participate, this fosters a culture of openness, and when new members of the team feel that they can contribute to the discussions as soon as they're ready, because it's all happening in front of them.

This is great. What I want to highlight here is that there's a lot of culture, automation, sharing that we paid attention to, to be seen as enablers instead of the traditional sentiment that security teams have of being gatekeepers and just making everything difficult for everyone.

As part of the security aspect, we had to basically continuously monitor the cloud-native services for security. And the reason to do that is that we needed to have visibility, because without visibility, how would we know, do we have data security? Do we have it now? And cloud is a big-ass API. So do we have the security now, or has someone's made a change which is problematic?

So we want to answer questions like: do we have EBS, Elastic Block Storage, with encryption? Because if we don't, then we may not be compliant. And in some cases, we ended up migrating older legacy unencrypted EBS by automating the copying of the volume snapshots.

Do we have incident response readiness? Are we ready to face security incidents? Is the process that we have based on the incident response guidance provided by AWS or the NIST Cybersecurity Framework? They kind of overlap.

Security alerts: you need visibility for that. What happens if a developer disables their 2FA to the AWS account? Do we get alerted? Should we get alerted? Is the alert working? Do we get alerted if an S3 bucket becomes public? If a new security group is open to the world with a management port available?

We also want to do some automated remediation using CloudWatch events, and this was great because we ended up doing this completely cloud-native with CloudWatch events and Lambdas and Fargate-based containers. What were we remediating for? Disable console access of user if no 2FA. Make public S3 bucket private if we don't have a specific kind of tag that we wanted. So primarily, we were monitoring for security, and we wanted to focus on that. That's why doing it the cloud-native way and not like anything else, which is readily available in the market.

We want to enforce security processes. You can call them guardrails or whatever it is, right? Guidelines which have to be enforced in runtime during operations. Can we make sure that no virtual machine, EC2 machine, can come up without IMDSv2, which prevents problematic attacks against applications like the server-side request forgery, right? It's a huge thing. It's a huge challenge.

We want to kind of do the coding standards check in the pipeline. We want to see if dev is committing secrets in the wrong place, right? And then getting committed in GitHub, and this is a problem because people have read access. Pushing vulnerabilities discovered to an open source vulnerability management software like DefectDojo, and using tags to ensure that resources had proper metadata or they're removed.

And we also ended up doing a lot of regular security audits, whether it was for the GitHub access control, the AWS scan for cloud perimeter, or the internal scans, or just auditing the privileges of IAM roles and policies regularly, right? The continuous part is the keyword.

This got reflected in many lessons we learned. We also had to get past some unique constraints that required new thinking. In India, we encountered the fact that teams were scattered over 25-plus states with varying levels of internet access and quality of access. The scale of this may be a bit difficult to imagine for the audience. I found this somewhere: that the entire European Union is just about 30% bigger than India. All those countries in the European Union, and the landmass is just 30% bigger. So it's massive in India, and the internet connection, the quality of access, where you are, if you're on mobile, 4G, all of that can have an impact.

So using the old-fashioned way of doing access control at network layer, the jump hosts, the multi-hop kind of bastions, which may require proprietary VPN, SSL software, was not really an option, right? We wanted to move fast. Everyone is working remote. We deployed WireGuard using infra as code whenever there was a need to provide, so that we could say that, hey, to the database that someone needs access to, we don't know what static IPs they'll be coming from. But by allowing access with WireGuard, we made sure that we catered to the low-bandwidth situations in multiple different internal services and databases, NoSQL databases, and even hosted Kafka were all provided with access control like this.

Thank you, Akash. We continue to grow in terms of number of engineers. Forty just were added in April. Applications that are being used by our business and the number of deployments are ever-growing.

We were in a unique position when it comes to the last one year. So our DevOps and the way to work with automation-first approach has taught us that this movement has been truly transformational for us. We are able to support the incredible goal of education for all. While a lot of people are in crisis all around us, we feel that it is our privilege and our duty to support and share what we were able to cause.

I'd like to thank Akash and Kloudle. They have provided us with the ability to monitor and take action without having to change how we do things. This makes it easy for us to teach new team members, and that has proven to be the biggest advantage that we have got out of them.

Thanks to Ramesh for sharing the business need and enlightening us with the thought process behind the business decisions that were taken in the last one year. Thanks to the organizers for giving us this big opportunity to present in front of all of you.

Thank you everyone for sticking around. We'll be happy to take questions on the Slack channel. See you all there.