Scaling DevSecOps adoption in a Large IT Services Firm

The session would highlight the experience of TCS, a large IT services organization, in deploying DevSecOps at scale. We have designed a comprehensive DevSecOps framework. This is abstracted into an assessment method for benchmarking the DevSecOps maturity of various projects. The assessment provides the quantified maturity score in various DevSecOps domains and dimensions. The assessment report includes the project-specific recommendations to elevate DevSecOps maturity. At the project level, the assessment findings are contextualized to come up with the improvement roadmap for implementation.


At the organization level, 530+ projects have already been assessed. The projects are on their DevSecOps improvement journey.

The session would describe the TCS journey, an overview of the assessment framework, and the approach of promoting improvements in the organization.

DC

Dhruba Chaudhuri

DevSecOps DevOps SRE Site Reliability Engineering Process Design Specialist Functional Lead, TCS

LP

Leena Pradhan

Delivery Excellence, TCS

Transcript

00:00:14

Good morning, good afternoon. And good evening everyone across the globe. It's a great pleasure for us to be sharing our experience in this great forum. The DevOps develop enterprise summit, we are from data consultancy services, a large it services firm. I'm Lena za, leading service reliability engineering practices within corporate delivery excellence group. The charter for our group is to enable our engagements to deliver reliability in their services to our end customers. Uh, we look at the best in class engineering practices, which are prevalent in the industry. We look at these practice implementation across our different, uh, customer engagements and then scale these practices across the organization.

00:00:57

Hello Tru here having 28 years of it experience and I'm currently working in corporate delivery excellence function. My focus areas are devs and <inaudible> practices. Data consultancy services is a it service company serving a very large customer base across geographies countries and industry verticals. Our each project is unique and a different use case in terms of domain technology, methodology, culture, maturity, and complexity. Today, we would like to share our experience story on how are we driving devs adaption at scale embracing diversity.

00:01:38

Before we go into the details of how we are scaling these depths practices across the organization, we would like to share what triggered this, uh, thought process in the year 2017. Uh, our leadership team set us one vision for the entire organization, which is to be enterprise agile by 2020. We were not looking at agile only in the it projects, but agile in every function. So the entire organization was driven by this one vision, which was set by our leadership team. Uh, we went about doing the transformation. Uh, we looked at the process transformation. We leaned as required. We did a people transformation to enable the teams to be able to invite agile behavior in all their practices. And we also did a workplace transformation, which enabled a seamless and collaborative working experience for our associates with a strong deployment focus rigor and a continuous leadership direction.

00:02:36

Uh, we achieved our vision and 2020 was the enterprise aha moment for us with this, uh, solid foundation on agile. We now have, uh, more than 80% of engagements being executed in agile and this, uh, transformation. It brought another objective in front of us to reinforce the engineering practices to go deeper into Devco practices, DevOps, uh, compliments a from a technology standpoint and thus, uh, this transformation brought DevOps into both prominence across the organization. Uh, this transformation also paved us the path to a faster realization of business value. We understood what it takes to be able to deliver faster and quicker. We are now able to help our customers to achieve their business objectives, to stay competitive. Uh, as against the other players in the industry, our customers are able to respond to the industry demand with much more agility. And this also brought in a lot of efficiency into the various processes.

00:03:38

Now that was all on the customer space. So if we look at what is happening on the technology space, uh, in the last few years, we know we have seen that there is a significant rise in the digital footprint in every business because small, uh, the industries have gone, uh, mobile. There is a lot of web presence in all the industries and the organizations are also looking at multi-channel, uh, approach for delivering their services to the end customers. And perhaps the pandemic has accelerated this movement across all the industries. Uh, this also, uh, saw an increased exposure to various cyber threats, which are there. So what it meant was that we require a very fine balance between speed, reliability, resilience, and quality, and hence deficits is becoming a default ask now. So what was the need in, uh, certain pockets or in certain, uh, industry or projects?

00:04:36

Now it has become a default need for every other industry. And, uh, around the same time, our customers were also, uh, getting more interested into dev SecOps and they were asking, what are the thesis's capabilities in implementing devs? And with this, uh, GI transformation, we also found that the accounts which were the early adopters of agile, they had already done multiple cycles of, uh, transformation, uh, for faster service delivery. Uh, they had done transformation in the terms of people process and the print technology or the automation enablers, and, uh, they were, uh, they have now moved from the earlier larger and less frequent releases of let's say yearly or half year to much smaller and much more frequent releases of quarterly, monthly, um, fortnightly, and some even have, uh, release on demand capabilities. Okay. But when we look deeper, we found that, uh, these are all diverse implementation and there is also to some extent, there is a difference in the understanding of what DebOps is among the different teams and the experience and learning this also mostly stayed within the teams and teams were kind of relearning and reinventing the wheel, even though the same practice has been already implemented elsewhere in the organization.

00:05:53

So what, uh, essentially it means that we were faced with a challenge of scalability and multiplicity in organization, uh, such a big organization. Like I like us. So for a process group like us, it was, uh, very imperative that we needed to have a standard framework to scale devs across the organization. That's when we came up with, uh, the thesis dev SecOps, uh, framework, it's a process, uh, ex exhaustive process framework, which helps, uh, to implement these, uh, dev SecOps practices, uh, across the different, uh, um, different, uh, uh, engagements. It helps us to institutionalize these practices across various teams, uh, with different diversity. And it also addresses the challenge of the higher incubation period or the higher, uh, readiness, uh, cycle time. It also gave us another opportunity to be able to benchmark our different accounts in terms of their depth across maturity implementation. Uh, this is an exhaustive com and comprehensive, uh, process framework, which helps us to strengthen our core capabilities in the engineering space.

00:07:03

Uh, the framework is a four layer, uh, structured framework. There are five domains, uh, practices, eight practice areas within them, then 40 plus themes, and more than 80 practices, the five domains are continuous planning, continuous development, continuous testing, release our deployment and continuous monitoring and improvement, continuous planning, uh, domain. It deals with the practices which are related to product conceptualization and product planning. It also looks at the team and the foundational readiness in terms of scaling and the automation capabilities, continuous development looks at the practices, which are related to development, peer reviews, security practices, and, uh, continuous integration. Continuous testing looks at the practices related to, uh, test design, uh, test strategy, test data enablement, and test environment enablement for the various functional and non-functional, uh, testing continuous release and deployment. It deals with the practices which are related to product validation, the release approval release process, and also the different, uh, zero downtime strategies for deployment, continuous monitoring and improvement.

00:08:13

This looks at the practices which are related to monitoring, uh, observability. And it also looks at what kind of insights can be derived, uh, based on the data which is collected throughout the delivery life cycle. And this framework has been designed, uh, with the best practices from Ts matured implementations, and also the practices which are more prevalent and upcoming ones in the industry. It's a multidimensional framework. Uh, it addresses the culture that addresses processes, the framework addresses people, the practices address the various aspects of quality security, reliability, resilience, and the framework has a special focus on the automation aspects of it, uh, for different, uh, practices such as for testing integration, uh, build deployment monitoring and so on. And the framework also covers in, in depth, the various KPIs, the pipeline KPIs via the software is being developed, the application performance, infrastructure, performance, and health, and also the product performance in terms of the business KPIs. These dimensions have been very carefully and thoughtfully selected the ones which are essential for, uh, achieving the goals of dev SecOps.

00:09:29

The framework covers the entire breadth and depth of the service delivery. Uh, all those it practices that I mentioned earlier, they cover the entire breadth of the service delivery life cycle. And for each of those it practices, we are going into the depth of each of them. When I say depth, it means, uh, who is practicing, uh, this practice, whether it is the product team, or it is some other team who is helping the product team in executing that practice. So that tells us about the self sufficiency or cross-functional liability of the product we are looking at. What is the readiness in terms of the skills which are required, and in terms of the automation capabilities, whether the practice is being in, uh, implemented offline, or it is a practice integrated as a part of the pipeline itself, we are looking at how early is the practice being done to understand the left shift culture of the team.

00:10:22

And if it is an automated practice, whether it is fully automated, or what is the extent of automation, which is, uh, there as an example of static application security testing, we are looking at whether the product development team is capable enough to, uh, implement this practice. Or there is an InfoSec team who is outside of the product team, and they are, uh, executing SAST, whether the product team, uh, is enabled to do the setup, to do the configuration of the tool. And they are also enabled to understand the tool output and infer from it, whether SAST is being triggered as a part of the CI bills, or it is executed as an offline, uh, process. And when we look at the left shift aspect, we are checking whether this, uh, SaaS tools are integrated as a part of the developer's ID, or are they a part of the CI bills or it is done just before the release. The framework also, uh, looks at whether the entire code base is, uh, being used for doing the SAS or it is done on, on a sampling basis. So the breadth and depth of each of these practice determine the maturity of the practice. And then in an aggregated way, these help us to determine the maturity of the team,

00:11:41

Just to summarize whatever we heard so far for a service industry like TCS, the devs adoption scenario is different. Why due to the huge scale and diversity, each project is a different instance and not comparable with other. So we have come up with this standard framework that integrates depth of school principles, practices, and culture into a software engineering as line to deliver reliable, resilient, and secure product increments at speed or on demand. The framework defines the TCS way of approaching devs projects aimed at achieving devs goals and objectives After the framework. But next, So now we wanted to benchmark our projects against this framework baseline to understand their current state of adoption and further opportunity for improvement, because we want all of them to elevate, elevate their deficit of maturity. The reason being we observed that maturity plays a very critical role in attending and maximizing desired benefits from devs adoption. So we started doing manual assessments to gauge the current state and opportunities. The method includes, as you can see, on my left hand side identification of stakeholders and assessment team conducting interviews and pipeline demo sessions, manual benchmarking against the framework elements, repair findings, conducting drug validation sessions with the team sharing final assessment report with detailed findings.

00:13:28

One sample report is shown as the outcome of this manual benchmarking method, which captures the adoption and practices again, 200 plus practice elements and its set of adoption using different color tools. Do you think this was scalable? No, because we could complete only 20 such assessments in seven months where we target to cover a multiple thousands. So needless to say, the method was neither scalable, nor sustainable due to many challenges, such as some of them, you can see uniformity, stakeholders, availability, Being very afford, intensive, consuming, very high cycle time of four to six weeks and so on. So how are you working this As the next step? We did abstract the framework into a technology agnostic, self based and digitized assessment method. Having four maturity levels, you can see as basic standard advanced and based in class,

00:14:41

It has over 500 plus responses designed to capture each practice element across five depth comments and associated various dimensions, such as people process technology, culture, automation, security integration, and so on. The maturity is based on breadth and depth of practice implementation and rigor. As explained earlier in a T shift model, it may also depend on the practice radiation such as for infrastructure as a code, the practice or the variation in deployment of this practice can vary widely. The starters can do these using these is the configurations only, whereas others, they can start doing provisioning and infrastructure component. Some are able to provision an entire environment like spinoff or tear down the environments as there will. And some are even more to orchestra the entire deployment production deployment automatically using auto rollback facilities.

00:15:55

This assessment has an integrated knowledge base to translate the responses and come up with a detailed report that is automated and send to the team within one day cycle 10. You can see one sample report, which is mainly having three parts. The fast part on your lift is the domain sub domain and various dimension wise maturity score and their levels. The second part, the middle one provides an executive summary on key aspects, such as deployment frequency, late time automation, security integration in the pipeline. And similar. The, the third part is the detailed findings on each and every strength opportunities, which are classified as observations and related recommendations based on the framework and the suggestions for some industry leading based practices to export such as hypothesis based testing, often known as Gauss engineering, red teaming. And so

00:17:00

Now we are showing one sample question just to show, you know, the kind of rigor we have into the assessment method and the responses that we have designed. You can see one continuous integration, build, check, get, uh, options. These are these possible responses. So the expectation persons responses, what we have experienced in various produces. And we have seen both the extremes like someone is doing a continuous integration for doing only compilation and linking different. Like while some other in that other extreme, they are able to integrate all these steps and having a robust check. Also the assessment and the reports are technology agnostic. So what we are doing with these reports, so what is the flow? The report findings are then interpreted in the context of the customer to come up with digital action plan with priority action owner and timeline. The context could be different like te technologies stack and limitations customers already investment into the technology and tools, agile maturity organization, and cultural rigidity, their alignment towards traditional and other technologies.

00:18:22

And so on. It is then shared with customer to further elevate the depth of maturity as the actions are being taken. Reassessment is recommended in four to six months or so to experience if the maturity elevation for these projects have happened. And if that is deflected in achieving the depth goal. So if the maturity elevation has not happened, then they need to continue with these actions and see the rigor of implementation. And if it has happened and the inflection within the goals are also visible, then they would go further next and maturity. And this is how the journey continues for every projects.

00:19:08

So the next question is we have a huge scale, and that is what that only the framework and the manual assessment could not sustain. So how, how are reaching to our different engagements? We have a strong ecosystem and enablement team in the form of delivery. Excellence said delivery, excellence, partners, unit, agile leaders, community of devs practitioners as neighborhood to facilitate this scaled adoption. Apart from this, we have been evangelizing devs adoption through various forums, such as webinars focused account, speak, knowledge sharing sessions. And so on. We also conduct orientation sessions at regular interval, both at business unit and organization level. Now to show a sample governance mechanism by different business unit, how the specific single point of contact or spot is driving rigor, in-depth circums adoption within his or her units. They often adopt various means such as coverage dashboard daily sync up to respond to different queries and removing impediments monthly Tylenol to answer all queries and to show up to the management. They often collaborate with us to get any help that is required to sustain this initiative. The focus is to bring everyone in this improvement journey to maximize both customer and organizational benefits.

00:20:45

Now talking about the impact, we have just explained how the framework and the assessment method are helping us to benchmark and to scale devs adoption at organization level, as we speak, we have already achieved 700 plus benchmarking assessments for 200 plus customers across business units and geographies customers are also very positive to see this proactive engagement as growth and transformation partners towards improving their devs maturity and thus helping and enabling them to realize their business benefits. This has reflected in their feedback too. You can see couple of sample given as a derived benefit at organization level. We are also having additional benefits because we have a rich information data set of 500 plus responses from each of these 700 plus assessments. So we are coming up with various additional organizational insights on each of these practice element and corresponding dimensions. This is helping us to identify specific focus areas to drive excellence in software engineering practices across our organization.

00:22:08

And even beyond <inaudible> projects to summarize a behavior as being exhibited by different maturity levels of projects at a very high level, we have been observing like advanced and best in class projects, exhibiting better ability in continuous delivery enabled by their maturity in test automation in front qualification, security integration, and so on. They also found to be more in zero down time and zero hands of production deployments. Often one click followed by real time observability and proactive event management. Additionally, based in class projects also show better adoption of advanced capabilities such as AIOps testing with focused user group, like every testing, dark launching failure, injection to test infra resilience and so on. Whereas the projects in standard maturity, they are capable of doing a continuous integration and now reaching for the next high level by improving and, um, on different areas, such as adaption of best practices, like test driven development, <inaudible> driven development, threat modeling, building cross-functional skill in test automation in fraud, automation, integrating security tools within the pipeline, improving real time pipeline visibility and more integration and proactive monitoring. And so on. We often have seen that cloud adoption is helping projects to reach the higher maturity levels, especially from info infrastructure, automation and continuous monitoring perspective.

00:23:56

So now how depth checkoffs adoption is helping PCSN customers. There have been customer delight and appreciations everywhere as it is helping them to realize their depths goals towards maximizing business benefits, such as faster delivery, high frequency delivers improved agility and responsiveness towards technology changes, business changes, events and disaster management, faster recovery from incidents and outages improve reliability. And so on. Now, the question could be, how are we refining the framework? Are we doing this framework and assessment method refin refining on enriching? Yes. The answer is yes. Obviously we are continuously working and to upgrading our knowledge on how the industry is trending. What are the new practices are emerging? What are our mature engagements are practicing? How are the practicing, what is the additional need? And that is how we are feeding back all our experiences into this Primark and assessment method to continuously enriching this with this, we'll come to the end of our presentation. So finally, thank you all having patience and listening to our experience towards skilled adoption, if circum adoption, and how is this different from a very large it services company perspective, we are sure you would be able to replicate this story in your organization as applicable. Thank you.

00:25:35

Thank you, and have a good day.