00:00:17

Hello, my name's Godwin USA, and I'm very happy to be here. And to be able to present you today, let's get started, but a little bit about me. So, so throughout my career, I have spent around nine years in society fields, dealing with databases, networks, and other CIS admin stuff. Then I spent another nine years working with one of the major telecom vendors. And then I had the opportunity to experience how the like home is much more rigid, the slow moving animal compared to other enterprise ID fields, and also had to experience what does that mean to their business and what problems does it take to now in my current role, difficult, I have the opportunity to work with the largest telecom providers and see the other side of the coin I have experienced in my mentor years. And this is what I want to talk about today.

00:01:22

How the telecom industry is thriving for a change in the operations, but vendors and providers being dependent on each other and both being highly regulated makes things difficult. Luckily, we start to see an out of this situation and things are in motion for the more, more agile operations, a bit closer collaboration and what some might even be able to call that. Whoops, that's going to start with what I have learned and what we did while I was working, uh, with the vendor while I was working in product development. We frequently encountered that from completion of release. Still it is actually taken into use at the CSB plant is significant and buses. It could have been anywhere between six to 18 months. This caused us problems in a multitude of phase, including going back to all the easiest to make what actions maintain environments to verify changes to these older easies and tasks compatibility.

00:02:37

And of course it happened many times that the person who created the original code has moved to another team or even to another company. So someone had to pick up sings and figure out what the original code was meant to do to begin with. As we started to investigate what are the root causes and what we could do about them. We have received many inputs from different stakeholders, and of course we made our own conclusions as well first. And probably the biggest problem we found was the release strategy. The company for its applications had maybe one or two releases per year, best case scenario. It was up to four, but, uh, it might as well, could have been much less as well. This meant that each leaves had a huge change content, dozens of features, even more corrections equally introduce significant changes on different interfaces or on the platform itself. This meant that the effort to take them into operational use was quite huge. And unfortunately, in many cases it was quite manual.

00:03:55

The different constellations of neighboring network elements had to be verified and CSP specific configurations tried and tested, of course, all these different scenarios, no on in these good tests, there's sometimes huge errors or uncovered in CSPs acceptance cycle dismantle long back and forth between CSB services and RMB or these two significant effort from all parties involved. In many cases, they're not even involved, but most importantly, this, this long cycle then eventually slowed down the development as quite an effort was tied down in correcting these, these volumes and, and even old adult S said, these acceptance cycles could take up to 12 or even 18 months in a support model that a release was maintained and supported for 76 months. This meant that continuous negotiations, but taking place. So the TSPs can utilize given release longer than it is granted by the support model, which let's do even more on doing the work that for example, customer pass had to be developed.

00:05:16

And the, of course it had to be tested as well. Also, businessman has this model and problems as well. It was rigid enough to not leave room for introducing new business models. Uh, instead of the existing and old way of permanent licensing, it was slowing down the R and D machinery enough that there was not enough capacity to experiment with new features, new capabilities and tied down enough sales and services. Your sources that new market opportunities had to slip away. And also that was not enough high level of automation or any digital support to boost the efficiency of sales services or even the R and D as we've been deeper in our investigation and map out what is actually happening when R and D creates a new software and the CSP buys it. We uncovered this quite obvious by the verbal nightmare. It's not the problem that you can be defined, painted with just be even scarier.

00:06:26

But by looking at this picture, we can see that there are an unreasonable amount of subsystems tools teams involved in new, simple order fulfillment. In most cases, these tools and even teams were disconnected, relying on manual data, transfer, manual data validation, people, triggering actions, and maybe even people executing them on average, this man that if somebody ordered an order to complete his software release, it would have been delivered on average. In 52 days, he would write roughly two months to get the software delivered. You can imagine that any given R and D engineer have already forgotten what he, she wrote in that single function by the time any user of the software would actually got their hands on it. Okay. So what we should do about this, what we should enable for things to be much better, we should enable that pretty much anything anytime can be delivered.

00:07:35

Any merge down to trunk could immediately be delivered to those customers who are willing and able to accept the software packages so that less changes would be delivered at mass quality assurance at customer side could be done parallel between the art and the internet testing. The feedback to any issue would be much more timely and relevant. And the software could go to production much sooner, all the wine, those customers who would not be willing or be able to take these frequent deliveries will still get the usual, big buying software ups, but already in a much more stable kind of battle-tested shape. So they would also encounter less problems and could go live much sooner with them. Of course, this meant process changes, not just technical enablement should be done, but, uh, that on its own would be an entirely different presentation. That's focused on the technical side for now.

00:08:43

And technically the decision was quite obvious. Let's make a pipeline, let's make a pipeline that extends to our customer pharmacist integrated. So digital software supply chain integrated digital components and let the software just guys through from plans to build and release, to deploy, and eventually operate all the why feedback and metrics are collected simple. Isn't it? Val of course not things are rarely this easy, especially in business to business scenarios immediately, there were a number of red flags along the pipeline plans, R and D work and CSP operations were in a decent shape. We could not complain about those, but how releases were managed made available, how it was getting ordered, delivered invoiced and paid for would need to significantly improve. And eventually how software is getting deployed and accepted or something that needed another bunch of significant improvement as order dimensioned.

00:09:55

So the supply chain as sales were highly manual and highly disconnected both in tooling and teams, but we also have problems. How releases were created, managed, distributed, and made available for our customers. These functions were also relying on people and manual actions, including manual data input for license key generation or download from an FTP server. Also deployment and acceptance was quite manual driven by VDF based method of procedures and test plans or HTML-based technical notes. So we got, we got three teams to us to do something about this. And we were given the metric to bring down the order fulfillment time from 52 days to something much more reasonable.

00:10:53

The teams got to working on three interworking subsystems, the first team verb on the digital sales, including the new sales platform together with the marketplace handling customer contract data, managing contract price cycles and providing inputs to supply chain. So that any deliveries we would do would be according to contract would be according to legal requirements and would comply with the global trade agreement. The second team worked on the software supply chain, including the industry and the operational data space for managing the releases, creating bellies bundles, creating license, keys, distributing these release bundles to the appropriate delivery endpoints, validating the customer access and pushing the league's bundle to the automation platform deployed on customer premise, just south team involved on the so-called that was platform, which was responsible for pulling the software release bundles, deploying the software and orchestrating the acceptance testing based on machine-readable input from the R and D is about to change content.

00:12:07

And from CSPs based on that individual requirements, since the subsystems were designed and developed as a joint effort, they were need for very little manual input. And there were highly reusable components shared between these solutions and where we got a bit dissolution. How did it before? Well, our main metric of delivery, as I said, was to bring down delivery time during the time of piloting phase. We van down from an average 52 days to an average around six minutes, six minutes from order to have the software ready and to be deployed and tested in any environment. Of course, this meant any feedback from CSPs came much more timely, R and D is, could receive field test results before closing is resulting in improvement in turnaround time for corrections and decrease in number of errors. Eventually the system made it possible that we could create a specific and generally currently use bundles targeting any customer group and with the different customer needs and the specific targeted release bundles, new business model, the different service levels could be introduced. So I system also enabled both application RNDs and our teams to continuously evolve and develop our solutions. Vico start forgetting the software is done. Now we see what problem situation you do near real time feedback and requirement management.

00:13:55

And this was the point during the piloting of this solution that I made a cardiac change to join ethical board and start working directly with major CSPs independent from any vendor. And this was the point that I realized and idealized Hockley, the Navy day of the vendor, that, that we had very imagined that we have now a functioning pipeline. We can pump software out swiftly and easily. So it's smooth sailing from here. The software can be delivered Depot tested easily. Everyone should be happy and we are on for a new mode of operation. But, uh, the reality is pretty much any major vendor has an army, oh, sorry. Pretty much any major CSP has an army of vendors or vendors delivering different software to them. And all of them create such a pipeline that is art is kind of guilty for the CSPs. You will, in one of our webinars last year, we had some balls and it's turned around out that major at the older CSPs have over 15 different pipelines comprise of over 50 different tools that they need to manage and maintain so that they can work with their vendor army. Of course, this does not include those vendors who are still mostly manual in the operations. CSPs have pipelines all over the place, sometimes serving highly specific purposes or, or even a single obligations.

00:15:41

What they would really need is something more like this they're their own flexible and adaptable pipeline is being fed by all the different vendor pipelines seems to be such set up. They could focus effort that is needed most like further improving their time to market. Also validating new service creation and roll out, or generally focused on higher volume adding work. And luckily I got the chance to work on just that the Dutch at home with whom we created the prototype of pipeline for the future way of working. They built the system where we split the different layers, which are required for 5g obligations to operate. We have defined the infrastructure, the virtual infrastructure, the core spas and the CNF or application layer that would require volumes networks, pulse nodes, et cetera, are created. And the needy tooling. And eventually the applications are being deployed for acceptance and rollout. And everything is declared in the single source of tools, single source of truth, originating from the architecture description, this architecture causes of the master template and the master configuration. And it's being broken down to each layers, individual descript, or of, of operating system image, data form, or Ansible script or hand chart with respectful configuration.

00:17:20

And then the process model distinguishes between four phases. We interrelated activities design is the first phase that the development of the automation templates definition of the Daisy, the configuration parameters and design of test activities, done blades based on the QA strategy. Our next phase is the continuous integration where the external software artifacts are being sold. The deployment artifacts are built and deployment templates and scripts along with the test activities are, and that integration is followed by continuous deployment, but the closed loop deployment of the different technology layers are happening followed by the execution of the dynamically defined test activities with checking the quality eight fulfillment and eventually resulting in a go no-go decision made for the goal life deployment phases followed by operations where continuous monitoring of the service quality and execution of different policies are made possible.

00:18:31

Architecture wise, first and foremost, the needy to expose videos, API APIs for vendors and the supply chain so that they can deliver the artifacts and can gain access to test systems for close collaboration. Then we have to secure this access and each vendor should be able to access their, and only their software for the delivered artifacts. We had to agree on the expected format and content. So they are machine readable, but human understandable. And we had to also set along branching tagging and the list of the G. So it can be controlled. What, and where can be deployed automatically without compromising production environment, uh, compromising the security of the environment, integrity or in general, the whole state of operations. Finally, we had to set up distribution of software and code to what your sides and subsidiaries from the scope to deliver software artifacts. And the format requires that vendors also do their due diligence.

00:19:43

And as such, I expect to have a long negotiation going on in the near or meet future. And to be honest, I would expect some form of standardization happening around these formats and delivery artifacts for the actual pipeline architecture. Have you mostly followed the approach, VD extensions for background management, uh, for test management security, scanning of the artifacts and deployment of infrastructure components, as well as additional monitoring and logging components. This way, we were able to accomplish that the artifacts arrive in the desired format, same tools, same process can apply then plates and configurations can be easily adopted to the given application separation of layers and tasks for specific tools also enables us to swap any of these tools. If the landscape were to change, we do set up. If we're able to create a solution that a vendor could have their own pipeline directly feeding into Dodge telecoms pipeline, providing them with the required speed and the feedback my DT can operate the pipeline that can handle applications from number of vendors without overhead of multiple pipelines, putting these two subsystems together from my vendor years, the delivery capability and the project together with , the deployment capability would enable the actual continuous deployment Intelequent actual continuous deployment that the creative software is pretty much immediately delivered to the ESP.

00:21:42

And pretty much immediately can be deployed binging down the acceptance cycle time significantly, but this definitely would not happen on its own. Neither the CSPs nor the vendors can complete such operation on their own. It definitely requires that close collaboration in shaping the future model for operations, for both parties to summarize what I have learned during these years. So first and foremost, CACD is unavoidable or part is experienced the pain of the decades, old, the agent mode of operations and strives to change that given 5g is momentum. This can be an ideal time to introduce such practices. Building a multi-vendor pipeline is not easy. If it is, then you surely miss something important. So go back and take a look, but a mindset it can be done, but definitely requires the collaboration of vendors and CSPs and the definition of handover points and four months, everyone building their own pipeline for their own purposes.

00:23:05

We are for short in the grand scheme of business to business service creation, but the flexible and other the pipeline can serve everyone well and too early, something up what has been achieved. And we can be proud of is that we can now near real time deliver software within minutes, software can be taken into use almost immediately effectively bringing down the infamous six to 12 to 18 months, acceptance cycle two weeks or even days. And finally, if you wish to hear more about this specific case, we will have a very now together be the, our colleagues from . So please visit our website for details. Thank you for the attention. You can find me on slack, and if you have any questions, please feel free to shoot.