DevIntOps

In the real world cloud deployment, it usually takes multiple months for cloud deployment, we have been thinking of how to deliver NFV with more agility. Based on the experience of China Mobile, including some trial deployment and pilot programs both in a Lab and in a production environment, we have summarized a collaboration pattern for telco operators and vendors which could lead to more agility and better quality in NFV delivery.

Integration is the Key to NFV Delivery

Network functions virtualization (NFV) revolutionized the way telco networks are being built and operated. With Softwarization, Virtualization & Orchestration, NFV replaces costly dedicated and purpose build hardware with generic servers that use software to provide a bunch of different virtualized network functions (VNFs). While decoupling NFV components, for example, in the decoupled scenario as the picture below, multiple vendors have to work together for a successful deployment, decoupling inevitably brings a lot of complexity, inter-operation, and integration between multiple vendors, which is the key step for Telco cloud deployment.

Telco cloud industry agrees with the opinion that integration is key to NFV delivery, for example, in 2019 ONS, one lean NFV session pointed out that NFV’s been too slow to arrive, and integration is the real problem. And in the 2020 China SDN/NFV/AI Conference, Wei Leping, executive deputy director of the science and Technology Commission of the Ministry of industry, pointed out that lack of integration ability slows NFV down. Also, China Mobile statistics show more than 60% of issues in NFV deployment were related to integration and inter-operation between multiple vendors

Multi-vendor solutions require much more integration and quality assurance for successful production deployment.

DevOps is perfect in the IT industry, however, it has limitations for operators. The hardware & software products are purchased from vendors, and telco operators are not actually participating in the development circle. For telco operators, we should spend more effort in the integration and verification stage. By utilizing the idea of DevOps, we are promoting a new collaboration pattern: DevIntOps, the idea is to encourage closer collaboration between vendors and operators. Operators should get involved in vendors’ development by continuous integration and testing.

Dev circle:

It represents the development circle of vendors when they receive any plans or requirements from the operators. They will do the software development process as usual.

Int circle:

This is the key step of DevIntOps, and an integration lab environment should be provided by the operators. Once vendors have a stable branch, instead of delivering directly to production developments, they will deliver to the Integration Lab instead. Then it should trigger an automatic deployment and integration with the latest builds in the integration lab.

The integration lab should continuously deploy and integrate new versions of software or tools or any dependent packages, to verify integration ability, and then continuously run automation tests for the integrated platform.

In this Int circle, we should give feedback to the vendors, like functions defects or optimized solutions, then the vendors will include it in the continuous development circle.

Ops circle:

Only verified software and integration solutions will be delivered to the production environment. With online monitoring and feedback, Integration lab should also support reproducing and verifying any issues in the production environment.

In the DevIntOps mode, continuously integrating products from different vendors, continuously running automation verification tests, continuous feedback & improvement, by pre-integration and pre-verification in the lab will lower the delivery risk, reduce the time cost of business deployment, which means more agility and better quality.

DevIntOps vs DevOps

DevIntOps is the collaboration pattern that refers to the idea of DevOps, and adjusts DevOps to suit telco-vendor collaboration patterns. It has some similarities and differences compared to DevOps.

Similarity: Both DevIntOps and DevOps are collaboration patterns for agile iteration and more efficient delivery. Both of them are encouraging more collaboration between different departments.

Difference: DevOps applies to IT industries that control their own development process and operation. While DevIntOps applies to companies that need to purchase software solutions from multiple vendors, such as Telco operators. By adding Int circle, operators need to improve technical ability and build a self-owned integration lab and self-controlled integration solution. Continuous integration pipeline and automation test are always needed in Int circle.

China Mobile’s Integration solution

Adapting network/IT organizations’ CI/CD methodologies, China Mobile has launched open CI pipelines to connect multi-vendor tools and operator tools for the E2E integration solutions.

There are 5 major stages for the CI pipeline: Plan&Design, Hardware test and configuration, Software Deployment , Integration, and software verification(acceptance test).

Plan & Design: Preposition the Design stage of both hardware and software design, refer to the idea of Infrastructure as Code(IaC), trigger NFV infrastructure by simple command line or configuration file. At the same time, define the standardized and unified data model for the resource pool, the unified data model will include all the hardware and software information, it is generated when designing each resource pool, the same data model will be used by automation test and installation tool.

Hardware: Automate hardware configuration and verification test step, then connect hardware automation tool to CI pipeline, including standard requirements from the operators and customize configuration from the vendors.

Deploy: Invoke vendor deployment tools by standard API for software installation.

Integration: Define standardized integration process, integration interface, and standardized configure files for inter-operation, develop automation tools to handle integration between components.

Verification: Operators should improve automation skill level, and develop and run automation tests against the environment based on the requirements.

By automating the steps above, with the benefits of the CI pipeline and automation tools, we will have an E2E solution for cloud deployment.

Open Source Technologies in DevIntOps

CMCC has a very large OpenStack-based cloud platform in the production environment where various services like 4G/5G VNFs & IT apps are running. All of our network clouds are based on OpenStack and KVM. Most of our tests are aligned with OpenStack specifications, and most of the integration of our components (such as NFVI-Storage) are referring to OpenStack standard API.

In the exploration of agility for NFV delivery, we have been using a lot of other open source tools, we have also shared some of our solutions/ideas/thoughts in open source, and we plan to continue to share updates or progress.

In the Plan&Design stage, we have been using a standardized Data Model for IaC purpose, configuration, and deployment, which is also called Pod Description File(PDF). We have updated the PDF definition in Reference Implementation(RI) in LFN-Anuket. We also plan to share our standardized OpenStack-Ceph integration process.

Our CI/CD pipeline is based on Jenkins and Docker, other open source tools included in our automation tools are Ansible, Django, fastapi, traefik, nginx, pytest, cucumber framework, etc.

We really appreciate the open source spirit. We tend to use open source technologies as much as possible, and contribute to open source when needed.

China Mobile DevIntOps experience

In a recent network cloud deployment project of China Mobile, 2 vendors(one from OpenStack and one from storage) did not have experience of integrate with each other, China mobile offered lab environment, so they can pre-deploy in integration lab, then they were able to figure out the specific steps for integration and updated their deployment & integration manual.

After the functional test, we found several function defects related to inter-operation. After rounds of discussion with vendors, they were able to work out a software patch with those bug fixes. Those issues were pre-exposed in the lab, so we had the opportunity to fix them before business deployment.

After they successfully integrated with each other, we gained the integration manual, to guide business deployment, automation script for integration, and software patches. They helped to optimize deployment in the production environment.

From lab continuously integration statistics, we are able to :

Trigger 6 software product deployment (NFVI/ distribution storage/MANO ) from CI pipeline;
Deploy NFVI within 5 hours, and deploy distribution storage within 30 mins;
Finished 4 pairs of NFVI + storage integration from different vendors, which follows the China Mobile unified integration standards;
Continuous iteration and testing of 4 versions for same software product within 1 month, more than thousands of cases were tested within 2 months

CMCC has populated this integration pattern(trigger cloud deployment by CI pipeline) to some production pools, which proves the lab integration solution can be copied and applied in production deployment.

Next Step:

In large-scale OpenStack based and multiple vendors’ software integrated network cloud deployment, China Mobile has been one of the first telco operators that dive into integration and production deployment and try to build standardized and automated integration solutions between highly customized software of multiple vendors. China Mobile has achieved zero-touch and fully automated hardware configuration, automatic NFVI+storage deployment and integration triggered by the CI pipeline. We will continue to improve integration and automation ability.

We have seen the benefit of the DevIntOps pattern, and we will collaborate with more vendors to work in DevIntOps mode and populate the solution to more cloud deployment scenarios, such as Edge or private 5GC.

We believe in deeply getting involved in production cloud deployment, fully automating the transition steps between hardware and software construction, and standardizing the steps and interface for inter-operation. We will achieve agility and better quality.

Here, we propose closer collaboration between vendors and operators, jointly defining and participating in DevIntOps.

Tags: China Mobile, devops, OpenStack