OpenStack 10 Years: A Revolution for Telco

On celebrating the 10 years’ birthday for OpenStack, we would like to share the revolution and transformation OpenStack has made for telco operators.

As the key technology for NFV (Network Function Virtualization), OpenStack is now deployed in more than 60,000 physical servers in China Mobile’s network clouds. This technology helps telco operators enjoy the benefit of low cost, high efficiency and agility. Since 2019, China Mobile launched its telco-cloud construction, which is also the world’s largest NFV project so far, with tens of thousands of servers in total, located in eight core regions across the country. More than 20 hardware and software suppliers are included in this telco cloud. Scalability and multi-vendor nature are the major challenges we have for our network cloud.

Telco operators used to build up network following the standards. However, when it comes to IT cloud, there is no specific standard to follow but lots of open-source technologies and best practice experience. By working with the OpenStack community, China Mobile adopts these technologies and experience, making full use of them to build up the network cloud. Over the years, OpenStack has become the de-facto standard for China Mobile’s network cloud infrastructure. Based on this de-facto standard, we also build up our automation testing platform, named “AUTO”, and cross-vendor CI/CD pipeline. AUTO makes full use of OpenStack SDK and existing testing tools to provide overall check and verification for our network cloud, especially on the scalability and performance.

We also build up a cross-vendor CI/CD pipeline, to provide iterate testing on multi-vendor’s cloud. Our CI/CD pipelines connect with Vendor Labs, including Ericsson, Huawei, H3C and ZTE. Any version updates from vendors will invoke automatic deployment and system testing in China Mobile’s lab and will feedback to vendor products. With the help of CI/CD pipeline, we are able to continuously deploy and test vendor OpenStack for more than 10 times per week, and each round only costs less than five hours. Such cross-vendor CI/CD helps our cloud to iterate fast and precisely.

When deploying such large scale cloud, we also realize it is important to improve the efficiency and quality of hardware integration, since any hardware defects will eventually influence software and will cause difficulties in sourcing errors. Therefore we also evolve the capability of AUTO to provide hardware configuration and test. So far, AUTO has been used in all the eight regions across the country and helps to reduce our construction time by 1/3. It only takes 20 minutes to configure all the devices using AUTO and 80 minutes for AUTO to finish testing on single resource pool of more than 1000 physical nodes. Based on AUTO, we accomplish full scale quality check. So far, more than 15,000 issues are found and solved. AUTO also helps us to successfully decrease the configuration fault rate from 30% to zero.

Our team has benefited a lot from open source in the past 10 years. All the automation magic can never happen without the open source nature of OpenStack and the de-facto standard interfaces it provides to NFV. We are also active contributors for OpenStack since 2013. So far, we have given one keynote and nine sessions in the OpenStack Summits and Open Infrastructure Summits, and have served in the programming committee for six times. We share and contribute our successful CI/CD and hardware automation experience to the community through these sessions and actively contributed to telco and Edge Computing WG. Besides, China Mobile is also one of the founding members and active contributors of OPNFV and CNTT. We engaged in building up the joint effort of these open source communities to conquer the integration challenges of the telco cloud.

On the 10 years’ birthday of OpenStack, we would like to thank this community for continuously being active and productive and evolving so fast to fulfill the requirement of telcos in such a short time. As telco operators, we see OpenStack as the key for multi-vendor cloud. We would like to continuously contribute and support this community, to make sure a sustainable open-source community is always behind to act as the de-facto standard for our cloud.

Tags: China Mobile, NFV, open source, OpenStack, telco