User Stories

Zuul case study: Packet Host

“Working with the OpenStack Infra team has really opened my eyes up to the capabilities of Zuul and the frameworks they’ve put together,” says John Studarus.

October 1, 2018
image

Zuul drives continuous integration, delivery and deployment systems with a focus on project gating and interrelated projects. In a series of interviews, Superuser asks users about why they chose it and how they’re using it.

John Studarus, OpenStack Ambassador, contributed this case study.

Contributing to open source projects, such as OpenStack, traditionally involves individuals and companies alike providing code contributions adding new features and fixing bugs. For almost two years, I’ve been running one-off OpenStack clouds for demos and labs at user group meetings across the United States on hardware donated from a bare metal service provider, Packet Host. Six months ago, they asked how they could make a larger donation to the community which led us to our path: building a community cloud to support OpenStack.

Every day, there are hundreds of code commits to the OpenStack code base that need to be tested as part of the continuous integration system managed by Zuul. Each commit runs through a series of tests (or gates) before a human review and then the gates run again before a code merge. All of these gates run across a pool of virtual machines instances (over 900 instances at peak times) donated by a number of public cloud providers. All of the OpenStack CI is dependent on donated computing resources. The OpenStack Infra team coordinates all of these cloud providers and served as our point of contact for donating these resources.

We set out to build a cloud where all the computing resources would be dedicated to the OpenStack Infra program. Building out our cloud, we had to meet the minimum requirements set by the OpenStack Infra team, namely support for a 100 concurrent VM instances each with 8 GB RAM, eight vCPUs and 80 GB storage. Packet Host allocated us 11 bare metal servers and an IPv4 /29 subnet to be used for floating IPs. With the computing and network resources in place, we moved ahead with the OpenStack architecture and implementation.

Since the test instance runs and the local mirror utilizes AFS, a distributed file system, all use ephemeral storage, the decision was made not to setup any persistent storage on the cloud. By eliminating the need to use Cinder storage service, more bare metal resources could be allocated to the Nova compute service thus supporting more concurrent test instances.

Working with the OpenStack Infra team has really opened my eyes up to the capabilities of Zuul and the frameworks they’ve put together. I had the opportunity to catch up with the OpenInfra Team at the most recent PTG. They realize that Zuul can put a strain on any cloud and are happy to work through issues that arise. Best of all, they run a great set of tools providing such metrics such as failed launch attempts and time to ready allowing me to identify issues as soon as possible.

A CI system, such as Zuul, puts an extreme load on a cloud environment as it continuously spins up and down virtual instances. While a typical instance might be up for weeks or months, a CI instance through Zuul lives, on average, a few hours. This means the control plane will always be busy stopping and starting services. Through the tools provided by the OpenStack Infra team, I was able to identify performance issues. In the first few months of operations, we quickly realized we had to upsize the control plane to handle the workload and reconfigure the image storage space to handle the disk images created daily by Zuul.

One of the limiting factors of this cloud is the availability of IPv4 addressing. Each test instance requires a floating IP address to communicate back to Zuul. Since we do have the compute resources, RAM and CPU, to group the cloud, we intend to start provisioning test instances with IPv6 addresses. Zuul and the OpenStack Infra project both already support IPv6.

While we’re continuing to improve this community run cloud, we’re also looking forward to what else we can provide across this donated hardware. Nodepool has driver capabilities to handle resources outside of OpenStack which we’re very interested in using automated bare metal support. We’re also hoping to extend CI resources to other open source projects through this same Zuul and Nodepool framework as well.

Technically and philanthropically, setting up and running this cloud has been rewarding! It’s been a great experience working with the OpenStack-Infra team and seeing everything they’re doing with Zuul. The knowledge I’ve gained running a cloud to support the OpenStack Infra team has far exceed my experiences running one-off clouds for user group demonstrations. If you’re an OpenStack cloud provider (public or private) and have an interest in donating resources to OpenStack, I encourage you to reach out to me or the OpenStack Infra team for more information.

BY NC

Tags: , ,

Superuser
Latest posts by Superuser (see all)