Bringing open source to the bare metal edge

Kubespray, a community project providing Ansible playbooks for the deployment and management of Kubernetes clusters, recently merged to support for bare metal cloud Packet. This new support allows Kubernetes clusters to be deployed across next generation edge locations including these cell-tower based micro-data centers.

Packet, which is unique in its bare metal focus, expands existing Kubespray’s support beyond the traditional AWS, GCE, Azure, OpenStack, vSphere, and Oracle Cloud infrastructure. Kubespray removes the complexities of standing up a Kubernetes clustering through automation via Terraform and Ansible. Terraform provisions the infrastructure and installs the prerequisites for the Ansible installation. Terraform provider plugins allow support for a variety of different cloud providers. The Ansible playbooks then deploys and configures Kubernetes.

Since there are already detailed instructions online for deploying with Kubespray on Packet, I’ll focus this post on why bare metal support is important for Kubernetes and what it takes to make it happen.

Why bare metal?

Historically, Kubernetes deployments have relied upon the “creature comforts” of a public cloud, or fully managed private cloud, to provide virtual machines and networking infrastructure upon which to run Kubernetes. This added a layer of abstraction (e.g. a hypervisor with virtual machines) that Kubernetes itself doesn’t necessarily need. In fact, Kubernetes began its life on bare metal as Google’s Borg.

As we move workloads closer to the end user (in the form of edge computing) and deploy to ever-more diverse environments (including hybrid and on-premises infrastructure of different architectures and sizes), relying on a homogenous public cloud substrate isn’t always possible or ideal. For instance, with edge locations being resource constrained, it’s more efficient and practical to run Kubernetes directly atop bare metal.

Mind the gaps!

Without a full-featured public cloud underneath a bare metal cluster, some traditional capabilities such as load balancing and storage orchestration will need to be managed within the Kubernetes cluster itself. Luckily, there are projects such as MetalLB and Rook that provide just such support for Kubernetes.

MetalLB, a Layer 2 and Layer 3 load balancer, has already been integrated into Kubespray. Support for Rook, which orchestrates Ceph to provide distributed and replicated storage for a Kubernetes cluster, can be easily installed onto a bare metal cluster. In addition to enabling full functionality, this “bring your own” approach to storage and load balancing removes the reliance upon specific cloud services, helping users to avoid lock-in with an approach that can be installed anywhere.

One gap that we don’t have to overcome is support for Arm64 processors, since that is already in place with Kubespray. The Arm architecture (which is starting to show up regularly in datacenter grade hardware, SmartNICs, and other custom accelerators) has a long history in mobile and embedded devices, making it well-suited for Edge deployments.

Going forward, I’m hoping to see deeper integration with MetalLB and Rook as well as bare metal CI of daily builds on a number of different hardware configurations. With access to automated bare metal at Packet, we now have the opportunity to test and maintain support across various processor types, storage options, and networking setups. This will help to ensure that Kubespray-powered Kubernetes can be deployed and managed confidently across public clouds, bare metal and edge environments.

It takes a village

As an open-source project driven by the community, it’s important that we thank the core developers and contributors to Kubespray, as well as the folks who assisted with the Packet integration. Notably, Maxime Guyot and Aivars Sterns for the initial commits and code reviews, Rong Zhang and Ed Vielmetti for document reviews, as well as Tomáš Karásek (who maintains the Packet Go library and Terraform provider) and John Studarus (who tries not to mess up Karásek’s code too much with his pull requests).

Learn More

At the upcoming Open Infrastructure Summit, there are over 20 sessions on edge computing including “The Open Micro Edge Data Center”, discussing using open source across bare metal edge locations.

About the author

John Studarus, president of JHL Consulting, provides cloud security product development and cloud security consulting services. Within the open source communities, he volunteers his time managing the community supported Packet CI cloud and running numerous user groups across the U.S. as an OpenStack Ambassador.

Tags: Ansible, containers, Kubernetes, Kubespray, Rook, Terraform