Case Study: Equinix Metal Builds on Flatcar

It started as a crazy idea.

Back in 2014, the cloud rush had just started. Compute workloads were moving to “virtual machines as a service”. This was the inevitable future of IT.

A small team of industry veterans, however, saw things differently. They believed there would be users who wanted the guaranteed performance, security and control of running their own bare metal, but with the experience of the cloud. The idea of a “bare metal cloud” was born, along with the company, Packet, that would define the category.

Fast forward to 2020. Equinix, the world’s largest data center and interconnection provider, acquires Packet. With 220+ facilities spread across 63 metros in 27 countries, Equinix has more than just scale: it also helps to create nearly 400,000 low-latency interconnections between its tenants, from Google Cloud to Zoom. For Packet —now known as Equinix Metal™ and with the resources of a $65 billion company behind it — their ambitious plans for redefining the next wave of cloud are accelerating. Over the course of just a few months, the team deploys the Metal platform in nearly 20 of Equinix’s most popular locations — the literal “center” of the internet.

There is just one problem. Sustaining growth at this pace would not be possible with the cloud management infrastructure built for Packet’s early startup days. A refactoring is required, to enable resilience, flexibility, and security — and scalability to support a control plane for the Metal platform in hundreds of locations. Working with Kinvolk, the team decides to build its next generation cloud control plane on Kubernetes, running atop Flatcar Container Linux.

Nicole Hubbard, principal engineer on the Equinix Metal delivery engineering team, puts it like this: “I need the security and management to be as simple as possible, and as automated as possible. Manually patching our Ubuntu hosts is hard enough when we have only 30 of them. In practice, that means they don’t get security updates. If we can’t do it at that scale, we definitely can’t do it as we scale to hundreds and thousands in the future.”

Nicole was already familiar with the concept of an immutable, auto-updating container Linux, having used CoreOS at various points in her career, which includes stints at Rackspace, WP Engine, and Microsoft. So Flatcar was “everything we expected it to be. The management overhead is massively reduced. Flatcar allows us to start thinking about our bare metal nodes like they’re cloud instances, we can think of them as more ephemeral. It’s really hard to explain, but once you’ve experienced what it’s like to run immutable infrastructure, you never want to go back.”

“Also important to us is the atomic nature of Flatcar updates, which comes with the ability to rollback easily,” continues Nicole. “For example, one point release included a systemd update that broke our Kubernetes networking. We could just rollback to the previous release while we diagnosed and fixed the problem. What might have been a major headache with a traditional Linux distro ended up being really simple with Flatcar.”

A positive surprise compared with her CoreOS experience is the Kinvolk Update Service, which “allows us to push updates to all our nodes via a Web UI, and gives us a single pane of glass to see the state of our nodes as they upgrade, which is kind of fun to watch!”

Given the mission-critical nature of Flatcar in the Equinix Metal control plane, Equinix views a support relationship with Kinvolk as essential. Says Nicole: “We have a great collaboration with the Kinvolk engineering team. All the communications have been outstanding and helpful.”

The bottom line? “Adopting Flatcar Container Linux, and relying on Kinvolk’s expert support, allows us to focus on the Kubernetes layer and what runs on it, which is what we really care about.”