Provisioning Flatcar Container Linux with Tinkerbell

While much of the world's computing has moved to virtual machines running in the cloud, there are still many reasons why users want to deploy their workloads on bare metal servers. But provisioning and managing physical servers has historically been a manual and resource-intensive process.

It doesn't have to be that way, though - once an oxymoron, the concept of "bare metal cloud" has been made possible by technologies for booting from the network, together with API-driven control planes for managing server and network configurations.

For some time, Equinix Metal (previously known as Packet) has been an innovator in the "bare metal cloud" space, and so we were excited when they open sourced their control plane. This new project, Tinkerbell, includes four major components:

• boots: a DHCP and iPXE server
• tink: a workflow engine
• OSIE: an in-memory operating system
• hegel: a metadata service

Combining these components together enables the automated bringing up of bare metal machines over a variety of platforms.

Of course, as Flatcar Container Linux is our preferred operating system for deploying container workloads, we wanted to see how easy it would be to use Tinkerbell to deploy Flatcar. The answer is that, with a few tweaks that we have now upstreamed into Tinkerbell, it is quite straightforward.

In this blog post, we'll walk through the steps to provision Flatcar Container Linux on bare metal machines using Tinkerbell.

We also filed a pull request to add a workflow example for Flatcar Container Linux to the Tinkerbell documentation.

To ease reproducibility, we'll use a local Vagrant setup that simulates a simple bare-metal setup.

Instructions

We'll use Vagrant to deploy a provisioner and a worker machine. The provisioner will run the Tinkerbell stack and the worker will be provisioned with Flatcar Container Linux using Tinkerbell.

We'll assume VirtualBox is the provider used for Vagrant.

Create VMs

First, we download, extract and enter v0.1.0 of the Tinkerbell sandbox repository.

wget https://github.com/tinkerbell/sandbox/archive/v0.1.0.tar.gz
tar xf v0.1.0.tar.gz
cd sandbox-0.1.0

Now we go into the Vagrant directory and run Vagrant to set up the Provisioner.

cd deploy/vagrant

vagrant up --provider virtualbox provisioner

When the provisioner is ready, you'll see the following message:

INFO: tinkerbell stack setup completed successfully on ubuntu server

Start Tinkerbell

SSH into the provisioner.

vagrant ssh provisioner

Bring up the Tinkerbell stack.

cd /vagrant && source .env && cd deploy
docker-compose up -d

Check all containers are up.

docker-compose ps

Create Flatcar Container Linux configuration

Tinkerbell uses containers to run the different actions needed for provisioning. For example, there can be one container to wipe the disk, one to partition the disk and another to install the root filesystem.

To install Flatcar Container Linux we only need one container that runs the official flatcar-install script. We'll push the Flatcar Container Linux installer container image to the Tinkerbell Docker registry.

Let's create our installer Docker image for Flatcar Container Linux and push it. This image includes the flatcar-install script to install Flatcar Container Linux to disk.

In the provisioner console:

TINKERBELL_HOST_IP=192.168.1.1 # This is the default so it doesn't need changing.

cat << EOF > Dockerfile
FROM ubuntu
RUN apt update && apt install -y udev gpg wget
RUN wget https://raw.githubusercontent.com/flatcar-linux/init/flatcar-master/bin/flatcar-install -O /usr/local/bin/flatcar-install && \
    chmod +x /usr/local/bin/flatcar-install
EOF

docker build -t $TINKERBELL_HOST_IP/flatcar-install .
docker push $TINKERBELL_HOST_IP/flatcar-install

Flatcar Container Linux uses a mechanism called Ignition to provision machines on the first boot. To be able to SSH into the worker machine, we need an Ignition config that adds our SSH public key to the authorized keys of the machine.

For simplicity, we generate a new SSH key on the provisioner and add that to the Ignition configuration.

ssh-keygen -f /home/vagrant/.ssh/id_rsa -N ""

Usually, we'll write a Container Linux config and then use the Container Linux Config Transpiler to convert it to a JSON Ignition config. In this case we're writing the Ignition config directly to avoid pulling in extra dependencies.

We write a minimal config and encode it in Base64, saving its value to a variable.

SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"

IGNITION_CONFIG=$(cat << EOF | base64 -w0; echo
{
   "ignition" : {
      "config" : {},
      "timeouts" : {},
      "version" : "2.1.0"
   },
   "networkd" : {},
   "passwd" : {
      "users" : [
         {
            "name" : "core",
            "sshAuthorizedKeys" : [
               "$SSH_KEY"
            ]
         }
      ]
   },
   "storage" : {},
   "systemd" : {}
}
EOF
)

Set up internet access for worker

The Flatcar Container Linux installer needs access to the internet to download the latest image, so we'll use the provisioner as a NAT gateway for the worker using iptables.

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
sudo iptables -A FORWARD -i eth1 -j ACCEPT
sudo iptables -A FORWARD -i eth0 -j ACCEPT

Create Tinkerbell workflow for Flatcar Container Linux

We now create a file named hardware-data.json with the hardware data for our worker.

For this example we'll use a hardcoded ID for the hardware data entry and the MAC is hardcoded in the Vagrant file.

ID="ce2e62ed-826f-4485-a39f-a82bb74338e2"
MAC="08:00:27:00:00:01"

cat << EOF > hardware-data.json
{
  "id": "$ID",
  "metadata": {
    "facility": {
      "facility_code": "onprem"
    },
    "instance": {},
    "state": ""
  },
  "network": {
    "interfaces": [
      {
        "dhcp": {
          "arch": "x86_64",
          "ip": {
            "address": "192.168.1.5",
            "gateway": "192.168.1.1",
            "netmask": "255.255.255.248"
          },
          "mac": "$MAC",
          "uefi": false
        },
        "netboot": {
          "allow_pxe": true,
          "allow_workflow": true
        }
      }
    ]
  }
}
EOF

We'll push the hardware data into the Tinkerbell database.

docker exec -i deploy_tink-cli_1 tink hardware push < ./hardware-data.json

Now we need to create a template for Flatcar Container Linux. We'll use the Ignition configuration we generated earlier.

cat << EOF > ./flatcar-template.yaml.tmpl
version: '0.1'
name: flatcar-install
global_timeout: 1800
tasks:
- name: "flatcar-install"
  worker: "{{.machine1}}"
  volumes:
  - /dev:/dev            # Needed for accessing the disk so that we can install Flatcar on it.
  - /statedir:/statedir  # Needed for passing the Ignition config JSON and running flatcar-install.
  actions:
  - name: "dump-ignition"
    image: flatcar-install
    command:
    - sh
    - -c
    - echo '$IGNITION_CONFIG' | base64 -d > /statedir/ignition.json # Decode Ignition config to a file.
  - name: "flatcar-install"
    image: flatcar-install
    command:
    - /usr/local/bin/flatcar-install
    - -s    # Install to smallest disk.
    - -i
    - /statedir/ignition.json
EOF

The next step is creating the Tinkerbell template.

docker exec -i deploy_tink-cli_1 tink template create --name flatcar-install < ./flatcar-template.yaml.tmpl

Note the resulting ID and use it to create a workflow for the worker. A workflow is an instantiation of a template that applies to one machine. To select the worker in our workflow, we need its MAC address again.

TEMPLATE_ID="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"

docker exec -i deploy_tink-cli_1 tink workflow create -t $TEMPLATE_ID -r "{\"machine1\": \"$MAC\"}"

Note the resulting workflow ID.

Start the Worker

Open a new terminal window and start the worker.

cd sandbox/deploy/vagrant

vagrant up --provider virtualbox worker

Note: This command will never exit successfully, we're using it only to start the worker and we'll be doing manual VirtualBox operations to continue the process.

The worker will boot up OSIE and then the workflow will run.

We can see the boot up process through the VirtualBox UI and, on the provisioner, we can check the workflow state.

WORKFLOW_ID="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"

watch docker exec -i deploy_tink-cli_1 tink workflow events $WORKFLOW_ID

Once the workflow shows ACTION_SUCCESS, Flatcar Container Linux is installed on the worker!

Boot into Flatcar Container Linux

To boot into Flatcar Container Linux we need to power off the worker machine through VirtualBox and make sure it will try to boot from its hard disk.

To do that, on a new terminal we get the name of the worker machine.

VBoxManage list vms | grep worker

WORKER_NAME="vagrant_worker_1603726186937_24058"

We stop the vm.

VBoxManage controlvm $WORKER_NAME poweroff

And then we enable disk boot as the second boot option.

VBoxManage modifyvm $WORKER_NAME --boot2=disk

And start the VM again.

VBoxManage startvm --type headless $WORKER_NAME

And after waiting a minute for the worker to boot, we can go back to the provisioner shell and connect to the worker.

ssh [email protected] # As configured in the workflow above.

We should get to the Flatcar Container Linux instance.

Flatcar Container Linux by Kinvolk stable (2605.6.0)
core@localhost ~ $

Clean up

To clean up everything, go to the deploy/vagrant directory and use Vagrant to destroy the VMs.

vagrant destroy

Conclusion

This blog post demonstrated how to provision Flatcar Container Linux with Tinkerbell. While we used a Vagrant setup for simplicity, we can follow the same process to provision bare metal nodes in any environment.

Join the discussion in the #tinkerbell channel on the Equinix Metal Community Slack. For inquiries about Flatcar Container Linux support, please do reach us at [email protected].