Auto-updating Flatcar Container Linux

Introduction

At the moment, Lokomotive supports only Flatcar Container Linux as the underlying operating system for nodes. While Flatcar can keep itself up to date, when running Kubernetes on top of it, we recommend using the Flatcar Linux Update Operator (FLUO) to perform updates to avoid rebooting too many nodes at a time which could cause a service outage.

FLUO is a supported Lokomotive component.

This guide will show how to install it and disable its auto-update feature for specific nodes, which might be useful when you run services, which require special care before shutting down (e.g. storage clusters).

NOTE: If you want to use the FLUO component on a non-Lokomotive cluster, make sure your Flatcar nodes have locksmithd.service systemd unit disabled to avoid nodes rebooting on their own. On Lokomotive, it is disabled by default.

Prerequisites

• A Lokomotive cluster accessible via kubectl.

Steps

Step 1: Disable auto-update for sensitive nodes

If you want to update the particular nodes manually in a controlled fashion, there is a way to disable automatic updates. Disabling updates can come in handy when the workloads run by these machines are storage or ingress network related, where applications cannot tolerate the abrupt reboot of node.

Please annotate the nodes as follows:

kubectl annotate node <node name> "flatcar-linux-update.v1.flatcar-linux.net/reboot-paused=true"

Step 2: Configure FLUO

Add the following content to your cluster configuration (e.g. in fluo.lokocfg file):

component "flatcar-linux-update-operator" {}

Step 3: Install FLUO

Execute the following command to deploy the flatcar-linux-update-operator component:

lokoctl component apply flatcar-linux-update-operator

Verify that pods in the reboot-coordinator namespace are running (this may take a few minutes):

kubectl -n reboot-coordinator get pods

Now that you have installed FLUO, nodes without annotation flatcar-linux-update.v1.flatcar-linux.net/reboot-paused=true will be updated automatically when a new version of Flatcar is available. One-by-one, the selected node is first drained before the reboot for an update.

Step 4: Test installation (optional)

You can annotate a node to trigger an automatic reboot:

export NODE="<node name>"
kubectl annotate node $NODE --overwrite \
    flatcar-linux-update.v1.flatcar-linux.net/reboot-needed="true"

You can also SSH into a node and trigger an update check by running update_engine_client -check_for_update or simulate a reboot is needed by running locksmithctl send-need-reboot.