Flatcar Container Linux developer SDK guide
These are the instructions for building Flatcar Container Linux itself. By the end of the guide you will build a developer image that you can run under KVM and have tools for making changes to the code.
Flatcar Container Linux is an open source project. All of the source for Flatcar Container Linux is available on github . If you find issues with these docs or the code please send a pull request.
Let’s get set up with an SDK chroot and build a bootable image of Flatcar Container Linux. The SDK chroot has a full toolchain and isolates the build process from quirks and differences between host OSes. The SDK must be run on an x86-64 Linux machine, the distro should not matter (Ubuntu, Fedora, etc).
System requirements to get started:
You also need a proper git setup:
git config --global user.email "[email protected]" git config --global user.name "Your Name"
NOTE: Do the git configuration as a normal user and not with sudo.
cork utility, included in the Flatcar Container Linux
project, is used to create and work with an SDK chroot.
First, download the cork utility and verify it with the signature:
curl -L -o cork https://github.com/kinvolk/mantle/releases/download/v0.15.2/cork-0.15.2-amd64 curl -L -o cork.sig https://github.com/kinvolk/mantle/releases/download/v0.15.2/cork-0.15.2-amd64.sig curl -LO https://www.flatcar-linux.org/security/image-signing-key/Flatcar_Image_Signing_Key.asc gpg --import Flatcar_Image_Signing_Key.asc rm Flatcar_Image_Signing_Key.asc gpg --verify cork.sig cork
gpg --verify command should output something like this:
gpg: Signature made Mon 07 Jan 2019 14:51:50 CET gpg: using RSA key 84C8E771C0DF83DFBFCAAAF03ADA89DEC2507883 gpg: Good signature from "Flatcar Application Signing Key <[email protected]>" [unknown] Primary key fingerprint: C1C0 B82A 2F75 90B2 E369 822B E52F 0DB3 9145 3C45 Subkey fingerprint: 84C8 E771 C0DF 83DF BFCA AAF0 3ADA 89DE C250 7883
Then proceed with the installation of the cork binary to a location on your path:
chmod +x cork mkdir -p ~/.local/bin mv cork ~/.local/bin export PATH=$PATH:$HOME/.local/bin
You may want to add the
PATH export to your shell profile (e.g.
Next, use the cork utility to create a project directory. This will hold all of your git repos and the SDK chroot. A few gigabytes of space will be necessary.
mkdir flatcar-sdk cd flatcar-sdk cork create # This will request root permisions via sudo cork enter # This will request root permisions via sudo
cork create will download the latest SDK used for the Alpha release which can be changed with
You can also change the SDK release later.
A manifest defines which source code branches are checked out. The default manifest uses the
main development branches for the next Alpha major version.
If you want to check out a specific release, you can pass
If you want to add changes to a major version that is propagated through the channels and gets maintenance releases increasing the minor version,
you need to use the maintenance branch
flatcar-MAJORVERSION (the major version is the x in x.y.z) which you can check out by passing
--manifest-branch flatcar-CHANNEL-x.y.z --manifest-name maintenance.xml for any release of that major version family (not for releases <= 2513.2.0).
You can also change the branches later with git.
Verify you are in the SDK chroot:
$ grep NAME /etc/os-release NAME="Flatcar Container Linux by Kinvolk"
To leave the SDK chroot, simply run
To use the SDK chroot in the future, run
cork enter from the above directory.
Building an image
Set up the chroot
When entering the SDK you are in the
~/trunk/src/scripts repository which can be seen as the build system.
It is one of the three repositories that define a Flatcar build. The other two are the
~/trunk/src/third_party/coreos-overlay repository. These contain Gentoo package build files. The
hosts build files for upstream Gentoo packages and the
coreos-overlay repository contains additional and patched packages.
You have to check out the right combination of branches if you don’t use the default. For example, checking out a feature branch
based on the
main branch of
coreos-overlay means that
scripts also need to point
main (or a feature branch based on it).
Any other repositories under
third_party/ are managed by commit references and you should not alter them.
The chroot you entered has the SDK packages installed. The actual image packages are built in a new chroot under
/build/arm64-usr/, called the board.
Both chroots use Gentoo’s portage to manage the packages (via
sudo emerge for the SDK and
emerge-BOARD for the boards).
To reduce the build time it is important that as many binary packages as possible are downloaded from a previous SDK release,
Flatcar release, or nightly build.
To reference the latest nightly builds as source for board packages, use the following command. It has to be rerun if you later want
to switch to a recent nightly version because it only dereferences the nightly of the day when you ran this command, to keep your
build environment reproducible. For
BRANCH use the branch you checked out or want to base your work on, e.g.,
the Alpha development branch
main or the
flatcar-MAJORVERSION maintenance branches.
./set_version --dev-board --board-version amd64-usr/BRANCH-nightly --dev-sdk --sdk-version sdk-main-nightly
To switch to a Flatcar release when you checked out release branches use the following:
./set_version --no-dev-board --board-version x.y.z --no-dev-sdk --sdk-version x.0.0 # --no-dev-board disables any previous --dev-board setting, same for --no-dev-sdk
Normally it is enough to switch to the first SDK release of a major version (as above) to ensure that you don’t have to rebuild many SDK packages. If you notice that no binary packages are used, try a later minor release.
Optional: Set a password for console login
After entering the chroot via
cork for the first time, you can set user
This is the password you will use to log into the console of images built and launched with the SDK.
You can also rely on SSH key detection which normally works well when booting the QEMU images.
Selecting the architecture to build
arm64-usr are the only targets supported by Flatcar.
When setting up a new build target you can select it as default architecture, otherwise you need to specify the architecture for each step.
64 bit AMD: The amd64-usr target
--board option can be set to one of a few known target architectures, or system “boards”, to build for a given CPU.
To create a root filesystem for the
amd64-usr target beneath the directory
./setup_board [--default] --board=amd64-usr
64 bit ARM: The arm64-usr target
The SDK runs on an amd64 host system, relying on cross-compilation to create arm64 binaries. Still, during the compilation phase many build systems perform compiler tests by running a compiled binary or invoke compiled helper binaries. This requires the host system to use binary translation for seamlessly running arm64 binaries.
On a Fedora/Debian host system there is a
qemu-user-static package that sets everything up.
If you have troubles or use a different host system, check that a statically compiled aarch64 qemu-user binary is referenced with the
:F flag for pinning it into the kernel so that it is not needed in every chroot:
$ cat /usr/lib/binfmt.d/qemu-aarch64-static.conf :qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F $ sudo systemctl restart systemd-binfmt.service
You can run
docker run --rm -ti arm64v8/alpine on your host system as an easy check to verify everything is ready.
Once in the SDK, create a root filesystem for the
arm64-usr target beneath the directory
./setup_board [--default] --board=arm64-usr
The SDK will set up the
QEMU_LD_PREFIX environment variable, allowing to run any binaries under
/build/arm64-usr/, without an additional
Compile and link system binaries
Build all of the target binary packages:
You can make sure that a package is rebuilt with your changes by running emerge manually.
For board packages run
emerge-amd64-usr PACKAGE and for SDK packages run
sudo emerge PACKAGE
PACKAGE being either
CATEGORY/PACKAGEFOLDER or just
Render the Flatcar Container Linux image
Build a production image based on the binary packages built above:
build_image completes, it prints commands for converting the raw bin into a bootable virtual machine. Run the
Once you build an image you can launch it with KVM (instructions will print out after
If you encounter errors with KVM, verify that virtualization is supported by your CPU by running
egrep '(vmx|svm)' /proc/cpuinfo. The
/dev/kvm directory will be in your host OS when virtualization is enabled in the BIOS.
./flatcar_production_qemu.sh file can be found in the
~/trunk/src/build/images/amd64-usr/latest directory inside the SDK chroot.
image_to_vm.sh completes, run
./flatcar_production_qemu.sh -curses to launch a graphical interface to log in to the Flatcar Container Linux VM.
You could instead use the
./flatcar_production_qemu.sh -nographic, which gives you the ability to switch from the VM to the QEMU monitor console by pressing CTRL+a and then c. To close the Flatcar Container Linux Guest OS VM, run
sudo systemctl poweroff inside the VM.
You could also log in via SSH by running
./flatcar_production_qemu.sh and then running
ssh [email protected] -p 2222 to enter the guest OS. Running without the
-p 2222 option will arise a ssh: connect to host 127.0.0.1 port 22: Connection refused or Permission denied (publickey,gssapi-keyex,gssapi-with-mic) warning. Additionally, you can log in via SSH keys or with a different ssh port by running this example
./flatcar_production_qemu.sh -a ~/.ssh/authorized_keys -p 2223 -- -curses. Refer to the
Booting with QEMU
guide for more information on this usage.
The default login username is
core and the
password is the one set in the
step of this guide. If you forget your password, you will need to rerun
./set_shared_user_password and then
git and repo
Flatcar Container Linux is managed by
repo, a tool built for the Android project that makes managing a large number of git repositories easier. From the repo announcement blog:
The repo tool uses an XML-based manifest file describing where the upstream repositories are, and how to merge them into a single working checkout. repo will recurse across all the git subtrees and handle uploads, pulls, and other needed items. repo has built-in knowledge of topic branches and makes working with them an essential part of the workflow.
(from the Google Open Source Blog )
You can find the full manual for repo by visiting android.com - Developing .
Updating repo manifests
The repo manifest for Flatcar Container Linux lives in a git repository in
.repo/manifests. If you need to update the manifest edit
in this directory.
repo uses a branch called ‘default’ to track the upstream branch you
repo init, this defaults to ‘origin/master’. Keep this in
mind when making changes, the origin git repository should not have a
Building release images
The production images document is unmaintained and out of date, but contains useful pointers as to how official release images are built.
Tips and tricks
We’ve compiled a list of tips and tricks that can make working with the SDK a bit easier.
Mantle is a collection of utilities used in testing and launching SDK images.