Flatcar Container Linux developer SDK guide
These are the instructions for building Flatcar Container Linux itself. By the end of the guide you will build a developer image that you can run under KVM and have tools for making changes to the code.
Flatcar Container Linux is an open source project. All of the source for Flatcar Container Linux is available on github . If you find issues with these docs or the code please send a pull request.
Direct questions and suggestions to the IRC channel or mailing list .
Getting started
Let’s get set up with an SDK chroot and build a bootable image of Flatcar Container Linux. The SDK chroot has a full toolchain and isolates the build process from quirks and differences between host OSes. The SDK must be run on an x86-64 Linux machine, the distro should not matter (Ubuntu, Fedora, etc).
Prerequisites
System requirements to get started:
- curl
- git
- bzip2
- gpg
- sudo
You also need a proper git setup:
git config --global user.email "[email protected]"
git config --global user.name "Your Name"
NOTE: Do the git configuration as a normal user and not with sudo.
Using Cork
The cork
utility, included in the Flatcar Container Linux
mantle
project, is used to create and work with an SDK chroot.
First, download the cork utility and verify it with the signature:
curl -L -o cork https://github.com/kinvolk/mantle/releases/download/v0.15.2/cork-0.15.2-amd64
curl -L -o cork.sig https://github.com/kinvolk/mantle/releases/download/v0.15.2/cork-0.15.2-amd64.sig
curl -LO https://www.flatcar-linux.org/security/image-signing-key/Flatcar_Image_Signing_Key.asc
gpg --import Flatcar_Image_Signing_Key.asc
rm Flatcar_Image_Signing_Key.asc
gpg --verify cork.sig cork
The gpg --verify
command should output something like this:
gpg: Signature made Mon 07 Jan 2019 14:51:50 CET
gpg: using RSA key 84C8E771C0DF83DFBFCAAAF03ADA89DEC2507883
gpg: Good signature from "Flatcar Application Signing Key <[email protected]>" [unknown]
Primary key fingerprint: C1C0 B82A 2F75 90B2 E369 822B E52F 0DB3 9145 3C45
Subkey fingerprint: 84C8 E771 C0DF 83DF BFCA AAF0 3ADA 89DE C250 7883
Then proceed with the installation of the cork binary to a location on your path:
chmod +x cork
mkdir -p ~/.local/bin
mv cork ~/.local/bin
export PATH=$PATH:$HOME/.local/bin
You may want to add the PATH
export to your shell profile (e.g. .bashrc
).
Next, use the cork utility to create a project directory. This will hold all of your git repos and the SDK chroot. A few gigabytes of space will be necessary.
mkdir flatcar-sdk
cd flatcar-sdk
cork create # This will request root permisions via sudo
cork enter # This will request root permisions via sudo
By default cork create
will download the latest SDK used for the Alpha release which can be changed with --sdk-version x.y.z
.
You can also change the SDK release later.
A manifest defines which source code branches are checked out. The default manifest uses the main
development branches for the next Alpha major version.
If you want to check out a specific release, you can pass --manifest-branch flatcar-CHANNEL-x.y.z
.
If you want to add changes to a major version that is propagated through the channels and gets maintenance releases increasing the minor version,
you need to use the maintenance branch flatcar-MAJORVERSION
(the major version is the x in x.y.z) which you can check out by passing
--manifest-branch flatcar-CHANNEL-x.y.z --manifest-name maintenance.xml
for any release of that major version family (not for releases <= 2513.2.0).
You can also change the branches later with git.
Verify you are in the SDK chroot:
$ grep NAME /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
To leave the SDK chroot, simply run exit
.
To use the SDK chroot in the future, run cork enter
from the above directory.
Building an image
Set up the chroot
When entering the SDK you are in the ~/trunk/src/scripts
repository which can be seen as the build system.
It is one of the three repositories that define a Flatcar build. The other two are the ~/trunk/src/third_party/portage-stable
and the ~/trunk/src/third_party/coreos-overlay
repository. These contain Gentoo package build files. The portage-stable
repository
hosts build files for upstream Gentoo packages and the coreos-overlay
repository contains additional and patched packages.
You have to check out the right combination of branches if you don’t use the default. For example, checking out a feature branch
based on the main
branch of coreos-overlay
means that portage-stable
and scripts
also need to point
to main
(or a feature branch based on it).
Any other repositories under third_party/
are managed by commit references and you should not alter them.
The chroot you entered has the SDK packages installed. The actual image packages are built in a new chroot under /build/amd64-usr/
or /build/arm64-usr/
, called the board.
Both chroots use Gentoo’s portage to manage the packages (via sudo emerge
for the SDK and emerge-BOARD
for the boards).
To reduce the build time it is important that as many binary packages as possible are downloaded from a previous SDK release,
Flatcar release, or nightly build.
To reference the latest nightly builds as source for board packages, use the following command. It has to be rerun if you later want
to switch to a recent nightly version because it only dereferences the nightly of the day when you ran this command, to keep your
build environment reproducible. For BRANCH
use the branch you checked out or want to base your work on, e.g.,
the Alpha development branch main
or the flatcar-MAJORVERSION
maintenance branches.
./set_version --dev-board --board-version amd64-usr/BRANCH-nightly --dev-sdk --sdk-version sdk-main-nightly
To switch to a Flatcar release when you checked out release branches use the following:
./set_version --no-dev-board --board-version x.y.z --no-dev-sdk --sdk-version x.0.0
# --no-dev-board disables any previous --dev-board setting, same for --no-dev-sdk
Normally it is enough to switch to the first SDK release of a major version (as above) to ensure that you don’t have to rebuild many SDK packages. If you notice that no binary packages are used, try a later minor release.
Optional: Set a password for console login
After entering the chroot via cork
for the first time, you can set user core
's password:
./set_shared_user_password.sh
This is the password you will use to log into the console of images built and launched with the SDK.
You can also rely on SSH key detection which normally works well when booting the QEMU images.
Selecting the architecture to build
amd64-usr
and arm64-usr
are the only targets supported by Flatcar.
When setting up a new build target you can select it as default architecture, otherwise you need to specify the architecture for each step.
64 bit AMD: The amd64-usr target
The --board
option can be set to one of a few known target architectures, or system “boards”, to build for a given CPU.
To create a root filesystem for the amd64-usr
target beneath the directory /build/amd64-usr/
:
./setup_board [--default] --board=amd64-usr
64 bit ARM: The arm64-usr target
The SDK runs on an amd64 host system, relying on cross-compilation to create arm64 binaries. Still, during the compilation phase many build systems perform compiler tests by running a compiled binary or invoke compiled helper binaries. This requires the host system to use binary translation for seamlessly running arm64 binaries.
On a Fedora/Debian host system there is a qemu-user-static
package that sets everything up.
If you have troubles or use a different host system, check that a statically compiled aarch64 qemu-user binary is referenced with the :F
flag for pinning it into the kernel so that it is not needed in every chroot:
$ cat /usr/lib/binfmt.d/qemu-aarch64-static.conf
:qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64-static:F
$ sudo systemctl restart systemd-binfmt.service
You can run docker run --rm -ti arm64v8/alpine
on your host system as an easy check to verify everything is ready.
Once in the SDK, create a root filesystem for the arm64-usr
target beneath the directory /build/arm64-usr/
:
./setup_board [--default] --board=arm64-usr
The SDK will set up the QEMU_LD_PREFIX
environment variable, allowing to run any binaries under /build/arm64-usr/
, without an additional chroot
command.
Compile and link system binaries
Build all of the target binary packages:
./build_packages [--board=...]
You can make sure that a package is rebuilt with your changes by running emerge manually.
For board packages run emerge-amd64-usr PACKAGE
and for SDK packages run sudo emerge PACKAGE
with PACKAGE
being either CATEGORY/PACKAGEFOLDER
or just PACKAGEFOLDER
.
Render the Flatcar Container Linux image
Build a production image based on the binary packages built above:
./build_image [--board=...]
After build_image
completes, it prints commands for converting the raw bin into a bootable virtual machine. Run the image_to_vm.sh
command.
Booting
Once you build an image you can launch it with KVM (instructions will print out after image_to_vm.sh
runs).
If you encounter errors with KVM, verify that virtualization is supported by your CPU by running egrep '(vmx|svm)' /proc/cpuinfo
. The /dev/kvm
directory will be in your host OS when virtualization is enabled in the BIOS.
The ./flatcar_production_qemu.sh
file can be found in the ~/trunk/src/build/images/amd64-usr/latest
directory inside the SDK chroot.
Boot Options
After image_to_vm.sh
completes, run ./flatcar_production_qemu.sh -curses
to launch a graphical interface to log in to the Flatcar Container Linux VM.
You could instead use the -nographic
option, ./flatcar_production_qemu.sh -nographic
, which gives you the ability to switch from the VM to the QEMU monitor console by pressing CTRL+a and then c. To close the Flatcar Container Linux Guest OS VM, run sudo systemctl poweroff
inside the VM.
You could also log in via SSH by running ./flatcar_production_qemu.sh
and then running ssh [email protected] -p 2222
to enter the guest OS. Running without the -p 2222
option will arise a ssh: connect to host 127.0.0.1 port 22: Connection refused or Permission denied (publickey,gssapi-keyex,gssapi-with-mic) warning. Additionally, you can log in via SSH keys or with a different ssh port by running this example ./flatcar_production_qemu.sh -a ~/.ssh/authorized_keys -p 2223 -- -curses
. Refer to the
Booting with QEMU
guide for more information on this usage.
The default login username is core
and the
password is the one set in the ./set_shared_user_password
step of this guide. If you forget your password, you will need to rerun ./set_shared_user_password
and then ./build_image
again.
Making changes
git and repo
Flatcar Container Linux is managed by repo
, a tool built for the Android project that makes managing a large number of git repositories easier. From the repo announcement blog:
The repo tool uses an XML-based manifest file describing where the upstream repositories are, and how to merge them into a single working checkout. repo will recurse across all the git subtrees and handle uploads, pulls, and other needed items. repo has built-in knowledge of topic branches and makes working with them an essential part of the workflow.
(from the Google Open Source Blog )
You can find the full manual for repo by visiting android.com - Developing .
Updating repo manifests
The repo manifest for Flatcar Container Linux lives in a git repository in
.repo/manifests
. If you need to update the manifest edit default.xml
in this directory.
repo
uses a branch called ‘default’ to track the upstream branch you
specify in repo init
, this defaults to ‘origin/master’. Keep this in
mind when making changes, the origin git repository should not have a
‘default’ branch.
Building release images
The production images document is unmaintained and out of date, but contains useful pointers as to how official release images are built.
Tips and tricks
We’ve compiled a list of tips and tricks that can make working with the SDK a bit easier.
Testing images
Mantle is a collection of utilities used in testing and launching SDK images.