Tips and tricks

    Finding all open pull requests and issues

    Searching all repo code

    Using repo grep you can search across all of the Git repos at once:

    repo grep CONFIG_EXTRA_FIRMWARE
    

    Note: this could take some time.

    Base system dependency graph

    Get a view into what the base system will contain and why it will contain those things with the emerge tree view:

    equery-amd64-usr depgraph --depth 1 coreos-base/coreos-dev
    

    Get a tree view of the SDK dependencies:

    equery depgraph --depth 1 coreos-base/hard-host-depends coreos-devel/sdk-depends
    

    Import ebuilds from Gentoo

    You can use scripts/update_ebuilds to fetch unmodified packages into src/third_party/portage-stable and add the files to git. The package argument should be in the format of category/package-name, e.g.:

    ~/trunk/src/scripts $ ./update_ebuilds sys-block/open-iscsi
    

    Modified packages must be moved out of src/third_party/portage-stable to src/third_party/coreos-overlay.

    If you know in advance that any files in the upstream package will need to be changed, the package can be fetched from upstream Gentoo directly into src/third_party/coreos-overlay. e.g.:

    ~/trunk/src/third_party/coreos-overlay $ mkdir -p sys-block/open-iscsi
    ~/trunk/src/third_party/coreos-overlay $ rsync -av rsync://rsync.gentoo.org/gentoo-portage/sys-block/open-iscsi/ sys-block/open-iscsi/
    

    The tailing / prevents rsync from creating the directory for the package so you don’t end up with sys-block/open-iscsi/open-iscsi. Remember to add any new files to git.

    To quickly test your new package(s), use the following commands:

    ~/trunk/src/scripts $ # Manually merge a package in the chroot
    ~/trunk/src/scripts $ emerge-amd64-usr packagename
    ~/trunk/src/scripts $ # Manually unmerge a package in the chroot
    ~/trunk/src/scripts $ emerge-amd64-usr --unmerge packagename
    ~/trunk/src/scripts $ # Remove a binary from the cache
    ~/trunk/src/scripts $ sudo rm /build/amd64-usr/packages/category/packagename-version.tbz2
    

    To recreate the chroot prior to a clean rebuild, exit the chroot and run:

    ~/flatcar-sdk $ cork create --replace
    

    To include the new package as a dependency of Flatcar Container Linux, add the package to the end of the RDEPEND environment variable in coreos-base/coreos/coreos-0.0.1.ebuild then increment the revision of Flatcar Container Linux by renaming the softlink (e.g.):

    ~/trunk/src/third_party/coreos-overly $ git mv coreos-base/coreos/coreos-0.0.1-r237.ebuild coreos-base/coreos/coreos-0.0.1-r238.ebuild
    

    The new package will now be built and installed as part of the normal build flow when you run build_packages again.

    If tests are successful, commit the changes, push to your GitHub fork and create a pull request.

    Packaging references

    References:

    Set a password for the core user (when building your own images)

    Your SSH keys should be detected and added automatically by the image build process. Optionally, you can set a password for the core user which you can use later for ssh authentication, should SSH pubkey authentication not work for you.

    After entering the chroot via cork for the first time, you can set user core's password:

    $ ./set_shared_user_password.sh
    

    This is the password you will use to log into the console of images built with the SDK.

    Creating SDK with different options

    To create SDK from a non-default manifest branch, for example, new-sdk:

    ~/flatcar-sdk $ cork create --manifest-branch=new-sdk
    

    To create SDK with a non-default SDK version, for example, 2229.0.0:

    ~/flatcar-sdk $ cork create --sdk-version=2229.0.0
    

    Use the nightly build SDK / Flatcar packages for the development branches

    While the most recent Alpha release should be a good base for the majority of changes (like package additions and kernel updates), sometimes more daring users might want to use work-in-progress branches of upcoming releases. Please note that while we run nightly builds and a limited set of automated tests on these branches, neither functionality nor stability are guaranteed.

    In general, there are two “flavours” of nightlies:

    1. Nightly builds of the maintenance branches of existing releases (“flatcar-MAJOR”). These cover the most recent Alpha, Beta, and Stable release versions and contain changed to be released with the next patch-level update.
    2. Nightly builds of the “main” branch - aka “Alpha-next”. This branch contains major changes for the next upcoming Alpha major release.

    For maintenance branches, use

    $ cork create --manifest-branch flatcar-CHANNEL-x.y.z --manifest-name maintenance.xml
    $ cork enter
    $  ./set_version --dev-board --board-version amd64-usr/flatcar-MAJOR-nightly --no-dev-sdk --sdk-version MAJOR.0.0
    

    For the main branch, use

    $ cork create
    $ cork enter
    $  ./set_version --dev-board --board-version amd64-usr/main-nightly --dev-sdk --sdk-version sdk-main-nightly
    

    Caching git https passwords

    Turn on the credential helper and git will save your password in memory for some time:

    git config --global credential.helper cache
    

    Note: You need git 1.7.10 or newer to use the credential helper

    Why doesn’t Flatcar Container Linux use SSH in the git remotes? Because we can’t do anonymous clones from GitHub with an SSH URL. This will be fixed eventually.

    SSH config

    You will be booting lots of VMs with on the fly ssh key generation. Add this in your $HOME/.ssh/config to stop the annoying fingerprint warnings.

    Host 127.0.0.1
      StrictHostKeyChecking no
      UserKnownHostsFile /dev/null
      User core
      LogLevel QUIET
    

    Hide loop devices from desktop environments

    By default desktop environments will diligently display any mounted devices including loop devices used to construct Flatcar Container Linux disk images. If the daemon responsible for this happens to be udisks then you can disable this behavior with the following udev rule:

    echo 'SUBSYSTEM=="block", KERNEL=="ram*|loop*", ENV{UDISKS_PRESENTATION_HIDE}="1", ENV{UDISKS_PRESENTATION_NOPOLICY}="1"' > /etc/udev/rules.d/85-hide-loop.rules
    udevadm control --reload
    

    Leaving developer mode

    Some daemons act differently in “dev mode”. For example update_engine refuses to auto-update or connect to HTTPS URLs. If you need to test something out of dev_mode on a vm you can do the following:

    mv /root/.dev_mode{,.old}
    

    If you want to permanently leave you can run the following:

    crossystem disable_dev_request=1; reboot
    

    Build everything from scratch

    If you want to build everything from scratch, but at the same time want to exclude several packages that take much time.

    emerge-amd64-usr --emptytree -1 -v --tree --exclude="dev-lang/rust sys-devel/gcc" coreos-base/coreos-dev
    

    Or if you want to do the rebuild by running build_packages, you should remove the binary package of coreos before rebuilding it:

    emerge-amd64-usr --unmerge coreos-base/coreos
    rm -f /build/amd64-usr/var/lib/portage/pkgs/coreos-base/coreos-0.0.1*.tbz2
    ./build_packages
    

    Modify or update invididual packages

    Before or after setting up the SDK with ./setup_board you can modify the package definitions in third_party/coreos-overlay/. Changes for toolchain packages like the compiler need to be done before running ./setup_board but any changes for the final image can be done before running ./build_packages && ./build_image. All build commands can be run multiple times but whether your last changes are picked up depends on whether the package revision was increased (by renaming the ebuild file) or the package uninstalled and the binary package removed (See the last commands in Build everything from scratch where it was done for the parent package coreos-base/coreos). Therefore, we recommend to run every build command only once in a fresh SDK to be sure that your most recent modification is used.

    For some packages, like the Linux kernel in coreos-source, coreos-kernel, and coreos-modules, it is enough to rename the ebuild file and it will download a new kernel version. Ebuilds for other packages under coreos-overlay/ reference a specific commit in CROS_WORKON_COMMIT which needs to be changed. If files of a package changed their hash sums, use ebuild packagename.ebuild manifest to recalculate the hashes for the Manifest file.

    Here is an example of updating an individual package to a newer version:

    git mv aaa-bbb/package/package-0.0.1-r1.ebuild aaa-bbb/package/package-0.0.1-r2.ebuild
    ebuild aaa-bbb/package/package-0.0.1-r2.ebuild manifest
    emerge-amd64-usr -1 -v aaa-bbb/package
    

    Do not forget about updating its version and revision in package.accept_keywords files in the profiles directory. In some cases such a file can pin an exact version of a specific package, which needs to be updated as well.

    Use binary packages from a shared build store

    Some packages like coreos-modules take a long time to build. Use:

    ./build_packages --getbinpkgver=$(gsutil cat gs://…/boards/amd64-usr/current-master/version.txt |& sed -n 's/^FLATCAR_VERSION=//p')
    

    to use packages from the another build store.

    Allow /usr to be remounted as read-write

    By default, in every Flatcar image, it is not possible to remount /usr partition as read-write. However, sometimes it is needed to mount the partition as read-write mainly for debugging purposes. To make such a debugging image, Use

    ./build_image --noenable_rootfs_verification
    

    Then it will create an image without dm-verity being enabled. So after booting with the image, you can simply run:

    sudo mount -o remount,rw /usr
    

    Known issues

    build_packages fails on coreos-base

    Sometimes coreos-dev or coreos builds will fail in build_packages with a backtrace pointing to epoll. This hasn’t been tracked down but running build_packages again should fix it. The error looks something like this:

    Packages failed:
    coreos-base/coreos-dev-0.1.0-r63
    coreos-base/coreos-0.0.1-r187
    

    Newly added package fails checking for kernel sources

    It may be necessary to comment out kernel source checks from the ebuild if the build fails, as Flatcar Container Linux does not yet provide visibility of the configured kernel source at build time. Usually this is not a problem, but may lead to warning messages.

    Emerging coreos-kernel (either manually or through build_packages) may fail with the error:

    /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.4/../../../../x86_64-pc-linux-gnu/bin/ld: scripts/kconfig/conf.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC scripts/kconfig/conf.o: error adding symbols: Bad value
    

    This indicates the ccache is corrupt. To clear the ccache, run:

    CCACHE_DIR=/var/tmp/ccache ccache -C
    

    To avoid corrupting the ccache, do not abort builds.

    build_image hangs while emerging packages after previously aborting a build

    Delete all *.portage_lockfiles in /build/<arch>/. To avoid stale lockfiles, do not abort builds.

    Constants and IDs

    Flatcar Container Linux app ID

    This UUID is used to identify Flatcar Container Linux to the update service and elsewhere:

    e96281a6-d1af-4bde-9a0a-97b76e56dc57
    

    GPT UUID types

    • Flatcar Container Linux Root: 5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6
    • Flatcar Container Linux Reserved: c95dc21a-df0e-4340-8d7b-26cbfa9a03e0
    • Flatcar Container Linux Raid Containing Root: be9067b9-ea49-4f15-b4f6-f36f8c9e1818