Debugging tools on Flatcar Container Linux
You can use common debugging tools like tcpdump or strace with Toolbox. Using the filesystem of a specified Docker container Toolbox will launch a container with full system privileges including access to system PIDs, network interfaces and other global information. Inside of the toolbox, the machine’s filesystem is mounted to
By default, Toolbox uses the stock Fedora Docker container. To start using it, simply run:
You’re now in the namespace of Fedora and can install any software you’d like via
dnf. For example, if you’d like to use
[[email protected] ~]# dnf -y install tcpdump [[email protected] ~]# tcpdump -i ens3 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens3, link-type EN10MB (Ethernet), capture size 65535 bytes
Specify a custom Docker image
.toolboxrc in the user’s home folder to use a specific Docker image:
$ cat .toolboxrc TOOLBOX_DOCKER_IMAGE=index.example.com/debug TOOLBOX_USER=root $ /usr/bin/toolbox Pulling repository index.example.com/debug ...
You can also specify this in a Container Linux Config:
storage: files: - path: /home/core/.toolboxrc filesystem: root mode: 0644 contents: inline: | TOOLBOX_DOCKER_IMAGE=index.example.com/debug TOOLBOX_DOCKER_TAG=v1 TOOLBOX_USER=root
Under the hood
Behind the scenes,
toolbox downloads, prepares and exports the container
image you specify (or the default
fedora image), then creates a container
from that extracted image by calling
systemd-nspawn. The exported
image is retained in
/var/lib/toolbox/[username]-[image name]-[image tag], e.g. the default
image run by the
core user is at
This means two important things:
- Changes made inside the container will persist between sessions
- The container filesystem will take up space on disk (a few hundred MiB
for the default
Spawn a toolbox with tmux in the background
toolbox can only be started once it is not straightforward to use
for long-running jobs or sharing a debugging session with someone else.
To keep user processes running in the background after logging out with SSH,
you need to start them via
systemd-run because process lingering is disabled
by default in logind and all non-service user processes are killed on logout.
Spawn a user service to persist the toolbox container with the
even when you log out with SSH.
The following command line will ensure
pidof are installed
in the container, then create a new
tmux session to which you can later attach,
and keep the service active by waiting with
strace until the
tmux process exits.
systemd-run --user toolbox sh -c 'dnf install -y tmux strace procps-ng; tmux new-session -d -s sharedsession; strace -p "$(pidof tmux)"'
-d we tell
tmux to not allocate a TTY now (needed for
systemd-run) but run a
new session in the background.
tmux forks away, we cannot use
wait in the shell to wait for children but need
strace to have a foreground process running that prevents
toolbox from quitting.
Once this is running you can can attach to the
tmux session as often as you want from any SSH connection.
sudo nsenter -t "$(pidof tmux | cut -d ' ' -f 1)" -a tmux a
As usual with
tmux you can attach and detach to the session as many times as you want because detaching
tmux running in the background. But keep in mind that if you exit the session, the process
systemd-run will terminate and you’ll have to start the service again with
SSH directly into a toolbox
Advanced users can SSH directly into a toolbox by setting up an
useradd bob -m -p '*' -s /usr/bin/toolbox -U -G sudo,docker,rkt
To test, SSH as bob:
ssh [email protected] Flatcar Container Linux by Kinvolk alpha (2671.0.0) Downloading sha256:ee7e8933710 [=============================] 63.4 MB / 63.4 MB Spawning container bob-fedora-latest on /var/lib/toolbox/bob-fedora-latest. Press ^] three times within 1s to kill container. [[email protected] ~]# dnf -y install emacs-nox [[email protected] ~]# emacs /media/root/etc/systemd/system/newapp.service