Troubleshooting OpenEBS

Introduction

This guide enlists various issues with OpenEBS and steps to mitigate those problems.

Prerequisites

iSCSI volume mount issue

If you are running Flatcar version 2605.9.0 and deploy OpenEBS with version 0.5.0 of Lokoctl, you might run into volume mount issue. The pod trying to mount the volume continues to stay in ContainerCreating state. To fix this issue follow the following steps. This volume mount issue is fixed in this PR in Lokomotive and is release as a part of 0.6.0. The nodes deployed using lokoctl version 0.5.0 or before need to do following fix.

Step 1: Fix the self-hosted Kubelet

NOTE: If any of the following commands fail then you might not be running a self-hosted Kubelet in your cluster.

Step 1.1: Update the image

kubectl -n kube-system set image ds kubelet kubelet=quay.io/kinvolk/kubelet:v1.19.4

Step 1.2: Remove the iSCSI mount

kubectl -n kube-system edit ds kubelet

Remove the following snippet from the kubelet Daemonset configuration:

- mountPath: /usr/sbin/iscsiadm
  name: iscsiadm

Step 2: Fix the iscsid service on hosts

SSH into all the worker nodes that will try to mount the volume created by OpenEBS. Run the following set of commands to fix the iscsid service.

Acquire root credentials:

sudo -i

Create the iscsid service dropin and restart the service with new config, copy paste the following commands:

mkdir /etc/systemd/system/iscsid.service.d
cat <<EOF > /etc/systemd/system/iscsid.service.d/00-iscsid.conf
[Service]
ExecStartPre=/bin/bash -c 'echo "InitiatorName=$(/sbin/iscsi-iname -p iqn.2020-01.io.kinvolk:01)" > /etc/iscsi/initiatorname.iscsi'
EOF
systemctl daemon-reload
systemctl restart iscsid
systemctl status --no-page iscsid
journalctl --no-pager -u iscsid

Step 3: Fix the bootstrap kubelet

Step 3.1 Remove all the occurrences of iscsiadm from kubelet service file

sed -i '/iscsiadm/d' /etc/systemd/system/kubelet.service

Step 3.2 Update kubelet version

  • Update the kubelet image URL by following this section .

  • Run the following commands to update the kubelet version:

    export latest_kubelet_version=v1.19.4
    sed -i "s|.*KUBELET_IMAGE_TAG.*|KUBELET_IMAGE_TAG=${latest_kubelet_version}|g" /etc/kubernetes/kubelet.env
    

Step 3.3 Restart kubelet service

systemctl daemon-reload
systemctl restart kubelet
systemctl status --no-page kubelet
journalctl --no-pager -u kubelet

Step 4: Verify the stuck pod

Check if the pod stuck in ContainerCreating state is in Running state now.