Troubleshooting OpenEBS

    Introduction

    This guide enlists various issues with OpenEBS and steps to mitigate those problems.

    Prerequisites

    iSCSI volume mount issue

    If you are running Flatcar version 2605.9.0 and deploy OpenEBS with version 0.5.0 of Lokoctl, you might run into volume mount issue. The pod trying to mount the volume continues to stay in ContainerCreating state. To fix this issue follow the following steps. This volume mount issue is fixed in this PR in Lokomotive and is release as a part of 0.6.0. The nodes deployed using lokoctl version 0.5.0 or before need to do following fix.

    Step 1: Fix the self-hosted Kubelet

    NOTE: If any of the following commands fail then you might not be running a self-hosted Kubelet in your cluster.

    Step 1.1: Update the image

    kubectl -n kube-system set image ds kubelet kubelet=quay.io/kinvolk/kubelet:v1.19.4

    Step 1.2: Remove the iSCSI mount

    kubectl -n kube-system edit ds kubelet

    Remove the following snippet from the kubelet Daemonset configuration:

    - mountPath: /usr/sbin/iscsiadm
  name: iscsiadm

    Step 2: Fix the iscsid service on hosts

    SSH into all the worker nodes that will try to mount the volume created by OpenEBS. Run the following set of commands to fix the iscsid service.

    Acquire root credentials:

    sudo -i

    Create the iscsid service dropin and restart the service with new config, copy paste the following commands:

    mkdir /etc/systemd/system/iscsid.service.d
cat <<EOF > /etc/systemd/system/iscsid.service.d/00-iscsid.conf
[Service]
ExecStartPre=/bin/bash -c 'echo "InitiatorName=$(/sbin/iscsi-iname -p iqn.2020-01.io.kinvolk:01)" > /etc/iscsi/initiatorname.iscsi'
EOF
systemctl daemon-reload
systemctl restart iscsid
systemctl status --no-page iscsid
journalctl --no-pager -u iscsid

    Step 3: Fix the bootstrap kubelet

    Step 3.1 Remove all the occurrences of iscsiadm from kubelet service file

    sed -i '/iscsiadm/d' /etc/systemd/system/kubelet.service

    Step 3.2 Update kubelet version

    • Update the kubelet image URL by following this section .

    • Run the following commands to update the kubelet version:

      export latest_kubelet_version=v1.19.4
sed -i "s|.*KUBELET_IMAGE_TAG.*|KUBELET_IMAGE_TAG=${latest_kubelet_version}|g" /etc/kubernetes/kubelet.env

    Step 3.3 Restart kubelet service

    systemctl daemon-reload
systemctl restart kubelet
systemctl status --no-page kubelet
journalctl --no-pager -u kubelet

    Step 4: Verify the stuck pod

    Check if the pod stuck in ContainerCreating state is in Running state now.

    github Edit this page github File documentation issue