Updating OpenEBS control plane and data plane components

Introduction

This guide provides step-by-step instructions on how to update OpenEBS control plane and data plane components from one version to another.

To get the current version installed, execute:

kubectl get pods -n openebs -o jsonpath='{.items[*].metadata.labels.openebs\.io/version}'

Refer to the Lokomotive release notes to get the version to update.

Make note of the current version installed and the version to update and set the values accordingly in the document.

As an example, in this document we are going to update from v2.6.0 to v2.10.0.

Prerequisites

To update the OpenEBS data plane components, we need the following:

A Kubernetes cluster accessible via kubectl.
OpenEBS component installed. You can check if OpenEBS is indeed in expected version:
```
kubectl get pods -n openebs -l openebs.io/version=2.6.0
```
Update process should not be disruptive, but it is recommended to schedule a downtime for the applications consuming the OpenEBS PV and make sure to take a backup of the data before starting the below update procedure in case some problem arises. Lokomotive provides Velero component for backup and restore.

Ensure the cluster and OpenEBS volumes are is in healthy state before proceeding.

$ lokoctl health
Node                    Ready    Reason          Message


alpha-large-worker-0    True     KubeletReady    kubelet is posting ready status
Name      Status    Message              Error




$ kubectl get cstorpools -n openebs # Status should be Healthy.


cstor-pool-openebs-replica-1-w3r7   6.98G       437G   444G       Healthy   false      striped   4d22h


$ kubectl get cstorvolume -n openebs # Status should be Healthy.


pvc-183dc3df-a7d9-4273-a8e2-7f66d2e19f4e   Healthy   3d20h   50Gi
pvc-5d4b4c2b-2ed5-4e75-aeb9-fbd2918ebb78   Healthy   3d20h   50Gi

Steps

Set the following environment variables in the terminal to assist in the update process:

export OPENEBS_OLD_VERSION=2.6.0
export OPENEBS_NEW_VERSION=2.10.0

Step 1: Update OpenEBS control plane components

Lokomotive provides an easy way of updating the OpenEBS control plane.

Execute the following command to update OpenEBS control plane:

lokoctl component apply openebs-operator

Doing so, terminates the OpenEBS resources associated with the old version and new resources are created for the new version.

Verify all the pods are in Running state before proceeding:

kubectl get pods -n openebs -l openebs.io/version=${OPENEBS_NEW_VERSION}

Step 2: Update OpenEBS data plane components

OpenEBS control plane and data plane components work independently. Even if the control plane components are updated to the new version, the data plane components continue to work with the older version.

Update cStor pools

Create a Job to update the existing cStor pools:

cat > update-cstor-pools-from-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}.yaml <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: cstor-spc-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}
  namespace: openebs
spec:
  backoffLimit: 4
  template:
    spec:
      serviceAccountName: openebs-operator
      containers:
      - name:  update
        args:
        - "cstor-spc"
        - "--v=4"
        - "--from-version=${OPENEBS_OLD_VERSION}"
        - "--to-version=${OPENEBS_NEW_VERSION}"
$(kubectl get spc -n openebs --no-headers -o custom-columns=":metadata.name" | sed 's/.*/        - "&"/')
        #DO NOT CHANGE BELOW PARAMETERS
        env:
        - name: OPENEBS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        tty: true
        image: openebs/m-upgrade:${OPENEBS_NEW_VERSION}
        imagePullPolicy: Always
      restartPolicy: OnFailure
EOF

Apply the job to start the update for cStor pools:

kubectl apply -f update-cstor-pools-from-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}.yaml

Ensure the job runs to completion:

kubectl describe job -n openebs cstor-spc-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}

# Should see a similar output of 'Completed'
Events:
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  5m52s  job-controller  Created pod: cstor-spc-2.6.0-to-2.10.0-t8qg8
  Normal  Completed         24s    job-controller  Job completed

Update cStor volumes

Create a Kubernetes job to update the existing cStor volumes:

cat > update-cstor-vols-from-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}.yaml <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: cstor-vol-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}
  namespace: openebs
spec:
  backoffLimit: 4
  template:
    spec:
      serviceAccountName: openebs-operator
      containers:
      - name:  update
        args:
        - "cstor-volume"
        - "--v=4"
        - "--from-version=${OPENEBS_OLD_VERSION}"
        - "--to-version=${OPENEBS_NEW_VERSION}"
$(kubectl get cstorvolumes -n openebs --no-headers -o custom-columns=":metadata.name" | sed 's/.*/        - "&"/')
        #DO NOT CHANGE BELOW PARAMETERS
        env:
        - name: OPENEBS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        tty: true
        image: openebs/m-upgrade:${OPENEBS_NEW_VERSION}
        imagePullPolicy: Always
      restartPolicy: OnFailure
EOF

kubectl apply -f update-cstor-vols-from-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}.yaml

Ensure the job runs to completion:

kubectl describe job -n openebs cstor-vol-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}

# Should see a similar output of 'Completed'
Events:
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  2m41s  job-controller  Created pod: cstor-vol-2.6.0-to-2.10.0-hjndp
  Normal  Completed         7s     job-controller  Job completed

Step 3: Verify

To check if the update process was successful, execute:

$ kubectl get pods -n openebs -l openebs.io/version=${OPENEBS_NEW_VERSION}
NAME                                                              READY   STATUS    RESTARTS   AGE
cstor-pool-openebs-replica-3-6qxo-bd59f5554-d97l4                 3/3     Running   0          27m
cstor-pool-openebs-replica-3-ag9v-78cfbb64b4-vzcs9                3/3     Running   0          26m
cstor-pool-openebs-replica-3-gt7g-8ddd9b457-jqwz4                 3/3     Running   0          25m
openebs-operator-admission-server-6b5fb6dff5-6dc8d                1/1     Running   2          34m
openebs-operator-apiserver-5c467bc588-fsl58                       1/1     Running   0          34m
openebs-operator-localpv-provisioner-74d76d55b-s8zwv              1/1     Running   0          33m
openebs-operator-ndm-cfnxp                                        1/1     Running   0          34m
openebs-operator-ndm-nsqkg                                        1/1     Running   0          33m
openebs-operator-ndm-operator-758fdbc5f4-5qqxx                    1/1     Running   0          34m
openebs-operator-ndm-x44d8                                        1/1     Running   0          34m
openebs-operator-provisioner-59c6dc5dfc-q9k65                     1/1     Running   0          34m
openebs-operator-snapshot-operator-bf49c5dc6-xqqlq                2/2     Running   0          34m
pvc-5b7d8ee3-59b8-4a02-bd6d-a4e66fbecf9f-target-68984f69b7l42sl   3/3     Running   0          22m
pvc-e8f8b268-b81c-4201-841a-283f557c44a7-target-7484b968c4pzld6   3/3     Running   0          20m

To check if all the StoragePoolClaims and CStorVolumes have been updated, execute:

kubectl get pods -n openebs -l openebs.io/version=${OPENEBS_OLD_VERSION}

No output should be displayed.

Summary

This guide helps to update the OpenEBS control plane and data plane components.

Additional resources

OpenEBS troubleshooting guide .

For additional information regarding the update steps, see OpenEBS update documentation .