Upgrading OpenEBS control plane and data plane components

Introduction

This guide provides step-by-step instructions on how to upgrade OpenEBS control plane and data plane components from one version to another.

To get the current version installed, execute:

kubectl get pods -n openebs -o jsonpath='{.items[*].metadata.labels.openebs\.io/version}'

Refer to the Lokomotive release notes to get the version to upgrade.

Make note of the current version installed and the version to upgrade and set the values accordingly in the document.

As an example, in this document we are going to upgrade from v1.12.0 to v2.2.0.

Prerequisites

To upgrade the OpenEBS data plane components, we need the following:

  • A Kubernetes cluster accessible via kubectl.

  • OpenEBS component installed. You can check if OpenEBS is indeed in expected version:

    $ kubectl get pods -n openebs -l openebs.io/version=1.12.0
    
  • Upgrade process should not be disruptive, but it is recommended to schedule a downtime for the applications consuming the OpenEBS PV and make sure to take a backup of the data before starting the below upgrade procedure in case some problem arises. Lokomotive provides Velero component for backup and restore.

  • Ensure the cluster and OpenEBS volumes are is in healthy state before proceeding.

    $ lokoctl health
    Node                    Ready    Reason          Message
    
    alpha-controller-0      True     KubeletReady    kubelet is posting ready status
    alpha-large-worker-0    True     KubeletReady    kubelet is posting ready status
    Name      Status    Message              Error
    
    etcd-0    True      {"health":"true"}
    
    
    $ kubectl get cstorpools -n openebs # Status should be Healthy.
    
    NAME                                ALLOCATED   FREE   CAPACITY   STATUS    READONLY   TYPE      AGE
    cstor-pool-openebs-replica-1-w3r7   6.98G       437G   444G       Healthy   false      striped   4d22h
    
    
    $ kubectl get cstorvolume -n openebs # Status should be Healthy.
    
    NAME                                       STATUS    AGE     CAPACITY
    pvc-183dc3df-a7d9-4273-a8e2-7f66d2e19f4e   Healthy   3d20h   50Gi
    pvc-5d4b4c2b-2ed5-4e75-aeb9-fbd2918ebb78   Healthy   3d20h   50Gi
    

Steps

Set the following environment variables in the terminal to assist in the upgrade process:

export OPENEBS_OLD_VERSION=1.12.0
export OPENEBS_NEW_VERSION=2.2.0

Step 1: Upgrade OpenEBS control plane components

Lokomotive provides an easy way of upgrading the OpenEBS control plane.

Execute the following command to upgrade OpenEBS control plane:

lokoctl component apply openebs-operator

Doing so, terminates the OpenEBS resources associated with the old version and new resources are created for the new version.

Verify all the pods are in Running state before proceeding:

$ kubectl get pods -n openebs -l openebs.io/version=${OPENEBS_NEW_VERSION}

Step 2: Upgrade OpenEBS data plane components

OpenEBS control plane and data plane components work independently. Even if the control plane components are upgraded to the new version, the data plane components continue to work with the older version.

Upgrade cStor pools

Create a Job to upgrade the existing cStor pools:

cat > upgrade-cstor-pools-from-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}.yaml <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: cstor-spc-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}
  namespace: openebs
spec:
  backoffLimit: 4
  template:
    spec:
      serviceAccountName: openebs-operator
      containers:
      - name:  upgrade
        args:
        - "cstor-spc"
        - "--v=4"
        - "--from-version=${OPENEBS_OLD_VERSION}"
        - "--to-version=${OPENEBS_NEW_VERSION}"
$(kubectl get spc -n openebs --no-headers -o custom-columns=":metadata.name" | sed 's/.*/        - "&"/')
        #DO NOT CHANGE BELOW PARAMETERS
        env:
        - name: OPENEBS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        tty: true
        image: openebs/m-upgrade:${OPENEBS_NEW_VERSION}
        imagePullPolicy: Always
      restartPolicy: OnFailure
EOF

Apply the job to start the upgrade for cStor pools:

kubectl apply -f upgrade-cstor-pools-from-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}.yaml

Ensure the job runs to completion:

kubectl describe job -n openebs cstor-spc-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}

# Should see a similar outpu of 'Completed'
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  2m35s  job-controller  Created pod: cstor-spc-1.12.0-to-2.2.0-hpcxl
  Normal  Completed         52s    job-controller  Job completed

Upgrade cStor volumes

Create a Kubernetes job to upgrade the existing cStor volumes:

cat > upgrade-cstor-vols-from-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}.yaml <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: cstor-vol-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}
  namespace: openebs
spec:
  backoffLimit: 4
  template:
    spec:
      serviceAccountName: openebs-operator
      containers:
      - name:  upgrade
        args:
        - "cstor-volume"
        - "--v=4"
        - "--from-version=${OPENEBS_OLD_VERSION}"
        - "--to-version=${OPENEBS_NEW_VERSION}"
$(kubectl get cstorvolumes -n openebs --no-headers -o custom-columns=":metadata.name" | sed 's/.*/        - "&"/')
        #DO NOT CHANGE BELOW PARAMETERS
        env:
        - name: OPENEBS_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        tty: true
        image: openebs/m-upgrade:${OPENEBS_NEW_VERSION}
        imagePullPolicy: Always
      restartPolicy: OnFailure
EOF
kubectl apply -f upgrade-cstor-vols-from-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}.yaml

Ensure the job runs to completion:

kubectl describe job -n openebs cstor-vol-${OPENEBS_OLD_VERSION}-to-${OPENEBS_NEW_VERSION}

# Should see a similar outpu of 'Completed'
  Type    Reason            Age    From            Message
  ----    ------            ----   ----            -------
  Normal  SuccessfulCreate  2m35s  job-controller  Created pod: cstor-vol-1.12.0-to-2.2.0-gtwsd
  Normal  Completed         1m36s  job-controller  Job completed

Step 3: Verify

To check if the upgrade process was successful, execute:

$ kubectl get pods -n openebs -l openebs.io/version=${OPENEBS_NEW_VERSION}
NAME                                                              READY   STATUS    RESTARTS   AGE
cstor-pool-openebs-replica-3-6qxo-bd59f5554-d97l4                 3/3     Running   0          27m
cstor-pool-openebs-replica-3-ag9v-78cfbb64b4-vzcs9                3/3     Running   0          26m
cstor-pool-openebs-replica-3-gt7g-8ddd9b457-jqwz4                 3/3     Running   0          25m
openebs-operator-admission-server-6b5fb6dff5-6dc8d                1/1     Running   2          34m
openebs-operator-apiserver-5c467bc588-fsl58                       1/1     Running   0          34m
openebs-operator-localpv-provisioner-74d76d55b-s8zwv              1/1     Running   0          33m
openebs-operator-ndm-cfnxp                                        1/1     Running   0          34m
openebs-operator-ndm-nsqkg                                        1/1     Running   0          33m
openebs-operator-ndm-operator-758fdbc5f4-5qqxx                    1/1     Running   0          34m
openebs-operator-ndm-x44d8                                        1/1     Running   0          34m
openebs-operator-provisioner-59c6dc5dfc-q9k65                     1/1     Running   0          34m
openebs-operator-snapshot-operator-bf49c5dc6-xqqlq                2/2     Running   0          34m
pvc-5b7d8ee3-59b8-4a02-bd6d-a4e66fbecf9f-target-68984f69b7l42sl   3/3     Running   0          22m
pvc-e8f8b268-b81c-4201-841a-283f557c44a7-target-7484b968c4pzld6   3/3     Running   0          20m

To check if all the StoragePoolClaims and CStorVolumes have been upgraded, execute:

$ kubectl get pods -n openebs -l openebs.io/version=${OPENEBS_OLD_VERSION}

No output should be displayed.

Summary

This guide helps to upgrade the OpenEBS control plane and data plane components.

Additional resources

OpenEBS troubleshooting guide .

For additional information regarding the upgrade steps, see OpenEBS upgrade documentation .