Backup Rook Ceph volume on S3 using Velero

    Introduction

    Rook is a component of Lokomotive which provides storage on Packet cloud. Taking regular backup of the data to a remote server is an essential strategy for disaster recovery.

    Velero is another component of Lokomotive which helps you to backup entire namespaces, including volume data in them.

    Learning objectives

    This document will walk you through the process of backing up a namespace including the volume in it.

    Prerequisites

    Steps

    Step 1: Deploy Velero

    Config

    Create a file velero.lokocfg with the following contents:

    component "velero" {
      provider = "restic"
    
      restic {
        credentials = file("./credentials-velero")
        backup_storage_location {
          provider = "aws"
          bucket   = "rook-ceph-backup"
          region   = "us-west-1"
        }
      }
    }
    

    In the above config region should match the region of bucket created previously using aws CLI.

    Replace the ./credentials-velero with path to AWS credentials file for the velero user.

    Deploy

    Execute the following command to deploy the velero component:

    lokoctl component apply velero
    

    Verify the pods in the velero namespace are in the Running state (this may take a few minutes):

    $ kubectl -n velero get pods
    NAME                     READY   STATUS    RESTARTS   AGE
    restic-c27rq             1/1     Running   0          2m
    velero-66d5d67b5-g54x7   1/1     Running   0          2m
    

    Step 2: Deploy sample workload

    If you already have an application you want to backup, then skip this step.

    Let us deloy a stateful application and save some demo data in it. Save the following YAML config in a file named stateful.yaml:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: demo-ns
    ---
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      labels:
        app: demo-app
      name: demo-app
      namespace: demo-ns
    spec:
      replicas: 1
      serviceName: "demo-app"
      selector:
        matchLabels:
          app: demo-app
      template:
        metadata:
          labels:
            app: demo-app
        spec:
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
            runAsGroup: 65534
          containers:
          - image: busybox:1
            name: app
            command: ["/bin/sh"]
            args:
            - -c
            - "echo 'sleeping' && sleep infinity"
            volumeMounts:
            - mountPath: "/data"
              name: data
      volumeClaimTemplates:
      - metadata:
          name: data
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 100Mi
    

    Execute the following command to deploy the application:

    kubectl apply -f stateful.yaml
    

    Verify the application is running fine:

    $ kubectl -n demo-ns get pods
    NAME         READY   STATUS    RESTARTS   AGE
    demo-app-0   1/1     Running   0          16s
    

    Execute the following command to generate some dummy data:

    kubectl -n demo-ns exec -it demo-app-0 -- /bin/sh -c \
        'dd if=/dev/zero of=/data/file.txt count=40 bs=1048576'
    

    Verify that the data is generated:

    $ kubectl -n demo-ns exec -it demo-app-0 -- /bin/sh -c 'du -s /data'
    40960   /data
    

    Step 3: Annotate pods

    Annotate the pods with volumes attached to them with their volume names so that Velero takes backup of volume data. Replace the pod name and the volume name as needed in the following command:

    kubectl -n demo-ns annotate pod demo-app-0 backup.velero.io/backup-volumes=data
    

    NOTE: Modify pod template in Deployment spec or StatefulSet spec to always backup persistent volumes attached to them. This permanent setting will render this step unnecessary.

    Step 4: Backup entire namespace

    Execute the following command to start the backup of the concerned namespace. In our demo application’s case it is demo-ns:

    velero backup create backup-demo-app-ns --include-namespaces demo-ns --wait
    

    Above operation may take some time, depending on the size of the data.

    Step 5: Restore Volumes

    Same Cluster

    If you plan to restore in the same cluster, then delete the namespace. In case of our demo application run the following command:

    kubectl delete ns demo-ns
    

    NOTE: If you are restoring a stateful component of Lokomotive like prometheus-operator, then delete the component namespace by running kubectl delete ns monitoring.

    Different Cluster

    In another cluster deploy components rook, rook-ceph and velero with the same configuration for a successful restore.

    Restore

    Execute the following command to start the restore:

    velero restore create --from-backup backup-demo-app-ns
    

    Verify if Velero restored the application successfully:

    $ kubectl -n demo-ns get pods
    NAME         READY   STATUS    RESTARTS   AGE
    demo-app-0   1/1     Running   0          51s
    

    NOTE: If you are restoring a stateful component of Lokomotive like prometheus-operator, then once pods in monitoring namespace are in Running state, then run lokoctl component apply prometheus-operator to ensure the latest configs are applied.

    Verify that the data is restored correctly:

    $ kubectl -n demo-ns exec -it demo-app-0 -- /bin/sh -c 'du -s /data'
    40960   /data
    

    Additional resources