Backup Rook Ceph volume on S3 using Velero

Introduction

Rook is a component of Lokomotive which provides storage on Packet cloud. Taking regular backup of the data to a remote server is an essential strategy for disaster recovery.

Velero is another component of Lokomotive which helps you to backup entire namespaces, including volume data in them.

Learning objectives

This document will walk you through the process of backing up a namespace including the volume in it.

Prerequisites

Steps

Step 1: Deploy Velero

Config

Create a file velero.lokocfg with the following contents:

component "velero" {
  restic {
    credentials = file("./credentials-velero")
    backup_storage_location {
      provider = "aws"
      bucket   = "rook-ceph-backup"
      region   = "us-west-1"
    }
  }
}

In the above config region should match the region of bucket created previously using aws CLI.

Replace the ./credentials-velero with path to AWS credentials file for the velero user.

Deploy

Execute the following command to deploy the velero component:

lokoctl component apply velero

Verify the pods in the velero namespace are in the Running state (this may take a few minutes):

$ kubectl -n velero get pods
NAME                     READY   STATUS    RESTARTS   AGE
restic-c27rq             1/1     Running   0          2m
velero-66d5d67b5-g54x7   1/1     Running   0          2m

Step 2: Deploy sample workload

If you already have an application you want to backup, then skip this step.

Let us deloy a stateful application and save some demo data in it. Save the following YAML config in a file named stateful.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: demo-ns
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: demo-app
  name: demo-app
  namespace: demo-ns
spec:
  replicas: 1
  serviceName: "demo-app"
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
      containers:
      - image: busybox:1
        name: app
        command: ["/bin/sh"]
        args:
        - -c
        - "echo 'sleeping' && sleep infinity"
        volumeMounts:
        - mountPath: "/data"
          name: data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Mi

Execute the following command to deploy the application:

kubectl apply -f stateful.yaml

Verify the application is running fine:

$ kubectl -n demo-ns get pods
NAME         READY   STATUS    RESTARTS   AGE
demo-app-0   1/1     Running   0          16s

Execute the following command to generate some dummy data:

kubectl -n demo-ns exec -it demo-app-0 -- /bin/sh -c \
    'dd if=/dev/zero of=/data/file.txt count=40 bs=1048576'

Verify that the data is generated:

$ kubectl -n demo-ns exec -it demo-app-0 -- /bin/sh -c 'du -s /data'
40960   /data

Step 3: Annotate pods

Annotate the pods with volumes attached to them with their volume names so that Velero takes backup of volume data. Replace the pod name and the volume name as needed in the following command:

kubectl -n demo-ns annotate pod demo-app-0 backup.velero.io/backup-volumes=data

NOTE: Modify pod template in Deployment spec or StatefulSet spec to always backup persistent volumes attached to them. This permanent setting will render this step unnecessary.

Step 4: Backup entire namespace

Execute the following command to start the backup of the concerned namespace. In our demo application’s case it is demo-ns:

velero backup create backup-demo-app-ns --include-namespaces demo-ns --wait

Above operation may take some time, depending on the size of the data.

Step 5: Restore Volumes

Same Cluster

If you plan to restore in the same cluster, then delete the namespace. In case of our demo application run the following command:

kubectl delete ns demo-ns

NOTE: If you are restoring a stateful component of Lokomotive like prometheus-operator, then delete the component namespace by running kubectl delete ns monitoring.

Different Cluster

In another cluster deploy components rook, rook-ceph and velero with the same configuration for a successful restore.

Restore

Execute the following command to start the restore:

velero restore create --from-backup backup-demo-app-ns

Verify if Velero restored the application successfully:

$ kubectl -n demo-ns get pods
NAME         READY   STATUS    RESTARTS   AGE
demo-app-0   1/1     Running   0          51s

NOTE: If you are restoring a stateful component of Lokomotive like prometheus-operator, then once pods in monitoring namespace are in Running state, then run lokoctl component apply prometheus-operator to ensure the latest configs are applied.

Verify that the data is restored correctly:

$ kubectl -n demo-ns exec -it demo-app-0 -- /bin/sh -c 'du -s /data'
40960   /data

Additional resources