Kubernetes storage with Rook Ceph on Packet cloud

Introduction

Ceph is a distributed storage provider which provides block, object and filesystem storage.

Rook is an aggregator of various storage providers via a common Kubernetes interface. Ceph is a first class citizen in Rook. Lokomotive, at the moment, supports Rook Ceph only.

This guide provides the steps for deploying a storage stack using the rook and rook-ceph Lokomotive component.

Learning objectives

At the end of this tutorial you will have a storage backed by rook and rook-ceph. Any application in the cluster can request storage using Kubernetes PVC. You will also learn how to access the Ceph dashboard, create a rook toolbox and access Ceph via command line and finally make storage provider rook ceph as default.

Prerequisites

  • A Lokomotive cluster deployed on a Packet cloud and accessible via kubectl.

Steps

Step 1: Deploy storage worker pool

Config

Deploy a cluster with at least one worker pool dedicated to rook-ceph. A dedicated worker pool configuration should look like the following:

cluster "packet" {
  ...

  worker_pool "storage" {
    count = 3
    node_type = "c2.medium.x86"

    labels = "storage.lokomotive.io=ceph"
    taints = "storage.lokomotive.io=ceph:NoSchedule"
  }
}
  • The number of machines provided using count should be an odd number greater than or equal to three.
  • Type of node, provided using node_type, should be one that has multiple disks like c2.medium.x86 or s1.large.x86. Find out more servers here .
  • To steer rook-ceph workload on these storage nodes provide labels.
  • Provide taints so that other workload can be steered away by default. This setting is not mandatory, but isolating storage workloads from others is recommended so that other workloads do not degrade the performance of the storage.

Deploy the worker pool

Execute the following command to deploy the storage worker pool:

lokoctl cluster apply -v --skip-components

Step 2: Deploy rook

Config

Create a file named storage.lokocfg with the following contents:

component "rook" {
  node_selector = {
    "storage.lokomotive.io" = "ceph"
  }

  toleration {
    key      = "storage.lokomotive.io"
    operator = "Equal"
    value    = "ceph"
    effect   = "NoSchedule"
  }

  agent_toleration_key    = "storage.lokomotive.io"
  agent_toleration_effect = "NoSchedule"

  discover_toleration_key    = "storage.lokomotive.io"
  discover_toleration_effect = "NoSchedule"
}
  • node_selector should match the labels attribute provided in the worker_pool.
  • toleration should match the taints attribute mentioned in the worker_pool.
  • agent_toleration_key and discover_toleration_key should match the key of the taints attribute provided in the worker_pool.
  • agent_toleration_effect and discover_toleration_effect should match the effect of the taints attribute provided in the worker_pool.

For more information on available configuration options for the rook component, visit the component’s configuration reference .

Deploy the component

Execute the following command to deploy the rook component:

lokoctl component apply rook

Verify the operator pod in the rook namespace is in the Running state (this may take a few minutes):

$ kubectl -n rook get pods -l app=rook-ceph-operator
NAME                                  READY   STATUS    RESTARTS   AGE
rook-ceph-operator-76d8687f95-6knf8   1/1     Running   0          2m

Step 3: Deploy rook-ceph

Config

Add following contents to the previously created file storage.lokocfg:

component "rook-ceph" {
  monitor_count = 3

  node_affinity {
    key      = "storage.lokomotive.io"
    operator = "Exists"
  }

  toleration {
    key      = "storage.lokomotive.io"
    operator = "Equal"
    value    = "ceph"
    effect   = "NoSchedule"
  }

  storage_class {
    enable = true
  }
}
  • monitor_count should be an odd number greater than three and not higher than the count attribute of workers in the worker_pool.
  • node_affinity should match the labels attribute provided in the worker_pool.
  • toleration should match the taints attribute provided in the worker_pool.

For more information on available configuration options for the rook-ceph component, visit the component’s configuration reference .

Deploy the component

Execute the following command to deploy the rook-ceph component:

lokoctl component apply rook-ceph

Verify the OSD pods in the rook namespace are in the Running state (this may take a few minutes):

$ kubectl -n rook get pods -l app=rook-ceph-osd
NAME                               READY   STATUS    RESTARTS   AGE
rook-ceph-osd-0-6d4f69dbf9-26kzl   1/1     Running   0          15m
rook-ceph-osd-1-86c9597b84-lmh94   1/1     Running   0          15m
rook-ceph-osd-2-6d97697897-7bprl   1/1     Running   0          15m
rook-ceph-osd-3-5bfb9d86b-rk6v4    1/1     Running   0          15m
rook-ceph-osd-4-5b76cb9675-cxkdw   1/1     Running   0          15m
rook-ceph-osd-5-8c86f5c6c-6qxtz    1/1     Running   0          15m
rook-ceph-osd-6-5b9cc479b7-vjc9v   1/1     Running   0          15m
rook-ceph-osd-7-7b84d6cc48-b46z9   1/1     Running   0          15m
rook-ceph-osd-8-5868969f97-2bn9r   1/1     Running   0          15m

Access the Ceph dashboard

Ceph dashboard provides valuable visual information. It is an essential tool to monitor the Ceph cluster. Here are the steps on how to access it.

Obtain the password for the admin Ceph user by running the following command:

kubectl -n rook get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

Execute the following command to forward port 8443 locally to the Ceph manager pod:

kubectl -n rook port-forward svc/rook-ceph-mgr-dashboard 8443

Now open the following URL: https://localhost:8443 and enter the username admin and the password obtained from the first step.

Enable and access toolbox

Ceph is a complex software system, and not everything that happens in the Ceph cluster is visible at the rook layer of abstraction. So the command-line interface to interact with Ceph cluster is useful to extract such hidden events and information. Ceph toolbox helps you access the ceph cluster using ceph CLI utility. Using the utility you can configure the Ceph cluster setting and debug the cluster.

To deploy the toolbox, the rook-ceph component config should set the attribute enable_toolbox to true.

component "rook-ceph" {
  enable_toolbox = true
  ...
}

Execute the following command to apply the changes:

lokoctl component apply rook-ceph

Verify the toolbox pod in the rook namespace is in the Running state (this may take a few minutes):

$ kubectl -n rook get deploy rook-ceph-tools
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
rook-ceph-tools   1/1     1            1           39s

Execute the following command to access the toolbox pod:

kubectl -n rook exec -it $(kubectl -n rook get pods -l app=rook-ceph-tools -o name) -- bash

Once inside the pod you can run usual ceph commands:

ceph status
ceph osd status
ceph df
rados df

Enable monitoring

Monitor rook and rook-ceph components using the prometheus-operator component. To enable your rook component config should have the attribute enable_monitoring set to true.

NOTE: Deploy the prometheus-operator component before. For more information follow this doc .

component "rook" {
  enable_monitoring = true
  ...
}

Execute the following command to apply the changes:

lokoctl component apply rook

Make default storage class

It is recommended to make the storage class as default if rook-ceph is the only storage provider in the cluster. This setting helps to provision volumes for the PVCs created by workloads. The rook-ceph component config should look like the following:

component "rook-ceph" {
  ...

  storage_class {
    enable  = true
    default = true
  }
}

Execute the following command to apply the changes:

lokoctl component apply rook-ceph

Verify the StorageClass is default:

$ kubectl get sc rook-ceph-block
NAME                        PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block (default)   rook.rbd.csi.ceph.com   Delete          Immediate           true                   8m17s

Additional resources