[Updated on 20-Jun-2020: Many changes in Rook Ceph in previous releases, so revisiting this blog article to accomodate the changes based on a ping in the slack 🙂 ]
In this article we will talk about, how to deploy Ceph ( a software-defined storage) cluster using a Kubernetes operator called ‘rook’.
Before we get into the step by step process on how to deploy and get it working, let me touch upon what is ceph and rook.
Obviously we should know something about Ceph and Rook before we try it 🙂
Ceph
Ceph storage is an open, massively scalable storage solution for modern workloads like cloud infrastructure, data analytics, media repositories, and backup and restore systems. It can: Free you from the expensive lock-in of proprietary, hardware-based storage solutions..
If you want to know more about ceph refer https://ceph.com/
Rook
Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments.
Rook turns storage software into self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities provided by the underlying cloud-native container management, scheduling and orchestration platform to perform its duties.
Rook integrates deeply into cloud-native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience…
etc
Refer# http://rook.io for more details.
Installation of Ceph Cluster in a working Kubernetes cluster has been made very easy by rook! I would say it is just a matter of 2/3 commands if you have a working Kubernetes cluster.
I have a Kubernetes cluster with 1 master node and 2 worker nodes.
Now, it is the turn to install rook and get the Ceph Cluster deployed in this kubernetes cluster.
Let us clone the rook repo first. Considering rook version 1.3 is the latest version available as of today, lets use that branch/release:
[root@node]# git clone https://github.com/rook/rook.git Cloning into ‘rook’… remote: Enumerating objects: 29, done. remote: Counting objects: 100% (29/29), done. remote: Compressing objects: 100% (29/29), done. Receiving objects: 37% (19965/53810), 11.44 MiB | 739.00 KiB/s0 KiB/s 3eceiving objects: 37% (19965/53810), 14.18 MiB | 396.00 KiB/s3.98 MiB | 392.00 KiB/s remote: Total 53810 (delta 6), reused 10 (delta 0), pack-reused 53781 Receiving objects: 100% (53810/53810), 34.29 MiB | 513.00 KiB/s, done. Resolving deltas: 100% (36995/36995), done. [root@node]# cd rook [root@node rook]# git checkout release-1.3 Branch ‘release-1.3’ set up to track remote branch ‘release-1.3’ from ‘origin’. Switched to a new branch ‘release-1.3’
Now the cloning is done, let’s go inside the “rook” directory and fetch contents as shown below:
As you already read, ‘rook’ is a Kubernetes operator that is capable of deploying other software including ceph. In this article we are
interested in Ceph – so let’s deploy ceph using rook operator.
[humble@node kubernetes]# pwd /root/rook/cluster/examples/kubernetes [humble@node kubernetes]#
[root@dhcp53-147 kubernetes]# ls cassandra ceph cockroachdb edgefs mysql.yaml nfs README.md wordpress.yaml yugabytedb
[humble@node]# cd ceph [humble@node]# pwd /root/rook/cluster/examples/kubernetes/ceph [humble@node ceph]# ls ceph-client.yaml dashboard-external-https.yaml nfs.yaml pool-test.yaml cluster-external-management.yaml dashboard-external-http.yaml object-bucket-claim-delete.yaml pool.yaml cluster-external.yaml dashboard-ingress-https.yaml object-bucket-claim-retain.yaml rgw-external.yaml cluster-on-pvc.yaml dashboard-loadbalancer.yaml object-ec.yaml scc.yaml cluster-test.yaml direct-mount.yaml object-openshift.yaml storageclass-bucket-delete.yaml cluster.yaml filesystem-ec.yaml object-test.yaml storageclass-bucket-retain-external.yaml common-external.yaml filesystem-test.yaml object-user.yaml storageclass-bucket-retain.yaml common.yaml filesystem.yaml object.yaml toolbox.yaml create-external-cluster-resources.py flex operator-openshift.yaml upgrade-from-v1.2-apply.yaml create-external-cluster-resources.sh import-external-cluster.sh operator.yaml upgrade-from-v1.2-crds.yaml csi monitoring pool-ec.yaml [humble@node ceph]#
Creation of Rook Operator is pretty easy as shown in below in 3 commands!
[humble@node ceph]# kubectl create -f operator.yaml configmap/rook-ceph-operator-config created deployment.apps/rook-ceph-operator created
[humble@node ceph]# kubectl get pod -n rook-ceph NAME READY STATUS RESTARTS AGE rook-ceph-operator-567d7945d6-2tlkf 1/1 Running 0 6s rook-discover-6m5mj 1/1 Running 0 5s rook-discover-csstp 1/1 Running 0 5s rook-discover-flflh 1/1 Running 0 5s [humble@node ceph]#
Next step is bringing up the Ceph Cluster! We have a sample/default template YAML in the same place and let’s list out the default configuration of the same
apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: ceph/ceph:v14.2.9 allowUnsupported: false dataDirHostPath: /var/lib/rook skipUpgradeChecks: false continueUpgradeAfterChecksEvenIfNotHealthy: false mon: count: 3 allowMultiplePerNode: false mgr: modules: – name: pg_autoscaler enabled: true dashboard: enabled: true ssl: true monitoring: enabled: false rulesNamespace: rook-ceph network: rbdMirroring: workers: 0 crashCollector: disable: false cleanupPolicy: confirmation: “”
annotations: resources: removeOSDsIfOutAndSafeToRemove: false useAllNodes: true useAllDevices: true config: disruptionManagement: managePodBudgets: false osdMaintenanceTimeout: 30 manageMachineDisruptionBudgets: false machineDisruptionBudgetNamespace: openshift-machine-api
Why to wait more? lets go ahead and create a Ceph Cluster.
Your Ceph Cluster is coming up!!! … Watch for new pods and see what’s coming up in your cluster.
NAME READY STATUS RESTARTS AGE csi-cephfsplugin-2qntx 3/3 Running 0 22m csi-cephfsplugin-8k2dx 3/3 Running 0 22m csi-cephfsplugin-provisioner-7469b99d4b-ghrvg 5/5 Running 0 22m csi-cephfsplugin-provisioner-7469b99d4b-tdhgd 5/5 Running 0 22m csi-cephfsplugin-rbjlr 3/3 Running 0 22m csi-rbdplugin-cv7b5 3/3 Running 0 3m28s csi-rbdplugin-ks4hj 3/3 Running 0 3m35s csi-rbdplugin-mpfc9 3/3 Running 0 3m35s csi-rbdplugin-provisioner-865f4d8d-bx888 6/6 Running 0 22m csi-rbdplugin-provisioner-865f4d8d-v846g 6/6 Running 0 22m rook-ceph-crashcollector-dhcp53-136.lab.eng.blr.redhat.comqpfx2 1/1 Running 0 21m rook-ceph-crashcollector-dhcp53-147.lab.eng.blr.redhat.comjqtp9 1/1 Running 0 21m rook-ceph-crashcollector-dhcp53-148.lab.eng.blr.redhat.com728kh 1/1 Running 0 21m rook-ceph-mgr-a-67b8954597-ljl4r 1/1 Running 0 21m rook-ceph-mon-a-6c9d54899d-k2gzh 1/1 Running 0 21m rook-ceph-mon-b-755dc649c7-gvwtg 1/1 Running 0 21m rook-ceph-mon-c-7db9b6fd5c-fd5nj 1/1 Running 0 21m rook-ceph-operator-5b6674cb6-hj9cq 1/1 Running 0 84m rook-ceph-osd-prepare-dhcp53-136.lab.eng.blr.redhat.com-s4dzw 0/1 Completed 0 20m rook-ceph-osd-prepare-dhcp53-147.lab.eng.blr.redhat.com-s6f4w 0/1 Completed 0 20m rook-ceph-osd-prepare-dhcp53-148.lab.eng.blr.redhat.com-lb46h 0/1 Completed 0 20m rook-discover-fzfpx 1/1 Running 0 64m rook-discover-jmcxz 1/1 Running 0 42m rook-discover-zrjt4 1/1 Running 0 84m
Additionally you can deploy ceph toolbox pod.
To list out the services deployed in this cluster:
In the above output you see, the monitor services and its cluster IPs :
Just to mention below are some details about the CephFS and RBD pods and containers deployed here:
CephFS -(csi-cephfsplugin-provisioner-*) : [csi-attacher csi-resizer csi-provisioner csi-cephfsplugin liveness-prometheus]
RBD – (csi-rbdplugin-provisioner-*) : [csi-attacher csi-resizer csi-provisioner csi-rbdplugin liveness-prometheus csi-snapshotter]
CephFS Node plugin pod (csi-rbdplugin-*) : [driver-registrar csi-cephfsplugin liveness-prometheus]
RBD Node plugin pod (csi-cephfsplugin-*) : [driver-registrar csi-rbdplugin liveness-prometheus]
The CephFS & RBD provisioner container logs looks like this:
[humble@node ceph]# kubectl logs csi-rbdplugin-provisioner-865f4d8d-bx888 -c csi-provisioner -n rook-ceph 0620 14:16:46.098838 1 csi-provisioner.go:98] Version: v1.4.0-0-g1d9bad3 I0620 14:16:46.098893 1 csi-provisioner.go:112] Building kube configs for running in cluster… I0620 14:16:46.106968 1 connection.go:151] Connecting to unix:///csi/csi-provisioner.sock W0620 14:16:56.107257 1 connection.go:170] Still connecting to unix:///csi/csi-provisioner.sock W0620 14:17:06.107378 1 connection.go:170] Still connecting to unix:///csi/csi-provisioner.sock I0620 14:17:12.145464 1 connection.go:261] Probing CSI driver for readiness I0620 14:17:12.147946 1 leaderelection.go:241] attempting to acquire leader lease rook-ceph/rook-ceph-rbd-csi-ceph-com… I0620 14:17:12.155246 1 leaderelection.go:251] successfully acquired lease rook-ceph/rook-ceph-rbd-csi-ceph-com I0620 14:17:12.155899 1 controller.go:770] Starting provisioner controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-865f4d8d-bx888_112ea0fe-2030-450b-8bdf-cf906b88300c! I0620 14:17:12.156169 1 volume_store.go:97] Starting save volume queue I0620 14:17:12.256701 1 controller.go:819] Started provisioner controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-865f4d8d-bx888_112ea0fe-2030-450b-8bdf-cf906b88300c!
[humble@node ceph]#
Now you have a working Ceph Cluster, You can create a Storage class and Persistent volume claim (PVC) from this cluster,
Make sure you have a storage class (SC) available to consume:
Once the SC is created, you can create a PVC which points to this SC and create a Claim!! I don’t want to repeat these steps in every blog article, so skipping it for now.1
1 thought on “Deploy a ceph cluster using Rook (rook.io) in kubernetes”
Comments are closed.