ceph-csi

CephFS snapshot, restore and cloning with Ceph CSI driver

by

In our previous article we discussed about the terminology side of Kubernetes volume snapshots and concepts around it. If you haven’t read through it, I would recommend to have a check on this article, may be before getting into this. But, I will leave that upto you.

This article describe about how CephFS snapshots can be created on CephFS volumes provisioned by the Ceph CSI driver. Once you have snapshots available you can also restore these snapshots to “NEW” Persistent Volume Claims!. We also have the ability to CLONE an existing PVC to a new Volumes! This covers a good amount of use cases related to Backup and Restore in many scenarios. The only requirement here is to have Ceph CSI version 3.1.0 running in your setup either directly by deploying Ceph CSI via templates from its project repository or by installing Rook version >= 1.4.1.

I have also captured a demo of this process in below screen cast.

Considering I have documented the process in the screen cast, its better to not duplicate it here. However if you have any queries, suggestions or comments on this, please place a comment here or reach out!

Kubernetes Volume Snapshot , Restore and PVC cloning introduction

by

As you know Snapshot is Point In time copy of a volume. In Kubernetes, a VolumeSnapshot represents a snapshot of a volume on a storage system.
Similar to how API resources like PersistentVolume and PersistentVolumeClaim are used to provision volumes for users and administrators in an kubernetes/OCP cluster, VolumeSnapshot and VolumeSnapshotContent are API resources that are provided to create volume snapshots.

For supporting or working with snapshot, we have mainly 3 API objects or Custom resources and to name those:
We have volumeSnapshot, VolumeSnapshotContent and VolumeSnapshot Class. These objects or workflow is analogues to Persistent Volume Claim (PVC) workflow if you are aware of.

A VolumeSnapshot is a request for snapshot of a volume by a user. This is a user property. It is similar to a PersistentVolumeClaim where user request a storage.

VolumeSnapshotClass is similar to storage class but for Snapshot requests. Its an admin property in an ocp cluster. It allows you to specify different attributes belonging to a VolumeSnapshot. These attributes may differ among snapshots taken from the same volume on the storage system and therefore cannot be expressed by using the same StorageClass of a PersistentVolumeClaim.

There is also volumesnapshotcontent object, A VolumeSnapshotContent is a snapshot taken from a volume in the cluster. It is a resource in the cluster just like a PersistentVolume is a cluster resource. IN dynamic provisioning world an admin/user dont have to worry about this object. That said, its getting created and managed by internal controllers.

In Slide 1, you could map volumesnapshot workflow and objects with persistent volume claim workflow or object.

I think its prety clear and we can see that its analogous to the persistent volume claim workflow. I have listed the yaml representation of volume Snapshot class, volume snapshot and volumesnapshotcontent in the slides so that you get some idea on how it looks like. Just to touch upon, a volumesnapshotClass object representation looks like this. Things to note here is the highlighted part where admin has to define the driver , deletion policy ..etc.
We just went through the volumesnapshotclass, so next is volumesnapshot and its the user input and how a user can request a snapshot from the cluster. If you look at the highlighted part, this is where user request saying, I want to create a snapshot from a PVC or volume called “pvc-test”. The spec.Source has been filled with the parent or source PVC. On right side you can also see volumesnapshotclass object. As mentioned earlier in dynamic provisioning case, an admin or user dont have to create this object rather its the responsibility of the controller. On a bound object you can see it as volumesnapshotclass got reference to the user request via VolumeSnapshotRefernce and also to the parent volume with the volume handle.

Thats mainly on the yamls. While working with snapshots, we should be knowing few things or some rules:

The parent/source PVC should be at bound state.
PVC should not be in use while snapshot is in progress.

There is also PVC protection available to ensure that in-use PersistentVolumeClaim API objects are not removed from the system while a snapshot is being taken from it (as this may result in data loss). Controller add a finalizer to make sure pvc is not getting removed.
Thats about snapshot , now lets think about Restore:


Restore:

While talking about restore, we are talking about restoring the snapshot to a completely NEW volume and we are not talking about In place restore or Roll Back or revert here. The lack of support for inplace restore or rollback has listed as a limitation in upstream kubernetes atm. So, We could expect this support in future Kubernetes releases.

if you look at the Yaml of a request to restore, you could see that, the highlighted part is where you mention you want to restore to a new volume from the volumesnapshot object named ‘new-snapshot-test”. Thats the only difference compared to the general persistent volume claim request.

In short restore input looks like a PVC request or IOW, its a PVC request with Data Source as ‘volumesnapshot’.

Clone:

Clone is about creating a new volume from an existing volume. Clone is defined as a duplicate of an existing Persistent Volume Claim/PVC. So, instead of creating a new Empty volume we are requesting that, we need a volume with prepopulated content from the source PVC. Its a simple interface.
If you look at the yaml representation its looks like a PVC request with a small difference that, we have Datasource field filled with an existing PVC name.

If you go back and compare restore and clone yamls, the only difference is that, the data source is “Snapshot” in case of “restore” and data source is “PVC/peristentvolumeclaim” in clone .
As we found in snapshot case, While working with Clone, we should be knowing that:

The source PVC must be BOUND and not in use
Only same namespace cloning is allowed , that means the source and destination PVC should be same namespace.
only within same storageclass cloning is allowed.

Cloning can only be performed between two volumes that use the same VolumeMode setting. When I say volumemode its nothing but kubernetes allow you to specify the PVC with either Filesystem or Block mode. So cloning is only allowed with same volumemode setting.
Thats about cloning…

So we went through an overview of snapshot, restore and clone features.

Things to Remember when working with these features:

Here i would like seek your attention mainly on below items/bullets.

When it comes to Data Consistency:

No snapshot consistency guarantees beyond any guarantees provided by the storage system (e.g. crash consistency). So its upto the storage vendor atm to provide the guarantee on the data consistency. You could provide more consistency but its upto the responsibility of higher-level APIs/controllers.

Inflight I/Os:
The API server does not enforce the source volume use ( in use or not ) while a user requests a snapshot/clone. Eventhough its mentioned as prerequisite or recommendation its just on the books/docs, no code in controller stop or prevent the request if its in use.

No inplace-restore:

As discussed some time back, at present It does not support reverting an existing volume to an earlier state represented by a snapshot (beta only supports provisioning a new volume from a snapshot).

User is free to delete the parent/source Volume:

Once the snapshot/clone is created, user is free to delete the source PVC/volume. This gives some challenges while we have Copy On Write based snapshots which is the case with many storage backends. Because the COW snapshots got a reference linking to parent volume and to enable the parent volume deletion we have to untangle the snap from the parent volume. Its bit of a work which CSI driver or other components depends on the storage system.

E2E WorkFlow

With that, let me move to the next slide which give some idea about how the snapshot request e2e works and the components involved in the path.

AS you can see in the slides there are 2 additional components which we havent discussed yet. One is the snapshot controller which is deployed with Kube/OCP platform and its main job is to watch for volumeSnapshot object, we also have a sidecar deployed with CSI bundle called csi snapshotter who watch for volumeSnapshotContent object.
The other primitives like volumesnapshotclass , volumesnapshot, and volumesnapshotcontent we already discussed in previous slides.
The volumesnapshotClass is available in the cluster and user can make a request for snapshot via volumesnapshot object, which is getting monitored by the snapshot controller and it create or populate the volumesnapshotcontent object which in turn monitored by the the csi-snapshotter sidecar. Once the CSI snapshotter sidecar container see there is a volumesnapshotContent object it talks to the CSI driver ( Example Ceph CSI driver) through the CSI endpoint and now its the the turn of CSI driver to talk to the backend or storage cluster and create a snapshot.

Just to repeat, we have 2 additional pods with controllers called

snapshotcontroller and csi snapshotter.

I hope this workflow is clear and helps to understand how the request originated from the user and land with snapshot creation in the backend.

Please let me know if you got any questions.

References:

Volume Snapshots:
https://kubernetes.io/docs/concepts/storage/volume-snapshots/

Restore from snapshots:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#volume-snapshot-and-restore-volume-from-snapshot-support

CSI Volume Cloning
https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/

Expand/Resize Ceph CSI Provisioned Volumes ( CephFS and RBD)

by

Its been a while this feature is supported in Ceph CSI driver and many folks are making use of it. This is a great feature and a useful one in many situations. For example think about a situation where we created a PVC and attached to an application pod, be it a database pod, monitoring pod, or any application pod which eventually consumed the entire space of the attached PVC.

Deleting data from this volume is NOT an option at all at times. If deletion is not an option, what other option we have? migrating the entire data to a new bigger PVC and attach to the same POD/workload? repeat this whenever we are going out of space ? It also need bringing down the workload or pod to attach a new PVC. Also planning how much size the application is going to consume upfront may not work always. But expansion feature is to the rescue! Without disturbing the workload or attached PVC /share/volume, just expand the PVC by a command and thats it! You get new volume size reflected in your POD without any delay and that too when POD is consuming the storage and writing data to it!! Awesome, Isn’t it ? thats what Ceph Volumes provisioned by Ceph CSI driver is capable of !

Before we start on this: I would like to mention about some prerequisites here.

1) You should have the csi-resizer sidecar running in your setup. ( This would be default when you deploy it via Rook or using ceph-csi templates)
2) The Storage class which you use to provision should have few parameters set in it: ( This would be also available if you deploy it with Rook or Ceph CSI templates)

For example:

AllowVolumeExpansion: True
csi.storage.k8s.io/controller-expand-secret-name=rook-csi-cephfs-provisioner
csi.storage.k8s.io/controlnd-secret-namespace=rook-ceph

Why to write and explain everything in this article ? rather let me show you the demo of Volume Expansion feature in Ceph CSI: We have two screencasts one for CephFS and one for RBD.

CephFS volume expansion Demo:

RBD volume expansion Demo:

To summarize, both CephFS and RBD volumes can be expanded online and Ceph CSI drivers are capable of doing it!

At times, due to some reason if the resize fails:

Recovering from Failure when Expanding Volumes

If expanding underlying storage fails, the cluster administrator can manually recover the Persistent Volume Claim (PVC) state and cancel the resize requests. Otherwise, the resize requests are continuously retried by the controller without administrator intervention.

Mark the PersistentVolume(PV) that is bound to the PersistentVolumeClaim(PVC) with Retain reclaim policy.
Delete the PVC. Since PV has Retain reclaim policy – we will not loose any data when we recreate the PVC.
Delete the claimRef entry from PV specs, so as new PVC can bind to it. This should make the PV Available.
Re-create the PVC with smaller size than PV and set volumeName field of the PVC to the name of the PV. This should bind new PVC to existing PV.
Don’t forget to restore the reclaim policy of the PV

I am preparing a demo of RBD volume expansion for the next article on this space, so watch out!

ceph csi v3.0.0 released-(Snapshot, Clone, Multi arch, ROX…)

by

We are excited to announce the third major release of ceph-csi, v3.0.0 !!
The Ceph-CSI team is excited that it has reached the next milestone with the release of v3.0.0! [1]. This release is not limited to features – many critical bug fixes, documentation updates are also part of this release. This is another great release with many improvements for Ceph and CSI integration to use in production with Kubernetes/openshift clusters. Of all the many features and bug fixes here are just a few of the highlights.


New Features:

Create/Delete snapshot for RBD
Create PVC from RBD snapshot
Create PVC from RBD PVC
Add support for multiple CephFS subvolume groups
Multi Architecture docker images(amd64 and arm64)
Support ROX(ReadOnlyMany) PVC for RBD
Support ROX(ReadOnlyMany) PVC for CephFS

Enhancement:

Move to go-ceph binding from RBD CLI
Move to go-ceph binding from RADOS CLI
Add Upgrade E2E testing from 2.1.2 to 3.0.0
Update Sidecars to the latest version
Improve locking to create a parallel clone and snapshot restore
Simplify Error Handling
Update golangci-lint version in CI
Update gosec version in CI
Add support to track cephfs PVC and subvolumes
Introduce build.env for configuration of the environment variables
Update go-ceph to v0.4.0
Update E2E testing to test with latest kubernetes versions
Split out CephFS and RBD E2E tests
Integration with Centos CI to run containerized builds
Update Rook to 1.2.7 for E2E testing
Disable reflink when creating xfs filesystem for RBD
Replace klog with klog v2
Reduce RBAC for kubernetes sidecar containers
Add option to compile e2e tests in containerized
Add commitlint bot in CI
Add Stale bot to the repo
Add E2E and documentation for CephFS PVC
Update kubernetes dependency to v1.18.6

Bug Fix:

Fix issue in CephFS Volume Not found

Breaking Changes

Remove support for v1.x.x PVC
Remove support for Mimic
Snapshot Alpha is no longer supported

Lets touch upon some more of the very cool features introduced in this release:

“Snapshot and Clone functionality made available with RBD”:

Since v1.0.0 release of Ceph CSI we had RBD snapshot in place but we marked it as “Alpha” for few reasons. One of them being the snapshot support in Kubernetes upstream CSI driver was also evolving and stabilizing on the API side of things. Its not only that, we had an issue of image lockup when we try to consume or untangle parent source volume from the snapshot volume/object. So we did a revamp in this version and we are happy to say that, we have it solved and with this release . From now on, RBD snapshot should work smooth.

Its capable of:

*) Creating a snapshot from a RBD volume
*) Restoring to a new volume from the existing snapshot
*) Deletion of snapshot and parent volume objects independently.

We also enabled a cool functionality here with the Volume Cloning feature. Thats nothing but the capability of provisioning a volume from another or existing PVC source.

Dont confuse this with “Restore of a snapshot”

The main difference here is that, While you restore from a snapshot, for the new PVC, the referred Datasource is “Snapshot” and if you do clone operation, the “DataSource” is an existing PVC.


Multiple subvolumegroup support

When CephFS CSI driver create a volume, we default to a subvolumegroup called “csi” , We place our subvolumes in this group. I mean the backend CephFS volumes which map to the PVC in openshift or kubernetes namespace. With this release, you are allowed to specify multiple subvolume groups! This comes handy in some setups where you could have a segregation of the subvolumes for various purposes!

ROX support for both RBD and CephFS

ROX is nothing but an access mode which you could define while requesting a PVC from Kubernetes/Openshift. What that means to an end user is that, the workload get a READONLY share. In a plain dynamic provisioning workflow it does not make much sense. The reasoning is that, you are requesting a volume of size “X” and the backend driver provision it from the storage cluster, but its an “empty” volume and if you attach this to a workload as “READONLY” , in nutshell its an empty volume and you cant write the data to it!! So the question was “Whats the use of ROX volumes” ?

BUT, with the snap and clone use case there is a big use case behind it. That said, think about a scenario where you have a “VM template” which you want to consume it as READONLY image! For such use case ROX support add much value! and here we are with that support!

Mutli arch image support for Ceph CSI

Our community users want to use multi arch images for amd64 and arm64 ..etc. It was indeed available for last few versions, however the manifest files were not carrying this properly and it is corrected in this release and images are available in quay.io.

Updated sidecars & Kuberenetes dependency lift to 1.18.6

We have upgraded the CSI community provided sidecars to latest versions and also brought the Kubernetes dependency chain to 1.18.6 version. This itself bring many bug fixes, improvements, features..etc! I dont want to list them here as its really huge!

Performance improvements – Especially on go ceph bindings

We kept on improving the performance of CSI driver and especially its connection with backend cluster by making use of latest go-ceph version ( v4.0). We have seen great improvement in the backend connection workflow and overall in the life cycle of volume management.. So worth a mention here!

Code Cleanup, Better E2E, what not?

Great amount of code cleanup, E2E improvement , Documentation update …etc are part of this release!

…so on

The container image is tagged with “v3.0.0” and its downloadable by #docker pull ..

Kudos to the Ceph CSI community for all the hard work to reach this critical milestone!
The Ceph-CSI project ( https://github.com/ceph/ceph-csi/), as well as its thriving community, has continued to grow and we are happy to share that, this is our 11th release since Jul 12, 2019!!

We are not stopping here, rather marching towards v3.1.0 (https://github.com/ceph/ceph-csi/issues/1272) with some more feature enhancement as tracked in the release issue.
One of the very important feature we are targeting with v3.1.0 release is that, CephFS snapshot and Clone functionality, watch this space for more update!

Reach us at slack (https://cephcsi.slack.com) or at github: https://github.com/ceph/ceph-csi/

Happy Hacking!

[1]
Release Issue: https://github.com/ceph/ceph-csi/issues/865
ceph-csi v3.0.0 tag: https://github.com/ceph/ceph-csi/releases/tag/v3.0.0,
Release Images: https://quay.io/repository/cephcsi/cephcsi?tab=tags

—

Failed to initialize CSINode: error updating CSINode annotation: timed out

by
May 06 21:41:42 node1.example.com systemd[26469]: I0506 21:41:42.386428 26469 nodeinfomanager.go:402] Failed to publish CSINode: the server could not find the requested resource May 06 21:41:42 node1.example.com systemd[26469]: E0506 21:41:42.386469 26469 csi_plugin.go:273] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: the server could not find the requested resource May 06 21:41:42 node1.example.com systemd[26469]: F0506 21:41:42.386474 26469 csi_plugin.go:287] Failed to initialize CSINode after retrying: timed out waiting for the condition

or the CSI plugin pods are crashing with an error similar to below and they are in CrashLoopBackState:

I0519 03:09:55.089594 1 main.go:110] Version: v1.2.0-0-g6ef000ae I0519 03:09:55.089644 1 connection.go:151] Connecting to unix:///csi/csi.sock I0519 03:09:55.090799 1 node_register.go:58] Starting Registration Server at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock I0519 03:09:55.091050 1 node_register.go:67] Registration Server started at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock I0519 03:09:55.937613 1 main.go:77] Received GetInfo call: &InfoRequest{} I0519 03:09:56.937580 1 main.go:77] Received GetInfo call: &InfoRequest{} I0519 03:09:58.343490 1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource,} E0519 03:09:58.343558 1 main.go:89] Registration process failed with error: RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource, restarting registration container.

Also:

csi-cephfsplugin-q6vk9 2/3 CrashLoopBackOff 26 8 15h csi-cephfsplugin-yy9bv 2/3 CrashLoopBackOff 26 8 15h csi-rbdplugin-88wn4 2/3 CrashLoopBackOff 199 15h csi-rbdplugin-8j8q9 2/3 CrashLoopBackOff 264 15h csi-rbdplugin-gz9t9 2/3 CrashLoopBackOff 26 8 15h

Whats happening here? The CSI node-driver-registrar is a sidecar container that fetches driver information (using NodeGetInfo) from a CSI endpoint and registers it with the kubelet on that node using the kubelet plugin registration mechanism.From there on kubelet directly talk to the CSI driver through the registration socket for driver calls like NodeGetInfo, NodeStageVolume, NodePublishVolume ..etc. After registration, there is an object called CSINOde which is managed by kubelet: While the plugin comes up the registration failed for some reason.

apiVersion: storage.k8s.io/v1 kind: CSINode metadata: name: node1 spec: drivers: – name: mycsidriver.example.com nodeID: storageNodeID1 topologyKeys: [‘mycsidriver.example.com/regions’, “mycsidriver.example.com/zones”]

Few troubleshooting tips I can share here:

Whats your Kubernetes client and server versions?

Are they running on the same version ( say both are 1.17 or 1.18 ? ) ? if not, can you make sure both are the same or at least they are under-supported version skew? This is mostly popped up after an upgrade of one component to Kubernetes v1.17.

Ceph CSI: “XFS Superblock has unknown read-only..” Or ” wrong fs type, bad …. on /dev/rbd., missing codepage or ..”

by

We announced ceph csi v2.1.0 release around 3 weeks back with many bug fixes and features. More details about this release can be found here https://github.com/ceph/ceph-csi/releases/tag/v2.1.0 and in this blog https://www.humblec.com/one-more-ceph-csi-release-yeah-v2-1-0-is-here/. From the CSI/Rook community interactions we know many folks have updated to this version in the last couple of weeks.

However, unfortunately, the community found a couple of issues with their app pod PVC mounting (XFS based) with this CSI version in their Kubernetes/openshift cluster.

There are mainly 2 errors/issues encountered in setups:

1) XFS: wrong fs type, bad option, bad superblock on /dev/rbd4, missing codepage or helper program, or other error
2) XFS: Superblock has unknown read-only compatible features (0x4) enabled

As you can see above, both of these issues are on XFS mounting. Ceph CSI plugin make use of mount.xfs binary at time of volume mounting and its part of “xfsprogs” package shipped by the CSI container. To understand this issue better, lets first see the change:

ceph v2.1.0 release: mkfs.xfs version 4.5.0

ceph v2.1.0 release: mkfs.xfs version 5.0.0

This change happened when we updated the Ceph base image in our containers from v14.2 to v15 in ceph CSI version 2.1.0 compared to 2.0.0

ceph v2.1.0 BASE_IMAGE=ceph/ceph:v14.2

ceph v2.1.0 BASE_IMAGE=ceph/ceph:v15

To give some more context about these issues:

The first one is more generic:

XFS: wrong fs type, bad option, bad superblock on /dev/rbd4, missing codepage or helper program, or other error

That said, this is seen when you have multiple PVCs with the same XFS UUID ( filesystem UUID) and the PVC mount fails when its attached to a pod.

The reason to have multiple PVCs with the same UUID arises when you have a snapshotted/cloned volume. Even though ceph CSI snapshot/clone support is still in Alpha state we have proactively fixed this issue [1].

The second issue:

XFS: Superblock has unknown read-only compatible features (0x4) enabled

pop up when the cluster nodes are running RHEL 7 based kernels ( for ex: redhat kernel 3.10.0-957 el7). The mkfs.xfs binary in the new CSI container image is not really compatible with this kernel version and you will encounter ‘unknown read-only compatible features (0x4) read-only error at the time of mounting’. To avoid this error the mkfs.xfs call would need to include -m reflink=0 which disable the incompatible copy-on-write reflink support.

More details about these issues can be found at https://github.com/ceph/ceph-csi/issues/966

[1] Solution:

Both of these issues have been addressed with ceph CSI v2.1.1 which got released today https://github.com/ceph/ceph-csi/releases/tag/v2.1.1. The immediate fix what we made with this new release is, reverting to older mkfs.xfs version. The real fix is known and we will make it soon. But, till then we don’t want to leave our awesome community with broken setup.

We advise you to update to v2.1.1 if you are using xfs as your PVC fsformat in your kubernetes/openshift cluster.

Special thanks to below community users for reporting this issue and helping us with the various stages of this debugging process!

https://github.com/chandr20
https://github.com/iExalt
https://github.com/fiveclubs
https://github.com/volvicoasis

Happy hacking and talk to us via slack ( http://cephcsi.slack.com) or github https://github.com/ceph/ceph-csi/.