ceph

CephFS snapshot, restore and cloning with Ceph CSI driver

by

In our previous article we discussed about the terminology side of Kubernetes volume snapshots and concepts around it. If you haven’t read through it, I would recommend to have a check on this article, may be before getting into this. But, I will leave that upto you.

This article describe about how CephFS snapshots can be created on CephFS volumes provisioned by the Ceph CSI driver. Once you have snapshots available you can also restore these snapshots to “NEW” Persistent Volume Claims!. We also have the ability to CLONE an existing PVC to a new Volumes! This covers a good amount of use cases related to Backup and Restore in many scenarios. The only requirement here is to have Ceph CSI version 3.1.0 running in your setup either directly by deploying Ceph CSI via templates from its project repository or by installing Rook version >= 1.4.1.

I have also captured a demo of this process in below screen cast.

Considering I have documented the process in the screen cast, its better to not duplicate it here. However if you have any queries, suggestions or comments on this, please place a comment here or reach out!

Kubernetes Volume Snapshot , Restore and PVC cloning introduction

by

As you know Snapshot is Point In time copy of a volume. In Kubernetes, a VolumeSnapshot represents a snapshot of a volume on a storage system.
Similar to how API resources like PersistentVolume and PersistentVolumeClaim are used to provision volumes for users and administrators in an kubernetes/OCP cluster, VolumeSnapshot and VolumeSnapshotContent are API resources that are provided to create volume snapshots.

For supporting or working with snapshot, we have mainly 3 API objects or Custom resources and to name those:
We have volumeSnapshot, VolumeSnapshotContent and VolumeSnapshot Class. These objects or workflow is analogues to Persistent Volume Claim (PVC) workflow if you are aware of.

A VolumeSnapshot is a request for snapshot of a volume by a user. This is a user property. It is similar to a PersistentVolumeClaim where user request a storage.

VolumeSnapshotClass is similar to storage class but for Snapshot requests. Its an admin property in an ocp cluster. It allows you to specify different attributes belonging to a VolumeSnapshot. These attributes may differ among snapshots taken from the same volume on the storage system and therefore cannot be expressed by using the same StorageClass of a PersistentVolumeClaim.

There is also volumesnapshotcontent object, A VolumeSnapshotContent is a snapshot taken from a volume in the cluster. It is a resource in the cluster just like a PersistentVolume is a cluster resource. IN dynamic provisioning world an admin/user dont have to worry about this object. That said, its getting created and managed by internal controllers.

In Slide 1, you could map volumesnapshot workflow and objects with persistent volume claim workflow or object.

I think its prety clear and we can see that its analogous to the persistent volume claim workflow. I have listed the yaml representation of volume Snapshot class, volume snapshot and volumesnapshotcontent in the slides so that you get some idea on how it looks like. Just to touch upon, a volumesnapshotClass object representation looks like this. Things to note here is the highlighted part where admin has to define the driver , deletion policy ..etc.
We just went through the volumesnapshotclass, so next is volumesnapshot and its the user input and how a user can request a snapshot from the cluster. If you look at the highlighted part, this is where user request saying, I want to create a snapshot from a PVC or volume called “pvc-test”. The spec.Source has been filled with the parent or source PVC. On right side you can also see volumesnapshotclass object. As mentioned earlier in dynamic provisioning case, an admin or user dont have to create this object rather its the responsibility of the controller. On a bound object you can see it as volumesnapshotclass got reference to the user request via VolumeSnapshotRefernce and also to the parent volume with the volume handle.

Thats mainly on the yamls. While working with snapshots, we should be knowing few things or some rules:

The parent/source PVC should be at bound state.
PVC should not be in use while snapshot is in progress.

There is also PVC protection available to ensure that in-use PersistentVolumeClaim API objects are not removed from the system while a snapshot is being taken from it (as this may result in data loss). Controller add a finalizer to make sure pvc is not getting removed.
Thats about snapshot , now lets think about Restore:


Restore:

While talking about restore, we are talking about restoring the snapshot to a completely NEW volume and we are not talking about In place restore or Roll Back or revert here. The lack of support for inplace restore or rollback has listed as a limitation in upstream kubernetes atm. So, We could expect this support in future Kubernetes releases.

if you look at the Yaml of a request to restore, you could see that, the highlighted part is where you mention you want to restore to a new volume from the volumesnapshot object named ‘new-snapshot-test”. Thats the only difference compared to the general persistent volume claim request.

In short restore input looks like a PVC request or IOW, its a PVC request with Data Source as ‘volumesnapshot’.

Clone:

Clone is about creating a new volume from an existing volume. Clone is defined as a duplicate of an existing Persistent Volume Claim/PVC. So, instead of creating a new Empty volume we are requesting that, we need a volume with prepopulated content from the source PVC. Its a simple interface.
If you look at the yaml representation its looks like a PVC request with a small difference that, we have Datasource field filled with an existing PVC name.

If you go back and compare restore and clone yamls, the only difference is that, the data source is “Snapshot” in case of “restore” and data source is “PVC/peristentvolumeclaim” in clone .
As we found in snapshot case, While working with Clone, we should be knowing that:

The source PVC must be BOUND and not in use
Only same namespace cloning is allowed , that means the source and destination PVC should be same namespace.
only within same storageclass cloning is allowed.

Cloning can only be performed between two volumes that use the same VolumeMode setting. When I say volumemode its nothing but kubernetes allow you to specify the PVC with either Filesystem or Block mode. So cloning is only allowed with same volumemode setting.
Thats about cloning…

So we went through an overview of snapshot, restore and clone features.

Things to Remember when working with these features:

Here i would like seek your attention mainly on below items/bullets.

When it comes to Data Consistency:

No snapshot consistency guarantees beyond any guarantees provided by the storage system (e.g. crash consistency). So its upto the storage vendor atm to provide the guarantee on the data consistency. You could provide more consistency but its upto the responsibility of higher-level APIs/controllers.

Inflight I/Os:
The API server does not enforce the source volume use ( in use or not ) while a user requests a snapshot/clone. Eventhough its mentioned as prerequisite or recommendation its just on the books/docs, no code in controller stop or prevent the request if its in use.

No inplace-restore:

As discussed some time back, at present It does not support reverting an existing volume to an earlier state represented by a snapshot (beta only supports provisioning a new volume from a snapshot).

User is free to delete the parent/source Volume:

Once the snapshot/clone is created, user is free to delete the source PVC/volume. This gives some challenges while we have Copy On Write based snapshots which is the case with many storage backends. Because the COW snapshots got a reference linking to parent volume and to enable the parent volume deletion we have to untangle the snap from the parent volume. Its bit of a work which CSI driver or other components depends on the storage system.

E2E WorkFlow

With that, let me move to the next slide which give some idea about how the snapshot request e2e works and the components involved in the path.

AS you can see in the slides there are 2 additional components which we havent discussed yet. One is the snapshot controller which is deployed with Kube/OCP platform and its main job is to watch for volumeSnapshot object, we also have a sidecar deployed with CSI bundle called csi snapshotter who watch for volumeSnapshotContent object.
The other primitives like volumesnapshotclass , volumesnapshot, and volumesnapshotcontent we already discussed in previous slides.
The volumesnapshotClass is available in the cluster and user can make a request for snapshot via volumesnapshot object, which is getting monitored by the snapshot controller and it create or populate the volumesnapshotcontent object which in turn monitored by the the csi-snapshotter sidecar. Once the CSI snapshotter sidecar container see there is a volumesnapshotContent object it talks to the CSI driver ( Example Ceph CSI driver) through the CSI endpoint and now its the the turn of CSI driver to talk to the backend or storage cluster and create a snapshot.

Just to repeat, we have 2 additional pods with controllers called

snapshotcontroller and csi snapshotter.

I hope this workflow is clear and helps to understand how the request originated from the user and land with snapshot creation in the backend.

Please let me know if you got any questions.

References:

Volume Snapshots:
https://kubernetes.io/docs/concepts/storage/volume-snapshots/

Restore from snapshots:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#volume-snapshot-and-restore-volume-from-snapshot-support

CSI Volume Cloning
https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/

Expand/Resize Ceph CSI Provisioned Volumes ( CephFS and RBD)

by

Its been a while this feature is supported in Ceph CSI driver and many folks are making use of it. This is a great feature and a useful one in many situations. For example think about a situation where we created a PVC and attached to an application pod, be it a database pod, monitoring pod, or any application pod which eventually consumed the entire space of the attached PVC.

Deleting data from this volume is NOT an option at all at times. If deletion is not an option, what other option we have? migrating the entire data to a new bigger PVC and attach to the same POD/workload? repeat this whenever we are going out of space ? It also need bringing down the workload or pod to attach a new PVC. Also planning how much size the application is going to consume upfront may not work always. But expansion feature is to the rescue! Without disturbing the workload or attached PVC /share/volume, just expand the PVC by a command and thats it! You get new volume size reflected in your POD without any delay and that too when POD is consuming the storage and writing data to it!! Awesome, Isn’t it ? thats what Ceph Volumes provisioned by Ceph CSI driver is capable of !

Before we start on this: I would like to mention about some prerequisites here.

1) You should have the csi-resizer sidecar running in your setup. ( This would be default when you deploy it via Rook or using ceph-csi templates)
2) The Storage class which you use to provision should have few parameters set in it: ( This would be also available if you deploy it with Rook or Ceph CSI templates)

For example:

AllowVolumeExpansion: True
csi.storage.k8s.io/controller-expand-secret-name=rook-csi-cephfs-provisioner
csi.storage.k8s.io/controlnd-secret-namespace=rook-ceph

Why to write and explain everything in this article ? rather let me show you the demo of Volume Expansion feature in Ceph CSI: We have two screencasts one for CephFS and one for RBD.

CephFS volume expansion Demo:

RBD volume expansion Demo:

To summarize, both CephFS and RBD volumes can be expanded online and Ceph CSI drivers are capable of doing it!

At times, due to some reason if the resize fails:

Recovering from Failure when Expanding Volumes

If expanding underlying storage fails, the cluster administrator can manually recover the Persistent Volume Claim (PVC) state and cancel the resize requests. Otherwise, the resize requests are continuously retried by the controller without administrator intervention.

Mark the PersistentVolume(PV) that is bound to the PersistentVolumeClaim(PVC) with Retain reclaim policy.
Delete the PVC. Since PV has Retain reclaim policy – we will not loose any data when we recreate the PVC.
Delete the claimRef entry from PV specs, so as new PVC can bind to it. This should make the PV Available.
Re-create the PVC with smaller size than PV and set volumeName field of the PVC to the name of the PV. This should bind new PVC to existing PV.
Don’t forget to restore the reclaim policy of the PV

I am preparing a demo of RBD volume expansion for the next article on this space, so watch out!

ceph csi v3.0.0 released-(Snapshot, Clone, Multi arch, ROX…)

by

We are excited to announce the third major release of ceph-csi, v3.0.0 !!
The Ceph-CSI team is excited that it has reached the next milestone with the release of v3.0.0! [1]. This release is not limited to features – many critical bug fixes, documentation updates are also part of this release. This is another great release with many improvements for Ceph and CSI integration to use in production with Kubernetes/openshift clusters. Of all the many features and bug fixes here are just a few of the highlights.


New Features:

Create/Delete snapshot for RBD
Create PVC from RBD snapshot
Create PVC from RBD PVC
Add support for multiple CephFS subvolume groups
Multi Architecture docker images(amd64 and arm64)
Support ROX(ReadOnlyMany) PVC for RBD
Support ROX(ReadOnlyMany) PVC for CephFS

Enhancement:

Move to go-ceph binding from RBD CLI
Move to go-ceph binding from RADOS CLI
Add Upgrade E2E testing from 2.1.2 to 3.0.0
Update Sidecars to the latest version
Improve locking to create a parallel clone and snapshot restore
Simplify Error Handling
Update golangci-lint version in CI
Update gosec version in CI
Add support to track cephfs PVC and subvolumes
Introduce build.env for configuration of the environment variables
Update go-ceph to v0.4.0
Update E2E testing to test with latest kubernetes versions
Split out CephFS and RBD E2E tests
Integration with Centos CI to run containerized builds
Update Rook to 1.2.7 for E2E testing
Disable reflink when creating xfs filesystem for RBD
Replace klog with klog v2
Reduce RBAC for kubernetes sidecar containers
Add option to compile e2e tests in containerized
Add commitlint bot in CI
Add Stale bot to the repo
Add E2E and documentation for CephFS PVC
Update kubernetes dependency to v1.18.6

Bug Fix:

Fix issue in CephFS Volume Not found

Breaking Changes

Remove support for v1.x.x PVC
Remove support for Mimic
Snapshot Alpha is no longer supported

Lets touch upon some more of the very cool features introduced in this release:

“Snapshot and Clone functionality made available with RBD”:

Since v1.0.0 release of Ceph CSI we had RBD snapshot in place but we marked it as “Alpha” for few reasons. One of them being the snapshot support in Kubernetes upstream CSI driver was also evolving and stabilizing on the API side of things. Its not only that, we had an issue of image lockup when we try to consume or untangle parent source volume from the snapshot volume/object. So we did a revamp in this version and we are happy to say that, we have it solved and with this release . From now on, RBD snapshot should work smooth.

Its capable of:

*) Creating a snapshot from a RBD volume
*) Restoring to a new volume from the existing snapshot
*) Deletion of snapshot and parent volume objects independently.

We also enabled a cool functionality here with the Volume Cloning feature. Thats nothing but the capability of provisioning a volume from another or existing PVC source.

Dont confuse this with “Restore of a snapshot”

The main difference here is that, While you restore from a snapshot, for the new PVC, the referred Datasource is “Snapshot” and if you do clone operation, the “DataSource” is an existing PVC.


Multiple subvolumegroup support

When CephFS CSI driver create a volume, we default to a subvolumegroup called “csi” , We place our subvolumes in this group. I mean the backend CephFS volumes which map to the PVC in openshift or kubernetes namespace. With this release, you are allowed to specify multiple subvolume groups! This comes handy in some setups where you could have a segregation of the subvolumes for various purposes!

ROX support for both RBD and CephFS

ROX is nothing but an access mode which you could define while requesting a PVC from Kubernetes/Openshift. What that means to an end user is that, the workload get a READONLY share. In a plain dynamic provisioning workflow it does not make much sense. The reasoning is that, you are requesting a volume of size “X” and the backend driver provision it from the storage cluster, but its an “empty” volume and if you attach this to a workload as “READONLY” , in nutshell its an empty volume and you cant write the data to it!! So the question was “Whats the use of ROX volumes” ?

BUT, with the snap and clone use case there is a big use case behind it. That said, think about a scenario where you have a “VM template” which you want to consume it as READONLY image! For such use case ROX support add much value! and here we are with that support!

Mutli arch image support for Ceph CSI

Our community users want to use multi arch images for amd64 and arm64 ..etc. It was indeed available for last few versions, however the manifest files were not carrying this properly and it is corrected in this release and images are available in quay.io.

Updated sidecars & Kuberenetes dependency lift to 1.18.6

We have upgraded the CSI community provided sidecars to latest versions and also brought the Kubernetes dependency chain to 1.18.6 version. This itself bring many bug fixes, improvements, features..etc! I dont want to list them here as its really huge!

Performance improvements – Especially on go ceph bindings

We kept on improving the performance of CSI driver and especially its connection with backend cluster by making use of latest go-ceph version ( v4.0). We have seen great improvement in the backend connection workflow and overall in the life cycle of volume management.. So worth a mention here!

Code Cleanup, Better E2E, what not?

Great amount of code cleanup, E2E improvement , Documentation update …etc are part of this release!

…so on

The container image is tagged with “v3.0.0” and its downloadable by #docker pull ..

Kudos to the Ceph CSI community for all the hard work to reach this critical milestone!
The Ceph-CSI project ( https://github.com/ceph/ceph-csi/), as well as its thriving community, has continued to grow and we are happy to share that, this is our 11th release since Jul 12, 2019!!

We are not stopping here, rather marching towards v3.1.0 (https://github.com/ceph/ceph-csi/issues/1272) with some more feature enhancement as tracked in the release issue.
One of the very important feature we are targeting with v3.1.0 release is that, CephFS snapshot and Clone functionality, watch this space for more update!

Reach us at slack (https://cephcsi.slack.com) or at github: https://github.com/ceph/ceph-csi/

Happy Hacking!

[1]
Release Issue: https://github.com/ceph/ceph-csi/issues/865
ceph-csi v3.0.0 tag: https://github.com/ceph/ceph-csi/releases/tag/v3.0.0,
Release Images: https://quay.io/repository/cephcsi/cephcsi?tab=tags

—

Failed to initialize CSINode: error updating CSINode annotation: timed out

by
May 06 21:41:42 node1.example.com systemd[26469]: I0506 21:41:42.386428 26469 nodeinfomanager.go:402] Failed to publish CSINode: the server could not find the requested resource May 06 21:41:42 node1.example.com systemd[26469]: E0506 21:41:42.386469 26469 csi_plugin.go:273] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: the server could not find the requested resource May 06 21:41:42 node1.example.com systemd[26469]: F0506 21:41:42.386474 26469 csi_plugin.go:287] Failed to initialize CSINode after retrying: timed out waiting for the condition

or the CSI plugin pods are crashing with an error similar to below and they are in CrashLoopBackState:

I0519 03:09:55.089594 1 main.go:110] Version: v1.2.0-0-g6ef000ae I0519 03:09:55.089644 1 connection.go:151] Connecting to unix:///csi/csi.sock I0519 03:09:55.090799 1 node_register.go:58] Starting Registration Server at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock I0519 03:09:55.091050 1 node_register.go:67] Registration Server started at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock I0519 03:09:55.937613 1 main.go:77] Received GetInfo call: &InfoRequest{} I0519 03:09:56.937580 1 main.go:77] Received GetInfo call: &InfoRequest{} I0519 03:09:58.343490 1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource,} E0519 03:09:58.343558 1 main.go:89] Registration process failed with error: RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource, restarting registration container.

Also:

csi-cephfsplugin-q6vk9 2/3 CrashLoopBackOff 26 8 15h csi-cephfsplugin-yy9bv 2/3 CrashLoopBackOff 26 8 15h csi-rbdplugin-88wn4 2/3 CrashLoopBackOff 199 15h csi-rbdplugin-8j8q9 2/3 CrashLoopBackOff 264 15h csi-rbdplugin-gz9t9 2/3 CrashLoopBackOff 26 8 15h

Whats happening here? The CSI node-driver-registrar is a sidecar container that fetches driver information (using NodeGetInfo) from a CSI endpoint and registers it with the kubelet on that node using the kubelet plugin registration mechanism.From there on kubelet directly talk to the CSI driver through the registration socket for driver calls like NodeGetInfo, NodeStageVolume, NodePublishVolume ..etc. After registration, there is an object called CSINOde which is managed by kubelet: While the plugin comes up the registration failed for some reason.

apiVersion: storage.k8s.io/v1 kind: CSINode metadata: name: node1 spec: drivers: – name: mycsidriver.example.com nodeID: storageNodeID1 topologyKeys: [‘mycsidriver.example.com/regions’, “mycsidriver.example.com/zones”]

Few troubleshooting tips I can share here:

Whats your Kubernetes client and server versions?

Are they running on the same version ( say both are 1.17 or 1.18 ? ) ? if not, can you make sure both are the same or at least they are under-supported version skew? This is mostly popped up after an upgrade of one component to Kubernetes v1.17.

Ceph CSI: “XFS Superblock has unknown read-only..” Or ” wrong fs type, bad …. on /dev/rbd., missing codepage or ..”

by

We announced ceph csi v2.1.0 release around 3 weeks back with many bug fixes and features. More details about this release can be found here https://github.com/ceph/ceph-csi/releases/tag/v2.1.0 and in this blog https://www.humblec.com/one-more-ceph-csi-release-yeah-v2-1-0-is-here/. From the CSI/Rook community interactions we know many folks have updated to this version in the last couple of weeks.

However, unfortunately, the community found a couple of issues with their app pod PVC mounting (XFS based) with this CSI version in their Kubernetes/openshift cluster.

There are mainly 2 errors/issues encountered in setups:

1) XFS: wrong fs type, bad option, bad superblock on /dev/rbd4, missing codepage or helper program, or other error
2) XFS: Superblock has unknown read-only compatible features (0x4) enabled

As you can see above, both of these issues are on XFS mounting. Ceph CSI plugin make use of mount.xfs binary at time of volume mounting and its part of “xfsprogs” package shipped by the CSI container. To understand this issue better, lets first see the change:

ceph v2.1.0 release: mkfs.xfs version 4.5.0

ceph v2.1.0 release: mkfs.xfs version 5.0.0

This change happened when we updated the Ceph base image in our containers from v14.2 to v15 in ceph CSI version 2.1.0 compared to 2.0.0

ceph v2.1.0 BASE_IMAGE=ceph/ceph:v14.2

ceph v2.1.0 BASE_IMAGE=ceph/ceph:v15

To give some more context about these issues:

The first one is more generic:

XFS: wrong fs type, bad option, bad superblock on /dev/rbd4, missing codepage or helper program, or other error

That said, this is seen when you have multiple PVCs with the same XFS UUID ( filesystem UUID) and the PVC mount fails when its attached to a pod.

The reason to have multiple PVCs with the same UUID arises when you have a snapshotted/cloned volume. Even though ceph CSI snapshot/clone support is still in Alpha state we have proactively fixed this issue [1].

The second issue:

XFS: Superblock has unknown read-only compatible features (0x4) enabled

pop up when the cluster nodes are running RHEL 7 based kernels ( for ex: redhat kernel 3.10.0-957 el7). The mkfs.xfs binary in the new CSI container image is not really compatible with this kernel version and you will encounter ‘unknown read-only compatible features (0x4) read-only error at the time of mounting’. To avoid this error the mkfs.xfs call would need to include -m reflink=0 which disable the incompatible copy-on-write reflink support.

More details about these issues can be found at https://github.com/ceph/ceph-csi/issues/966

[1] Solution:

Both of these issues have been addressed with ceph CSI v2.1.1 which got released today https://github.com/ceph/ceph-csi/releases/tag/v2.1.1. The immediate fix what we made with this new release is, reverting to older mkfs.xfs version. The real fix is known and we will make it soon. But, till then we don’t want to leave our awesome community with broken setup.

We advise you to update to v2.1.1 if you are using xfs as your PVC fsformat in your kubernetes/openshift cluster.

Special thanks to below community users for reporting this issue and helping us with the various stages of this debugging process!

https://github.com/chandr20
https://github.com/iExalt
https://github.com/fiveclubs
https://github.com/volvicoasis

Happy hacking and talk to us via slack ( http://cephcsi.slack.com) or github https://github.com/ceph/ceph-csi/.

One more ceph-csi release ? Yeah, v2.1.0 is here!

by

The Ceph-CSI team is excited that it has reached the next milestone with the release of v2.1.0! [1] This is another great release with many improvements for Ceph and CSI integration to use in production with Kubernetes/openshift clusters. Of all the many features and bug fixes here are just a few of the highlights.

Release Issue # https://github.com/ceph/ceph-csi/issues/816


# Changelog or Highlights:

## Features:
Add support for rbd static PVC
Move cephfs subvolume support from Alpha to Beta.
Added support for rbd topology-based provisioning.
Support externally managed configmap.
Updated Base image to ceph Octopus
Added csiImageKey to keep track of the image name in RADOS omap
Added E2E for helm charts
...

## Enhancements:
Implement CreateVolume with go-ceph which boosts performance.
Migrated from dep to go modules
Updated Kubernetes version to v1.18.0
Updated golang version to 1.13.9
Updated Kubernetes sidecar containers to latest version
E2E: Add Ability to test with non root user
...

## Bug Fixes:
Log an error message if cephfs mounter fails during init
Aligned with klog standards for logging
Added support in to run E2E in a different namespace
Removed cache functionality for cephfs plugin restart
rbd: fallback to inline image deletion if adding it as a task fails
code cleanup for static errors and unwanted code blocks
Fix mountoption issue in rbd
travis: re-enable running on arm64
...

## Deprecated:
GRPC metrics in cephcsi

## Documentation:
Added Document to cleanup stale resources
Updated ceph-csi support matrix
dev-guide: add reference to required go-ceph dependencies
Update upgrade doc for node hang issue
....

Many other bug fixes, code improvements, README updates are also part of this release. The container image is tagged with “v2.1.0” and its downloadable by #docker pull quay.io/cephcsi/cephcsi:v2.1.0

Kudos to the Ceph CSI community for all the hard work to reach this critical milestone!
The Ceph-CSI project ( https://github.com/ceph/ceph-csi/), as well as its thriving community, has continued to grow and we are happy to share that, this is our 7th release since Jul 12, 2019!!

Happy Hacking!

Release Issue #
https://github.com/ceph/ceph-csi/issues/806
[1] https://github.com/ceph/ceph-csi/releases/tag/v2.1.0

Ceph-CSI v2.0.0 released with multi arch support, encryption, expansion..etc!

by

We are excited to announce the second major release of ceph-CSI, v2.0.0 !!

In the last few months, ceph-csi community has been tirelessly working to improve the project with many new features, bug fixes, usability improvement..etc. Some of the important features of this release include the capability of resizing RBD and CephFS volumes on demand, Encryption with LUKS support for RBD PVCs, Multi arch support (ceph-csi Arm64 image), Compatibility with kube 1.17, Upgraded sidecar containers..etc. This release is not limited features – many critical bug fixes, documentation updates are also part of this release [1][2].

[1]https://github.com/ceph/ceph-csi/issues/557
[2]https://github.com/ceph/ceph-csi/releases/tag/v2.0.0

An excerpt from the changelog:


Added dynamic resize support for CephFS PVCs
Added dynamic resize support for RBD PVCs
Added encryption with LUKS support for RBD PVCs
Mutli arch support ( ceph-csi Arm64 image)
Upgrade documentation from v1.2.2 to v2.0.0
Updated code base to kube v1.17
leader election enabled in deployment
Added Version flag to cephcsi
Removed Kubernetes 1.13.x support with 2.0.0 release
CSI: run all containers as privileged in daemonset pods
Upgrade: csi-attacher sidecar from v1.2.0 to v2.1.0
Upgrade: csi-snapshotter sidecar from v1.2.1 to v1.2.2
Upgrade: csi-node-driver-registrar sidecar from v1.1.0 to v1.2.0
Upgrade: csi-resizer from sidecar v0.3.0 to v0.4.0
Update csi-provisioner from sidecar v1.3.0 to v1.4.0
Remove deprecated containerized flag in rbd
Discard umount error if the directory is not mounted
Use EmptyDir to store provisioner socket
Add ContentSource to the CreateVolume response
Rbd: only load nbd module if not available yet
Enhance scripts to deploy ceph cluster using rook
Add e2e tests for RBD resizer
Update minikube to latest released version
Update golangci-lint version to v1.21.0
Fix to use kubectl create not kubectl apply in the e2e
Add volume size roundoff for expand request
Add E2E for cephfs resize functionality
Add Documentation for PVC resize
Fix block resize issue in RBD
Add 13.0.0 Mimic supported version to the readme
update Metrics supported version in Readme
Remove hard-coded UpdateStrategy from templates
Add E2E for block PVC resize
Enable logging in E2E if the test fails
Enable Block E2E for rbd
Add ID-based logging for ExpandVolume
Validate rbd image name in NodeExpand

There are some features we deferred from this release to next. Reach out to us via https://github.com/ceph/ceph-csi/issues/806 if you would like to see any feature/bugfix/update as part of the next ceph csi release. The CSI community has been growing since our first release ( v1.0.0) and we are glad to share that, many new contributors have come together to make ceph csi a production-grade CSI driver with this release!

Thanks to all!

Ceph CSI v1.2.0 , v1.2.1 releases and so on

by

If you care about CSI and Ceph plugin, you would have noticed a massive improvement that is going on in the ceph CSI repo for last 4/5 months!! We started to engage this repo heavily to address upstream user issues, many many bug fixes, improvements, streamlining the communication in the community, release planning and rolling out releases, integration with Rook project (https://rook.io/) to make it as default storage driver in Rook…etc. With all these efforts, we are hearing lots of very positive feedback from community users.

We were able to roll out the very first release of CSI – v1.1.0 around 3 months back with many changes and solid code in the project. We were continuing our efforts to release CSI v1.2.0 series and at present we have rolled out v1.2.0 and v1.2.1 !!

I would like to list down some of the highlights of these releases here:

Ceph CSI v1.2.0 Release

Release Issue: https://github.com/ceph/ceph-csi/issues/393

Changelog or highlights:

*) Cephfs: Use ceph kernel client if kernel version >= 4.17
*) implement grpc metrics for ceph-csi
*) Add xfs fstype as default type in storageclass
*) Add support to use ceph manager rbd command to delete an image
*) e2e: correct log format in execCommandInPod()
*) Add 'gosec' to the static-checks
*) switch to cephfs, utils, and csicommon to new loging system
*) utility to trace backend volume from RBD pvc
*) Implement context-based logging
*) implement klog wrapper
*) unmap rbd image if connection timeout.
*) start controller or node server based on config
*) fix: Adds liveness sidecar to v1.14+ helm charts
*) Prometheus liveness probe sidecar
*) Wrap error if failed to fetch mon
*) provisioners: add reconfiguring of PID limit
*) Use "rbd device list" to list and find rbd images and their device paths
*) Update Unstage transaction to undo steps done in Stage
*) Move mounting staging instance to a sub-path within staging path
*) e2e: do not fail to delete resources when "resource not found"
*) remove post validation of rbd device

Many other bug fixes, code improvements, README updates are also part of this release.


CSI v1.2.1 Release.

Release Issue # https://github.com/ceph/ceph-csi/issues/600

*) Change the recommended/default FS for RBD to ext4
*) Use nodiscard option while formatting RBD devices.
*) Use provisioner socket while probing liveness.
*) Reject request if the operation is in progress
*) Fix pod termination issue due to stale mount after node plugin restart.
….etc

As you can see in the changelog, a great amount of features, bug fixes.. etc are part of v1.2 release series.
We are not stopping here, rather marching towards v1.2’s next minor release without much delay. You can track the release items from https://github.com/ceph/ceph-csi/issues/639 .

Kudos to the Ceph CSI community for all the hard work to reach this critical milestone.

I would like to summarize this article by mentioning that, your participation is highly encouraged for future releases of Ceph CSI!

Talk to us via GitHub issues/PRs or Slack https://cephcsi.slack.com/ or other channels.

Happy hacking.

Forcefully delete stuck rook-ceph namespace ?

by

I have seen scenarios where I or someone, who deployed rook-ceph cluster want to clean up or delete “rook-ceph” namespace which hosts most of the ceph cluster and rook operator pods. However, this deletion got stuck or IOW, some of the pods get into “terminating” state forever which is not good There are many threads or discussions in various forums on, how to tackle this scenario or how to solve this and delete all stuck pods. I would like to summarize the command worked for me or helped me to successfully delete the pods and thus namespace when I encountered this issue.

If you are in this scenario, please try this out!

#kubectl -n rook-ceph patch cephclusters.ceph.rook.io rook-ceph -p ‘{“metadata”:{“finalizers”: []}}’ –type=merge

More details on how we landed on this situation..etc can be found here:

https://github.com/rook/rook/issues/2668

If you are still unlucky, check the detail of this namespace like below and then remove the finalizer or problematic marker.


[root@ ceph]# kubectl get ns/rook-ceph -oyaml
apiVersion: v1
kind: Namespace
metadata:
creationTimestamp: "2021-03-19T10:06:57Z"
deletionTimestamp: "2021-03-22T11:38:35Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:phase: {}
manager: kubectl-create
operation: Update
time: "2021-03-19T10:06:57Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:conditions:
.: {}
k:{"type":"NamespaceContentRemaining"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionContentFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionDiscoveryFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceDeletionGroupVersionParsingFailure"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"NamespaceFinalizersRemaining"}:
.: {}
f:lastTransitionTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
manager: kube-controller-manager
operation: Update
time: "2021-03-22T11:38:40Z"
name: rook-ceph
resourceVersion: "1121184"
uid: b962942b-8f2d-4fac-96df-22ff18a77143
spec:
finalizers:
- kubernetes
status:
conditions:
- lastTransitionTime: "2021-03-22T11:38:40Z"
message: All resources successfully discovered
reason: ResourcesDiscovered
status: "False"
type: NamespaceDeletionDiscoveryFailure
- lastTransitionTime: "2021-03-22T11:38:40Z"
message: All legacy kube types successfully parsed
reason: ParsedGroupVersions
status: "False"
type: NamespaceDeletionGroupVersionParsingFailure
- lastTransitionTime: "2021-03-22T11:39:40Z"
message: All content successfully deleted, may be waiting on finalization
reason: ContentDeleted
status: "False"
type: NamespaceDeletionContentFailure
- lastTransitionTime: "2021-03-22T11:38:40Z"
message: 'Some resources are remaining: cephfilesystems.ceph.rook.io has 1 resource
instances'
reason: SomeResourcesRemain
status: "True"
type: NamespaceContentRemaining
- lastTransitionTime: "2021-03-22T11:38:40Z"
message: 'Some content in the namespace has finalizers remaining: cephfilesystem.ceph.rook.io
in 1 resource instances'
reason: SomeFinalizersRemain
status: "True"
type: NamespaceFinalizersRemaining
phase: Terminating
[root@ ceph]# kubectl edit cephfilesystems.ceph.rook.io -n rook-ceph
cephfilesystem.ceph.rook.io/myfs edited