As you know Snapshot is Point In time copy of a volume. In Kubernetes, a VolumeSnapshot represents a snapshot of a volume on a storage system.
Similar to how API resources like PersistentVolume and PersistentVolumeClaim are used to provision volumes for users and administrators in an kubernetes/OCP cluster, VolumeSnapshot and VolumeSnapshotContent are API resources that are provided to create volume snapshots.
For supporting or working with snapshot, we have mainly 3 API objects or Custom resources and to name those:
We have volumeSnapshot, VolumeSnapshotContent and VolumeSnapshot Class. These objects or workflow is analogues to Persistent Volume Claim (PVC) workflow if you are aware of.
A VolumeSnapshot is a request for snapshot of a volume by a user. This is a user property. It is similar to a PersistentVolumeClaim where user request a storage.
VolumeSnapshotClass is similar to storage class but for Snapshot requests. Its an admin property in an ocp cluster. It allows you to specify different attributes belonging to a VolumeSnapshot. These attributes may differ among snapshots taken from the same volume on the storage system and therefore cannot be expressed by using the same StorageClass of a PersistentVolumeClaim.
There is also volumesnapshotcontent object, A VolumeSnapshotContent is a snapshot taken from a volume in the cluster. It is a resource in the cluster just like a PersistentVolume is a cluster resource. IN dynamic provisioning world an admin/user dont have to worry about this object. That said, its getting created and managed by internal controllers.
In Slide 1, you could map volumesnapshot workflow and objects with persistent volume claim workflow or object.
I think its prety clear and we can see that its analogous to the persistent volume claim workflow. I have listed the yaml representation of volume Snapshot class, volume snapshot and volumesnapshotcontent in the slides so that you get some idea on how it looks like. Just to touch upon, a volumesnapshotClass object representation looks like this. Things to note here is the highlighted part where admin has to define the driver , deletion policy ..etc.
We just went through the volumesnapshotclass, so next is volumesnapshot and its the user input and how a user can request a snapshot from the cluster. If you look at the highlighted part, this is where user request saying, I want to create a snapshot from a PVC or volume called “pvc-test”. The `spec.Source` has been filled with the parent or source PVC. On right side you can also see volumesnapshotclass object. As mentioned earlier in dynamic provisioning case, an admin or user dont have to create this object rather its the responsibility of the controller. On a bound object you can see it as volumesnapshotclass got reference to the user request via VolumeSnapshotRefernce and also to the parent volume with the volume handle.
Thats mainly on the yamls. While working with snapshots, we should be knowing few things or some rules:
The parent/source PVC should be at bound state.
PVC should not be in use while snapshot is in progress.
There is also PVC protection available to ensure that in-use PersistentVolumeClaim API objects are not removed from the system while a snapshot is being taken from it (as this may result in data loss). Controller add a finalizer to make sure pvc is not getting removed.
Thats about snapshot , now lets think about Restore:
Restore:
While talking about restore, we are talking about restoring the snapshot to a completely NEW volume and we are not talking about In place restore or Roll Back or revert here. The lack of support for inplace restore or rollback has listed as a limitation in upstream kubernetes atm. So, We could expect this support in future Kubernetes releases.
if you look at the Yaml of a request to restore, you could see that, the highlighted part is where you mention you want to restore to a new volume from the volumesnapshot object named ‘new-snapshot-test”. Thats the only difference compared to the general persistent volume claim request.
In short restore input looks like a PVC request or IOW, its a PVC request with Data Source as ‘volumesnapshot’.
Clone:
Clone is about creating a new volume from an existing volume. Clone is defined as a duplicate of an existing Persistent Volume Claim/PVC. So, instead of creating a new Empty volume we are requesting that, we need a volume with prepopulated content from the source PVC. Its a simple interface.
If you look at the yaml representation its looks like a PVC request with a small difference that, we have Datasource field filled with an existing PVC name.
If you go back and compare restore and clone yamls, the only difference is that, the data source is “Snapshot” in case of “restore” and data source is “PVC/peristentvolumeclaim” in clone .
As we found in snapshot case, While working with Clone, we should be knowing that:
The source PVC must be BOUND and not in use
Only same namespace cloning is allowed , that means the source and destination PVC should be same namespace.
only within same storageclass cloning is allowed.
Cloning can only be performed between two volumes that use the same VolumeMode setting. When I say volumemode its nothing but kubernetes allow you to specify the PVC with either Filesystem or Block mode. So cloning is only allowed with same volumemode setting.
Thats about cloning…
So we went through an overview of snapshot, restore and clone features.
Things to Remember when working with these features:
Here i would like seek your attention mainly on below items/bullets.
When it comes to Data Consistency:
No snapshot consistency guarantees beyond any guarantees provided by the storage system (e.g. crash consistency). So its upto the storage vendor atm to provide the guarantee on the data consistency. You could provide more consistency but its upto the responsibility of higher-level APIs/controllers.
Inflight I/Os:
The API server does not enforce the source volume use ( in use or not ) while a user requests a snapshot/clone. Eventhough its mentioned as prerequisite or recommendation its just on the books/docs, no code in controller stop or prevent the request if its in use.
No inplace-restore:
As discussed some time back, at present It does not support reverting an existing volume to an earlier state represented by a snapshot (beta only supports provisioning a new volume from a snapshot).
User is free to delete the parent/source Volume:
Once the snapshot/clone is created, user is free to delete the source PVC/volume. This gives some challenges while we have Copy On Write based snapshots which is the case with many storage backends. Because the COW snapshots got a reference linking to parent volume and to enable the parent volume deletion we have to untangle the snap from the parent volume. Its bit of a work which CSI driver or other components depends on the storage system.
E2E WorkFlow
With that, let me move to the next slide which give some idea about how the snapshot request e2e works and the components involved in the path.
AS you can see in the slides there are 2 additional components which we havent discussed yet. One is the snapshot controller which is deployed with Kube/OCP platform and its main job is to watch for volumeSnapshot object, we also have a sidecar deployed with CSI bundle called csi snapshotter who watch for volumeSnapshotContent object.
The other primitives like volumesnapshotclass , volumesnapshot, and volumesnapshotcontent we already discussed in previous slides.
The volumesnapshotClass is available in the cluster and user can make a request for snapshot via volumesnapshot object, which is getting monitored by the snapshot controller and it create or populate the volumesnapshotcontent object which in turn monitored by the the csi-snapshotter sidecar. Once the CSI snapshotter sidecar container see there is a volumesnapshotContent object it talks to the CSI driver ( Example Ceph CSI driver) through the CSI endpoint and now its the the turn of CSI driver to talk to the backend or storage cluster and create a snapshot.
Just to repeat, we have 2 additional pods with controllers called
snapshotcontroller and csi snapshotter.
I hope this workflow is clear and helps to understand how the request originated from the user and land with snapshot creation in the backend.
Please let me know if you got any questions.
References:
Volume Snapshots:
https://kubernetes.io/docs/concepts/storage/volume-snapshots/
Restore from snapshots:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#volume-snapshot-and-restore-volume-from-snapshot-support
CSI Volume Cloning
https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/