Kubernetes Volume Snapshot , Restore and PVC cloning introduction

As you know Snapshot is Point In time copy of a volume. In Kubernetes, a VolumeSnapshot represents a snapshot of a volume on a storage system.
Similar to how API resources like PersistentVolume and PersistentVolumeClaim are used to provision volumes for users and administrators in an kubernetes/OCP cluster, VolumeSnapshot and VolumeSnapshotContent are API resources that are provided to create volume snapshots.

For supporting or working with snapshot, we have mainly 3 API objects or Custom resources and to name those:
We have volumeSnapshot, VolumeSnapshotContent and VolumeSnapshot Class. These objects or workflow is analogues to Persistent Volume Claim (PVC) workflow if you are aware of.

A VolumeSnapshot is a request for snapshot of a volume by a user. This is a user property. It is similar to a PersistentVolumeClaim where user request a storage.

VolumeSnapshotClass is similar to storage class but for Snapshot requests. Its an admin property in an ocp cluster. It allows you to specify different attributes belonging to a VolumeSnapshot. These attributes may differ among snapshots taken from the same volume on the storage system and therefore cannot be expressed by using the same StorageClass of a PersistentVolumeClaim.

There is also volumesnapshotcontent object, A VolumeSnapshotContent is a snapshot taken from a volume in the cluster. It is a resource in the cluster just like a PersistentVolume is a cluster resource. IN dynamic provisioning world an admin/user dont have to worry about this object. That said, its getting created and managed by internal controllers.

In Slide 1, you could map volumesnapshot workflow and objects with persistent volume claim workflow or object.

I think its prety clear and we can see that its analogous to the persistent volume claim workflow. I have listed the yaml representation of volume Snapshot class, volume snapshot and volumesnapshotcontent in the slides so that you get some idea on how it looks like. Just to touch upon, a volumesnapshotClass object representation looks like this. Things to note here is the highlighted part where admin has to define the driver , deletion policy ..etc.
We just went through the volumesnapshotclass, so next is volumesnapshot and its the user input and how a user can request a snapshot from the cluster. If you look at the highlighted part, this is where user request saying, I want to create a snapshot from a PVC or volume called “pvc-test”. The `spec.Source` has been filled with the parent or source PVC. On right side you can also see volumesnapshotclass object. As mentioned earlier in dynamic provisioning case, an admin or user dont have to create this object rather its the responsibility of the controller. On a bound object you can see it as volumesnapshotclass got reference to the user request via VolumeSnapshotRefernce and also to the parent volume with the volume handle.

Thats mainly on the yamls. While working with snapshots, we should be knowing few things or some rules:

The parent/source PVC should be at bound state.
PVC should not be in use while snapshot is in progress.

There is also PVC protection available to ensure that in-use PersistentVolumeClaim API objects are not removed from the system while a snapshot is being taken from it (as this may result in data loss). Controller add a finalizer to make sure pvc is not getting removed.
Thats about snapshot , now lets think about Restore:


Restore:

While talking about restore, we are talking about restoring the snapshot to a completely NEW volume and we are not talking about In place restore or Roll Back or revert here. The lack of support for inplace restore or rollback has listed as a limitation in upstream kubernetes atm. So, We could expect this support in future Kubernetes releases.

if you look at the Yaml of a request to restore, you could see that, the highlighted part is where you mention you want to restore to a new volume from the volumesnapshot object named ‘new-snapshot-test”. Thats the only difference compared to the general persistent volume claim request.

In short restore input looks like a PVC request or IOW, its a PVC request with Data Source as ‘volumesnapshot’.

Clone:

Clone is about creating a new volume from an existing volume. Clone is defined as a duplicate of an existing Persistent Volume Claim/PVC. So, instead of creating a new Empty volume we are requesting that, we need a volume with prepopulated content from the source PVC. Its a simple interface.
If you look at the yaml representation its looks like a PVC request with a small difference that, we have Datasource field filled with an existing PVC name.

If you go back and compare restore and clone yamls, the only difference is that, the data source is “Snapshot” in case of “restore” and data source is “PVC/peristentvolumeclaim” in clone .
As we found in snapshot case, While working with Clone, we should be knowing that:

The source PVC must be BOUND and not in use
Only same namespace cloning is allowed , that means the source and destination PVC should be same namespace.
only within same storageclass cloning is allowed.

Cloning can only be performed between two volumes that use the same VolumeMode setting. When I say volumemode its nothing but kubernetes allow you to specify the PVC with either Filesystem or Block mode. So cloning is only allowed with same volumemode setting.
Thats about cloning…

So we went through an overview of snapshot, restore and clone features.

Things to Remember when working with these features:

Here i would like seek your attention mainly on below items/bullets.

When it comes to Data Consistency:

No snapshot consistency guarantees beyond any guarantees provided by the storage system (e.g. crash consistency). So its upto the storage vendor atm to provide the guarantee on the data consistency. You could provide more consistency but its upto the responsibility of higher-level APIs/controllers.

Inflight I/Os:
The API server does not enforce the source volume use ( in use or not ) while a user requests a snapshot/clone. Eventhough its mentioned as prerequisite or recommendation its just on the books/docs, no code in controller stop or prevent the request if its in use.

No inplace-restore:

As discussed some time back, at present It does not support reverting an existing volume to an earlier state represented by a snapshot (beta only supports provisioning a new volume from a snapshot).

User is free to delete the parent/source Volume:

Once the snapshot/clone is created, user is free to delete the source PVC/volume. This gives some challenges while we have Copy On Write based snapshots which is the case with many storage backends. Because the COW snapshots got a reference linking to parent volume and to enable the parent volume deletion we have to untangle the snap from the parent volume. Its bit of a work which CSI driver or other components depends on the storage system.

E2E WorkFlow

With that, let me move to the next slide which give some idea about how the snapshot request e2e works and the components involved in the path.

AS you can see in the slides there are 2 additional components which we havent discussed yet. One is the snapshot controller which is deployed with Kube/OCP platform and its main job is to watch for volumeSnapshot object, we also have a sidecar deployed with CSI bundle called csi snapshotter who watch for volumeSnapshotContent object.
The other primitives like volumesnapshotclass , volumesnapshot, and volumesnapshotcontent we already discussed in previous slides.
The volumesnapshotClass is available in the cluster and user can make a request for snapshot via volumesnapshot object, which is getting monitored by the snapshot controller and it create or populate the volumesnapshotcontent object which in turn monitored by the the csi-snapshotter sidecar. Once the CSI snapshotter sidecar container see there is a volumesnapshotContent object it talks to the CSI driver ( Example Ceph CSI driver) through the CSI endpoint and now its the the turn of CSI driver to talk to the backend or storage cluster and create a snapshot.

Just to repeat, we have 2 additional pods with controllers called

snapshotcontroller and csi snapshotter.

I hope this workflow is clear and helps to understand how the request originated from the user and land with snapshot creation in the backend.

Please let me know if you got any questions.

References:

Volume Snapshots:
https://kubernetes.io/docs/concepts/storage/volume-snapshots/

Restore from snapshots:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#volume-snapshot-and-restore-volume-from-snapshot-support

CSI Volume Cloning
https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/

Install a kubernetes cluster ( using kind) in your laptop/anywhere in just 2 minutes

Before we start, let me introduce to `kind` which is the tool I will be using here to bring up a kube cluster, so whats `kind`?:

From : https://github.com/kubernetes-sigs/kind/

kind is a tool for running local Kubernetes clusters using Docker container "nodes".
kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

If you have go (1.11+) and docker installed GO111MODULE="on" go get sigs.k8s.io/kind@v0.8.0 && kind create cluster is all you need!

We can use it to create multi-node or multi-control-plane Kubernetes clusters, however, in this article I will bring up a single node cluster which may be the test bed for quickly test or learn something. Yes, multi-node cluster has its own advantages, so I will write next post about the same.[Please see the reference section for multi-node cluster deployment]

Why I blogged about `kind` today is mainly because of the new `kind` release happened today ( `v0.8.0`) which include one of the fixes which I have been waiting for long time ie Clusters that continue working when you reboot your host / docker :

https://github.com/kubernetes-sigs/kind/releases/tag/v0.8.0


KIND v0.8.0 is here with:
- Clusters that continue working when you reboot your host / docker :tada:
- Experimental podman support
- Improved networking (It's always been DNS! & a fix for the pesky iptables :arrow_right: nftables)
- Enhanced errors :pray:

Without much delay lets bring up a kubernetes cluster with version `1.18.x` using kind:

[terminal user=”humblec” computer=”localhost” cwd=”/home/humblec”]

$curl -Lo kind https://github.com/kubernetes-sigs/kind/releases/latest/download/kind-Linux-amd64
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 147 100 147 0 0 26 0 0:00:05 0:00:05 –:–:– 34
100 629 100 629 0 0 103 0 0:00:06 0:00:06 –:–:– 103
100 9896k 100 9896k 0 0 58671 0 0:02:52 0:02:52 –:–:– 42278

$chmod +x kind

$sudo mv kind /usr/local/bin/

$kind create cluster –image=docker.io/kindest/node:1.18.0
Creating cluster “kind” …
✓ Ensuring node image (kindest/node:v1.18.2) ?
✓ Preparing nodes ?
✓ Writing configuration ?
✓ Starting control-plane ?️
✓ Installing CNI ?
✓ Installing StorageClass ?
Set kubectl context to “kind-kind”
You can now use your cluster with:

kubectl cluster-info –context kind-kind

Have a nice day! ?

[/terminal]

Above bootstrapped a Kubernetes cluster using a pre-built node image (https://kind.sigs.k8s.io/docs/design/node-image) – you can find it on docker hub `kindest/node`. If you desire to build the node image yourself, it’s possible too, refer kind documentation for the same. To specify another image to bring up your cluster, you can use the `–image` flag with the `create cluster` command.

By default, the cluster will be given the name kind. Use the –name flag to assign the cluster a different context name.

Just to go some more in details about the `kind` options and how to use it, let’s see below examples which can be used in `kind` created cluster:

[terminal]

$sudo /usr/local/bin/kind export kubeconfig
Set kubectl context to “kind-kind”

$kubectl config use-context kind-kind
Switched to context “kind-kind”.

$sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready master 15m v1.18.2
$sudo kubectl get pods –all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-c92t6 1/1 Running 0 18m
kube-system coredns-66bff467f8-kg7zl 1/1 Running 0 18m
kube-system etcd-kind-control-plane 1/1 Running 0 18m
kube-system kindnet-zz6dt 1/1 Running 0 18m
kube-system kube-apiserver-kind-control-plane 1/1 Running 0 18m
kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 18m
kube-system kube-proxy-sjbcw 1/1 Running 0 18m
kube-system kube-scheduler-kind-control-plane 1/1 Running 0 18m
local-path-storage local-path-provisioner-bd4bb6b75-n28xd 1/1 Running 0 18m

$ sudo docker exec kind-control-plane crictl images

IMAGE TAG IMAGE ID SIZE
docker.io/kindest/kindnetd 0.5.4 2186a1a396deb 113MB
docker.io/rancher/local-path-provisioner v0.0.12 db10073a6f829 42MB
k8s.gcr.io/coredns 1.6.7 67da37a9a360e 43.9MB
k8s.gcr.io/debian-base v2.0.0 9bd6154724425 53.9MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90d 290MB
k8s.gcr.io/kube-apiserver v1.18.2 7df05884b1e25 147MB
k8s.gcr.io/kube-controller-manager v1.18.2 31fd71c85722f 133MB
k8s.gcr.io/kube-proxy v1.18.2 312d3d1cb6c72 133MB
k8s.gcr.io/kube-scheduler v1.18.2 121edc8356c58 113MB
k8s.gcr.io/pause 3.2 80d28bedfe5de 686kB

[/terminal]

If you want to get details about your `kind` cluster:

[terminal]
$ sudo kind help
kind creates and manages local Kubernetes clusters using Docker container ‘nodes’

Usage:
kind [command]

Available Commands:
build Build one of [node-image]
completion Output shell completion code for the specified shell (bash, zsh or fish)
create Creates one of [cluster]
delete Deletes one of [cluster]
export Exports one of [kubeconfig, logs]
get Gets one of [clusters, nodes, kubeconfig]
help Help about any command
load Loads images into nodes
version Prints the kind CLI version

Flags:
-h, –help help for kind
–loglevel string DEPRECATED: see -v instead
-q, –quiet silence all stderr output
-v, –verbosity int32 info log verbosity

Use “kind [command] –help” for more information about a command.

$ sudo /usr/local/bin/kind get nodes
kind-control-plane

$ sudo /usr/local/bin/kind get kubeconfig
apiVersion: v1
clusters:
– cluster:
certificate-authority-data: ………
server: https://127.0.0.1:38831
name: kind-kind
contexts:
– context:
cluster: kind-kind
user: kind-kind
name: kind-kind
current-context: kind-kind
kind: Config
preferences: {}
users:
– name: kind-kind
user:
client-certificate-data: ………………….
client-key-data: ………….
$

$ sudo /usr/local/bin/kind load docker-image alpine
Image: “alpine” with ID “sha256:f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a” not yet present on node “kind-control-plane”, loading…

$ sudo docker exec kind-control-plane crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/kindest/kindnetd 0.5.4 2186a1a396deb 113MB
docker.io/library/alpine latest f70734b6a266d 5.88MB
docker.io/rancher/local-path-provisioner v0.0.12 db10073a6f829 42MB
k8s.gcr.io/coredns 1.6.7 67da37a9a360e 43.9MB
k8s.gcr.io/debian-base v2.0.0 9bd6154724425 53.9MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90d 290MB
k8s.gcr.io/kube-apiserver v1.18.2 7df05884b1e25 147MB
k8s.gcr.io/kube-controller-manager v1.18.2 31fd71c85722f 133MB
k8s.gcr.io/kube-proxy v1.18.2 312d3d1cb6c72 133MB
k8s.gcr.io/kube-scheduler v1.18.2 121edc8356c58 113MB
k8s.gcr.io/pause 3.2 80d28bedfe5de 686kB
[/terminal]

At times, we may fail to create an application pod in the above `kind` cluster due to an error or a behavior where it fails to pull the docker image of the applications deployed. In those cases, the workaround I followed was to create a kind cluster by nullifying the http_proxy configurations as shown below

[terminal]
sudo https_proxy=”” http_proxy=”” all_proxy=”” /usr/local/bin/kind create cluster
[/terminal]

Now, go ahead and set up your cluster in 2 mins!! then play with it.

Update:

There was a question about whether `kind` has loadbalancer type service support?

Answer:

you can do `load balancer` with metallb [1] on Linux, but docker for Mac / windows Mac/windows have very limited network connectivity. There is an upstream issue that tracks the `load balancer` support https://github.com/kubernetes-sigs/kind/issues/702 .

Please take a look at https://github.com/kubernetes-sigs/kind/issues/702#issuecomment-624561998 which talks about how the metallb can be used to deploy a load balancer service in Linux setup.

[1] https://mauilion.dev/posts/kind-metallb/

Reference:

If you really got excited about kind and if you would like to deploy multi node cluster, please check this article.

Install a multi node kubernetes cluster using `kind` (kind.sigs.k8s.io)