CephFS snapshot, restore and cloning with Ceph CSI driver

by

In our previous article we discussed about the terminology side of Kubernetes volume snapshots and concepts around it. If you haven’t read through it, I would recommend to have a check on this article, may be before getting into this. But, I will leave that upto you.

This article describe about how CephFS snapshots can be created on CephFS volumes provisioned by the Ceph CSI driver. Once you have snapshots available you can also restore these snapshots to “NEW” Persistent Volume Claims!. We also have the ability to CLONE an existing PVC to a new Volumes! This covers a good amount of use cases related to Backup and Restore in many scenarios. The only requirement here is to have Ceph CSI version 3.1.0 running in your setup either directly by deploying Ceph CSI via templates from its project repository or by installing Rook version >= 1.4.1.

I have also captured a demo of this process in below screen cast.

Considering I have documented the process in the screen cast, its better to not duplicate it here. However if you have any queries, suggestions or comments on this, please place a comment here or reach out!

Kubernetes Volume Snapshot , Restore and PVC cloning introduction

by

As you know Snapshot is Point In time copy of a volume. In Kubernetes, a VolumeSnapshot represents a snapshot of a volume on a storage system.
Similar to how API resources like PersistentVolume and PersistentVolumeClaim are used to provision volumes for users and administrators in an kubernetes/OCP cluster, VolumeSnapshot and VolumeSnapshotContent are API resources that are provided to create volume snapshots.

For supporting or working with snapshot, we have mainly 3 API objects or Custom resources and to name those:
We have volumeSnapshot, VolumeSnapshotContent and VolumeSnapshot Class. These objects or workflow is analogues to Persistent Volume Claim (PVC) workflow if you are aware of.

A VolumeSnapshot is a request for snapshot of a volume by a user. This is a user property. It is similar to a PersistentVolumeClaim where user request a storage.

VolumeSnapshotClass is similar to storage class but for Snapshot requests. Its an admin property in an ocp cluster. It allows you to specify different attributes belonging to a VolumeSnapshot. These attributes may differ among snapshots taken from the same volume on the storage system and therefore cannot be expressed by using the same StorageClass of a PersistentVolumeClaim.

There is also volumesnapshotcontent object, A VolumeSnapshotContent is a snapshot taken from a volume in the cluster. It is a resource in the cluster just like a PersistentVolume is a cluster resource. IN dynamic provisioning world an admin/user dont have to worry about this object. That said, its getting created and managed by internal controllers.

In Slide 1, you could map volumesnapshot workflow and objects with persistent volume claim workflow or object.

I think its prety clear and we can see that its analogous to the persistent volume claim workflow. I have listed the yaml representation of volume Snapshot class, volume snapshot and volumesnapshotcontent in the slides so that you get some idea on how it looks like. Just to touch upon, a volumesnapshotClass object representation looks like this. Things to note here is the highlighted part where admin has to define the driver , deletion policy ..etc.
We just went through the volumesnapshotclass, so next is volumesnapshot and its the user input and how a user can request a snapshot from the cluster. If you look at the highlighted part, this is where user request saying, I want to create a snapshot from a PVC or volume called “pvc-test”. The `spec.Source` has been filled with the parent or source PVC. On right side you can also see volumesnapshotclass object. As mentioned earlier in dynamic provisioning case, an admin or user dont have to create this object rather its the responsibility of the controller. On a bound object you can see it as volumesnapshotclass got reference to the user request via VolumeSnapshotRefernce and also to the parent volume with the volume handle.

Thats mainly on the yamls. While working with snapshots, we should be knowing few things or some rules:

The parent/source PVC should be at bound state.
PVC should not be in use while snapshot is in progress.

There is also PVC protection available to ensure that in-use PersistentVolumeClaim API objects are not removed from the system while a snapshot is being taken from it (as this may result in data loss). Controller add a finalizer to make sure pvc is not getting removed.
Thats about snapshot , now lets think about Restore:


Restore:

While talking about restore, we are talking about restoring the snapshot to a completely NEW volume and we are not talking about In place restore or Roll Back or revert here. The lack of support for inplace restore or rollback has listed as a limitation in upstream kubernetes atm. So, We could expect this support in future Kubernetes releases.

if you look at the Yaml of a request to restore, you could see that, the highlighted part is where you mention you want to restore to a new volume from the volumesnapshot object named ‘new-snapshot-test”. Thats the only difference compared to the general persistent volume claim request.

In short restore input looks like a PVC request or IOW, its a PVC request with Data Source as ‘volumesnapshot’.

Clone:

Clone is about creating a new volume from an existing volume. Clone is defined as a duplicate of an existing Persistent Volume Claim/PVC. So, instead of creating a new Empty volume we are requesting that, we need a volume with prepopulated content from the source PVC. Its a simple interface.
If you look at the yaml representation its looks like a PVC request with a small difference that, we have Datasource field filled with an existing PVC name.

If you go back and compare restore and clone yamls, the only difference is that, the data source is “Snapshot” in case of “restore” and data source is “PVC/peristentvolumeclaim” in clone .
As we found in snapshot case, While working with Clone, we should be knowing that:

The source PVC must be BOUND and not in use
Only same namespace cloning is allowed , that means the source and destination PVC should be same namespace.
only within same storageclass cloning is allowed.

Cloning can only be performed between two volumes that use the same VolumeMode setting. When I say volumemode its nothing but kubernetes allow you to specify the PVC with either Filesystem or Block mode. So cloning is only allowed with same volumemode setting.
Thats about cloning…

So we went through an overview of snapshot, restore and clone features.

Things to Remember when working with these features:

Here i would like seek your attention mainly on below items/bullets.

When it comes to Data Consistency:

No snapshot consistency guarantees beyond any guarantees provided by the storage system (e.g. crash consistency). So its upto the storage vendor atm to provide the guarantee on the data consistency. You could provide more consistency but its upto the responsibility of higher-level APIs/controllers.

Inflight I/Os:
The API server does not enforce the source volume use ( in use or not ) while a user requests a snapshot/clone. Eventhough its mentioned as prerequisite or recommendation its just on the books/docs, no code in controller stop or prevent the request if its in use.

No inplace-restore:

As discussed some time back, at present It does not support reverting an existing volume to an earlier state represented by a snapshot (beta only supports provisioning a new volume from a snapshot).

User is free to delete the parent/source Volume:

Once the snapshot/clone is created, user is free to delete the source PVC/volume. This gives some challenges while we have Copy On Write based snapshots which is the case with many storage backends. Because the COW snapshots got a reference linking to parent volume and to enable the parent volume deletion we have to untangle the snap from the parent volume. Its bit of a work which CSI driver or other components depends on the storage system.

E2E WorkFlow

With that, let me move to the next slide which give some idea about how the snapshot request e2e works and the components involved in the path.

AS you can see in the slides there are 2 additional components which we havent discussed yet. One is the snapshot controller which is deployed with Kube/OCP platform and its main job is to watch for volumeSnapshot object, we also have a sidecar deployed with CSI bundle called csi snapshotter who watch for volumeSnapshotContent object.
The other primitives like volumesnapshotclass , volumesnapshot, and volumesnapshotcontent we already discussed in previous slides.
The volumesnapshotClass is available in the cluster and user can make a request for snapshot via volumesnapshot object, which is getting monitored by the snapshot controller and it create or populate the volumesnapshotcontent object which in turn monitored by the the csi-snapshotter sidecar. Once the CSI snapshotter sidecar container see there is a volumesnapshotContent object it talks to the CSI driver ( Example Ceph CSI driver) through the CSI endpoint and now its the the turn of CSI driver to talk to the backend or storage cluster and create a snapshot.

Just to repeat, we have 2 additional pods with controllers called

snapshotcontroller and csi snapshotter.

I hope this workflow is clear and helps to understand how the request originated from the user and land with snapshot creation in the backend.

Please let me know if you got any questions.

References:

Volume Snapshots:
https://kubernetes.io/docs/concepts/storage/volume-snapshots/

Restore from snapshots:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#volume-snapshot-and-restore-volume-from-snapshot-support

CSI Volume Cloning
https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/

Ceph CSI v3.1.0 out with CephFS snapshot , restore and clone, RADOS namespace support!

by

We are excited to announce one more feature packed release of ceph-csi, v3.1.0 !!. Couple of weeks ago we released v3.0.0, and we are proud to bring one more release pretty close with important features and enhancements.

This release enables you to take a snapshot of a CephFS volume, Restore Snapshot to new volume and also do PVC clone on CephFS PV. We also got RADOS namespace support for RBD plugin from a community contributor in this release.!

More details about the release is available here @ https://github.com/ceph/ceph-csi/releases/tag/v3.1.0. The container image is tagged with “v3.1.0” and is downloadable by #docker pull .

Changelog or Highlights:
New Features:

Create/Delete snapshot for CephFS
Create PVC from CephFS snapshot
Create PVC from CephFS PVC
Add RADOS namespace support for RBD

Enhancement:

E2E migration from Travis minikube to CentOS CI
Use common logging functions in utils
Add Upgrade E2E testing from 3.0.0 to 3.1.0
Update Sidecars to the latest version
Add vault creation to rbd driver deployment
Update E2E testing to test with latest kubernetes versions
Update Rook to 1.3.8 for E2E testing
Add backend validation of cephfs PVC in E2E
Make the number of CPUs for minikube VM configurable
Use rbdVolume.resize() with go-ceph
Introduce new makefile target run-e2e
upgrade to minikube 1.12.1
Debugging tool (tracevol.py) enhancement

Bug Fix:

Fix tracevol.py to work with the dynamic value of fsname
Fix tracevol.py to take config map namespace as an option

Breaking Changes

None.

NOTE: At present, there is a limit of 400 snapshots per cephFS filesystem.
Also PVC cannot be deleted if it’s having snapshots. Make sure all the snapshots
on the PVC are deleted before you delete the PVC.

Many other bug fixes, code improvements are also part of this release.

Just to touch upon the new features:

CephFS snapshot, restore and clone:

This functionlity allows a user to create VolumeSnapshot Objects on top of CephFS PVCs which is nothing but the snapshots. Snapshots have many use cases and its an important feature in many areas to make sure you have the point in time copy of the data. Once you have the snapshot you can restore this to a new PVCs and attach or use it in the workload. This operation is totally independent to Parent Volume or snapshot objects. Clone functionality is bit different, that said, it allows you to create a clone of an existing PVC to a new PVC, so that you get another copy of the existing PVC which can be independently used!

RADOS namespace support

This would allow multiple kubernetes clusters to share one unique ceph cluster without creating a pool per kubernetes cluster. Pools in ceph can be a computationally expensive as stated in documentation and namespaces are the recommended way to segregate tenants/users. The kubernetes admin can specify a ceph namespace to use in the helm chart and all ceph’s operations done in the backend are automatically using –namespace=$mynamespace. Hence, all rbd operations are going to the specified namespace.
A different kubernetes cluster using a different ceph client, could be assigned to a different ceph’s namespace and would not be able to see the image done by the first cluster.

This is an important feature for multi tenant use cases!!

Please read more about Ceph Namespaces here https://docs.ceph.com/docs/master/rados/operations/user-management/#namespace

Kubernetes and sidecar updates

The kubernetes dependency and all the sidecars ( external provisioner, attacher, resizer..etc) are updated to latest versions available in upstream which got many bug fixes, code improvements..etc

Few other enhancements

There are many code cleanups, E2E enhancements..etc with this release. One of the notable or worth to mention change here is the migration of CI from Travis to CentOS CI!
Improvements in tracevol.py to make the debugging easier is also part of this release.

Thanks to Rook community to include this CSI release without any delay and make it available with Rook 1.4.1 (https://github.com/rook/rook/releases/tag/v1.4.1)!

Kudos to the Ceph CSI community for all the hard work to reach this critical milestone!

The Ceph-CSI project ( https://github.com/ceph/ceph-csi/), as well as its thriving community, has continued to grow and the contribution from community increases with each release in terms of code contribution, discussion on the features/issues..etc.

We are marching towards v3.2.0 and at a stage where collecting a wishlist from the community for the features we want to pull in the upcoming release.
Please engage and share your thoughts/ideas here : https://github.com/ceph/ceph-csi/issues/1340

Reach us at slack (cephcsi.slack.com) or at github: https://github.com/ceph/ceph-csi/

Happy Hacking!

Expand/Resize Ceph CSI Provisioned Volumes ( CephFS and RBD)

by

Its been a while this feature is supported in Ceph CSI driver and many folks are making use of it. This is a great feature and a useful one in many situations. For example think about a situation where we created a PVC and attached to an application pod, be it a database pod, monitoring pod, or any application pod which eventually consumed the entire space of the attached PVC.

Deleting data from this volume is NOT an option at all at times. If deletion is not an option, what other option we have? migrating the entire data to a new bigger PVC and attach to the same POD/workload? repeat this whenever we are going out of space ? It also need bringing down the workload or pod to attach a new PVC. Also planning how much size the application is going to consume upfront may not work always. But expansion feature is to the rescue! Without disturbing the workload or attached PVC /share/volume, just expand the PVC by a command and thats it! You get new volume size reflected in your POD without any delay and that too when POD is consuming the storage and writing data to it!! Awesome, Isn’t it ? thats what Ceph Volumes provisioned by Ceph CSI driver is capable of !

Before we start on this: I would like to mention about some prerequisites here.

1) You should have the csi-resizer sidecar running in your setup. ( This would be default when you deploy it via Rook or using ceph-csi templates)
2) The Storage class which you use to provision should have few parameters set in it: ( This would be also available if you deploy it with Rook or Ceph CSI templates)

For example:

AllowVolumeExpansion: True
csi.storage.k8s.io/controller-expand-secret-name=rook-csi-cephfs-provisioner
csi.storage.k8s.io/controlnd-secret-namespace=rook-ceph

Why to write and explain everything in this article ? rather let me show you the demo of Volume Expansion feature in Ceph CSI: We have two screencasts one for CephFS and one for RBD.

CephFS volume expansion Demo:

RBD volume expansion Demo:

To summarize, both CephFS and RBD volumes can be expanded online and Ceph CSI drivers are capable of doing it!

At times, due to some reason if the resize fails:

Recovering from Failure when Expanding Volumes

If expanding underlying storage fails, the cluster administrator can manually recover the Persistent Volume Claim (PVC) state and cancel the resize requests. Otherwise, the resize requests are continuously retried by the controller without administrator intervention.

Mark the PersistentVolume(PV) that is bound to the PersistentVolumeClaim(PVC) with Retain reclaim policy.
Delete the PVC. Since PV has Retain reclaim policy – we will not loose any data when we recreate the PVC.
Delete the claimRef entry from PV specs, so as new PVC can bind to it. This should make the PV Available.
Re-create the PVC with smaller size than PV and set volumeName field of the PVC to the name of the PV. This should bind new PVC to existing PV.
Don’t forget to restore the reclaim policy of the PV

I am preparing a demo of RBD volume expansion for the next article on this space, so watch out!

ceph csi v3.0.0 released-(Snapshot, Clone, Multi arch, ROX…)

by

We are excited to announce the third major release of ceph-csi, v3.0.0 !!
The Ceph-CSI team is excited that it has reached the next milestone with the release of v3.0.0! [1]. This release is not limited to features – many critical bug fixes, documentation updates are also part of this release. This is another great release with many improvements for Ceph and CSI integration to use in production with Kubernetes/openshift clusters. Of all the many features and bug fixes here are just a few of the highlights.


New Features:

Create/Delete snapshot for RBD
Create PVC from RBD snapshot
Create PVC from RBD PVC
Add support for multiple CephFS subvolume groups
Multi Architecture docker images(amd64 and arm64)
Support ROX(ReadOnlyMany) PVC for RBD
Support ROX(ReadOnlyMany) PVC for CephFS

Enhancement:

Move to go-ceph binding from RBD CLI
Move to go-ceph binding from RADOS CLI
Add Upgrade E2E testing from 2.1.2 to 3.0.0
Update Sidecars to the latest version
Improve locking to create a parallel clone and snapshot restore
Simplify Error Handling
Update golangci-lint version in CI
Update gosec version in CI
Add support to track cephfs PVC and subvolumes
Introduce build.env for configuration of the environment variables
Update go-ceph to v0.4.0
Update E2E testing to test with latest kubernetes versions
Split out CephFS and RBD E2E tests
Integration with Centos CI to run containerized builds
Update Rook to 1.2.7 for E2E testing
Disable reflink when creating xfs filesystem for RBD
Replace klog with klog v2
Reduce RBAC for kubernetes sidecar containers
Add option to compile e2e tests in containerized
Add commitlint bot in CI
Add Stale bot to the repo
Add E2E and documentation for CephFS PVC
Update kubernetes dependency to v1.18.6

Bug Fix:

Fix issue in CephFS Volume Not found

Breaking Changes

Remove support for v1.x.x PVC
Remove support for Mimic
Snapshot Alpha is no longer supported

Lets touch upon some more of the very cool features introduced in this release:

“Snapshot and Clone functionality made available with RBD”:

Since v1.0.0 release of Ceph CSI we had RBD snapshot in place but we marked it as “Alpha” for few reasons. One of them being the snapshot support in Kubernetes upstream CSI driver was also evolving and stabilizing on the API side of things. Its not only that, we had an issue of image lockup when we try to consume or untangle parent source volume from the snapshot volume/object. So we did a revamp in this version and we are happy to say that, we have it solved and with this release . From now on, RBD snapshot should work smooth.

Its capable of:

*) Creating a snapshot from a RBD volume
*) Restoring to a new volume from the existing snapshot
*) Deletion of snapshot and parent volume objects independently.

We also enabled a cool functionality here with the Volume Cloning feature. Thats nothing but the capability of provisioning a volume from another or existing PVC source.

Dont confuse this with “Restore of a snapshot”

The main difference here is that, While you restore from a snapshot, for the new PVC, the referred `Datasource` is “Snapshot” and if you do clone operation, the “DataSource” is an existing PVC.


Multiple subvolumegroup support

When CephFS CSI driver create a volume, we default to a subvolumegroup called “csi” , We place our subvolumes in this group. I mean the backend CephFS volumes which map to the PVC in openshift or kubernetes namespace. With this release, you are allowed to specify multiple subvolume groups! This comes handy in some setups where you could have a segregation of the subvolumes for various purposes!

ROX support for both RBD and CephFS

ROX is nothing but an access mode which you could define while requesting a PVC from Kubernetes/Openshift. What that means to an end user is that, the workload get a READONLY share. In a plain dynamic provisioning workflow it does not make much sense. The reasoning is that, you are requesting a volume of size “X” and the backend driver provision it from the storage cluster, but its an “empty” volume and if you attach this to a workload as “READONLY” , in nutshell its an empty volume and you cant write the data to it!! So the question was “Whats the use of ROX volumes” ?

BUT, with the snap and clone use case there is a big use case behind it. That said, think about a scenario where you have a “VM template” which you want to consume it as READONLY image! For such use case ROX support add much value! and here we are with that support!

Mutli arch image support for Ceph CSI

Our community users want to use multi arch images for `amd64` and `arm64` ..etc. It was indeed available for last few versions, however the manifest files were not carrying this properly and it is corrected in this release and images are available in `quay.io`.

Updated sidecars & Kuberenetes dependency lift to 1.18.6

We have upgraded the CSI community provided sidecars to latest versions and also brought the Kubernetes dependency chain to `1.18.6` version. This itself bring many bug fixes, improvements, features..etc! I dont want to list them here as its really huge!

Performance improvements – Especially on go ceph bindings

We kept on improving the performance of CSI driver and especially its connection with backend cluster by making use of latest go-ceph version ( v4.0). We have seen great improvement in the backend connection workflow and overall in the life cycle of volume management.. So worth a mention here!

Code Cleanup, Better E2E, what not?

Great amount of code cleanup, E2E improvement , Documentation update …etc are part of this release!

…so on

The container image is tagged with “v3.0.0” and its downloadable by #docker pull ..

Kudos to the Ceph CSI community for all the hard work to reach this critical milestone!
The Ceph-CSI project ( https://github.com/ceph/ceph-csi/), as well as its thriving community, has continued to grow and we are happy to share that, this is our 11th release since Jul 12, 2019!!

We are not stopping here, rather marching towards v3.1.0 (https://github.com/ceph/ceph-csi/issues/1272) with some more feature enhancement as tracked in the release issue.
One of the very important feature we are targeting with v3.1.0 release is that, CephFS snapshot and Clone functionality, watch this space for more update!

Reach us at slack (https://cephcsi.slack.com) or at github: https://github.com/ceph/ceph-csi/

Happy Hacking!

[1]
Release Issue: https://github.com/ceph/ceph-csi/issues/865
ceph-csi v3.0.0 tag: https://github.com/ceph/ceph-csi/releases/tag/v3.0.0,
Release Images: https://quay.io/repository/cephcsi/cephcsi?tab=tags

—

Persistent Volume and Claim ( PV and PVC) status in kubernetes

by

Most of the time, a user or an admin in a Kubernetes or Openshift cluster is confused about what is meant by the persistent volume and persistent volume claim status field.

Just to make sure we are on same page, this is about the “STATUS” field in the output captured below:

[terminal]
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datastore-demo-sts-0 Bound pvc-5a541780-dda4-4462-9dde-c143ece76341 1Gi RWO gp2 33d
datastore-demo-sts-1 Bound pvc-686efec4-fb35-4afe-a175-2f2e064b74fb 1Gi RWO gp2 33d
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
postgres-pv-volume 5Gi RWX Retain Bound bsk-n1-dev3/postgres-pv-claim manual 7d22h
pvc-5a541780-dda4-4462-9dde-c143ece76341 1Gi RWO Delete Bound default/datastore-demo-sts-0 gp2 33d
pvc-686efec4-fb35-4afe-a175-2f2e064b74fb 1Gi RWO Delete Bound default/datastore-demo-sts-1 gp2 33d
[/terminal]

Persistent Volume States: PV states:

It can be any of:

“Pending”, “Available”, “Bound”, “Released” or “Failed”

[terminal]
Pending : Used for PersistentVolumes that are not available.

Available: Used for PersistentVolumes that are not yet bound to a PVC.

Bound : Used for PersistentVolumes that are bound with a PVC.

Released : Used for PersistentVolumes where the bound PersistentVolumeClaim was deleted. Released volumes must be recycled before becoming available again.This phase is used by the persistent volume claim binder to signal to another process to reclaim the resource

Failed : Used for PersistentVolumes that failed to be correctly recycled or deleted after being released from a claim
[/terminal]

Persistent Volume Claim States: PVC states:

It can be any of:

“Pending”, “Bound”, or “Lost”

[terminal]
Pending: Used for PersistentVolumeClaims that are not yet bound

Bound: Used for PersistentVolumeClaims that are bound

Lost: Used for PersistentVolumeClaims that lost their underlying PersistentVolume. The claim was bound to a PersistentVolume and this volume does not exist any longer and all data on it was lost.
[/terminal]

Failed to initialize CSINode: error updating CSINode annotation: timed out

by

[terminal]
May 06 21:41:42 node1.example.com systemd[26469]: I0506 21:41:42.386428 26469 nodeinfomanager.go:402] Failed to publish CSINode: the server could not find the requested resource
May 06 21:41:42 node1.example.com systemd[26469]: E0506 21:41:42.386469 26469 csi_plugin.go:273] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: the server could not find the requested resource
May 06 21:41:42 node1.example.com systemd[26469]: F0506 21:41:42.386474 26469 csi_plugin.go:287] Failed to initialize CSINode after retrying: timed out waiting for the condition
[/terminal]

or the CSI plugin pods are crashing with an error similar to below and they are in `CrashLoopBackState`:

[terminal]

I0519 03:09:55.089594 1 main.go:110] Version: v1.2.0-0-g6ef000ae
I0519 03:09:55.089644 1 connection.go:151] Connecting to unix:///csi/csi.sock
I0519 03:09:55.090799 1 node_register.go:58] Starting Registration Server at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
I0519 03:09:55.091050 1 node_register.go:67] Registration Server started at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
I0519 03:09:55.937613 1 main.go:77] Received GetInfo call: &InfoRequest{}
I0519 03:09:56.937580 1 main.go:77] Received GetInfo call: &InfoRequest{}
I0519 03:09:58.343490 1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource,}
E0519 03:09:58.343558 1 main.go:89] Registration process failed with error: RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource, restarting registration container.

[/terminal]

Also:
[terminal]

csi-cephfsplugin-q6vk9 2/3 CrashLoopBackOff 26 8 15h
csi-cephfsplugin-yy9bv 2/3 CrashLoopBackOff 26 8 15h
csi-rbdplugin-88wn4 2/3 CrashLoopBackOff 199 15h
csi-rbdplugin-8j8q9 2/3 CrashLoopBackOff 264 15h
csi-rbdplugin-gz9t9 2/3 CrashLoopBackOff 26 8 15h
[/terminal]

Whats happening here? The CSI node-driver-registrar is a sidecar container that fetches driver information (using `NodeGetInfo`) from a CSI endpoint and registers it with the kubelet on that node using the kubelet plugin registration mechanism.From there on kubelet directly talk to the CSI driver through the registration socket for driver calls like NodeGetInfo, NodeStageVolume, NodePublishVolume ..etc. After registration, there is an object called CSINOde which is managed by kubelet: While the plugin comes up the registration failed for some reason.

[terminal]
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
name: node1
spec:
drivers:
– name: mycsidriver.example.com
nodeID: storageNodeID1
topologyKeys: [‘mycsidriver.example.com/regions’, “mycsidriver.example.com/zones”]
[/terminal]

Few troubleshooting tips I can share here:

Whats your Kubernetes client and server versions?

Are they running on the same version ( say both are 1.17 or 1.18 ? ) ? if not, can you make sure both are the same or at least they are under-supported version skew? This is mostly popped up after an upgrade of one component to Kubernetes v1.17.

Install a multi node kubernetes cluster using `kind` (kind.sigs.k8s.io)

by

In an earlier blog post we discussed on how we can deploy a Kubernetes cluster using kind tool in less than 2 mins! If you missed it, please check this article before proceeding here to understand what `kind` is all about and its basic usage.

To summarize:

kind is a tool for running local Kubernetes clusters using Docker container “nodes”.
kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

In this article we will see how can we deploy a multi-node cluster ( 3 masters + 3 workers) again less than 2 mins :). Because at times we have to play with features like taints, tolerations and node affinities. To really dive deep into these features or if you want to play with such configurations, you need to have more than one node available.

The prerequisites are really minimal to install a multi-node cluster and I would say it’s pretty much that you just need below `YAML` file.

For ex:

[terminal]
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
– role: control-plane
– role: control-plane
– role: control-plane
– role: worker
– role: worker
– role: worker
[/terminal]

Even though the entries in the above YAML file is self-explanatory, let me touch upon those. Actually the `role` part says the master and worker node configuration. That said, 3 master and 3 worker nodes.

Without much delay lets create a multi-node cluster. I have saved above YAML configuration file in `multi.yaml`, which is going to be the configuration file path we pass with `–config` option below.

[terminal]
[humble@localhost ~]$ sudo kind create cluster –name=multi –config=multi.yaml
Creating cluster “multi” …
✓ Ensuring node image (kindest/node:v1.18.2) ?
✓ Preparing nodes ? ? ? ? ? ?
✓ Configuring the external load balancer ⚖️
✓ Writing configuration ?
✓ Starting control-plane ?️
✓ Installing CNI ?
✓ Installing StorageClass ?
✓ Joining more control-plane nodes ?
✓ Joining worker nodes ?
Set kubectl context to “kind-multi”
You can now use your cluster with:

kubectl cluster-info –context kind-multi

Not sure what to do next? ? Check out https://kind.sigs.k8s.io/docs/user/quick-start/
[humble@localhost ~]$

[/terminal]

WoW! Is it the effort to deploy a multi-node Kubernetes cluster? yeah! Amazing.

Just to confirm quickly we have the nodes available and running:

[terminal]
[humble@localhost ~]$ sudo kubectl config use-context kind-multi
Switched to context “kind-multi”.
[humble@localhost ~]$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
multi-control-plane Ready master 18m v1.18.2
multi-control-plane2 Ready master 17m v1.18.2
multi-control-plane3 Ready master 15m v1.18.2
multi-worker Ready 14m v1.18.2
multi-worker2 Ready 14m v1.18.2
multi-worker3 Ready 14m v1.18.2
[humble@localhost ~]$
[/terminal]

It is not just nodes other resources are also available in this cluster, for example: storage class.

[terminal]
[humble@localhost ~]$ sudo kubectl describe sc
Name: standard
IsDefaultClass: Yes
Annotations: kubectl.kubernetes.io/last-applied-configuration={“apiVersion”:”storage.k8s.io/v1″,”kind”:”StorageClass”,”metadata”:{“annotations”:{“storageclass.kubernetes.io/is-default-class”:”true”},”name”:”standard”},”provisioner”:”rancher.io/local-path”,”reclaimPolicy”:”Delete”,”volumeBindingMode”:”WaitForFirstConsumer”}
,storageclass.kubernetes.io/is-default-class=true
Provisioner: rancher.io/local-path
Parameters:
AllowVolumeExpansion:
MountOptions:
ReclaimPolicy: Delete
VolumeBindingMode: WaitForFirstConsumer
Events:
[humble@localhost ~]$
[/terminal]

Lets try to get some more details about the nodes:

[terminal]
[humble@localhost ~]$ sudo kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
multi-control-plane Ready master 26m v1.18.2 172.18.0.8 Ubuntu 19.10 5.6.8-200.fc31.x86_64 containerd://1.3.3-14-g449e9269
multi-control-plane2 Ready master 25m v1.18.2 172.18.0.7 Ubuntu 19.10 5.6.8-200.fc31.x86_64 containerd://1.3.3-14-g449e9269
multi-control-plane3 Ready master 23m v1.18.2 172.18.0.4 Ubuntu 19.10 5.6.8-200.fc31.x86_64 containerd://1.3.3-14-g449e9269
multi-worker Ready 22m v1.18.2 172.18.0.9 Ubuntu 19.10 5.6.8-200.fc31.x86_64 containerd://1.3.3-14-g449e9269
multi-worker2 Ready 22m v1.18.2 172.18.0.6 Ubuntu 19.10 5.6.8-200.fc31.x86_64 containerd://1.3.3-14-g449e9269
multi-worker3 Ready 22m v1.18.2 172.18.0.5 Ubuntu 19.10 5.6.8-200.fc31.x86_64 containerd://1.3.3-14-g449e9269
[humble@localhost ~]$
[/terminal]

Don’t get confused with the kernel listing, just to make it clear, `#uname -r output `5.6.8-200.fc31.x86_64` from my system matches with the above listing.

Let us get some more information about this cluster:

[terminal]
[humble@localhost ~]$ sudo kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:46223
KubeDNS is running at https://127.0.0.1:46223/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’.
[humble@localhost ~]$
[/terminal]

Below shows some more details about these cluster nodes, I have truncated below output to show just one master and one worker node:
[terminal]
humble@localhost ~]$ sudo kubectl describe nodes
Name: multi-control-plane
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=multi-control-plane
kubernetes.io/os=linux
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 10 May 2020 20:15:40 +0530
Taints: node-role.kubernetes.io/master:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: multi-control-plane
AcquireTime:
RenewTime: Sun, 10 May 2020 23:21:57 +0530
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
—- —— —————– —————— —— ——-
MemoryPressure False Sun, 10 May 2020 23:19:27 +0530 Sun, 10 May 2020 20:15:40 +0530 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sun, 10 May 2020 23:19:27 +0530 Sun, 10 May 2020 20:15:40 +0530 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sun, 10 May 2020 23:19:27 +0530 Sun, 10 May 2020 20:15:40 +0530 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sun, 10 May 2020 23:19:27 +0530 Sun, 10 May 2020 20:16:31 +0530 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.18.0.8
Hostname: multi-control-plane
Capacity:
cpu: 4
ephemeral-storage: 103077688Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 19885692Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 103077688Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 19885692Ki
pods: 110
System Info:
Machine ID: 090f76515dde4e02a5edf4dd513db6bb
System UUID: c594630a-9d52-4dbe-b70d-aaec20124dc8
Boot ID: 51e95ed3-91b9-492d-9b1b-6ff87def8d10
Kernel Version: 5.6.8-200.fc31.x86_64
OS Image: Ubuntu 19.10
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.3.3-14-g449e9269
Kubelet Version: v1.18.2
Kube-Proxy Version: v1.18.2
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
——— —- ———— ———- ————— ————- —
kube-system coredns-66bff467f8-4djpt 100m (2%) 0 (0%) 70Mi (0%) 170Mi (0%) 3h5m
kube-system coredns-66bff467f8-szq2g 100m (2%) 0 (0%) 70Mi (0%) 170Mi (0%) 3h6m
kube-system etcd-multi-control-plane 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h6m
kube-system kindnet-d5qts 100m (2%) 100m (2%) 50Mi (0%) 50Mi (0%) 3h5m
kube-system kube-apiserver-multi-control-plane 250m (6%) 0 (0%) 0 (0%) 0 (0%) 3h6m
kube-system kube-controller-manager-multi-control-plane 200m (5%) 0 (0%) 0 (0%) 0 (0%) 3h6m
kube-system kube-proxy-bbbfl 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h6m
kube-system kube-scheduler-multi-control-plane 100m (2%) 0 (0%) 0 (0%) 0 (0%) 3h6m
local-path-storage local-path-provisioner-bd4bb6b75-8w5d5 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h5m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
——– ——– ——
cpu 850m (21%) 100m (2%)
memory 190Mi (0%) 390Mi (2%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
……

Name: multi-worker
Roles:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=multi-worker
kubernetes.io/os=linux
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sun, 10 May 2020 20:19:57 +0530
Taints:
Unschedulable: false
Lease:
HolderIdentity: multi-worker
AcquireTime:
RenewTime: Sun, 10 May 2020 23:21:57 +0530
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
—- —— —————– —————— —— ——-
MemoryPressure False Sun, 10 May 2020 23:19:29 +0530 Sun, 10 May 2020 20:19:57 +0530 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sun, 10 May 2020 23:19:29 +0530 Sun, 10 May 2020 20:19:57 +0530 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sun, 10 May 2020 23:19:29 +0530 Sun, 10 May 2020 20:19:57 +0530 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sun, 10 May 2020 23:19:29 +0530 Sun, 10 May 2020 20:21:12 +0530 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 172.18.0.9
Hostname: multi-worker
Capacity:
cpu: 4
ephemeral-storage: 103077688Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 19885692Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 103077688Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 19885692Ki
pods: 110
System Info:
Machine ID: 24e26961e57440958576287c7b29145a
System UUID: a50a5bd5-81b4-4789-8180-0f4f986c09b8
Boot ID: 51e95ed3-91b9-492d-9b1b-6ff87def8d10
Kernel Version: 5.6.8-200.fc31.x86_64
OS Image: Ubuntu 19.10
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.3.3-14-g449e9269
Kubelet Version: v1.18.2
Kube-Proxy Version: v1.18.2
PodCIDR: 10.244.4.0/24
PodCIDRs: 10.244.4.0/24
Non-terminated Pods: (2 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
——— —- ———— ———- ————— ————- —
kube-system kindnet-95jdp 100m (2%) 100m (2%) 50Mi (0%) 50Mi (0%) 3h2m
kube-system kube-proxy-5vv58 0 (0%) 0 (0%) 0 (0%) 0 (0%) 3h2m
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
——– ——– ——
cpu 100m (2%) 100m (2%)
memory 50Mi (0%) 50Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
………..
[/terminal]

It’s the time to deploy a POD and see whether it’s working 🙂

Let us get a job.yaml:

[terminal]
[humble@localhost ~]$ cat job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: hello
spec:
template:
# This is the pod template
spec:
containers:
– name: hello
image: busybox
command: [‘sh’, ‘-c’, ‘echo “Hello, Kubernetes!” && sleep 3600’]
restartPolicy: OnFailure
# The pod template ends here
[humble@localhost ~]$
[/terminal]

Let us create the above `job` resource in this cluster.

[terminal]
[humble@localhost ~]$ sudo kubectl create -f job.yaml
job.batch/hello created
[humble@localhost ~]$ sudo kubectl get pods -n default -w
NAME READY STATUS RESTARTS AGE
hello-mm7h2 0/1 ContainerCreating 0 18s
[humble@localhost ~]$ sudo kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
hello-mm7h2 1/1 Running 0 79s

[humble@localhost ~]$ sudo kubectl get jobs
NAME COMPLETIONS DURATION AGE
hello 1/1 61m 16h
[humble@localhost ~]$ sudo kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
hello-mm7h2 0/1 Completed 0 16h
[humble@localhost ~]$ sudo kubectl logs hello-mm7h2 -n default
Hello, Kubernetes!
[humble@localhost ~]$

[/terminal]

voila!! so this cluster works and you can deploy applications in it.

Ceph CSI: “XFS Superblock has unknown read-only..” Or ” wrong fs type, bad …. on /dev/rbd., missing codepage or ..”

by

We announced ceph csi v2.1.0 release around 3 weeks back with many bug fixes and features. More details about this release can be found here https://github.com/ceph/ceph-csi/releases/tag/v2.1.0 and in this blog https://www.humblec.com/one-more-ceph-csi-release-yeah-v2-1-0-is-here/. From the CSI/Rook community interactions we know many folks have updated to this version in the last couple of weeks.

However, unfortunately, the community found a couple of issues with their app pod PVC mounting (XFS based) with this CSI version in their Kubernetes/openshift cluster.

There are mainly 2 errors/issues encountered in setups:

1) XFS: wrong fs type, bad option, bad superblock on /dev/rbd4, missing codepage or helper program, or other error
2) XFS: Superblock has unknown read-only compatible features (0x4) enabled

As you can see above, both of these issues are on XFS mounting. Ceph CSI plugin make use of `mount.xfs` binary at time of volume mounting and its part of “xfsprogs” package shipped by the CSI container. To understand this issue better, lets first see the change:

[terminal]
ceph v2.1.0 release: mkfs.xfs version 4.5.0

ceph v2.1.0 release: mkfs.xfs version 5.0.0
[/terminal]

This change happened when we updated the Ceph base image in our containers from v14.2 to v15 in ceph CSI version 2.1.0 compared to 2.0.0

[terminal]
ceph v2.1.0
BASE_IMAGE=ceph/ceph:v14.2

ceph v2.1.0
BASE_IMAGE=ceph/ceph:v15
[/terminal]

To give some more context about these issues:

The first one is more generic:

XFS: wrong fs type, bad option, bad superblock on /dev/rbd4, missing codepage or helper program, or other error

That said, this is seen when you have multiple PVCs with the same XFS UUID ( filesystem UUID) and the PVC mount fails when its attached to a pod.

The reason to have multiple PVCs with the same UUID arises when you have a snapshotted/cloned volume. Even though ceph CSI snapshot/clone support is still in `Alpha` state we have proactively fixed this issue [1].

The second issue:

XFS: Superblock has unknown read-only compatible features (0x4) enabled

pop up when the cluster nodes are running RHEL 7 based kernels ( for ex: redhat kernel 3.10.0-957 el7). The `mkfs.xfs` binary in the new CSI container image is not really compatible with this kernel version and you will encounter ‘unknown read-only compatible features (0x4) read-only error at the time of mounting’. To avoid this error the `mkfs.xfs` call would need to include `-m reflink=0` which disable the incompatible copy-on-write reflink support.

More details about these issues can be found at https://github.com/ceph/ceph-csi/issues/966

[1] Solution:

Both of these issues have been addressed with ceph CSI v2.1.1 which got released today https://github.com/ceph/ceph-csi/releases/tag/v2.1.1. The immediate fix what we made with this new release is, reverting to older `mkfs.xfs` version. The real fix is known and we will make it soon. But, till then we don’t want to leave our awesome community with broken setup.

We advise you to update to v2.1.1 if you are using `xfs` as your PVC `fsformat` in your kubernetes/openshift cluster.

Special thanks to below community users for reporting this issue and helping us with the various stages of this debugging process!

https://github.com/chandr20
https://github.com/iExalt
https://github.com/fiveclubs
https://github.com/volvicoasis

Happy hacking and talk to us via slack ( http://cephcsi.slack.com) or github https://github.com/ceph/ceph-csi/.

Install a kubernetes cluster ( using kind) in your laptop/anywhere in just 2 minutes

by

Before we start, let me introduce to `kind` which is the tool I will be using here to bring up a kube cluster, so whats `kind`?:

From : https://github.com/kubernetes-sigs/kind/

kind is a tool for running local Kubernetes clusters using Docker container "nodes".
kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.

If you have go (1.11+) and docker installed GO111MODULE="on" go get sigs.k8s.io/kind@v0.8.0 && kind create cluster is all you need!

We can use it to create multi-node or multi-control-plane Kubernetes clusters, however, in this article I will bring up a single node cluster which may be the test bed for quickly test or learn something. Yes, multi-node cluster has its own advantages, so I will write next post about the same.[Please see the reference section for multi-node cluster deployment]

Why I blogged about `kind` today is mainly because of the new `kind` release happened today ( `v0.8.0`) which include one of the fixes which I have been waiting for long time ie Clusters that continue working when you reboot your host / docker :

https://github.com/kubernetes-sigs/kind/releases/tag/v0.8.0


KIND v0.8.0 is here with:
- Clusters that continue working when you reboot your host / docker :tada:
- Experimental podman support
- Improved networking (It's always been DNS! & a fix for the pesky iptables :arrow_right: nftables)
- Enhanced errors :pray:

Without much delay lets bring up a kubernetes cluster with version `1.18.x` using kind:

[terminal user=”humblec” computer=”localhost” cwd=”/home/humblec”]

$curl -Lo kind https://github.com/kubernetes-sigs/kind/releases/latest/download/kind-Linux-amd64
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 147 100 147 0 0 26 0 0:00:05 0:00:05 –:–:– 34
100 629 100 629 0 0 103 0 0:00:06 0:00:06 –:–:– 103
100 9896k 100 9896k 0 0 58671 0 0:02:52 0:02:52 –:–:– 42278

$chmod +x kind

$sudo mv kind /usr/local/bin/

$kind create cluster –image=docker.io/kindest/node:1.18.0
Creating cluster “kind” …
✓ Ensuring node image (kindest/node:v1.18.2) ?
✓ Preparing nodes ?
✓ Writing configuration ?
✓ Starting control-plane ?️
✓ Installing CNI ?
✓ Installing StorageClass ?
Set kubectl context to “kind-kind”
You can now use your cluster with:

kubectl cluster-info –context kind-kind

Have a nice day! ?

[/terminal]

Above bootstrapped a Kubernetes cluster using a pre-built node image (https://kind.sigs.k8s.io/docs/design/node-image) – you can find it on docker hub `kindest/node`. If you desire to build the node image yourself, it’s possible too, refer kind documentation for the same. To specify another image to bring up your cluster, you can use the `–image` flag with the `create cluster` command.

By default, the cluster will be given the name kind. Use the –name flag to assign the cluster a different context name.

Just to go some more in details about the `kind` options and how to use it, let’s see below examples which can be used in `kind` created cluster:

[terminal]

$sudo /usr/local/bin/kind export kubeconfig
Set kubectl context to “kind-kind”

$kubectl config use-context kind-kind
Switched to context “kind-kind”.

$sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready master 15m v1.18.2
$sudo kubectl get pods –all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-c92t6 1/1 Running 0 18m
kube-system coredns-66bff467f8-kg7zl 1/1 Running 0 18m
kube-system etcd-kind-control-plane 1/1 Running 0 18m
kube-system kindnet-zz6dt 1/1 Running 0 18m
kube-system kube-apiserver-kind-control-plane 1/1 Running 0 18m
kube-system kube-controller-manager-kind-control-plane 1/1 Running 0 18m
kube-system kube-proxy-sjbcw 1/1 Running 0 18m
kube-system kube-scheduler-kind-control-plane 1/1 Running 0 18m
local-path-storage local-path-provisioner-bd4bb6b75-n28xd 1/1 Running 0 18m

$ sudo docker exec kind-control-plane crictl images

IMAGE TAG IMAGE ID SIZE
docker.io/kindest/kindnetd 0.5.4 2186a1a396deb 113MB
docker.io/rancher/local-path-provisioner v0.0.12 db10073a6f829 42MB
k8s.gcr.io/coredns 1.6.7 67da37a9a360e 43.9MB
k8s.gcr.io/debian-base v2.0.0 9bd6154724425 53.9MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90d 290MB
k8s.gcr.io/kube-apiserver v1.18.2 7df05884b1e25 147MB
k8s.gcr.io/kube-controller-manager v1.18.2 31fd71c85722f 133MB
k8s.gcr.io/kube-proxy v1.18.2 312d3d1cb6c72 133MB
k8s.gcr.io/kube-scheduler v1.18.2 121edc8356c58 113MB
k8s.gcr.io/pause 3.2 80d28bedfe5de 686kB

[/terminal]

If you want to get details about your `kind` cluster:

[terminal]
$ sudo kind help
kind creates and manages local Kubernetes clusters using Docker container ‘nodes’

Usage:
kind [command]

Available Commands:
build Build one of [node-image]
completion Output shell completion code for the specified shell (bash, zsh or fish)
create Creates one of [cluster]
delete Deletes one of [cluster]
export Exports one of [kubeconfig, logs]
get Gets one of [clusters, nodes, kubeconfig]
help Help about any command
load Loads images into nodes
version Prints the kind CLI version

Flags:
-h, –help help for kind
–loglevel string DEPRECATED: see -v instead
-q, –quiet silence all stderr output
-v, –verbosity int32 info log verbosity

Use “kind [command] –help” for more information about a command.

$ sudo /usr/local/bin/kind get nodes
kind-control-plane

$ sudo /usr/local/bin/kind get kubeconfig
apiVersion: v1
clusters:
– cluster:
certificate-authority-data: ………
server: https://127.0.0.1:38831
name: kind-kind
contexts:
– context:
cluster: kind-kind
user: kind-kind
name: kind-kind
current-context: kind-kind
kind: Config
preferences: {}
users:
– name: kind-kind
user:
client-certificate-data: ………………….
client-key-data: ………….
$

$ sudo /usr/local/bin/kind load docker-image alpine
Image: “alpine” with ID “sha256:f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a” not yet present on node “kind-control-plane”, loading…

$ sudo docker exec kind-control-plane crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/kindest/kindnetd 0.5.4 2186a1a396deb 113MB
docker.io/library/alpine latest f70734b6a266d 5.88MB
docker.io/rancher/local-path-provisioner v0.0.12 db10073a6f829 42MB
k8s.gcr.io/coredns 1.6.7 67da37a9a360e 43.9MB
k8s.gcr.io/debian-base v2.0.0 9bd6154724425 53.9MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90d 290MB
k8s.gcr.io/kube-apiserver v1.18.2 7df05884b1e25 147MB
k8s.gcr.io/kube-controller-manager v1.18.2 31fd71c85722f 133MB
k8s.gcr.io/kube-proxy v1.18.2 312d3d1cb6c72 133MB
k8s.gcr.io/kube-scheduler v1.18.2 121edc8356c58 113MB
k8s.gcr.io/pause 3.2 80d28bedfe5de 686kB
[/terminal]

At times, we may fail to create an application pod in the above `kind` cluster due to an error or a behavior where it fails to pull the docker image of the applications deployed. In those cases, the workaround I followed was to create a kind cluster by nullifying the http_proxy configurations as shown below

[terminal]
sudo https_proxy=”” http_proxy=”” all_proxy=”” /usr/local/bin/kind create cluster
[/terminal]

Now, go ahead and set up your cluster in 2 mins!! then play with it.

Update:

There was a question about whether `kind` has loadbalancer type service support?

Answer:

you can do `load balancer` with metallb [1] on Linux, but docker for Mac / windows Mac/windows have very limited network connectivity. There is an upstream issue that tracks the `load balancer` support https://github.com/kubernetes-sigs/kind/issues/702 .

Please take a look at https://github.com/kubernetes-sigs/kind/issues/702#issuecomment-624561998 which talks about how the metallb can be used to deploy a load balancer service in Linux setup.

[1] https://mauilion.dev/posts/kind-metallb/

Reference:

If you really got excited about kind and if you would like to deploy multi node cluster, please check this article.

Install a multi node kubernetes cluster using `kind` (kind.sigs.k8s.io)