openshift

[GlusterFS dynamic Provisioner] New endpoint and service name format for GlusterFS volumes!

by

What is this all about? I would like to keep a short blog article about glusterfs endpoint and service creation happening under the hood when you dynamically provision glusterfs volumes in kubernetes/Openshift cluster.


1) Provisioner Take Request
2) Call Heketi Volume Create API
3) Heketi Talk to Gluster and create a Volume
4) Upon successful completion of Volume creation, Provisioner name endpoint/service of form:
glusterfs-dynamic-PVCNAME
5) Then provisioner try to create Endpoint/Service
6) Upon successful endpoint/service creation, PVC move to BOUND state

Let us pay attention to Step “4”, here, the endpoint and service name is created with a prefix “glusterfs-dynamic” to PVC name.
However if you create a PVC with latest Kubernetes Version ( 1.13) , you would see that the endpoint and services are in below format:

If I list endpoints and service in my system:

[root@Node ~]# kubectl get ep NAME ENDPOINTS AGE

glusterfs-dynamic-8e155928-f77f-11e8-b229-005056a5c1c4 10.70.46.80:1,10.70.47.189:1,10.70.47.68:1 3h glusterfs-dynamic-a71fa382-f77d-11e8-b229-005056a5c1c4 10.70.46.80:1,10.70.47.189:1,10.70.47.68:1 4h heketi-db-storage-endpoints 10.70.46.80:1,10.70.47.189:1,10.70.47.68:1 5d heketi-storage 10.130.0.6:8080 5d [root@Node ~]#

[root@Node ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

glusterfs-dynamic-8e155928-f77f-11e8-b229-005056a5c1c4 ClusterIP 172.31.221.49 <none> 1/TCP 3h glusterfs-dynamic-a71fa382-f77d-11e8-b229-005056a5c1c4 ClusterIP 172.31.147.132 <none> 1/TCP 4h heketi-db-storage-endpoints ClusterIP 172.31.202.20 <none> 1/TCP 5d heketi-storage ClusterIP 172.31.183.17 <none> 8080/TCP 5d


Here I would like to map the endpoint/service for claim1, How would I do that?

In short:

*) Fetch PVC UID
*) Look for this UID in endpoint/service output

[root@Node ~]# kubectl describe pvc claim1 Name: claim1 Namespace: app-storage StorageClass: glusterfs-storage Status: Bound Volume: pvc-a71fa382-f77d-11e8-b229-005056a5c1c4 Labels: <none> Annotations: pv.kubernetes.io/bind-completed=yes pv.kubernetes.io/bound-by-controller=yes volume.beta.kubernetes.io/storage-class=glusterfs-storage volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/glusterfs Finalizers: [kubernetes.io/pvc-protection] Capacity: 2Gi Access Modes: RWO Events: <none>

“claim1” PVC got UID “a71fa382-f77d-11e8-b229-005056a5c1c4” and endpoint/service “glusterfs-dynamic-a71fa382-f77d-11e8-b229-005056a5c1c4″ belong to this claim!!

Let us wrap it up. Why this change?

This change was introduced mainly because of:

1) There is an issue if you delete and recreate a PVC with the same name without any delay. Due to a race scenario, the second PVC creation would end up without endpoint and service..

2) If user want to create a PVC with a name in range of 45-63 characters, it was not possible to create endpoint and service due to the prefix ( glusterfs-dynamic) added to it. The API server will reject the lengthy endpoint and service name.

Disadvantages:

It was very easy to fetch an endpoint/service in a namespace with PVC name with naming format glusterfs-dynamic-pvcname, but now it need some more steps.

As always, please let me know your thoughts!!

[glusterfs] An external file provisioner to dynamically provision Gluster File Volumes.

by

If your application pod use dynamically provisioned glusterfs volumes in your Kubernetes/openshift cluster, it could be that, the volumes are provisioned using intree gluster provisioner. Are you confused when I say “intree” ? by intree I am talking about the plugin/driver compiled and shipped with kubernetes/openshift release. While using intree drivers in the cluster, admin is not worried about how to deploy it, the reason being, by default it is running or registered with Kube API server and listening for persistent volume actions. However Kubernetes sig-storage is also maintaining an external-storage repo which hosts other controllers and provisioners.

External storage repo is not just hosting external provisioners, it also hosts other controllers like the snapshot controller. This repo is hosted under incubator and the beauty is that, the flexibility to update the code! If you ever contributed a patch to Kubernetes you would know how difficult it is :), you need to take care of many things including release timelines of kubernetes..etc.

We also had a requirement which has to be quickly satisfied for integrating gluster with one other important project, getting things done at dependent project’s timeline was bit difficult, I would explain what feature we were chasing in my next blog, till then let it hide. To achieve the same we started to implement an external file provisioner for gluster file. We already have an external gluster block provisioner in this same repo, if you are already using gluster block provisioner you know what is external provisioner and how to use it?

The external provisioner for gluster file was merged in external storage repo a couple of months back! You can pull this provisioner and run it in kubernetes/openshift as a pod! The workflow of creating and deleting the persistent volumes remains the same as you use intree plugins or drivers today. The only difference is that you need to mention or use ‘external provisioner’ name in the storage class instead of ‘intree provisioner name’

For example, in Kubernetes/openshift to provision gluster file volumes you would have used provisioner name kubernetes.io/glusterfs however in this case, the provisioner name is different and you could use gluster.org/glusterfile. The latter name or the external provisioner name is configurable and you can give any name. It’s a configuration parameter which has to be set in your pod template, the provisioner will listen for the requests which match the name specified in the storage class.

Below demo shows how to provision and delete volumes using this external file provisioner. All the artifacts which are used in this demo is available here . Follow this doc for file provisioner deployment .

Demo:

All good, you used external file provisioner and provisioned/deleted gluster file volumes, however, you may have a question comes up to your mind that, why we have one more provisioner for gluster file when we already have intree plugins for provisioner and delete volumes? The answer will be covered in the next blog post on this same topic 🙂

glusterfs: “[heketi] failed to create volume: signature is invalid”

by

Oh. You got that error and want to know why? Short answer: The heketi credentials you submitted does not match with the credential specified in heketi configuration file. Long answer: 1) How did you deploy heketi ? As a service using service file ? If yes, check the entries in /etc/heketi/heketi.json file for admin password. …

Read more

glusterfs: Decode heketi configuration or topology file entries from heketi pod.

by

Let’s start hacking!! Most of the time, one needs to know the heketi configuration and topology he is running with. I do this at times and today thought of sharing with a wider audience, there are more hacking that can be done once you got a handle of the things, but I put a disclaimer …

Read more

[Alpha] GlusterFS CSI ( Container Storage Interface) Driver for Container Orchestrators!

by

Every Container or Cloud space storage vendor wants a standard interface for an unbiased solution development which then will not require them to have a non-trivial testing matrix . “Container Storage Interface” (CSI) is a proposed new Industry standard for cluster-wide volume plugins. CSI will enable storage vendors (SP) to develop a plugin once and …

Read more

[GlusterFS Dynamic Provisioner] Online Resizing of GlusterFS PVs in kubernetes v>= 1.8 !!!

by

While the kubernetes storage-sig keeps adding new features in each and every release, we also listen to our user feedback to improve the existing storage interfaces in solving the existing limitations. Previously, the admin had to do proper capacity planning in order to use persistent Volumes as the microservices or the application pods wanted to …

Read more

“Retain” ( PV claim policy) GlusterFS Dynamically Provisioned Persistent Volumes in Kubernetes >=1.8

by

Since introduction, the dynamic provisioning feature of Kube storage defaulted with reclaim policy as Delete. It was not specific to glusterfs PVs rather common for all PVs which are dynamically provisioned in kube storage. However from kube >= v1.8, we could specify retain policy in storage class! A much-needed functionality in SC for different use …

Read more

[Gluster-Block] Provision Gluster Block Volumes In Kubernetes/Openshift – ( Part 1)

by

Voila…. Its almost a couple of years GlusterFS keep rocking in Container Storage Space which brought the buzz word “Container Native Storage” to the emerging market!! If you dont know about what I mean by “Container Native Storage” , check out this link GlusterFS was one of the plugin in kubernetes tree which implemented Dynamic …

Read more

[GlusterFS Dynamic Provisioner] Set GlusterFS Volume Options via StorageClass parameter

by

Much awaited RFE/Enhancement to the GlusterFS provisioner in Kubernetes !! Its been long time kubernetes/Openshift users are looking for a feature/functionality to enable various GlusterFS volume options via storage class. We discussed on few methods to implement this in Heketi. One thought was to wrap options under ‘classified’ strings, for eg# when someone asked for …

Read more

Kubernetes 101 for Bangalore Kubernauts .

by

We keep conducting Kubernetes and Openshift meetups as part of https://www.meetup.com/kubernetes-openshift-India-Meetup/ .

We see great momentum in this meetup group and lots of enthusiam around this. The last events were well received and lots of requests came in to have a hands on training on Kubernetes and Openshift. If you are still to catch up with these emerging technologies, dont delay, just join us in next event planned on 20-May-2017.

More details about this event can be found @ https://www.meetup.com/kubernetes-openshift-India-Meetup/events/239381714/

This will be a beginner oriented workshop. As a pre requisite, you need to install linux in your laptop, thats it.

We are also looking for volunteers to help and venue to conduct this event.

Please RSVP and let us know if you would like to help us on organizing this event.