Introducing “gluster-kubernetes” repo && “gk-deploy” script .

The information to bring Gluster into kubernetes was kind of scattered among few repos and we noticed few concerns on lagging single place to look into! it made the launch of new repo under gluster github account called “glusterfs-kubernetes” . This repo introduced a new tool called “gk-deploy” which helps to deploy the entire solution in kubernetes or Openshift environment.

An excerpt from its run.

On a successful run, you will have running Gluster pods, Heketi service..etc.

New features ( GID support, Cluster support ) added to GlusterFS dynamic provisioner in Kubernetes.

Always there are asks for RFEs and we are bringing it into GlusterFS dynamic provisioner one by one!

I would like to introduce 2 new features available now in upstream Kube tree which enhance the functionality of GlusterFS provisioner.

1) The GID ( Group ID ) support for dynamically provisioned volume
2) The support for specifying the “Cluster” from which an admin want to provision the volume.

Lets look into these options in detail.

The GID support:Till now, the gluster dynamic provisioner was creating volumes with USERID and GROUPID as root (UID 0 : GID 0 ). So, the access to the volume is restricted to root user. However with the addition of GID support, we now have a GID allocator internal to the provisioner. This allows a storage admin of Openshift/Kubernetes cluster specify a POOL of GIDs for a particular storage class. That said, the provisioner introduced 2 more new parameters called gidMin and gidMax in Storage class. These are optional parameters.

With the above storage class configuration, if you provision PVs using PV Claim requests, the PVs are created and given access to GID value for ex: 2001 which is a value between mentioned gidMin and gidMax value. If the gidMin and gidMax value are not provided, then the dynamic provisioned volumes will have the GID between 2000 and 2147483647.

If you attach this PVClaim to a pod, the pod get the new GID value in its supplemental group, thus get access to the volume. How this GID is passed to the supplemental group is internally taken care by the provisioner. You can validate that the GID has been reflected in Supplemental Group ID of the pod using ‘id’ command inside the container.

For ex: inside the container where this PVC is attached:

Now you should be able to write to the volume with the access of this Group. Once the Claim is deleted, previously allocated GID will go back to the GID pool.

Nice feature. Isnt it ?
lets look into the second feature ( Cluster Support ):

As yet, when you dynamically provision a volume, Heketi-the server who create volumes pick any of the cluster according to certain criteria. However there was a request from few users on, there should be a provision to specify a cluster or group of clusters for a storage class , so we added that support. According to me, this can help in scenarios for example, where storage admin want to specify a particular cluster for Dev and another cluster for TEST/QE departments, may be based on the hardware he has allocated for these departments and so on. That said, an admin is left with the choice of grouping nodes with faster disks ( SSDs) in one cluster and thus a storage class called ‘fast’ and other slow speed harddisks in another cluster and a subjected storage class ‘slow’. The PVs will be created based on this classification if you use ‘clusterid’ parameter in the storage class. With that it is possible now in dynamic provisioner to select a cluster from which you want to provision volume. Cool feature, Isnt it ?

* clusterid: 630372ccdc720a92c681fb928f27b53f is the ID of the cluster which will be used by Heketi when provisioning the volume. It can also be a list of clusterids, for ex:”8452344e2becec931ece4e33c4674e4e,42982310de6c63381718ccfa6d8cf397″. This is an optional parameter.

Please feel free to share your thoughts on these new features and also let me know if you would like to see any further enhancements for the gluster dynamic provisioner.!! I am watching this space for your comments.

[How to] GlusterFS Dynamic Volume Provisioner in Kubernetes (>= v1.4 ) / Openshift.

I am happy to share that Gluster Dynamic Volume Provisioner is available in kubernetes tree since 1.4 release !!

This is one of the feature which enables the User/Developer in Kubernetes/Openshift to have Persistent Volume Claim ( PVC ) request to be satisfied dynamically without an admin intervention. IMHO, it gives a nice user experience and its a good feature to have in container orchestrators. Till this feature came in, the persistent volumes in the store was created statically.

The static provisioning workflow looks like this:

Eventhough it was easy to perform the static provisioning of volumes, it has few limitations in my view.

*) The admin has to create the persistent volumes upfront and keep it in persistent store.

*) When a claim request comes to the controller, it check for the size of the request against the available PVs in the pool and if the ( available PV size >= size of the request ) it bind the claim.

The latter can lead to wastage of storage in most of the case.

These kind of limitations have been lifted with Dynamic provisioning. Now the admin define the storage classes and the user/developer request the persistent volumes using the storage class reference in the PVC request. The storage classes can pass the parameters of the plugin using the storage class key value pairs.

I have created the following diagram to simply the workflow of dynamic (Ex: GlusterFS) provisioning.

As you can see in the above diagram, the GlusterFS plugin in Kubernetes/Openshift make use of “Heketi” to provision GlusterFS volumes. If you want to know more about Heketi and how it can be used to manage gluster clusters, please refer this Wiki. In short, Heketi is a volume manager for GlusterFS clusters. It manage Gluster trusted pool and create volumes based on demand.

Lets start from Storage Class which allows us to do dynamic provisioning in kubernetes.

Here is an example of the storage class parameters:


resturl : Gluster REST service/Heketi service url which provision gluster volumes on demand. The general format should be IPaddress:Port and this is a mandatory parameter for GlusterFS dynamic provisioner. If Heketi service is exposed as a routable service in openshift/kubernetes setup, this can have a format similar to where the fqdn is a resolvable heketi service url.

restuser : Gluster REST service/Heketi user who has access to create volumes in the Gluster Trusted Pool.

secretNamespace + secretName : Identification of Secret instance that containes user password to use when talking to Gluster REST service. These parameters are optional, empty password will be used when both secretNamespace and secretName are omitted.

for more details on these parameters please refer

To summarize, user/developer request the persistent storage using the claim and mention the storage class which need to be used/mapped with the claim. As soon as the claim request comes in, the GlusterFS plugin in kubernetes create a volume with the requested size and BIND the persistent volume to Claim. When there is a request to delete the claim, the subjected volume is deleted from the backend gluster trusted pool. The glusterfs plugin in kubernetes make use of ‘Heketi’ to provision a volume dynamically.

Here is the demo video of Dynamic GlusterFS provisioner in Kubernetes.

As always comments/suggestions/questions are welcome.. 🙂

[ Berlin ] Gluster Summit 2016 – An awesome event

This year’s gluster summit was held at Berlin ( Oct 6th and 7th – 2016) and I am happy to be part of it. An awesome city and great folks around.

There were lots of presentations about Gluster and its upcoming features. If you would like to catch up with all the presentations please check this account in slideshare #

I gave a presentation in the same event with Luis on contenarized storage for cloud applications and received really good feedback about this. \o/

Here is the slide of the presentation.

[slideshare id=67284723&doc=glusterfssummit2016-161017120703]

Gluster Client Containers or containers capable of mounting Gluster Volumes.

I have been receiving lots of queries on whether we can mount GlusterFS from the container ? or Gluster Client containers are available ?
Yes, it is. Today I refreshed this image to fedora24, so the blog.
The process to use gluster client containers are simple as shown below:

[Update: the same docker image can be found at ]

Then run the container as

Make sure “FUSE” device exist in the container as shown below.

I have a gluster volume exported from another server and would like to mount it inside this container.

Thats it.

I will automate the build in docker hub and move this image to gluster official account soon.

Gluster Container Demo Videos on “Gluster as a Persistent Data Store for Containers…. “

Recently I got a chance to consolidate the Demo videos which covers how GlusterFS can be used in Docker, kubernetes and Openshift. I have placed everything in single channel for better tracking.

Here is a brief description about the available videos:

How to run a Gluster Container :

Reference # Blog – Presentation –

This demo covers the standard process to run Gluster Docker Containers in a Linux system. It shows which directories can be exported when spawning Gluster Containers to make sure data persistence. This demo also talks about the usage of official Gluster Container image.

How to form a Gluster trusted pool using Gluster Containers:

This demo shows how we can create a Gluster trusted pool (cluster) among Gluster containers. This is one of the important building blocks of Running storage in container or to be precise, “Storage as a Service” in my terms.

Gluster Container deployment in Openshift using templates-DaemonSets

Reference #

Once we are confident that we can run gluster containers and form a trusted pool, the next step would be deploying it in PaaS like offering. Obviously I chose Openshift PaaS for this demo because it is an awesome PaaS based on Kubernetes.

“OpenShift is Red Hat’s Platform-as-a-Service (PaaS) that allows developers to quickly develop, host, and scale applications in a cloud environment. With OpenShift you have a choice of offerings, including online, on-premise, and open source project options.” .

OpenShift has a deployment model called ‘template deployment’ which will be a part of this demo as well. This demo also covers one of the deployment option ‘DaemonSets’ to deploy gluster containers/pods in Openshift.

Gluster Persistent Storage in Kubernetes/Openshift OR PV and PVC using GlusterFS plugin

Reference #

Blog – Presentation –

This demo shows us a way to use Gluster trusted pool and Gluster volumes in an OpenShift or Kubernetes environment. The trusted pool may or may not be hosted in containers. The demo covers the Endpoint, Service, Persistent Volume and Persistent Volume Claim creation based on GlusterFS trusted pool and volumes. Once we have a ‘BOUND” claim, one can use it in application pod/container for Persistent Data Store.

Gluster Pods deployment with PetSets for consistent pod name

In general the pods get a random name in an Openshift or Kubernetes environment. At times we may need to assign defined names or ordered names for the Gluster Pods. There is a new attribute ‘PetSets‘ in kubernetes which enables this functionality. Here we use “PetSets’ in this demo to show how we can get defined names for Gluster Containers and to show how this PetSet pods can be controlled or scaled as well.

Stay tuned, there’s more to come in this space.