[GlusterFS Dynamic Provisioner] Online Resizing of GlusterFS PVs in kubernetes v>= 1.8 !!!

While the kubernetes storage-sig keeps adding new features in each and every release, we also listen to our user feedback to improve the existing storage interfaces in solving the existing limitations. Previously, the admin had to do proper capacity planning in order to use persistent Volumes as the microservices or the application pods wanted to …

Read more

“Retain” ( PV claim policy) GlusterFS Dynamically Provisioned Persistent Volumes in Kubernetes >=1.8

Since introduction, the dynamic provisioning feature of Kube storage defaulted with reclaim policy as Delete. It was not specific to glusterfs PVs rather common for all PVs which are dynamically provisioned in kube storage. However from kube >= v1.8, we could specify retain policy in storage class! A much-needed functionality in SC for different use …

Read more

[Gluster-Block] Provision Gluster Block Volumes In Kubernetes/Openshift – ( Part 1)

Voila…. Its almost a couple of years GlusterFS keep rocking in Container Storage Space which brought the buzz word “Container Native Storage” to the emerging market!! If you dont know about what I mean by “Container Native Storage” , check out this link GlusterFS was one of the plugin in kubernetes tree which implemented Dynamic …

Read more

[GlusterFS Dynamic Provisioner] Set GlusterFS Volume Options via StorageClass parameter

Much awaited RFE/Enhancement to the GlusterFS provisioner in Kubernetes !! Its been long time kubernetes/Openshift users are looking for a feature/functionality to enable various GlusterFS volume options via storage class. We discussed on few methods to implement this in Heketi. One thought was to wrap options under ‘classified’ strings, for eg# when someone asked for …

Read more

Kubernetes 101 for Bangalore Kubernauts .

We keep conducting Kubernetes and Openshift meetups as part of https://www.meetup.com/kubernetes-openshift-India-Meetup/ .

We see great momentum in this meetup group and lots of enthusiam around this. The last events were well received and lots of requests came in to have a hands on training on Kubernetes and Openshift. If you are still to catch up with these emerging technologies, dont delay, just join us in next event planned on 20-May-2017.

More details about this event can be found @ https://www.meetup.com/kubernetes-openshift-India-Meetup/events/239381714/

This will be a beginner oriented workshop. As a pre requisite, you need to install linux in your laptop, thats it.

We are also looking for volunteers to help and venue to conduct this event.

Please RSVP and let us know if you would like to help us on organizing this event.

[2020 updated] ISCSI multipath support in Kubernetes (v1.6) or Openshift.

Multipathing is an important feature in most of the storage setups, so for ISCSI based storage systems.
In most of the ISCSI based storages, multiple paths can be defined as the same IQN shared with more than one portal IPs.
Even if one of the network interface/portal is down, the share/target can be accessed via other active interfaces/portals.
This is indeed a good feature considering I/O from ISCSI initiator to Target and high availability of data path.
However the ISCSI plugin in Kubernetes was not capable of making use of multipath feature and it
was always just one path configured by default in Kubernetes. If that path goes down, the target can not be accessed.

Recently I added multipath support to ISCSI kubernetes plugin with this Pull Request.

With this functionality, a Kubernetes user/admin can specify the other Target Portal IPs in a new field called portals in ISCSI volume. That’s the only change required from admin side!!. If there are multiple portals, admin can mention these additional target portals in the portals field as shown below.

The new structure will look like this.

iscsi: targetPortal: 10.0.2.15:3260 + portals: [‘10.0.2.16:3260’, ‘10.0.2.17:3260’] iqn: iqn.2001-04.com.example:storage.kube.sys1.xyz lun: 0 fsType: ext4 readOnly: true

If you are directly using the above volume definition in POD spec, your pod spec may look like this.

apiVersion: v1 kind: Pod metadata: name: iscsipd spec: containers: – name: iscsipd-rw image: kubernetes/pause volumeMounts: – mountPath: “/mnt/iscsipd” name: iscsipd-rw volumes: – name: iscsipd-rw iscsi: targetPortal: 10.0.2.15:3260 portals: [‘10.0.2.16:3260’, ‘10.0.2.17:3260’, ‘10.0.2.18:3260’] iqn: iqn.2016-04.test.com:storage.target00 lun: 0 fsType: ext4 readOnly: true

Once the pod is up and running, you could check and verify below outputs from the Kubernetes Host where the pod is running:

# multipath -ll mpatha (360014059bafbe58ba644b2889c34903f) dm-2 LIO-ORG ,disk01 size=20G features=’0′ hwhandler=’0′ wp=rw |-+- policy=’service-time 0′ prio=1 status=active | – 44:0:0:0 sdb 8:16 active ready running -+- policy=’service-time 0′ prio=1 status=enabled `- 42:0:0:0 sdc 8:32 active ready running -+- policy=’service-time 0′ prio=1 status=enabled `- 46:0:0:0 sdd 8:48 active ready running -+- policy=’service-time 0′ prio=1 status=enabled `- 48:0:0:0 sde 8:64 active ready running

ISCSI session looks like below:

# iscsiadm -m session tcp: [10] 10.0.2.15:3260,1 iqn.2016-04.test.com:storage.target00 (non-flash) tcp: [12] 10.0.2.16:3260,1 iqn.2016-04.test.com:storage.target00 (non-flash) tcp: [14] 10.0.2.17:3260,1 iqn.2016-04.test.com:storage.target00 (non-flash) tcp: [16] 10.0.2.18:3260,1 iqn.2016-04.test.com:storage.target00 (non-flash)

The device paths:

# ll /dev/disk/by-path/ lrwxrwxrwx. 1 root root 9 Feb 16 15:58 ip-10.0.2.15:3260-iscsi-iqn.2016-04.test.com:storage.target00-lun-0 -> ../../sdb lrwxrwxrwx. 1 root root 9 Feb 16 15:41 ip-10.0.2.16:3260-iscsi-iqn.2016-04.test.com:storage.target00-lun-0 -> ../../sdc lrwxrwxrwx. 1 root root 9 Feb 16 15:58 ip-10.0.2.17:3260-iscsi-iqn.2016-04.test.com:storage.target00-lun-0 -> ../../sdd lrwxrwxrwx. 1 root root 9 Feb 16 15:41 ip-10.0.2.18:3260-iscsi-iqn.2016-04.test.com:storage.target00-lun-0 -> ../../sde

I believe, this is a nice feature added to Kubernetes ISCSI storage. Please let me know your comments/feedback/suggestions on this.

[Gluster Dynamic Provisioner ] “glusterfs: failed to get endpoints..” , Do I need to create an endpont/service ?

Let me give more details about the endpoints or service wrt GlusterFS plugin in Kubernetes or Openshift. GlusterFS plugin is designed in such a way that it need a mandatory parameter ‘endpoint’ in its spec. Endpoint is same instance of

api.endpoint in kube/openshift, ie it need IP addresses. In gluster plugin, we carry IP address of gluster nodes in the endpoint. When we manually create a PV we also need to create an endpoint and a headless service for the PV. I could call this state as 'static provisioning'. This is tedious, as admin want to fetch the nodes and then create endpoints and keep it for the developer/user. We also heard same concern from community users about the difficulty of creating endpoints in each namespace where we want to run the app pods which use the gluster PVs. At the same time, there was some user reports where they liked the isolation it brings. 

We tried to avoid this dependency of endpoints and thought about different designs to overcome this. It was bit difficult due to reasons like backward compatibility, security concerns..etc. But when we introduced dynamic provisioning of Gluster PVs which is available from Kube 1.4 version or from Openshift 3.4 version, this situation has changed. It is no longer a pain for admin to create

an endpoint/service based on the volume which just got created. The dynamic provisioner will create the endpoint and service automatically based on the cluster where the volume is created. 

The entire workflow has been made easy from user/admin point of view. The endpoint and service will be created in PVC namespace. If a user want to make use of this PVC in his application pod, the endpoint/service has to be in this namespace which is available with dynamic provisioning without any extra effort. In short, user/developer dont have to worry about the PV which should have the mention of endpoint.

I hope this helps.

Please let me know if you need more details on this or any comments on this..

Support for “Volume Types” option in Kubernetes GlusterFS dynamic provisioner.

Till now, there was no option to specify various volume types and specifications of dynamically provisioned volumes in Kubernetes or Openshift. The main reason for that being we always recommend to have a ‘replica 3′ volume whenever it provision.

However there were requests from users of GlusterFS dynamic provisioner to have this functionality for various use cases, for example, someone want to setup a quick demo by having a replica 1 or replica 2 volume in a small cluster. Sometimes they want to create an EC volume via storage class…etc.

This functionality has been added some time back to Kubernetes upstream and provide the choice to select volume type in kubernetes/Openshift via Storage Class parameter.

This has been added in below format to the gluster plugins’ storage class.

volumetype : The volume type and it’s parameters can be configured with this optional value. If the volume type is not mentioned, it’s up to the provisioner to decide the volume type.

For example: ‘Replica volume’: volumetype: replicate:3 where ‘3’ is replica count.

‘Disperse/EC volume’: volumetype: disperse:4:2 where ‘4’ is data and ‘2’ is the redundancy count.

‘Distribute volume’: volumetype: none

Based on above mention, the volumes will be created in the trusted storage pool .

The storage class file will look like this:

apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: fast provisioner: kubernetes.io/glusterfs parameters: resturl: “http://127.0.0.1:8081” restuser: “admin” secretNamespace: “default” secretName: “heketi-secret” volumetype: “replicate:2”

Please let me know if you have any comments/suggestions/feedback about this functionality or if you would like to see any other enhancements to Gluster Dynamic Provisioner in Kubernetes/Openshift.

“libgfapi-python” is now available as Fedora rpm. ( python-glusterfs-api)

I wanted to see this as distro package for long time, but it happened very recently. Eventhough there had a package review request, I couldnt follow up and get it completed. There was few other thoughts on this which also caused the delay. Any way, with the help of kaleb Keithely and Prasanth Pai it is now available in Fedora !.

There were many users wanted this rpm/package in distributions like fedora to make use of libgfapi python bindings and to become consumers of this api client.

[root@dhcp35-111 ]# yum install python-glusterfs-api
Redirecting to '/usr/bin/dnf install python-glusterfs-api' (see 'man yum2dnf')

Last metadata expiration check: 0:12:02 ago on Mon Jan 30 00:57:52 2017.
Dependencies resolved.
==============================================================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================================================
Installing:
python2-glusterfs-api noarch 1.1-2.fc24 updates 48 k

Transaction Summary
==============================================================================================================================================================================================
Install 1 Package

Total download size: 48 k
Installed size: 182 k
Is this ok [y/N]: y
Downloading Packages:
python2-glusterfs-api-1.1-2.fc24.noarch.rpm 13 kB/s | 48 kB 00:03
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 8.8 kB/s | 48 kB 00:05
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Installing : python2-glusterfs-api-1.1-2.fc24.noarch 1/1
Verifying : python2-glusterfs-api-1.1-2.fc24.noarch 1/1

Installed:
python2-glusterfs-api.noarch 1.1-2.fc24

Complete!
[root@dhcp35-111 ]# rpm -ql python-glusterfs-api
package python-glusterfs-api is not installed
[root@dhcp35-111 ]# rpm -ql python2-glusterfs-api
/usr/lib/python2.7/site-packages/gfapi-1.1-py2.7.egg-info
/usr/lib/python2.7/site-packages/gfapi-1.1-py2.7.egg-info/PKG-INFO
/usr/lib/python2.7/site-packages/gfapi-1.1-py2.7.egg-info/SOURCES.txt
/usr/lib/python2.7/site-packages/gfapi-1.1-py2.7.egg-info/dependency_links.txt
/usr/lib/python2.7/site-packages/gfapi-1.1-py2.7.egg-info/top_level.txt
/usr/lib/python2.7/site-packages/gluster
/usr/lib/python2.7/site-packages/gluster/gfapi
/usr/lib/python2.7/site-packages/gluster/gfapi/__init__.py
/usr/lib/python2.7/site-packages/gluster/gfapi/__init__.pyc
/usr/lib/python2.7/site-packages/gluster/gfapi/__init__.pyo
/usr/lib/python2.7/site-packages/gluster/gfapi/api.py
/usr/lib/python2.7/site-packages/gluster/gfapi/api.pyc
/usr/lib/python2.7/site-packages/gluster/gfapi/api.pyo
/usr/lib/python2.7/site-packages/gluster/gfapi/exceptions.py
/usr/lib/python2.7/site-packages/gluster/gfapi/exceptions.pyc
/usr/lib/python2.7/site-packages/gluster/gfapi/exceptions.pyo
/usr/lib/python2.7/site-packages/gluster/gfapi/gfapi.py
/usr/lib/python2.7/site-packages/gluster/gfapi/gfapi.pyc
/usr/lib/python2.7/site-packages/gluster/gfapi/gfapi.pyo
/usr/lib/python2.7/site-packages/gluster/gfapi/utils.py
/usr/lib/python2.7/site-packages/gluster/gfapi/utils.pyc
/usr/lib/python2.7/site-packages/gluster/gfapi/utils.pyo
/usr/share/doc/python2-glusterfs-api
/usr/share/doc/python2-glusterfs-api/README.rst
/usr/share/licenses/python2-glusterfs-api
/usr/share/licenses/python2-glusterfs-api/COPYING-GPLV2
/usr/share/licenses/python2-glusterfs-api/COPYING-LGPLV3

[root@dhcp35-111 ]# rpm -qi python2-glusterfs-api
Name : python2-glusterfs-api
Version : 1.1
Release : 2.fc24
Architecture: noarch
Install Date: Mon 30 Jan 2017 01:10:04 AM IST
Group : System Environment/Libraries
Size : 186261
License : GPLv2 or LGPLv3+
Signature : RSA/SHA256, Thu 19 Jan 2017 08:10:04 PM IST, Key ID 73bde98381b46521
Source RPM : python-glusterfs-api-1.1-2.fc24.src.rpm
Build Date : Thu 19 Jan 2017 08:03:32 PM IST
Build Host : buildvm-07.phx2.fedoraproject.org
Relocations : (not relocatable)
Packager : Fedora Project
Vendor : Fedora Project
URL : https://github.com/gluster/libgfapi-python
Summary : Python2 bindings for GlusterFS libgfapi
Description :
libgfapi is a library that allows applications to natively access
GlusterFS volumes. This package contains python bindings to libgfapi.

See http://libgfapi-python.rtfd.io/ for more details.

Please give a try and let us know your feedback!

Why GlusterFS is contenarized ? Advantages ?

First of all GlusterFS is a userspace file system. Containers are designed for ‘user-space applications’ , Isn’t it? Once you containerize your user space application, you get many advantages, so GlusterFS containers.

If I quote the advantages of Container ( for ex: docker ) from this link:

Docker brings in an API for container management, an image format, and a possibility to use a remote registry for sharing containers. This scheme benefits both developers and system administrators with advantages such as:

Rapid application deployment – containers include the minimal runtime requirements of the application, reducing their size and allowing them to be deployed quickly.

Portability across machines – an application and all its dependencies can be bundled into a single container that is independent from the host version of Linux kernel, platform distribution, or deployment model. This container can be transferred to another machine that runs Docker and executed there without compatibility issues.

Version control and component reuse – you can track successive versions of a container, inspect differences, or roll-back to previous versions. Containers reuse components from the preceding layers, which makes them noticeably lightweight.

Sharing – you can use a remote repository to share your container with others. Red Hat provides a registry for this purpose, and it is also possible to configure your own private repository.

Lightweight footprint and minimal overhead – Docker images are typically very small, which facilitates rapid delivery and reduces the time to deploy new application containers.

Simplified maintenance – Docker reduces effort and risk of problems with application dependencies.

Apart from above, we closely work on Container Operating systems like “Atomic host” which designed as a container platform/OS to run your application containers, so GlusterFS can. It is not possible to use package managers like rpm and set up the system in your own way if you use this stripped container OSs. If we want to take advantage of these OSs, we have to be containerized.

If the application is in a container, we get many advantages which come with the container orchestration software like Kubernetes/Openshift. So, let Gluster take that advantage as well. For ex: if you deploy GlusterFS in a bare metal system there is no piece of code that monitors GlusterFS and if something goes wrong, admin intervention is required to bring it back, but if the gluster is containerized the orchestrator does it for you like any other application.

Let us look at the deployment part, if you have to deploy GlusterFS in a new Kubernetes/Openshift node, you don’t have to worry about the ‘preparation/setup’, ie Setting up the repositories, Installation of packages ..etc rather just label ( in case of DeamonSet deployment model in Kube/Openshift ) a node, You got new Gluster Node within seconds.

This list goes, but I have to stop.

I would like to wrap this article saying, the attempt of gluster containerization has been justified by the massive download of GlusterFS containers from docker hub.

https://hub.docker.com/r/gluster/gluster-centos/