Why GlusterFS is contenarized ? Advantages ?

First of all GlusterFS is a userspace file system. Containers are designed for ‘user space applications’ , Isnt it? Once you contenarize your user space application, you get many advantages, so GlusterFS containers.

If I quote the advantages of Container ( for ex: docker ) from this link:

Docker brings in an API for container management, an image format and a possibility to use a remote registry for sharing containers. This scheme benefits both developers and system administrators with advantages such as:

Rapid application deployment – containers include the minimal runtime requirements of the application, reducing their size and allowing them to be deployed quickly.

Portability across machines – an application and all its dependencies can be bundled into a single container that is independent from the host version of Linux kernel, platform distribution, or deployment model. This container can be transfered to another machine that runs Docker, and executed there without compatibility issues.

Version control and component reuse
– you can track successive versions of a container, inspect differences, or roll-back to previous versions. Containers reuse components from the preceding layers, which makes them noticeably lightweight.

Sharing – you can use a remote repository to share your container with others. Red Hat provides a registry for this purpose, and it is also possible to configure your own private repository.

Lightweight footprint and minimal overhead
– Docker images are typically very small, which facilitates rapid delivery and reduces the time to deploy new application containers.

Simplified maintenance – Docker reduces effort and risk of problems with application dependencies.

Apart from above, we closely work on Container Operating systems like “Atomic host” which designed as container platform/OS to run your application containers, so GlusterFS can. It is not possible to use package managers like rpm and setup the system in your own way if you use this stripped container OSs. If we want to take advantage of these OSs, we have to be contenarized.

If the application is in a container, we get many advantages which comes with the container orchestration software like Kubernetes/Openshift. So, let Gluster take that advantage as well. For ex: if you deploy GlusterFS in a baremetal system there is no piece of code which monitor GlusterFS and if something goes wrong, admin intervention is required to bring it back, but if the gluster is contenarized the orchestrator does it for you like any other application.

Let us look at the deployment part, if you have to deploy GlusterFS in a new Kubernetes/Openshift node, you dont have to worry about the ‘preparation/setup’, ie Setting up the repositories, Installation of packages ..etc rather just label ( in case of DeamonSet deployment model in Kube/Openshift ) a node, You got new Gluster Node within seconds.

This list goes, but I have to stop.

I would like to wrap this article saying, the attempt of gluster containerization has been justified by the massive download of GlusterFS containers from docker hub.

hub.docker.com/r/gluster/gluster-centos/

Persistent Volume and Claim in OpenShift and Kubernetes using GlusterFS Volume Plugin

OpenShift is a platform as a service product from Red Hat. The software that runs the service is open-sourced under the name OpenShift Origin, and is available on GitHub.

OpenShift v3 is a layered system designed to expose underlying Docker and Kubernetes concepts as accurately as possible, with a focus on easy composition of applications by a developer. For example, install Ruby, push code, and add MySQL.

Docker is an open platform for developing, shipping, and running applications. With Docker you can separate your applications from your infrastructure and treat your infrastructure like a managed application. Docker does this by combining kernel containerization features with workflows and tooling that help you manage and deploy your applications. Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. Available on GitHub.

Kubernetes is an open-source system for automating deployment, operations, and scaling of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon a decade and a half of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community. Available on GitHub.

GlusterFS is a scalable network filesystem. Using common off-the-shelf hardware, you can create large, distributed storage solutions for media streaming, data analysis, and other data- and bandwidth-intensive tasks. GlusterFS is free and open source software. Available on GitHub.

Hope you know a little bit of all the above Technologies, now we jump right into our topic which is Persistent Volume and Persistent volume claim in Kubernetes and Openshift v3 using GlusterFS volume. So what is Persistent Volume? Why do we need it? How does it work using GlusterFS Volume Plugin?

In Kubernetes, Managing storage is a distinct problem from managing compute. The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this we introduce two new API resources in kubernetes: PersistentVolume and PersistentVolumeClaim.

A PersistentVolume (PV) is a piece of networked storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g, can be mounted once read/write or many times read-only).

In simple words, Containers in Kubernetes Cluster need some storage which should be persistent even if the container goes down or no longer needed. So Kubernetes Administrator creates a Storage(GlusterFS storage, In this case) and creates a PV for that storage. When a Developer (Kubernetes cluster user) needs a Persistent Volume in a container, creates a Persistent Volume claim. Persistent Volume Claim will contain the options which Developer needs in the pods. So from list of Persistent Volume the best match is selected for the claim and Binded to the claim. Now the developer can use the claim in the pods.


Prerequisites:

Need a Kubernetes or Openshift cluster, My setup is one master and three nodes.

Note: you can use kubectl in place of oc, oc is openshift controller which is a wrapper around kubectl. I am not sure about the difference.

2) Have a GlusterFS cluster setup, Create a GlusterFS Volume and start the GlusterFS volume.

3) All nodes in kubernetes cluster must have GlusterFS-Client Package installed.

Now we have the prerequisites \o/ …

In Kube-master administrator has to write required yaml file which will be given as input to the kube cluster.

There are three files to be written by administrator and one by Developer.

Service
Service Keeps the endpoint to be persistent or active.
Endpoint
Endpoint is the file which points to the GlusterFS cluster location.
PV
PV is Persistent Volume where the administrator will define the gluster volume name, capacity of volume and access mode.
PVC
PVC is persistent volume claim where developer defines the type of storage as needed.

STEP 1: Create a service for the gluster volume.

Verify:

STEP 2: Create an Endpoint for the gluster service

The ip here is the glusterfs cluster ip.

STEP 3: Create a PV for the gluster volume.

Note : path here is the gluster volume name. Access mode specifies the way to access the volume. Capacity has the storage size of the GlusterFS volume.

STEP 4: Create a PVC for the gluster PV.

Note: the Developer request for 8 Gb of storage with access mode rwx.

Here the pvc is bounded as soon as created, because it found the PV that satisfies the requirement. Now lets go and check the pv status

See now the PV has been bound to “default/glusterfs-claim”. In this state developer has the Persistent Volume Claim bounded successfully, now the developer can use the pv claim like below.

STEP 5: Use the persistent Volume Claim in a Pod defined by the Developer.

The above pod definition will pull the humble/gluster-client image(some private image) and start init script. The gluster volume will be mounted on the host machine by the GlusterFS volume Plugin available in the kubernetes and then bind mounted to the container’s /home. So all the Kubernetes cluster nodes must have glusterfs-client packages.

Lets try running.

Wow its running… lets go and check where it is running.

Found the Pod running successfully on one of the Kubernetes node.

On the host:

I can see the gluster volume being mounted on the host \o/. Lets check inside the container. Note the random number is the container-id from the docker ps command.

Yippy the GlusterFS volume has been mounted inside the container on /home as mentioned in the pod definition. Lets try writing something to it

Since the AccessMode is RWX I am able to write to the mount point.

That’s all Folks.

GlusterFS containers in Kubernetes Cluster for persistent data store !!

Everything is containerized, so Gluster . As you know, Gluster Container images are available for long time ( for both CentOS and Fedora ) in Docker hub. In previous blog posts, we saw how to build/run Gluster Containers. In this setup, we will try to setup a kubernetes cluster with Gluster containers. If you dont know much about kubernetes , please go through this . In short, kubernetes is an orchestration software for container environment which brings the services like scheduling, service discovery..etc. We will deploy a kubernetes cluster in couple of atomic nodes. Then run Gluster containers on these atomic hosts via kubernetes. Once the gluster containers are running, we will form a trusted pool out of these gluster containers and export a volume, so that other application containers can make use of this volume to store its data in persistent way!! .

kubernetes

Sounds interesting ? Yes, let us start.

NOTE: This article also discuss the steps to configure etcd server ( a key value store).
. For this particular setup we may not need to configure etcd. However your environment may need, for example to configure flannel.

Setup

Three centos ( You can also use fedora/RHEL) atomic hosts :


centos-atomic-KubeMaster
centos-atomic-Kubenode1
centos-atomic-Kubenode2

To configure/install CentOS atomic hosts, please follow the steps mentioned here.
and the atomic images can be downloaded from here

Then start the atomic installation, if cloud init is configured, it will come into play and ask for “atomic host” login.

username: centos
password: atomic

Note: Above is based on the cloud-init configuration. If you have customized the cloud init configuration for different username and password, please supply the same. (wait till the vm to completely load meta-data and user-data. else it will throw invalid login till its completely loaded)

At this stage we have three atomic hosts.:


10.70.42.184 centos-atomic-KubeMaster
10.70.42.29 centos-atomic-Kubenode1
10.70.43.88 centos-atomic-Kubenode2

If you already have this setup, make sure all the machines are able to talk to each other.

First things first,

-bash-4.2# atomic host upgrade

Upgrade your system to latest docker, etcd, kubernetes..etc, in all nodes.
With the three systems in place, the next thing is to set up Kubernetes. Setting up Kubernetes on the Master, select any system to be master.

1. Etcd configuration:
Edit the /etc/etcd/etcd.conf. The etcd service needs to be configured to listen on all interfaces to ports 2380. (ETCD_LISTEN_PEER_URLS) and port 2379 (ETCD_LISTEN_CLIENT_URLS), and listen on 2380 on localhost (ETCD_LISTEN_PEER_URLS)

-bash-4.2# cat /etc/etcd/etcd.conf | grep -v "#"
ETCD_NAME=default
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

2. Kubernetes Configuration:

Edit the /etc/kubernetes/config file and change the KUBE_MASTER line to identify the location of your master server (it points to 127.0.0.1, by default). Leave other settings as they are.

KUBE_MASTER="--master=http://10.70.42.184:8080"

3. Kubernetes apiserver Configuration:

Edit the /etc/kubernetes/apiserver and add a new KUBE_ETCD_SERVERS line (as shown below), then review and change other lines in the apiserver configuration file. Change KUBE_API_ADDRESS to listen on all network addresses(0.0.0.0), instead of just localhost. Set an address range for the KUBE_SERVICE_ADDRESS that Kubernetes can use to assign to services (see a description of this address below). Finally, remove the term “ServiceAccount” from the KUBE_ADMISSION_CONTROL instruction.


-bash-4.2# cat /etc/kubernetes/apiserver | grep -v "#"
KUBE_API_ADDRESS="--address=0.0.0.0"
KUBE_ETCD_SERVERS="--etcd_servers=http://10.70.42.184:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.100.0/24"
KUBE_ADMISSION_CONTROL="--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
KUBE_API_ARGS=""

4. Start master services:

To run the Kubernetes master services, you need to enable and start several systemd services. From the master, run the following for loop to start and enable Kubernetes systemd services on the master:


-bash-4.2# for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do
systemctl restart $SERVICES;
systemctl enable $SERVICES;
systemctl status $SERVICES;
done

5. Setting up Kubernetes on the Nodes

On each of the two Kubernetes nodes, you need to edit several configuration files and start and enable several Kubernetes systemd services:

1.Edit /etc/kubernetes/config:

Edit the KUBE_MASTER line in this file to identify the location of your master (it is 127.0.0.1, by default). allow_privileged must be set to true. Leave other settings as they are.


KUBE_ALLOW_PRIV="--allow_privileged=true"
KUBE_MASTER="--master=http://10.70.42.184:8080"

2.Edit /etc/kubernetes/kubelet:

In this file on each node, modify KUBELET_ADDRESS (0.0.0.0 to listen on all network interfaces), KUBELET_HOSTNAME (replace hostname_override with the hostname or IP address of the local system). You may leave this blank to use the actual hostname, set KUBELET_ARGS, and KUBELET_API_SERVER as below. --host-network-sources=* is specified to use the host networking option of docker(–net=host). You can use any networking mode of docker. However in this setup, we use --net=host option to make sure we get maximum performance.

-bash-4.2# cat /etc/kubernetes/kubelet | grep -v "#"
KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_HOSTNAME="--hostname_override="
KUBELET_API_SERVER="--api_servers=http://10.70.42.184:8080"
KUBELET_ARGS="--register-node=true --host-network-sources=*"

3. Edit /etc/kubernetes/proxy:
No settings are required in this file. If you have set KUBE_PROXY_ARGS, you can comment it out:

-bash-4.2# cat /etc/kubernetes/proxy
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
#KUBE_PROXY_ARGS="--master=http://master.example.com:8080"

4. Start the Kubernetes nodes systemd services:

On each node, you need to start several services associated with a Kubernetes node:


-bash-4.2# for SERVICES in docker kube-proxy.service kubelet.service; do
systemctl restart $SERVICES;
systemctl enable $SERVICES;
systemctl status $SERVICES; done

5. Check the services:
Run the netstat command on each of the three systems to check which ports the services are running on. The etcd service should only be running on the master.

From master:


-bash-4.2# netstat -tulnp | grep -E "(kube)|(etcd)"
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 17805/kube-schedule
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 17764/kube-controll
tcp6 0 0 :::6443 :::* LISTEN 17833/kube-apiserve
tcp6 0 0 :::2379 :::* LISTEN 17668/etcd
tcp6 0 0 :::2380 :::* LISTEN 17668/etcd
tcp6 0 0 :::8080 :::* LISTEN 17833/kube-apiserve

From nodes:


-bash-4.2# netstat -tulnp | grep kube
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 104398/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 104331/kube-proxy
tcp6 0 0 :::10250 :::* LISTEN 104398/kubelet
tcp6 0 0 :::57421 :::* LISTEN 104331/kube-proxy
tcp6 0 0 :::10255 :::* LISTEN 104398/kubelet
tcp6 0 0 :::34269 :::* LISTEN 104331/kube-proxy
tcp6 0 0 :::58239 :::* LISTEN 104331/kube-proxy
tcp6 0 0 :::4194 :::* LISTEN 104398/kubelet
Continue reading "GlusterFS containers in Kubernetes Cluster for persistent data store !!"

GlusterFS Bangalore Workshop

An introductory workshop on GlusterFS was held at Red Hat, Bangalore office (www.meetup.com/glusterfs-India/events/225770608/ ) on 31st October. Around 15 people turned up for the event. There were admins, DevOps and a few developers. As requested on the meetup page, the participants had turned up with their laptops to experience the awesomeness of Gluster!!

We started with a self introduction. Then CentOS VM images containing pre-installed gluster packages were distributed. While the participants were deploying VM images, a brief introductory talk on GlusterFS was held. Prashanth Pai explained basic Gluster terminologies, various types of volumes, replication, geo-replication etc.

Gluster_Wokshop

This was followed by a basic demo by Poornima which taught the participants to create a cluster, create different types of volumes, mount volumes etc.

Gluster_workshop

Few participants were keen on using GlusterFS volumes over NFS and also in other cloud use-cases such as setting up GlusterFS + NFS HA cluster in AWS deployment. Further, Aravinda explained the advantages of using geo-replication for async replication.

gluster_2

The workshop was overall well received and we were glad to receive requests to conduct more GlusterFS Workshops!! You can read some feedback about this event here. Thanks to all speakers & attendees.

gluster_1

ps# If you would like to receive notifications about the upcoming GlusterFS meetups/workshops happening in India, join our meetup group www.meetup.com/glusterfs-India.

Gluster volume plugin of docker !!

Does gluster volume plugin available for docker?

Yes, its available here .

This article talks about how to use this plugin and make use of gluster volume when spawning docker containers.

For the gluster volume plugin to work, we need experimental build of docker which can be fetched from docker github. If you dont have the experimental binary of docker running in your system get it from docker github.

github.com/docker/docker/tree/master/experimental have instructions on how to run docker experimental binary.

Once your docker daemon is running from experimental build, pull gluster volume plugin from github source.


[root@dhcp35-20 go]# go get github.com/calavera/docker-volume-glusterfs

As mentioned in the README file in github, you need to execute ‘docker-volume-glusterfs’ as shown below. That said, here the IP, “10.70.1.100” is my gluster server which export a replica volume called ‘test-vol’. For more details on gluster volume types and configuration please refer gluster.readthedocs.org/en/latest/ .


[root@dhcp35-20 go]# docker-volume-glusterfs -servers 10.70.1.100


[root@dhcp35-20 check1]# ps aux |grep docker
root 7674 0.0 0.0 7612 1596 pts/13 Sl+ 12:47 0:00 ./docker-volume-glusterfs -servers 10.70.1.100
root 8169 0.0 0.3 558828 29924 pts/14 Sl 12:52 0:00 ./docker-latest daemon

Once its done, we can spawn containers as shown below, where ‘test-vol’ is the gluster volume name and “/b1” is the mount point in spawned container, ‘docker.io/fedora’ is the image name.

‘touch /b1/second” create a file called ‘second’ in “/b1” mount point.

[root@]# ./docker-latest run -it –volume-driver glusterfs –volume test-vol:/b1 docker.io/fedora touch /b1/second


INFO[4891] POST /v1.21/containers/create
INFO[4892] POST /v1.21/containers/b3b61146188db97e3b2c96e1ae38dc53478287d557e24a26b0dcbf09be68140a/attach?stderr=1&stdin=1&stdout=1&stream=1
INFO[4892] POST /v1.21/containers/b3b61146188db97e3b2c96e1ae38dc53478287d557e24a26b0dcbf09be68140a/start
INFO[4892] POST /v1.21/containers/b3b61146188db97e3b2c96e1ae38dc53478287d557e24a26b0dcbf09be68140a/resize?h=46&w=190
INFO[4892] GET /v1.21/containers/b3b61146188db97e3b2c96e1ae38dc53478287d557e24a26b0dcbf09be68140a/json

Lets verify whether the file creation worked successfully and the new file (second) is available in brick path of gluster server node.

From ‘test-vol’ volume details, we can see that “/home/test-brick1” is one leg of replica volume in my setup.


Volume Name: test-vol
Type: Replicate
Volume ID: 2cebb33f-e849-40c1-9344-939025f80b1f
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.70.1.100:/home/test-brick1
Brick2: 10.70.1.101:/home/test-brick2
Options Reconfigured:
performance.readdir-ahead: on
You have new mail in /var/spool/mail/root
[root@dhcp1-100 test-brick1]#

[root@dhcp1-100 test-brick1]# pwd
/home/test-brick1
[root@dhcp1-100 test-brick1]# ls
second

Awesome !! Isnt it ?

Thanks github.com/calavera for the plugin & Thanks neependra for pointers.

Gluster 3.7 upgrade in EL systems.

If you are upgrading your Gluster 3.6 EL systems ( RHEL, SL..etc) to Gluster 3.7, you may come into package dependency issues.

For ex: in EL6 system:

error: Failed dependencies:
pkgconfig(sqlite3) is needed by
glusterfs-api-devel-3.7.0-1.el6.x86_64
pkgconfig(uuid) is needed by
glusterfs-api-devel-3.7.0-1.el6.x86_64
python-argparse is needed by glusterfs-server-3.7.0-1.el6.x86_64
pyxattr is needed by glusterfs-server-3.7.0-1.el6.x86_64
liburcu-bp.so.1()(64bit) is needed by
glusterfs-server-3.7.0-1.el6.x86_64
liburcu-cds.so.1()(64bit) is needed by
glusterfs-server-3.7.0-1.el6.x86_64

The above listed dependencies ( pyxattr, python-argparse, userspace-rcu..etc) are part of EPEL repo. You can enable EPEL repo in your system by executing below command.

rpm -ivh http://dl.fedoraproject.org/pub/epel/epel-release-latest-${RHEL_VERSION}.noarch.rpm

NOTE: Replace ‘RHEL_VERSION’ with ‘5’,’6′, or ‘7’.

Once EPEL repo is enabled it will resolve the dependencies.