GlusterFS containers in Kubernetes Cluster for persistent data store !!

Everything is containerized, so Gluster . As you know, Gluster Container images are available for long time ( for both CentOS and Fedora ) in Docker hub. In previous blog posts, we saw how to build/run Gluster Containers. In this setup, we will try to setup a kubernetes cluster with Gluster containers. If you dont know much about kubernetes , please go through this . In short, kubernetes is an orchestration software for container environment which brings the services like scheduling, service discovery..etc. We will deploy a kubernetes cluster in couple of atomic nodes. Then run Gluster containers on these atomic hosts via kubernetes. Once the gluster containers are running, we will form a trusted pool out of these gluster containers and export a volume, so that other application containers can make use of this volume to store its data in persistent way!! .

kubernetes

Sounds interesting ? Yes, let us start.

NOTE: This article also discuss the steps to configure etcd server ( a key value store).
. For this particular setup we may not need to configure etcd. However your environment may need, for example to configure flannel.

Setup

Three centos ( You can also use fedora/RHEL) atomic hosts :


centos-atomic-KubeMaster
centos-atomic-Kubenode1
centos-atomic-Kubenode2

To configure/install CentOS atomic hosts, please follow the steps mentioned here.
and the atomic images can be downloaded from here

Then start the atomic installation, if cloud init is configured, it will come into play and ask for “atomic host” login.

username: centos
password: atomic

Note: Above is based on the cloud-init configuration. If you have customized the cloud init configuration for different username and password, please supply the same. (wait till the vm to completely load meta-data and user-data. else it will throw invalid login till its completely loaded)

At this stage we have three atomic hosts.:


10.70.42.184 centos-atomic-KubeMaster
10.70.42.29 centos-atomic-Kubenode1
10.70.43.88 centos-atomic-Kubenode2

If you already have this setup, make sure all the machines are able to talk to each other.

First things first,

-bash-4.2# atomic host upgrade

Upgrade your system to latest docker, etcd, kubernetes..etc, in all nodes.
With the three systems in place, the next thing is to set up Kubernetes. Setting up Kubernetes on the Master, select any system to be master.

1. Etcd configuration:
Edit the /etc/etcd/etcd.conf. The etcd service needs to be configured to listen on all interfaces to ports 2380. (ETCD_LISTEN_PEER_URLS) and port 2379 (ETCD_LISTEN_CLIENT_URLS), and listen on 2380 on localhost (ETCD_LISTEN_PEER_URLS)

-bash-4.2# cat /etc/etcd/etcd.conf | grep -v "#"
ETCD_NAME=default
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="http://0.0.0.0:2380"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

2. Kubernetes Configuration:

Edit the /etc/kubernetes/config file and change the KUBE_MASTER line to identify the location of your master server (it points to 127.0.0.1, by default). Leave other settings as they are.

KUBE_MASTER="--master=http://10.70.42.184:8080"

3. Kubernetes apiserver Configuration:

Edit the /etc/kubernetes/apiserver and add a new KUBE_ETCD_SERVERS line (as shown below), then review and change other lines in the apiserver configuration file. Change KUBE_API_ADDRESS to listen on all network addresses(0.0.0.0), instead of just localhost. Set an address range for the KUBE_SERVICE_ADDRESS that Kubernetes can use to assign to services (see a description of this address below). Finally, remove the term “ServiceAccount” from the KUBE_ADMISSION_CONTROL instruction.


-bash-4.2# cat /etc/kubernetes/apiserver | grep -v "#"
KUBE_API_ADDRESS="--address=0.0.0.0"
KUBE_ETCD_SERVERS="--etcd_servers=http://10.70.42.184:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.100.0/24"
KUBE_ADMISSION_CONTROL="--admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ResourceQuota"
KUBE_API_ARGS=""

4. Start master services:

To run the Kubernetes master services, you need to enable and start several systemd services. From the master, run the following for loop to start and enable Kubernetes systemd services on the master:


-bash-4.2# for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do
systemctl restart $SERVICES;
systemctl enable $SERVICES;
systemctl status $SERVICES;
done

5. Setting up Kubernetes on the Nodes

On each of the two Kubernetes nodes, you need to edit several configuration files and start and enable several Kubernetes systemd services:

1.Edit /etc/kubernetes/config:

Edit the KUBE_MASTER line in this file to identify the location of your master (it is 127.0.0.1, by default). allow_privileged must be set to true. Leave other settings as they are.


KUBE_ALLOW_PRIV="--allow_privileged=true"
KUBE_MASTER="--master=http://10.70.42.184:8080"

2.Edit /etc/kubernetes/kubelet:

In this file on each node, modify KUBELET_ADDRESS (0.0.0.0 to listen on all network interfaces), KUBELET_HOSTNAME (replace hostname_override with the hostname or IP address of the local system). You may leave this blank to use the actual hostname, set KUBELET_ARGS, and KUBELET_API_SERVER as below. --host-network-sources=* is specified to use the host networking option of docker(–net=host). You can use any networking mode of docker. However in this setup, we use --net=host option to make sure we get maximum performance.

-bash-4.2# cat /etc/kubernetes/kubelet | grep -v "#"
KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_HOSTNAME="--hostname_override="
KUBELET_API_SERVER="--api_servers=http://10.70.42.184:8080"
KUBELET_ARGS="--register-node=true --host-network-sources=*"

3. Edit /etc/kubernetes/proxy:
No settings are required in this file. If you have set KUBE_PROXY_ARGS, you can comment it out:

-bash-4.2# cat /etc/kubernetes/proxy
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
#KUBE_PROXY_ARGS="--master=http://master.example.com:8080"

4. Start the Kubernetes nodes systemd services:

On each node, you need to start several services associated with a Kubernetes node:


-bash-4.2# for SERVICES in docker kube-proxy.service kubelet.service; do
systemctl restart $SERVICES;
systemctl enable $SERVICES;
systemctl status $SERVICES; done

5. Check the services:
Run the netstat command on each of the three systems to check which ports the services are running on. The etcd service should only be running on the master.

From master:


-bash-4.2# netstat -tulnp | grep -E "(kube)|(etcd)"
tcp 0 0 127.0.0.1:10251 0.0.0.0:* LISTEN 17805/kube-schedule
tcp 0 0 127.0.0.1:10252 0.0.0.0:* LISTEN 17764/kube-controll
tcp6 0 0 :::6443 :::* LISTEN 17833/kube-apiserve
tcp6 0 0 :::2379 :::* LISTEN 17668/etcd
tcp6 0 0 :::2380 :::* LISTEN 17668/etcd
tcp6 0 0 :::8080 :::* LISTEN 17833/kube-apiserve

From nodes:


-bash-4.2# netstat -tulnp | grep kube
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 104398/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 104331/kube-proxy
tcp6 0 0 :::10250 :::* LISTEN 104398/kubelet
tcp6 0 0 :::57421 :::* LISTEN 104331/kube-proxy
tcp6 0 0 :::10255 :::* LISTEN 104398/kubelet
tcp6 0 0 :::34269 :::* LISTEN 104331/kube-proxy
tcp6 0 0 :::58239 :::* LISTEN 104331/kube-proxy
tcp6 0 0 :::4194 :::* LISTEN 104398/kubelet

6. Test the etcd service:
Use the curl command from any of the three systems to check that the etcd service is running and accessible: use the master node IP or hostname, run on all nodes.


-bash-4.2# sudo curl -s -L http://10.70.42.184:2379/version

7.Check the nodes and its status:

From the master,


-bash-4.2# kubectl get nodes
NAME LABELS STATUS
atomic-node1 kubernetes.io/hostname=atomic-node1 Ready
atomic-node2 kubernetes.io/hostname=atomic-node2 Ready

Pay attention to the status field of above command, it should be in ready status to run any containers to run. At this stage, we can say you have a working kubernetes cluster.

Cool, Isnt it ?

Few things to do before we start our containers.

GlusterFS peers will need access to local storage that can be used as the GlusterFS peer’s brick. We have to create a mount point, for example: /mnt/brick1 in nodes where we want to run the gluster containers. To specify the node where to run the containers in Kubernetes, we have to mention NodeSelector label to ensure that it is always scheduled on a specific node. As you know the container pod definition file will have these mentioned in it. Run the below command for your nodes to work with the yaml file configuration.

It sets the node names as a label to the nodes( in below command, key is name and value is worker-1). atomic-node1 is hostname of my system please use your respective name of the node.

atomic-node1 is my node1
atomic-node2 is my node2

In master:


-bash-4.2# kubectl label node atomic-node1 name=worker-1
-bash-4.2# kubectl label node atomic-node2 name=worker-2

Copy these yaml file: The sample yaml files are available here

-bash-4.2# cat gluster-1.yaml
apiVersion: v1
kind: Pod
metadata:
name: gluster-1
labels:
name: gluster-1
spec:
hostNetwork: true
nodeSelector:
name: worker-1
containers:
- name: glusterfs
image: gluster/gluster-centos
ports:
- name: web
containerPort: 80
volumeMounts:
- name: brickpath
mountPath: "/mnt/brick1"
securityContext:
capabilities: {}
privileged: true
volumes:
- name: brickpath
hostPath:
path: "/mnt/brick1"

-bash-4.2# cat gluster-2.yaml
apiVersion: v1
kind: Pod
metadata:
name: gluster-2
labels:
name: gluster-2
spec:
hostNetwork: true
nodeSelector:
name: worker-2
containers:
- name: glusterfs
image: gluster/gluster-centos
ports:
- name: web
containerPort: 80
volumeMounts:
- name: brickpath
mountPath: "/mnt/brick1"
securityContext:
capabilities: {}
privileged: true
volumes:
- name: brickpath
hostPath:
path: "/mnt/brick1"

Now that you have the yaml files which specifies the pods structure. If you dont know what is pod in kubernetes, please refer this url.

Finally it is the time for us to create Gluster containers!

In master:


-bash-4.2# kubectl create -f gluster-1.yaml
pods/gluster-1

-bash-4.2# kubectl create -f gluster-2.yaml
pods/gluster-2

Awesome ! gluster containers started in kubernetes nodes. Lets verify its status by running below command.


-bash-4.2# kubectl get pods
NAME READY STATUS RESTARTS AGE
gluster-1 1/1 Running 0 35s
gluster-2 1/1 Running 0 3s

So the containers are started Successfully , lets check on the nodes.

Go to Node1:


-bash-4.2# sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
91ac2654c1f8 gluster/gluster-centos "/usr/sbin/init" 5 minutes ago Up 5 minutes k8s_glusterfs.49424e8b_gluster-1_default_cbbcdc6c-a491-11e5-ac43-525400c3df83_a69c7952

Go to Node2:


-bash-4.2# sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b030df94c037 gluster/gluster-centos "/usr/sbin/init" 3 minutes ago Up 3 minutes k8s_glusterfs.49424e8b_gluster-2_default_261157a1-a492-11e5-ac43-525400c3df83_c47753a3

So we have two containers running in different atomic hosts, to form a trusted pool, these containers have to be peer probed. From one container we are going to peer probe to another container, since we are using host configuration ( –net=host option in docker run enabled via kubernetes configuration ) give the centos atomic host’s IP to peer probe. If you are using some other SDN ( for ex: flannel) use the containers IP to peer probe.


[root@atomic-node1 /]# gluster peer probe 10.70.43.88
peer probe: success.
[root@atomic-node1 /]# gluster peer status
Number of Peers: 1

Hostname: 10.70.43.88
Uuid: 1a76df91-4e8f-4d4c-87b8-6254e3337c6d
State: Peer in Cluster (Connected)

Now that we have the Trusted storage pool with node1 and node2, We shall create a volume in the container.

[root@atomic-node1 /]# gluster volume create newvol replica 2 10.70.42.29:/mnt/brick1/new 10.70.43.88:/mnt/brick1/new
volume create: newvol: success: please start the volume to access data
[root@atomic-node1 /]# gluster volume start newvol
volume start: newvol: success
[root@atomic-node1 /]# gluster volume status
Status of volume: newvol
Gluster process TCP Port RDMA Port Online Pid
——————————————————————————
Brick 10.70.42.29:/mnt/brick1/new 49152 0 Y 550
Brick 10.70.43.88:/mnt/brick1/new 49152 0 Y 492
NFS Server on localhost N/A N/A N N/A
Self-heal Daemon on localhost N/A N/A Y 577
NFS Server on 10.70.43.88 N/A N/A N N/A
Self-heal Daemon on 10.70.43.88 N/A N/A Y 520

Task Status of Volume newvol
——————————————————————————
There are no active volume tasks

Great, we have our volume exported from the trusted pool formed between the gluster containers. Now that we have our volume running , lets mount our volume to the master node. As shown above in the picture, this volume can be mounted in any of the system, it can be a micro service or application container or atomic host or some other bare metal system. The only requirement is that the gluster server should be reachable from the mount client system. For ex: if there is a application container runnning in your cluster, you can mount the gluster volume inside it and use it for storing data.

In this setup, we are mounting it inside the kubernetes master node.

In Master,


-bash-4.2# mount -t glusterfs 10.70.42.29:/newvol /mnt/brick
-bash-4.2# ps aux | grep newvol
root 40579 0.1 1.5 461008 28704 ? Ssl 13:34 0:00 /usr/sbin/glusterfs --volfile-server=10.70.42.29 --volfile-id=/newvol /var/mnt/brick

So We have mounted the volume!!

In this article, we have created a Kubernetes cluster on atomic hosts and ran Gluster Containers and created a trusted storage pool out of these containers, then created and exported a volume successfully for the client to use it as a persistent data store.

PS# If you any queries/comments please leave a comment.

Authors: Mohammed Ashiq and Humble Chirammal

Digiprove sealCopyright secured by Digiprove © 2017 Humble Chirammal

Leave a Reply

Your email address will not be published. Required fields are marked *