As you know, docker and GlusterFS are widely adopted by IT world.
The integration of glusterfs and docker technologies should be done without delay, so as to benefit from the fast paced advances in technology.
For those who already know these terms Docker & GlusterFS , please skip below introduction , and start from “DOCKIT” section:
If you didnt reach DOCKIT section:
GlusterFS :
“GlusterFS is an open source, distributed file system capable of scaling to several petabytes (actually, 72 brontobytes!) and handling thousands of clients. GlusterFS clusters together storage building blocks over Infiniband RDMA or TCP/IP interconnect, aggregating disk and memory resources and managing data in a single global namespace. GlusterFS is based on a stackable user space design and can deliver exceptional performance for diverse workloads…. continue reading at http://gluster.org/documentation/About_Gluster/”
Docker :
“Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications. Consisting of Docker Engine, a portable, lightweight runtime and packaging tool, and Docker Hub, a cloud service for sharing applications and automating workflows, Docker enables apps to be quickly assembled from components and eliminates the friction between development, QA, and production environments. As a result, IT can ship faster and run the same app, unchanged, on laptops, data center VMs, and any cloud…. continue reading at https://www.docker.com/whatisdocker/. “
DOCKIT:
Dockit is an application which is developed for:
Easy deployment of Containers from any docker image.
Easy deployment of GlusterFS containers with any version of GlusterFS binary
& auto configuration of GlusterFS Volumes.
More details can be fetched from https://github.com/humblec/dockit/blob/master/README.md
However let me put some more examples and use cases here:
Assume you dont have much idea about Docker and GlusterFS, however would like to deploy GlusterFS in containers with specific version of gluster binary. Then it is as easy as running below command in your system:
[root@HOST]dockit -p -r humble -i f20-glusterfs-source -s -n 2 -g -t latest –gv –gi 3.5 -c configfile
Let me explain above options, but you can see, ‘dockit –help’ already has information about these options 🙂
[root@HOST]# dockit –help
————————————————————
M A I N – M E N U – O F – DOCKIT
————————————————————
Invoke dockit with any of (-d , -p, -b, -s) options
1. Install and Run Docker deamon (-d) –> dryrun
2. Pull image from docker repo and Run containers (-p) –> requires -i
3. Build from dockerfile and Run Containers (-b) –> requires -f
4. Run container from existing image (-s) –> requires -i
Optional:
Create and start gluster containers (-g) –> Effective only with -s option
(–gi) –> To install Gluster From Source
(–gv) –> To auto configure Gluster Volume
————————————————————
None
dockit : INFO Dockit starting.. Process logs are available at:/var/log/dockit/dockit.log
Usage: dockit [options]
Options:
-h, –help show this help message and exit
-d, –dry_run Do dry run – dont try to install any packages
-p, –pullimage Whether to pull from the docker repo ? Need to specify
dockerrepo and image name
-s, –startc Whether to start from an image ? Need to specify image
and tag
-b, –buildimage Whether to build image from the dockerfile? Need to
specify dockerfile path, and imagetag
-g, –gluster_mode Configure gluster volume in containers
-i IMAGE, –image=IMAGE
Image name – Containers will be based on this image
-t IMAGETAG, –imgtag=IMAGETAG
Image tag name – Containers will be assigned this tag
-n COUNT, –count=COUNT
Number of containers to start –
-c CONFIGFILE, –configfile=CONFIGFILE
COnfig file path to read gluster configuration –
-f DOCKERFILE, –dockerfile=DOCKERFILE
Docker file path to build the container –
-r DOCKERREPO, –dockerrepo=DOCKERREPO
Docker repository name with a trailing blackslash –
–gv, –glustervolume
Gluster Volume Creation inside containers – Valid
with -g option
–gi=GLUSTERVERSION, –glusterinstall=GLUSTERVERSION
Install gluster inside containers – Valid with -g
option
Any way, here is the conclusion of above command :
Pull (-p) an image ( -i) called “f20-glusterfs-source” with tag ( -t latest) from docker repo (-r) ‘humble’ and start ( -s ) 2 containers (-n 2 ): Also work in gluster mode ( -g) and install “3.5” version of gluster binary ( –gi 3.5 ) inside these containers and auto configure gluster Volume (–gv) by fetching details from configuration file called configfile ( -c configfile) .
Thats it 🙂
What is inside my configfile ? It has similar entries as shown here:
BRICKS=”/brick6,/brick7,/brick8,/brick9″
VOLNAME=”testvol”
VOL_TYPE=”2x2x1″
SERVER_EXPORT_DIR=”/rhs_bricks”
I think the only entry which need some explanation here is VOL_TYPE – It is “distribute-replica-stripe” format.
Lets see what happens if we run previously mentioned command?
[root@dell-pe-c5220-6 ~]# dockit -p -r humble -i f20-glusterfs-source -s -n 2 -g -t latest –gv –gi 3.5 -c configfile
————————————————————
M A I N – M E N U – O F – DOCKIT
————————————————————
Invoke dockit with any of (-d , -p, -b, -s) options
1. Install and Run Docker deamon (-d) –> dryrun
2. Pull image from docker repo and Run containers (-p) –> requires -i
3. Build from dockerfile and Run Containers (-b) –> requires -f
4. Run container from existing image (-s) –> requires -i
Optional:
Create and start gluster containers (-g) –> Effective only with -s option
(–gi) –> To install Gluster From Source
(–gv) –> To auto configure Gluster Volume
————————————————————
None
dockit : INFO Dockit starting.. Process logs are available at:/var/log/dockit/dockit.log
Do you want to continue (y/n)y
dockit : INFO Proceeding
dockit : INFO Need to install gluster inside containers
dockit : INFO
Need to configure gluster volume..
dockit : INFO Reading gluster configuration from config file
{‘GLUSTER_VERSION’: ‘3.5’, ‘BRICKS’: ‘/brick6,/brick7,/brick8,/brick9’, ‘SERVER_EXPORT_DIR’: ‘/rhs_bricks’, ‘VOL_TYPE’: ‘2x2x1’, ‘VOLNAME’: ‘testvol’}
dockit : INFO No of gluster containers to spawn:4
Do you want to continue (y/n):y
dockit : INFO Configuration read from configuration file
dockit : INFO {‘GLUSTER_VERSION’: ‘3.5’, ‘BRICKS’: ‘/brick6,/brick7,/brick8,/brick9’, ‘SERVER_EXPORT_DIR’: ‘/rhs_bricks’, ‘VOL_TYPE’: ‘2x2x1’, ‘VOLNAME’: ‘testvol’}
dockit : INFO Distribution:fedora
dockit : INFO Distribution:fedora Required [‘docker-io’, ‘python-docker-py’] packages
Making yum transactions
Loaded plugins: langpacks, refresh-packagekit
dockit : INFO docker-io -> Installed
dockit : INFO python-docker-py -> not installed
dockit : INFO Pre-requisites are installed
dockit : INFO Requested process: docker -d is running
dockit : INFO Successfully connected to docker deamon:
pull/build/start containers accordingly.
dockit : INFO Trying to pull f20-glusterfs-source from docker repo:humble …
This can take some time, please wait…
dockit : INFO Successfully pulled docker image:humble/f20-glusterfs-source
dockit : INFO Done with pulling.. continuing
dockit : INFO Going to run the containers
dockit : INFO Create and start containers with image :humble/f20-glusterfs-source:latest
dockit : INFO Bricks will be using in order:[‘/brick6’, ‘/brick7’, ‘/brick8’, ‘/brick9′]
root : INFO Enable Gluster Volume :1
…
…
…
…
dockit : INFO Information about running containers
dockit : INFO Containers are running successfully.. please login and work!!!!
————————————————————
dockit : INFO Details about running containers..
dockit : INFO Container IPs : [u’172.17.0.2′, u’172.17.0.3′, u’172.17.0.4′, u’172.17.0.5′]
dockit : INFO Container Ids : [u’a58bedf9a7a66745429f47b6b886e533b246ce4b6b12ca5ef6e312e584d11adf’, u’a58bedf9a7a66745429f47b6b886e533b246ce4b6b12ca5ef6e312e584d11adf’, u’a58bedf9a7a66745429f47b6b886e533b246ce4b6b12ca5ef6e312e584d11adf’, u’a58bedf9a7a66745429f47b6b886e533b246ce4b6b12ca5ef6e312e584d11adf’]
————————————————————
dockit : INFO Trying to install gluster on [u’172.17.0.2′, u’172.17.0.3′, u’172.17.0.4′, u’172.17.0.5′] nodes
dockit : INFO Configuring/installing on node:172.17.0.2
dockit : INFO Continuing ..
dockit : INFO Successfully configured GlusterFS binary on node:172.17.0.2
dockit : INFO Configuring/installing on node:172.17.0.3
dockit : INFO Continuing ..
dockit : INFO Successfully configured GlusterFS binary on node:172.17.0.3
dockit : INFO Configuring/installing on node:172.17.0.4
dockit : INFO Continuing ..
dockit : INFO Successfully configured GlusterFS binary on node:172.17.0.4
dockit : INFO Configuring/installing on node:172.17.0.5
dockit : INFO Continuing ..
dockit : INFO Successfully configured GlusterFS binary on node:172.17.0.5
dockit : INFO Successful Gluster Package Installation and GlusterFS Binary installation on all the nodes!
dockit : INFO nodes are [u’172.17.0.2′, u’172.17.0.3′, u’172.17.0.4′, u’172.17.0.5′]
dockit : INFO Number of nodes: 4
dockit : INFO number of bricks:4
volume create: testvol: success: please start the volume to access data
root : INFO Gluster Volume operations done! Please mount volume :testvol in your client
All good, how-ever lets start with proofs even though dockit claimed everything was perfect:
First of all, are the newly created containers running ?
[root@HOSt ~]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e2b889b78263 humble/f20-glusterfs-source:latest /usr/bin/supervisord 26 minutes ago Up 25 minutes 0.0.0.0:49159->22/tcp, 0.0.0.0:49160->80/tcp compassionate_pare
04f468d03288 humble/f20-glusterfs-source:latest /usr/bin/supervisord 26 minutes ago Up 26 minutes 0.0.0.0:49157->22/tcp, 0.0.0.0:49158->80/tcp naughty_mayer
5b6ef3f3be9a humble/f20-glusterfs-source:latest /usr/bin/supervisord 26 minutes ago Up 26 minutes 0.0.0.0:49155->22/tcp, 0.0.0.0:49156->80/tcp nostalgic_mccarthy
a58bedf9a7a6 humble/f20-glusterfs-source:latest /usr/bin/supervisord 26 minutes ago Up 26 minutes 0.0.0.0:49153->22/tcp, 0.0.0.0:49154->80/tcp romantic_blackwell
Well, lets login to one of the containers using password ‘redhat‘:
From dockit output we know container IPs are :[u’172.17.0.2′, u’172.17.0.3′, u’172.17.0.4′, u’172.17.0.5′]
[root@HOST~]# ssh root@172.17.0.2
The authenticity of host ‘172.17.0.2 (172.17.0.2)’ can’t be established.
ECDSA key fingerprint is d7:7f:c7:1f:7d:b0:e4:2e:31:de:ca:b6:82:b3:61:b5.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘172.17.0.2’ (ECDSA) to the list of known hosts.
root@172.17.0.2’s password:
[root@a58bedf9a7a6 ~]# [root@a58bedf9a7a6 ~]# rpm -qa |grep gluster
[root@a58bedf9a7a6 ~]# glusterd –version
glusterfs 3.5git built on Jul 24 2014 21:39:25
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc.
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
[root@a58bedf9a7a6 ~]# ps aux |grep gluster
root 24242 0.1 0.1 418560 19072 ? Ssl 21:57 0:00 glusterd
root 24619 0.0 0.1 506104 20724 ? Ssl 21:59 0:00 /usr/local/sbin/glusterfsd -s 172.17.0.2 –volfile-id testvol.172.17.0.2.rhs_bricks-testvol_brick0 -p /var/lib/glusterd/vols/testvol/run/172.17.0.2-rhs_bricks-testvol_brick0.pid -S /var/run/ca30e21b539dd0d45f71d3502cf2ff40.socket –brick-name /rhs_bricks/testvol_brick0 -l /usr/local/var/log/glusterfs/bricks/rhs_bricks-testvol_brick0.log –xlator-option *-posix.glusterd-uuid=1e3a7c3f-5688-4689-89fd-7df4c3dea9c6 –brick-port 49152 –xlator-option testvol-server.listen-port=49152
root 24634 0.0 0.1 336760 28956 ? Ssl 21:59 0:00 /usr/local/sbin/glusterfs -s localhost –volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /usr/local/var/log/glusterfs/glustershd.log -S /var/run/a0d128a7f6f707df5490769a84eb1f70.socket –xlator-option *replicate*.node-uuid=1e3a7c3f-5688-4689-89fd-7df4c3dea9c6
root 24665 0.0 0.0 112668 968 pts/0 S+ 22:02 0:00 grep –color=auto gluster
[root@a58bedf9a7a6 ~]#
[root@a58bedf9a7a6 ~]# gluster pool list
UUID Hostname State
9b093590-0879-4201-8869-b40bc0dcc717 172.17.0.3 Connected
767018ba-1f01-4d14-b8ee-51c0907edf3b 172.17.0.4 Connected
7b8eca79-5c75-4b25-95de-707e2639b339 172.17.0.5 Connected
1e3a7c3f-5688-4689-89fd-7df4c3dea9c6 localhost Connected
[root@a58bedf9a7a6 ~]#
[root@a58bedf9a7a6 ~]# gluster volume info
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 3523d724-2bcc-47ae-9ee4-d6006c726ba8
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 172.17.0.2:/rhs_bricks/testvol_brick0
Brick2: 172.17.0.3:/rhs_bricks/testvol_brick1
Brick3: 172.17.0.4:/rhs_bricks/testvol_brick2
Brick4: 172.17.0.5:/rhs_bricks/testvol_brick3
[root@a58bedf9a7a6 ~]# mount|grep rhs_bricks
/dev/mapper/fedora_dell–pe–c5220–6-root on /rhs_bricks type ext4 (rw,relatime,seclabel,data=ordered)
[root@a58bedf9a7a6 ~]#
[root@a58bedf9a7a6 ~]# logout
Connection to 172.17.0.2 closed.
Considering the HOST system as gluster client , lets try to mount the Volume created by Dockit!
[root@HOST ~]# mount -t glusterfs 172.17.0.3:/testvol /mnt
[root@HOST ~]# mount |grep glusterfs
172.17.0.3:/testvol on /mnt type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root@HOST ~]#
[root@HOST ~]# cd /mnt/
[root@HOST mnt]# ls
[root@HOST mnt]# echo “I am created by dockit” > gluster-dockit-file
[root@HOST mnt]# ls
gluster-dockit-file
[root@HOST mnt]#
[root@HOST ~]# ssh root@172.17.0.3
root@172.17.0.3’s password:
[root@5b6ef3f3be9a ~]# cd /rhs_bricks/
[root@5b6ef3f3be9a rhs_bricks]# ls
testvol_brick1
[root@5b6ef3f3be9a rhs_bricks]# cd testvol_brick1/
[root@5b6ef3f3be9a testvol_brick1]# ls
gluster-dockit-file
[root@5b6ef3f3be9a testvol_brick1]# cat gluster-dockit-file
I am created by dockit
[root@5b6ef3f3be9a testvol_brick1]#
Now, let me ask following questions :
Did you install docker packages in your system ?
Did you start docker daemon ?
Did you download the image from docker repository?
Did you create brick directories in your filesystem?
Did you run containers with exported bricks from host?
Did you install GlusterFS on these containers from gluster source ?
Did you configure a gluster Volume across these container nodes ?
…etc
Everything was taken care by “Dockit” !!
Apart from above,
Dockit can be used build containers from Dockerfile and spawn containers using the built image
.You can operate in gluster mode on these containers as well.
Isnt it worth to give a try ? Criticize ? Point Bugs ? Fill TODOs ..etc ? Please do so. 🙂
NOTE: If you have used “-p” in your command options and if it failed at first attempt, give it a try once again. I have seen manual pulling with docker commands fail at last second , even-though everything was successful.
If you are NOT using the images listed under below repo, please make sure “SSH deamon is running on the containers and root password is “redhat”. UPDATE: now the password is asked by dockit at time of deployment.
https://hub.docker.com/u/humble/
Both Images in above repository has ssh deamon running and root password configured as “redhat”:
fed20-gluster : This image has 3.5 gluster packages in it
f20-glusterfs-source : This image has all prerequisites packages installed for GlusterFS source compilation
PS #
The “dockit” project is available from github and gluster forge:
https://github.com/humblec/dockit
https://forge.gluster.org/dockit
Even though its undergoing active development, the current version is pretty stable. Please let me know if you would like to see any features included in the same.