I will walk you through the setup to deploy GlusterFS containers in an atomic host environment with Flannel as overlay networking.
In this implementation, we will deploy 2 GlusterFS containers on two different atomic hosts respectively. The networking between the atomic hosts will be happening through the tunnel created by “Flannel” overlay networking. The “etcd” ( etcd is an open-source distributed key-value store that provides shared configuration and service discovery) service is used as a key-value store for this setup. The flannel daemon will be running in both atomic hosts. Flannel contact etcd server and fetch the networking configuration.
Once flanneld is configured in atomic hosts, the default docker network will fall into the same network of flannel.
The containers spawned on these atomic hosts get IP addresses from the same network of its flannel subnet. It assures the communication between the containers which runs on atomic hosts works.
In this configuration, we are trying to persist GlusterFs configuration data by exporting host filesystem to GlusterFS in containers. ie the directories ( ex: /etc/glusterfs, /var/lib/glusterd, /var/log/glusterfs) from atomic hosts are mounted in containers to make sure persistence of trusted pool metadata. Also, the container bind mount atomic host filesystem mount point ( for ex:/mnt/brick1 ) which serve as the brick for gluster volume created in this trusted pool. Once the gluster volume is created, the glusterfs clients will be able to mount the volume using FUSE and NFS protocols.
Setup:
*) CentOS 7.1 atomic hosts and the atomic hosts are deployed inside KVM VMs.
Below diagram gives more details about this setup.
NOTE: If you already have an atomic host setup, skip ‘Section 1’ and proceed from Section 2.
Section 1: Configuration of atomic hosts :
PLATFORM/ HOST OS :
[root@humble-server]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.122.7 atomicetcd → Atomic ETCD server 192.168.122.133 atomictest1 → Atomic HOST1 192.168.122.188 atomictest2 → Atomic HOST2 [root@humble-server]#
First of all, “Cloud-Init” has to be configured for atomic installation. Cloud-Init iso requires {user,meta}-data files.
Create a meta-data file with your desired hostname and instance-id.
$ vi meta-data instance-id: atomic-test1 local-hostname: atomictest1
Create a user-data file. The #cloud-config directive at the beginning of the file is mandatory, not a comment. If you have multiple admins and ssh keys you’d like to access the default user, you can add a new ssh-rsa line.
$ vi user-data #cloud-config password: atomic ssh_pwauth: True chpasswd: { expire: False }
ssh_authorized_keys: – ssh-rsa … ..
After creating the user-data and meta-data files, generate an ISO file. Make sure the user running libvirt has the proper permissions to read the generated image.
I: -input-charset not specified, using utf-8 (detected in locale settings) Total translation table size: 0 Total rockridge attributes bytes: 331 Total directory bytes: 0 Path table size(bytes): 10 Max brk space used 0 183 extents written (0 MB)
NOTE: This example run on CentOS 7 atomic host, the qcow2 image of the same can be downloaded from https://wiki.centos.org/SpecialInterestGroup/Atomic/Download/
If you are creating atomic hosts in KVM VMs , please follow below process.
Creating with virt-manager
Here’s how to get started with Atomic on your machine using virt-manager on Linux. The instructions below are for running virt-manager on Fedora 21 or above. The steps may vary slightly when running older distributions of virt-manager.
Select File -> New Virtual Machine from the menu bar. The New VM dialog box will open.
Select the Import existing disk image option and click Forward. The next page in the New VM dialog will appear.
Click Browse. The Locate or create storage volume dialog will open.
Click Browse Local. The Locate existing storage dialog will open.
Navigate to the downloaded virtual machine file, select it, and click Open.
In the New VM dialog, select Linux for the OS type, Fedora 21 (or later) for the Version, and click Forward.
Adjust the VM’s RAM and CPU settings (if needed) and click Forward.
Select the checkbox next to Customize configuration before install and click Forward. This will allow you to add the metatdata ISO device before booting the VM.
Note: When running virt-manager on Red Hat Enterprise Linux 6 or CentOS 6, the VM will not boot until the disk storage format is changed from raw to qcow2.
Adding the CD-ROM device for the metadata source;
In the virt-manager GUI, click to open your Atomic machine. Then on the top bar click View > Details
Click on Add Hardware on the bottom left corner.
Choose Storage, and Select managed or other existing storage. Browse and select the init.iso image you created. Change the Device type to CD-ROM device. Click on Finish to create and attach this storage.
Then start the atomic installation, the cloud init will come into play and it will ask for “atomic host” login.
username: centos
password: atomic
Note: Above is based on the cloud-init configuration. If you have customized the cloud init configuration for different username and password, please supply the same.
Once you login in Server : Atomic host1:
[centos@atomictest1 ~]$ hostname atomictest1.localdomain
[centos@atomictest1 ~]$ ip a |grep inet inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host inet 192.168.122.133/24 brd 192.168.122.255 scope global dynamic eth0 inet6 fe80::5054:ff:fe49:3b95/64 scope link inet 172.17.42.1/16 scope global docker0
[centos@atomictest1 ~]$ sudo rpm-ostree upgrade Updating from: centos-atomic-host:centos-atomic-host/7/x86_64/standard No upgrade available. [centos@atomictest1 ~]$ rpm -qa |egrep ‘docker|flannel|atomic’ docker-1.7.1-108.el7.centos.x86_64 atomic-1.0-108.el7.centos.x86_64 flannel-0.2.0-10.el7.x86_64 …. [centos@atomictest1 ~]$ brctl show bridge name bridge id STP enabled interfaces docker0 8000.56847afe9799 no [centos@atomictest1 ~]$ ps aux |grep docker root 1929 0.0 1.2 289084 12644 ? Ssl 18:47 0:00 /usr/bin/docker -d –selinux-enabled –storage-driver devicemapper –storage-opt dm.fs=xfs –storage-opt dm.thinpooldev=/dev/mapper/atomicos-docker–pool –storage-opt dm.use_deferred_removal=true
Server: Atomic host 2
[centos@atomictest2 ~]$ cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) [centos@atomictest2 ~]$ hostname atomictest2.localdomain [centos@atomictest2 ~]$ ip a |grep inet inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host inet 192.168.122.188/24 brd 192.168.122.255 scope global dynamic ens3 inet6 fe80::5054:ff:fec6:4241/64 scope link inet 172.17.42.1/16 scope global docker0 [centos@atomictest2 ~]$
Flannel requires etcd server , in this example we are configuring etcd in another atomic host which runs same CentOS atomic host image.
Section 2: Configuration of etcd server
Server : Atomic-etcd:
[centos@atomicetcd ~]$ cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core)
[centos@atomicetcd ~]$ rpm -qa |grep etcd etcd-2.0.13-2.el7.x86_64
[centos@atomicetcd ~]$ ifconfig |grep inet
inet 172.17.42.1 netmask 255.255.0.0 broadcast 0.0.0.0
inet 192.168.122.7 netmask 255.255.255.0 broadcast 192.168.122.255
inet6 fe80::5054:ff:fef6:5c5e prefixlen 64 scopeid 0x20
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
[centos@atomicetcd ~]$ sudo systemctl start etcd [centos@atomicetcd ~]$ sudo systemctl status etcd etcd.service – Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled) Active: active (running) since Sat 2015-09-26 10:11:11 UTC; 5s ago Main PID: 10512 (etcd) CGroup: /system.slice/etcd.service └─10512 /usr/bin/etcd ….
Section 3: Configure flannel for overlay networking
Make sure “etcd” service has started successfully. Once ‘etcd’ is running, create a flannel configuration json file to feed etcd.
[centos@atomicetcd ~]$ cat flannel-config.json { “Network”: “10.0.0.0/16”, “SubnetLen”: 24, “Backend”: { “Type”: “vxlan”, “VNI”: 1 } }
By default etcd will be listening on “localhost” port 2379, make etcd to listen on all the interfaces, so that other atomic hosts can reach etcd and fetch flannel configuration data.
Default configuration of etcd looks like below:
Change above file to reflect etcd server IP in client URLs.
Set the network key in etcd server via curl.
{“action”:”set”,”node”:{“key”:”/atomic01/network/config”,”value”:”{\n\”Network\”: \”10.0.0.0/16\”,\n\”SubnetLen\”: 24,\n\”Backend\”: {\n\”Type\”: \”vxlan\”,\n\”VNI\”: 1\n }\n}\n\n”,”modifiedIndex”:3,”createdIndex”:3}} [centos@atomicetcd ~]$
Retrieve the data from etcd server to make sure its recorded properly.
cu % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 222 100 222 0 0 1468 0 –:–:– –:–:– –:–:– 1470 { “action”: “get”, “node”: { “createdIndex”: 3, “key”: “/atomic01/network/config”, “modifiedIndex”: 3, “value”: “{\n\”Network\”: \”10.0.0.0/16\”,\n\”SubnetLen\”: 24,\n\”Backend\”: {\n\”Type\”: \”vxlan\”,\n\”VNI\”: 1\n }\n}\n\n” } }
NOTE: Its *not* required to configure flannel in ETCD server, however
in this setup we are configuring flannel in ETCD server so that if needed,
this server can be used as a client for gluster volumes. We can mount the gluster volume and perform tests.
Set the flanneld network configuration service file as shown below.
[Service] EnvironmentFile=/run/flannel/subnet.env ExecStartPre=-/usr/sbin/ip link del docker0 ExecStart= ExecStart=/usr/bin/docker -d \ –bip=${FLANNEL_SUBNET} \ –mtu=${FLANNEL_MTU} \ $OPTIONS \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $INSECURE_REGISTRY
[centos@atomicetcd ~]$ cat /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD=”http://192.168.122.7:2379″ # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY=”/atomic01/network” # Any additional options that you want to pass FLANNEL_OPTIONS=”–iface=eth0 -ip-masq=true”
[centos@atomicetcd glusterd]$ cat /etc/sysconfig/docker |grep -v “#”
OPTIONS=’–selinux-enabled –ip-masq=false’ DOCKER_CERT_PATH=/etc/docker
Make flannel, etcd to load at boot time.
After reboot, check docker0 bridge and flannel network configuration and validate both are in same network configuration.
eth0: flags=4163
flannel.1: flags=4163
……..
If both are in same network, we are good to proceed 🙂
Now, lets make sure the other 2 atomic hosts can connect and fetch the ETCD configuration data from atomicetcd server:
FROM atomictest1:
FROM atomictest2:
[centos@atomictest2 ~]$ curl -L http://atomicetcd:2379/v2/keys/atomic01/network/config {“action”:”get”,”node”:{“key”:”/atomic01/network/config”,”value”:”{\n\”Network\”: \”10.0.0.0/16\”,\n\”SubnetLen\”: 24,\n\”Backend\”: {\n\”Type\”: \”vxlan\”,\n\”VNI\”: 1\n }\n}\n\n”,”modifiedIndex”:3,”createdIndex”:3}} [centos@atomictest2 ~]$
Make sure flanneld running on “atomictest{1,2} servers” contact “atomicetcd” server (192.168.122.7) to fetch the flannel configuration.
[centos@atomictest2 ~]$ cat /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD=”http://192.168.122.7:2379″ # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY=”/atomic01/network” # Any additional options that you want to pass FLANNEL_OPTIONS=”–iface=eth0 -ip-masq=true”
NOTE: Due to an issue with glusterd to form a trusted pool when flannel is configured as overlay networking solution, we need a hack in flannel configuration and docker configuration file for ip masquerading. Please note that, in above FLANNEL_OPTIONS value, “eth0” should be replaced with the network interface name of your atomic server and “-ip-masq” option should be set to ‘true’ to overcome above mentioned limitation. We also have to configure docker option as shown below.
[centos@atomictest2 glusterd]$ cat /etc/sysconfig/docker |grep -v “#”
OPTIONS=’–selinux-enabled –ip-masq=false’ DOCKER_CERT_PATH=/etc/docker
Enable flanneld in both servers:
[centos@atomictest1 ~]$ sudo systemctl enable flanneld ln -s ‘/usr/lib/systemd/system/flanneld.service’ ‘/etc/systemd/system/docker.service.requires/flanneld.service’ [centos@atomictest1 ~]$
[centos@atomictest2 ~]$ sudo systemctl enable flanneld ln -s ‘/usr/lib/systemd/system/flanneld.service’ ‘/etc/systemd/system/docker.service.requires/flanneld.service’ [centos@atomictest2 ~]$
Repeat below steps in ‘atomictest 1’ and ‘atomictest 2’ :
[centos@atomictest2 ~]$ sudo mkdir -p /etc/systemd/system/docker.service.d/
[centos@atomictest2 ~]$ cat /etc/systemd/system/docker.service.d/10-flanneld-network.conf [Unit] After=flanneld.service Requires=flanneld.service
[Service] EnvironmentFile=/run/flannel/subnet.env ExecStartPre=-/usr/sbin/ip link del docker0 ExecStart= ExecStart=/usr/bin/docker -d \ –bip=${FLANNEL_SUBNET} \ –mtu=${FLANNEL_MTU} \ $OPTIONS \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $INSECURE_REGISTRY [centos@atomictest2 ~]$
Reboot ‘atomictest1’ and ‘atomictest2’ servers , once these servers are back, both ‘docker’ and ‘flanneld’ services should be up and running and should see ‘docker0’ and ‘flannel’ are in same network.
[centos@atomictest1 ~]$ systemctl status docker flanneld docker.service – Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled) Drop-In: /etc/systemd/system/docker.service.d └─10-flanneld-network.conf /usr/lib/systemd/system/docker.service.d └─flannel.conf Active: active (running) since Sat 2015-09-26 10:57:12 … flanneld.service – Flanneld overlay address etcd agent Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled) Active: active (running) since Sat 2015-09-26 10:57:08 ……
[centos@atomictest1 ~]$ ifconfig |grep flags -A 2
docker0: flags=4099
—
flannel.1: flags=4163
—
lo: flags=73
You can validate that, the atomic hosts subnets are allocated properly via below command from any of the nodes:
centos@atomicetcd ~]$ curl -L http://atomicetcd:2379/v2/keys/atomic01/network/subnets | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 908 100 908 0 0 5987 0 –:–:– –:–:– –:–:– 6013 { “action”: “get”, “node”: { “createdIndex”: 5, “dir”: true, “key”: “/atomic01/network/subnets”, “modifiedIndex”: 5, “nodes”: [ { “createdIndex”: 5, “expiration”: “2015-09-27T10:38:13.318286933Z”, “key”: “/atomic01/network/subnets/10.0.12.0-24”, “modifiedIndex”: 5, “ttl”: 84248, “value”: “{\”PublicIP\”:\”192.168.122.7\”,\”BackendType\”:\”vxlan\”,\”BackendData\”:{\”VtepMAC\”:\”96:1e:41:4a:aa:ce\”}}” }, { “createdIndex”: 6, “expiration”: “2015-09-27T10:57:08.763771526Z”, “key”: “/atomic01/network/subnets/10.0.80.0-24”, “modifiedIndex”: 6, “ttl”: 85384, “value”: “{\”PublicIP\”:\”192.168.122.133\”,\”BackendType\”:\”vxlan\”,\”BackendData\”:{\”VtepMAC\”:\”de:c1:3a:e4:64:fc\”}}” }, { “createdIndex”: 7, “expiration”: “2015-09-27T11:09:55.54906845Z”, “key”: “/atomic01/network/subnets/10.0.55.0-24”, “modifiedIndex”: 7, “ttl”: 86150, “value”: “{\”PublicIP\”:\”192.168.122.188\”,\”BackendType\”:\”vxlan\”,\”BackendData\”:{\”VtepMAC\”:\”c2:29:e0:13:d9:40\”}}” } ] } } Section 4: Run RHGS containers
Once flannel is configured and it’s up, we have to pull the RHGS image from Red Hat internal registry via docker pull command as shown below. In order to obtain this image, one has to: