containers.

[root@node]# cat claim1clone.yaml kind: PersistentVolumeClaim apiVersion: v1 metadata: name: claim1clone annotations: volume.beta.kubernetes.io/storage-class: “glusterfile” “k8s.io/CloneRequest”: “claim1” spec: accessModes: – ReadWriteMany resources: requests: storage: 1Gi

[root@humble-server]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.122.7 atomicetcd → Atomic ETCD server 192.168.122.133 atomictest1 → Atomic HOST1 192.168.122.188 atomictest2 → Atomic HOST2 [root@humble-server]#

First of all, “Cloud-Init” has to be configured for atomic installation. Cloud-Init iso requires {user,meta}-data files.

Create a meta-data file with your desired hostname and instance-id.

$ vi meta-data instance-id: atomic-test1 local-hostname: atomictest1

Create a user-data file. The #cloud-config directive at the beginning of the file is mandatory, not a comment. If you have multiple admins and ssh keys you’d like to access the default user, you can add a new ssh-rsa line.

$ vi user-data #cloud-config password: atomic ssh_pwauth: True chpasswd: { expire: False }

ssh_authorized_keys: – ssh-rsa … ..

I: -input-charset not specified, using utf-8 (detected in locale settings) Total translation table size: 0 Total rockridge attributes bytes: 331 Total directory bytes: 0 Path table size(bytes): 10 Max brk space used 0 183 extents written (0 MB)

[centos@atomictest1 ~]$ hostname atomictest1.localdomain

[centos@atomictest1 ~]$ ip a |grep inet inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host inet 192.168.122.133/24 brd 192.168.122.255 scope global dynamic eth0 inet6 fe80::5054:ff:fe49:3b95/64 scope link inet 172.17.42.1/16 scope global docker0

[centos@atomictest1 ~]$ sudo rpm-ostree upgrade Updating from: centos-atomic-host:centos-atomic-host/7/x86_64/standard No upgrade available. [centos@atomictest1 ~]$ rpm -qa |egrep ‘docker|flannel|atomic’ docker-1.7.1-108.el7.centos.x86_64 atomic-1.0-108.el7.centos.x86_64 flannel-0.2.0-10.el7.x86_64 …. [centos@atomictest1 ~]$ brctl show bridge name bridge id STP enabled interfaces docker0 8000.56847afe9799 no [centos@atomictest1 ~]$ ps aux |grep docker root 1929 0.0 1.2 289084 12644 ? Ssl 18:47 0:00 /usr/bin/docker -d –selinux-enabled –storage-driver devicemapper –storage-opt dm.fs=xfs –storage-opt dm.thinpooldev=/dev/mapper/atomicos-docker–pool –storage-opt dm.use_deferred_removal=true

Server: Atomic host 2

[centos@atomictest2 ~]$ cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core) [centos@atomictest2 ~]$ hostname atomictest2.localdomain [centos@atomictest2 ~]$ ip a |grep inet inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host inet 192.168.122.188/24 brd 192.168.122.255 scope global dynamic ens3 inet6 fe80::5054:ff:fec6:4241/64 scope link inet 172.17.42.1/16 scope global docker0 [centos@atomictest2 ~]$

Flannel requires etcd server , in this example we are configuring etcd in another atomic host which runs same CentOS atomic host image.

Section 2: Configuration of etcd server

Server : Atomic-etcd:

[centos@atomicetcd ~]$ cat /etc/redhat-release CentOS Linux release 7.1.1503 (Core)

[centos@atomicetcd ~]$ rpm -qa |grep etcd etcd-2.0.13-2.el7.x86_64

[centos@atomicetcd ~]$ ifconfig |grep inet inet 172.17.42.1 netmask 255.255.0.0 broadcast 0.0.0.0 inet 192.168.122.7 netmask 255.255.255.0 broadcast 192.168.122.255 inet6 fe80::5054:ff:fef6:5c5e prefixlen 64 scopeid 0x20 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10

[centos@atomicetcd ~]$ sudo systemctl start etcd [centos@atomicetcd ~]$ sudo systemctl status etcd etcd.service – Etcd Server Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled) Active: active (running) since Sat 2015-09-26 10:11:11 UTC; 5s ago Main PID: 10512 (etcd) CGroup: /system.slice/etcd.service └─10512 /usr/bin/etcd ….

Section 3: Configure flannel for overlay networking

Make sure “etcd” service has started successfully. Once ‘etcd’ is running, create a flannel configuration json file to feed etcd.

[centos@atomicetcd ~]$ cat flannel-config.json { “Network”: “10.0.0.0/16”, “SubnetLen”: 24, “Backend”: { “Type”: “vxlan”, “VNI”: 1 } }

{“action”:”set”,”node”:{“key”:”/atomic01/network/config”,”value”:”{\n\”Network\”: \”10.0.0.0/16\”,\n\”SubnetLen\”: 24,\n\”Backend\”: {\n\”Type\”: \”vxlan\”,\n\”VNI\”: 1\n }\n}\n\n”,”modifiedIndex”:3,”createdIndex”:3}} [centos@atomicetcd ~]$

cu % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 222 100 222 0 0 1468 0 –:–:– –:–:– –:–:– 1470 { “action”: “get”, “node”: { “createdIndex”: 3, “key”: “/atomic01/network/config”, “modifiedIndex”: 3, “value”: “{\n\”Network\”: \”10.0.0.0/16\”,\n\”SubnetLen\”: 24,\n\”Backend\”: {\n\”Type\”: \”vxlan\”,\n\”VNI\”: 1\n }\n}\n\n” } }

[Service] EnvironmentFile=/run/flannel/subnet.env ExecStartPre=-/usr/sbin/ip link del docker0 ExecStart= ExecStart=/usr/bin/docker -d \ –bip=${FLANNEL_SUBNET} \ –mtu=${FLANNEL_MTU} \ $OPTIONS \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $INSECURE_REGISTRY

[centos@atomicetcd ~]$ cat /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD=”http://192.168.122.7:2379″ # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY=”/atomic01/network” # Any additional options that you want to pass FLANNEL_OPTIONS=”–iface=eth0 -ip-masq=true”

[centos@atomicetcd glusterd]$ cat /etc/sysconfig/docker |grep -v “#”

OPTIONS=’–selinux-enabled –ip-masq=false’ DOCKER_CERT_PATH=/etc/docker

FROM atomictest2:

[centos@atomictest2 ~]$ curl -L http://atomicetcd:2379/v2/keys/atomic01/network/config {“action”:”get”,”node”:{“key”:”/atomic01/network/config”,”value”:”{\n\”Network\”: \”10.0.0.0/16\”,\n\”SubnetLen\”: 24,\n\”Backend\”: {\n\”Type\”: \”vxlan\”,\n\”VNI\”: 1\n }\n}\n\n”,”modifiedIndex”:3,”createdIndex”:3}} [centos@atomictest2 ~]$

Make sure flanneld running on “atomictest{1,2} servers” contact “atomicetcd” server (192.168.122.7) to fetch the flannel configuration.

[centos@atomictest2 ~]$ cat /etc/sysconfig/flanneld # Flanneld configuration options # etcd url location. Point this to the server where etcd runs FLANNEL_ETCD=”http://192.168.122.7:2379″ # etcd config key. This is the configuration key that flannel queries # For address range assignment FLANNEL_ETCD_KEY=”/atomic01/network” # Any additional options that you want to pass FLANNEL_OPTIONS=”–iface=eth0 -ip-masq=true”

NOTE: Due to an issue with glusterd to form a trusted pool when flannel is configured as overlay networking solution, we need a hack in flannel configuration and docker configuration file for ip masquerading. Please note that, in above FLANNEL_OPTIONS value, “eth0” should be replaced with the network interface name of your atomic server and “-ip-masq” option should be set to ‘true’ to overcome above mentioned limitation. We also have to configure docker option as shown below.

[centos@atomictest2 glusterd]$ cat /etc/sysconfig/docker |grep -v “#”

OPTIONS=’–selinux-enabled –ip-masq=false’ DOCKER_CERT_PATH=/etc/docker

Enable flanneld in both servers:

[centos@atomictest1 ~]$ sudo systemctl enable flanneld ln -s ‘/usr/lib/systemd/system/flanneld.service’ ‘/etc/systemd/system/docker.service.requires/flanneld.service’ [centos@atomictest1 ~]$

[centos@atomictest2 ~]$ sudo systemctl enable flanneld ln -s ‘/usr/lib/systemd/system/flanneld.service’ ‘/etc/systemd/system/docker.service.requires/flanneld.service’ [centos@atomictest2 ~]$

Repeat below steps in ‘atomictest 1’ and ‘atomictest 2’ :

[centos@atomictest2 ~]$ sudo mkdir -p /etc/systemd/system/docker.service.d/

[centos@atomictest2 ~]$ cat /etc/systemd/system/docker.service.d/10-flanneld-network.conf [Unit] After=flanneld.service Requires=flanneld.service

[Service] EnvironmentFile=/run/flannel/subnet.env ExecStartPre=-/usr/sbin/ip link del docker0 ExecStart= ExecStart=/usr/bin/docker -d \ –bip=${FLANNEL_SUBNET} \ –mtu=${FLANNEL_MTU} \ $OPTIONS \ $DOCKER_STORAGE_OPTIONS \ $DOCKER_NETWORK_OPTIONS \ $INSECURE_REGISTRY [centos@atomictest2 ~]$

Reboot ‘atomictest1’ and ‘atomictest2’ servers , once these servers are back, both ‘docker’ and ‘flanneld’ services should be up and running and should see ‘docker0’ and ‘flannel’ are in same network.

[centos@atomictest1 ~]$ systemctl status docker flanneld docker.service – Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled) Drop-In: /etc/systemd/system/docker.service.d └─10-flanneld-network.conf /usr/lib/systemd/system/docker.service.d └─flannel.conf Active: active (running) since Sat 2015-09-26 10:57:12 … flanneld.service – Flanneld overlay address etcd agent Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled) Active: active (running) since Sat 2015-09-26 10:57:08 ……

[centos@atomictest1 ~]$ ifconfig |grep flags -A 2 docker0: flags=4099 mtu 1500 inet 10.0.80.1 netmask 255.255.255.0 broadcast 0.0.0.0 — eth0: flags=4163 mtu 1500 inet 192.168.122.133 netmask 255.255.255.0 broadcast 192.168.122.255

— flannel.1: flags=4163 mtu 1450 inet 10.0.80.0 netmask 255.255.0.0 broadcast 0.0.0.0

— lo: flags=73 mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 [centos@atomictest1 ~]$

You can validate that, the atomic hosts subnets are allocated properly via below command from any of the nodes:

centos@atomicetcd ~]$ curl -L http://atomicetcd:2379/v2/keys/atomic01/network/subnets | python -m json.tool

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 908 100 908 0 0 5987 0 –:–:– –:–:– –:–:– 6013 { “action”: “get”, “node”: { “createdIndex”: 5, “dir”: true, “key”: “/atomic01/network/subnets”, “modifiedIndex”: 5, “nodes”: [ { “createdIndex”: 5, “expiration”: “2015-09-27T10:38:13.318286933Z”, “key”: “/atomic01/network/subnets/10.0.12.0-24”, “modifiedIndex”: 5, “ttl”: 84248, “value”: “{\”PublicIP\”:\”192.168.122.7\”,\”BackendType\”:\”vxlan\”,\”BackendData\”:{\”VtepMAC\”:\”96:1e:41:4a:aa:ce\”}}” }, { “createdIndex”: 6, “expiration”: “2015-09-27T10:57:08.763771526Z”, “key”: “/atomic01/network/subnets/10.0.80.0-24”, “modifiedIndex”: 6, “ttl”: 85384, “value”: “{\”PublicIP\”:\”192.168.122.133\”,\”BackendType\”:\”vxlan\”,\”BackendData\”:{\”VtepMAC\”:\”de:c1:3a:e4:64:fc\”}}” }, { “createdIndex”: 7, “expiration”: “2015-09-27T11:09:55.54906845Z”, “key”: “/atomic01/network/subnets/10.0.55.0-24”, “modifiedIndex”: 7, “ttl”: 86150, “value”: “{\”PublicIP\”:\”192.168.122.188\”,\”BackendType\”:\”vxlan\”,\”BackendData\”:{\”VtepMAC\”:\”c2:29:e0:13:d9:40\”}}” } ] } } Section 4: Run RHGS containers