Virtualization – Page 3 – My Humble Abode

Open vSwitch (OVS) is a software switch. It consist of user space tools provided by openvswitch rpm and kernel modules provided by kmod-openvswitch rpm. Unlike the bridge available in old versions of distros ( fedora, rhel..etc ) it provides many advanced configuration options like tagged vlan. Full details available at http://www.openvswitch.org/

Open vSwitch is a network switch; at its lowest level, it is concerned with routing packets between interfaces. It is aimed at virtualization users, so, naturally, it is used in the creation of virtual networks. A switch can be set up with a number of virtual network interfaces, most of which are used by virtual machines to communicate with each other and the wider world. These virtual networks can be connected across hosts and across physical networks. One of the key features of Open vSwitch appears to be the ability to easily migrate virtual machines between physical hosts and have their network configuration (addresses, firewall rules, open connections, etc.) seamlessly follow.

Below Process will outline the steps required in CentOS/RHEL..etc

STARTING THE SERVICE

modprobe -r bridge (OVS kernel modules conflict with the “bridge” module)

modprobe openvswitch_mod

modprobe brcompat_mod (OVS bridge compatibility support module. Without this lbvirt cannot use OVS)

Once kernel module part is loaded, lets try to start the subjected service as shown below:

/etc/init.d/openvswitch start

/etc/openvswitch/conf.db does not exist … (warning).
Creating empty database /etc/openvswitch/conf.db           [ OK ]
Starting ovsdb-server                                      [ OK ]
Configuring Open vSwitch system IDs                        [ OK ]
Starting ovs-vswitchd                                      [ OK ]

So, ovs-vswitchd service is started successfully..

Below command needs ‘brcompat_mod’ and will enable libvirt to use brctl to manage OVS.

ovs-brcompatd –pidfile –detach

Below are the steps which I followed to configure vlan ..

HOW TO CREATE TAGGED VLANS

On the HOST

ovs-vsctl add-br br0

ovs-vsctl add-port br0 vlan10 tag=10 — set interface vlan10 type=internal [creating a taged vlan 10]

ifconfig vlan10 192.168.10.254 netmask 255.255.255.0

ovs-vsctl add-port br0 vlan20 tag=20 — set interface vlan20 type=internal [creating a tagged vlan 20]

ifconfig vlan20 192.168.20.254 netmask 255.255.255.0

ovs-vsctl show

Now, lets list the bridge configuration in the system via ‘brctl’ command..

brctl show

If something goes wrong (easiest way ) stop openvswitch service, delete /etc/openvswitch/conf.db and start the service again. Do the configuration again.

To test: Execute below steps in VM1

vconfig add eth0 10 [add vlan interface]

ifconfig eth0.10 192.168.10.100 netmask 255.255.255.0 broadcast 192.168.10.255 up [assign IP]

vconfig add eth0 20

ifconfig eth0.20 192.168.20.100 netmask 255.255.255.0 broadcast 192.168.20.255 up

VM2

vconfig add eth0 10 [add vlan interface]

ifconfig eth0.10 192.168.10.101 netmask 255.255.255.0 broadcast 192.168.10.255 up [assign IP]

vconfig add eth0 20

ifconfig eth0.20 192.168.20.101 netmask 255.255.255.0 broadcast 192.168.20.255 up

Now try pinging VM1 to VM2

***************************************************************************************************************
Below Process will outline Fedora 17, 18, 19 Versions:

Unlike centos/rhel , the openvswitch support is added in libvirt. So, you dont need ‘brcompat_mod’ mentioned in first/above section to use it with libvirt.

Install these packages

[root@Xman 3.9.9-301.fc19.x86_64]# rpm -qa |grep openvswitch
openvswitch-controller-1.10.0-1.fc19.x86_64
openvswitch-1.10.0-1.fc19.x86_64
openvswitch-test-1.10.0-1.fc19.noarch
python-openvswitch-1.10.0-1.fc19.noarch
[root@Xman 3.9.9-301.fc19.x86_64]#

Load openvswitch module and enable the service.

[root@Xman 3.9.9-301.fc19.x86_64]# modprobe openvswitch
[root@Xman 3.9.9-301.fc19.x86_64]# lsmod |grep openvswitch
openvswitch 43393 0

[root@Xman 3.9.9-301.fc19.x86_64]# systemctl enable openvswitch
ln -s ‘/usr/lib/systemd/system/openvswitch.service’ ‘/etc/systemd/system/multi-user.target.wants/openvswitch.service’

[root@Xman 3.9.9-301.fc19.x86_64]# systemctl start openvswitch
[root@Xman 3.9.9-301.fc19.x86_64]# systemctl status openvswitch
openvswitch.service – Open vSwitch
Loaded: loaded (/usr/lib/systemd/system/openvswitch.service; enabled)
Active: active (exited) since Fri 2013-08-02 08:26:25 EDT; 5s ago
Process: 31565 ExecStart=/usr/share/openvswitch/scripts/openvswitch.init start (code=exited, status=0/SUCCESS)
CGroup: name=systemd:/system/openvswitch.service
├─31590 ovsdb-server: monitoring pid 31591 (healthy)
├─31591 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info –remote=punix:/var/run/openvswitch/db.sock –private-key=db:Open_vSwitch,SSL,private_key …
├─31599 ovs-vswitchd: monitoring pid 31600 (healthy)
├─31600 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info –mlockall –no-chdir –log-file=/var/log/openvswitch/ovs-vswitchd.log –pidfile=…
└─31601 ovs-vswitchd: worker process for pid 31600

Aug 02 08:26:25 Xman openvswitch.init[31565]: /etc/openvswitch/conf.db does not exist … (warning).
Aug 02 08:26:25 Xman openvswitch.init[31565]: Creating empty database /etc/openvswitch/conf.db [ OK ]
Aug 02 08:26:25 Xman openvswitch.init[31565]: Starting ovsdb-server [ OK ]
Aug 02 08:26:25 Xman ovs-vsctl[31592]: 00001|vsctl|INFO|Called as ovs-vsctl –no-wait — init — set Open_vSwitch . db-version=6.12.0
Aug 02 08:26:25 Xman ovs-vsctl[31597]: 00001|vsctl|INFO|Called as ovs-vsctl –no-wait set Open_vSwitch . ovs-version=1.10.0 “external-ids:system-id…unknown\””
Aug 02 08:26:25 Xman openvswitch.init[31565]: Configuring Open vSwitch system IDs [ OK ]
Aug 02 08:26:25 Xman openvswitch.init[31565]: Starting ovs-vswitchd [ OK ]
Aug 02 08:26:25 Xman openvswitch.init[31565]: Enabling remote OVSDB managers [ OK ]
Aug 02 08:26:25 Xman openvswitch.init[31565]: Enabling gre with iptables [ OK ]
Aug 02 08:26:25 Xman systemd[1]: Started Open vSwitch.
[root@Xman 3.9.9-301.fc19.x86_64]#

[root@Xman 3.9.9-301.fc19.x86_64]# ps aux |grep openv
root 31591 0.0 0.0 43592 1996 ? S< 08:26 0:00 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor root 31600 0.0 0.1 43880 9028 ? S mtu 1500
inet 192.168.10.254 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::e062:e7ff:fe73:5a0 prefixlen 64 scopeid 0x20 ether e2:62:e7:73:05:a0 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6 bytes 468 (468.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

vlan20: flags=67 mtu 1500
inet 192.168.20.254 netmask 255.255.255.0 broadcast 192.168.20.255
inet6 fe80::8b3:e6ff:fe5a:a2e3 prefixlen 64 scopeid 0x20 ether 0a:b3:e6:5a:a2:e3 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 6 bytes 468 (468.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[root@Xman 3.9.9-301.fc19.x86_64]#

Now, before starting a guest please edit the guest configuration as shown below:

–> It should be filled with the mac–> This should be the ‘bridge’ you configured for ovs
–> This add ovs support from libvirt layer

Once you started the guest please follow the same process mentioned ( configuration of vlan networks in guest) in section 1 ( inside Centos/RHEL..etc)

You should be good to go.

Reference:

http://openvswitch.org/support/

http://lwn.net/Articles/469775/

openvswitch faq..

First of all, what are hugepages ? I would love to explain this, but it is already did by someone here, so I am just quoting some bits of it..

Thats it.. Hope this helps..

[root@humbles-lap qemu_upstream]# cat /proc/meminfo |grep -i huge
AnonHugePages: 434176 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB ======; 2M
[root@humbles-lap qemu_upstream]#

[root@humbles-lap qemu_upstream]# sysctl -a |grep -i huge
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.hugetlb_shm_group = 0
vm.hugepages_treat_as_movable = 0
vm.nr_overcommit_hugepages = 0

My guest have ‘512M’ of memory , so I decided to give “256” ( total 256*2M=512) as the number of hugepages

[root@humbles-lap qemu_upstream]# sysctl vm.nr_hugepages=256
vm.nr_hugepages = 256
[root@humbles-lap qemu_upstream]#

[root@humbles-lap qemu_upstream]# sysctl -a |grep -i huge
vm.nr_hugepages = 256
vm.nr_hugepages_mempolicy = 256
vm.hugetlb_shm_group = 0
vm.hugepages_treat_as_movable = 0
vm.nr_overcommit_hugepages = 0
[root@humbles-lap qemu_upstream]#

So, hugepage assignment reflected here:

[root@humbles-lap qemu_upstream]# cat /proc/meminfo |grep -i huge
AnonHugePages: 450560 kB
HugePages_Total: 256
HugePages_Free: 256
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@humbles-lap qemu_upstream]#

Once you assigned hugepages, you need to restart ‘libvirtd” service…

[root@humbles-lap hpages]# service libvirtd restart
Stopping libvirtd daemon: [ OK ]
Starting libvirtd daemon: [ OK ]
[root@humbles-lap hpages]#

Check mount point for ‘existence’ of a directory as shown below:

[root@humbles-lap qemu_upstream]# cd /dev/hpages/
[root@humbles-lap hpages]# ls
libvirt
[root@humbles-lap hpages]# cd libvirt/
[root@humbles-lap libvirt]# ls
qemu
[root@humbles-lap libvirt]# cd qemu/
[root@humbles-lap qemu]# ls
[root@humbles-lap qemu]#
With the current configuration, I have NOT specified to have ‘hugepage’ backed memory for my guest..

[root@humbles-lap qemu_upstream]# cat /etc/libvirt/qemu/rhel5.4-x86_64-kvm.xml |grep -i memory
memory;524288/memory;
currentMemory;524288/currentMemory;
[root@humbles-lap qemu_upstream]#

I edited my guest xml configuration to say, this guest memory is backed with “HUGEPAGES”..

[root@humbles-lap qemu_upstream]# cat /etc/libvirt/qemu/rhel5.4-x86_64-kvm.xml|grep -i memory -A 2
memory;524288/memory;
currentMemory;524288/currentMemory;
;memoryBacking;
hugepages/;
/memoryBacking;
vcpu;2/vcpu;
os;
[root@humbles-lap qemu_upstream]#

Started guest:

[root@humbles-lap qemu_upstream]# virsh create /etc/libvirt/qemu/rhel5.4-x86_64-kvm.xml
Domain rhel5.4-x86_64-kvm created from /etc/libvirt/qemu/rhel5.4-x86_64-kvm.xml

[root@humbles-lap qemu_upstream]# virsh list
Id Name State
———————————-
1 rhel5.4-x86_64-kvm running

[root@humbles-lap qemu_upstream]#

Now, check ‘meminfo’ in HOST system to know whether “HUGEPAGES” are used with GUESTS..

[root@humbles-lap qemu_upstream]# cat /proc/meminfo |grep -i huge
AnonHugePages: 483328 kB
HugePages_Total: 256
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
[root@humbles-lap qemu_upstream]#

Ovirt 3.2 is out , catch it …

How to share ‘a disk’ to more than one VM using libvirt? Or What ‘sharable’ flag do in libvirt ?

Is it possible to do online resizing of guest block devices ( Or without shutdown ?)

QMP ( qemu monitor protocol ) and Different ways of accessing it

‘lscpu’ command to check ‘virtualization’ support of CPU/processor

virtio-scsi implementation details, documentation & other references..

“error: internal error Process exited while reading console log output: Supported machines are:” …

Ovirt Videos ( ovirt architecture, gui , hypervisor/node) ..etc

Configure openvswitch in virtualization environment and use it for simple and complex (ex: vlan) testing in centos/rhel/fedora..etc

Give hugepage memory for guests in KVM virtualization