Nested virtualization with KVM/VMX OR Guest inside guest in kvm in fedora linux

I would like to explain “nested Virtualization” in my own terms . How-ever it is already documented , so I am just quoting some bits from the same.

On Intel processors, KVM uses Intel’s VMX (Virtual-Machine eXtensions)
to easily and efficiently run guest operating systems. Normally, these guests
*cannot* themselves be hypervisors running their own guests, because in VMX,
guests cannot use VMX instructions.

The “Nested VMX” feature adds this missing capability – of running guest
hypervisors (which use VMX) with their own nested guests. It does so by
allowing a guest to use VMX instructions, and correctly and efficiently
emulating them using the single level of VMX available in the hardware.


Single-level virtualization has two levels – the host (KVM) and the guests.
In nested virtualization, we have three levels: The host (KVM), which we call
L0, the guest hypervisor, which we call L1, and its nested guest, which we
call L2.

Running nested VMX

The nested VMX feature is disabled by default. It can be enabled by giving
the “nested=1” option to the kvm-intel module.

No modifications are required to user space (qemu). However, qemu’s default
emulated CPU type (qemu64) does not list the “VMX” CPU feature, so it must be
explicitly enabled, by giving qemu one of the following options:

     -cpu host              (emulated CPU has all features of the real CPU)

     -cpu qemu64,+vmx       (add just the vmx feature to a named CPU type)

Instead of explaining a lot of theory about it, I will demo its use:


//Checking whether “kvm_intel” will support this parameter?

[root@humbles-lap Humble]# modinfo kvm_intel
filename:       /lib/modules/
license:        GPL
author:         Qumranet
depends:        kvm
intree:         Y
vermagic: SMP mod_unload
parm:           vpid:bool
parm:           flexpriority:bool
parm:           ept:bool
parm:           unrestricted_guest:bool
parm:           emulate_invalid_guest_state:bool
parm:           vmm_exclusive:bool
parm:           yield_on_hlt:bool
parm:           fasteoi:bool
parm:           nested:bool
parm:           ple_gap:int
parm:           ple_window:int

// This feature is “disabled” by default..

[root@humbles-lap Humble]# cat /sys/module/kvm_intel/parameters/nested

// Remove the module

[root@humbles-lap Humble]# rmmod kvm_intel
[root@humbles-lap Humble]# lsmod |grep kvm_intel
[root@humbles-lap Humble]#

// Make the ‘nested’ boolean “ON” and reload:

[root@humbles-lap Humble]# insmod /lib/modules/ nested=Y
[root@humbles-lap Humble]# lsmod |grep intel
kvm_intel             132406  0
snd_hda_intel          33276  3
snd_hda_codec         115767  3 snd_hda_codec_hdmi,snd_hda_codec_realtek,snd_hda_intel
snd_pcm                97170  4 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec
snd                    78908  14 snd_hda_codec_hdmi,snd_hda_codec_realtek,snd_hda_intel,snd_hda_codec,snd_hwdep,snd_seq,snd_seq_device,snd_pcm,snd_timer
kvm                   407039  1 kvm_intel
snd_page_alloc         18101  2 snd_hda_intel,snd_pcm
intel_ips              18304  0

[root@humbles-lap Humble]# cat /sys/module/kvm_intel/parameters/nested
[root@humbles-lap Humble]#



Now, if you are configuring a virtual machine or guest as another hypervisor, you should have virtualization support in its processor. For that you have to export your CPU as “vmx” enabled one. You can achieve this by editing guest configuration file and then starting the vm.

Below is my guest configuration file wrt to the CPU session . The important field here is “”



[root@humbles-lap Humble]# cat /etc/libvirt/qemu/rhel5.4-x86_64-kvm.xml |grep -w cpu -A 13
  <cpu match='exact'>
    <feature policy='require' name='est'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='tm'/>



//Then I started this VM via “virsh” command..

[root@humbles-lap Humble]# virsh create /etc/libvirt/qemu/rhel5.4-x86_64-kvm.xml
Domain rhel5.4-x86_64-kvm created from /etc/libvirt/qemu/rhel5.4-x86_64-kvm.xml

[root@humbles-lap Humble]#

// Check the ‘qemu-kvm’ process for “+vmx” flag

[root@humbles-lap Humble]# ps aux |grep -i qemu-kvm |grep vmx
qemu      8389 50.6  1.3 872040 53908 ?        Sl   23:26   0:06 /usr/bin/qemu-kvm -S -M pc-0.13 -cpu core2duo,+lahf_lm,+rdtscp,+aes,+popcnt,+sse4.2,+sse4.1,+xtpr,+cx16,+tm2,+est,+vmx,+ds_cpl,+pbe,+tm,+ht,+ss,+acpi,+ds -enable-kvm -m 512 -mem-prealloc -mem-path /dev/hugepages/libvirt/qemu -smp 2,sockets=2,cores=1,threads=1 -name rhel5.4-x86_64-kvm -uuid b2d13137-ff17-d505-0a6b-fb79c867584a -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel5.4-x86_64-kvm.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -boot c -drive file=/misc/guest-images/rhel5.4-x86_64-kvm.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=21,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=54:52:00:6c:6e:70,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -vnc -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
[root@humbles-lap Humble]#

Once “VM” started, try to login to that VM and see whether its processor support virtualization..



[Humble@humbles-lap ~]$ ssh root@
root@‘s password:
Last login: Thu Jun 14 23:28:20 2012
[root@dhcp208-238 ~]# cat /proc/cpuinfo |grep -i vmx
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc pni vmx ssse3 cx16 sse4_1 sse4_2 popcnt lahf_lm
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc pni vmx ssse3 cx16 sse4_1 sse4_2 popcnt lahf_lm


Now you have to believe me that this VM can act as a host/hypervisor system for another VM.


As you do in a host system, try to create VM inside this guest and play.


Reference# nested-vmx.txt in kernel source

2 Replies to “Nested virtualization with KVM/VMX OR Guest inside guest in kvm in fedora linux”

  1. I’ve been exploring for a little for any high-quality articles or blog posts in this sort of house . Exploring in Yahoo I at last stumbled upon this website. Reading this info So i’m happy to convey that I have a very good uncanny feeling I came upon exactly what I needed. I most certainly will make sure to do not disregard this site and give it a look on a constant basis.

Leave a Reply

Your email address will not be published. Required fields are marked *