Jump to content
Welcome to our new Citrix community!
  • 0

Nested Hypervisors in Xenserver 7

John Fullbright


Hi All,


I installed Xenserver 7 yesterday.  Today I think I want to play with nested hypervisors.  I was looking on xen.org, and it seems I need to add two parameters and change 1.  I have some prior experience making templates, and this looks doable. http://wiki.xen.org/wiki/Nested_Virtualization_in_Xen



Has anyone actually tried it yet?  If so, would you care to share your experiences?






Link to comment

10 answers to this question

Recommended Posts

  • 0

It's doable, known to work, but is of course unsupported. Have not played with (yet) myself, but may. Be cautious about trying to run newer version of XenServer under of older ones as not all functionality may be recognized in terms of how in particular connectivity is handled up the chain.


Link to comment
  • 0

Keep in mind that when running a hypervisor under another hypervisor CPU virtualization extensions are not available for the 2nd hypervisor. So you can only run a 2nd hypervisor that is able to work without those virtualization extensions. Xen for instance can do this but it losses certain capabilities that will limit its functionality and lower its speed.


There are also some hypervisors that can emulate those CPU virtualization extensions for VM's. So these will provide full fucntionality to the 2nd hypervisor though with reduced performance.


So running a hypervisor under another hypervisor is never recommended for production use, there are just too many downsides. For testing it might be fine. (I sometimes run XenServer under Vmware player/Virtualbox on my Windows desktop, to find out if a certain driver is available, or how to make yum install work under XenServer 7.0) For real testing scenarios I keep some old servers and storage devices though.

Link to comment
  • 0

Hi Willem,


"Keep in mind that when running a hypervisor under another hypervisor CPU virtualization extensions are not available for the 2nd hypervisor."


This is what has changed, and what I want to experiment with.  It's based on hardware (EPT and VT-d) and has been in "tech preview" since xen 4.4.  Recently, it is possible to do the same with Hyper-v in Windows Server 2016 TP5. https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/user_guide/nesting  I have tried this on Windows Server 2016 TP5 and it does work.


In either case, you expose the virtualization extensions to the guest.  For the Xen hypervisor, prior testing is documented here http://wiki.xen.org/wiki/Nested_Virtualization_in_Xen


In both cases, it's not for production and is considered experimental. I wanted to try on Xenserver 7 since it uses the 4.6 hypervisor.   In Xen, the configuration looks like:

    hap=1    nestedhvm=1    cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']

Well, I know where to mask the CPU bits in a xenserver template, and am fairly familiar with manipulating them or building them.  I had to make for FreeBSD last year, and it wasn't really that hard.  hap= and nestedhvm= look like QEMU parameters that get passed through, so I just need to figure out where in the template to add them.



Link to comment
  • 0

I think I figured it out.  At least I can see vmx in flags of cpuinfo in my Ubuntu guest. 


You enable it in the vm with xe vm-param-set uuid=<uuid> platform...


I had to dig through the source code to figure out which platform setting, but it appears to be working.  I'll try some various guest hypervisors tomorrow and see which ones work/which ones don't.



Link to comment
  • 0



It does work.


L0 = Fujistu Primergy Rx200 S7 - Xenserver 7

L1 = Xenserver 7

L2 = Ubuntu Xenial 16.04


I used the Centos 7 template with the xenserver 7 ISO to create the L1.  Adjust the boot LUN size to whatever you want.  Uncheck start the VM.  Then, from the console, find the uuid of the vm and:


xe vm-param-set uuid=<your vm uuid> platform:exp-nested-hvm=true


The start the VM and load xenserver 7 in the L1.  I added the L1 to my pool.  I was then able to access all the shared SRs and create an ubuntu xenial L2.


I tried just ubuntu with Xen 4.6 as an L1, and that worked.  I tried ubuntu with KVM in the L1, and that appeared to work as well.  I then tried Windows Server 2016 TP5 in the L1.  Windows loads, you can see the hypervisor, vmx, and ept are there, you can install the hyper-v role, but when you go to launch the L2 hyper-v errors.  It's event ID 46.  MSR index 48d has a value that is not within the accepted capabilities range for Hyper-v essentially. 


coreinfo says


Intel® Xeon® CPU E5-2620 0 @ 2.00GHz

Intel64 Family 6 Model 45 Stepping 7, GenuineIntel

Microcode signature: 00000710

HYPERVISOR      *       Hypervisor is present

VMX             *       Supports Intel hardware-assisted virtualization

EPT             *       Supports Intel extended page tables (SLAT)

PS C:\>


Probably something wrong with the way the CPUid bits are masked.  In any event, I have not tried vmware as an L1 yet.


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...