Blog posts tagged with 'xenserver'

Well, the big launch of XenServer 5 has gone smoothly, and with it have arrived a flood of questions about how exactly the new High Availability functionality works.  I'll use this post to explain the overall architecture of HA in XenServer 5, and also how some of the fault detection and failure planning works.

Fundamentally, HA is about making sure important VMs are always running on a resource pool. There are two aspects to this: reliably detecting host failure, and computing a failure plan to deal with swift recovery.

Detecting host failure reliably is difficult since you need to remotely distinguish between a host disappearing for a while versus exploding in a ball of flames.  If we mistakenly decide that a master host has broken down and elect a new master in its place, there may be unpredictable results if the original host were to make a comeback!   Similarly, if there is a network issue and a resource pool splits into two equal halves, we need to ensure that only one half accesses the shared storage and not both simultaneously.

Heartbeating for availability

We solve all these problems in XenServer by having two mechanisms: a storage heartbeat and a network heartbeat. When you enable HA in a pool, you must nominate an iSCSI or FC storage repository to be the heartbeat SR. XenServer automatically creates a couple of small virtual disks in this SR. The first disk is used by every physical host in the resource pool as a shared quorum disk. Each host allocates itself a unique block in the shared disk and regularly writes to the block to indicate that it is alive.

I asked Dave 'highly available' Scott, the principal engineer behind HA about the startup process:

"When HA starts up, all hosts exchange data over both network and storage channels, indicating which hosts *they* can see over both channels; i.e. which I/O paths are working and which are not.  This liveness information is exchanged until a fixed point is reached and all of the hosts are satisfied that they are in agreement about what they can see.  When this happens, the HA functionality is 'armed' and the pool is protected."

This HA arming process can take a few minutes to settle for larger pools, but is only required when HA is first enabled.

Once HA is active, each host regularly writes storage updates to the heartbeat virtual disk, and network packets over the management interface.  It is vital to ensure that network adapters are bonded for resilience, and that storage interfaces are using dynamic multipathing where supported.  This will ensure that any single adapter or wiring failures do not result in any availability issues.

The worst-case scenario for HA is the situation where a host is thought to be off-line but is actually still writing to the shared storage, since this can result in corruption of persistent data.  In order to prevent this situation without requiring active power strip control, we implemented hypervisor-level fencing.  This is a Xen modification which will hard-power the host off at a very low-level if it doesn't hear regularly from a watchdog process running in the control domain.  Since it is implemented at a very low-level, this also covers the case where the control domain becomes unresponsive for any reason.

Hosts will self-fence (i.e. power off and restart) in the event of any heartbeat failure unless any of the following hold true:

• The storage heartbeat is present for all hosts but the network has partitioned (so that there are now two groups of hosts).  In this case, all of the hosts which are members of the largest network partition stay running, and the hosts in the smaller network partition self-fence.  The assumption here is that the network outage has isolated the VMs, and they ought to be restarted on a host with working networking.  If the network partitions are exactly the same size, then only one of them will self-fence according to a stable selection function.
• If the storage heartbeat goes away but the network heartbeat remains, then the hosts check to see if they can see all other hosts over the network.  If this condition holds true, then the hosts remain running on the assumption that the storage heartbeat server has gone away.  This doesn't compromise VM safety, but any network glitches will result in fencing since that would mean both heartbeats have disappeared.

Planning for failure

The heartbeat system gives us reliable notification of host failure, and so we move onto the second step of HA: capacity planning for failure.

A resource pool consists of several physical hosts (say, 16), each with potentially different amounts of host memory and a different number of running VMs.  In order to ensure that no single host failure will result in the VMs on that host being unrestartable (e.g. due to insufficient memory on any other host), the XenServer pool dynamically computes a failure plan which calculates the actions that would be taken on any host failure.

But there's one more complexity... a single host failure plan does not cover more advanced cases such as network partitions which take out entire groups of hosts.  It would be very useful to be able to create a plan that could tolerate more than a single host failure, so that administrators could ignore the first host failure and be safe in the knowledge that (for example) three more hosts could fail before the pool runs out of spare capacity.

That's exactly what we do in XenServer... the resource pool dynamically computes a failure plan which considers the "number of host failures to tolerate" (or nhtol).  This represents the number of disposable servers in a pool for a given set of protected VMs.

The planning algorithms are pretty complex, since doing a brute force search of all possible failures across all hosts across all VMs is an exponential problem.  We apply heuristics to ensure we can compute a plan in a reasonably small time:

• for up to 3 host failures, we do a comprehensive search which tries almost all permutations.  This covers corner cases such as having hosts or VMs with very different amounts of memory (e.g. 4GB vs 128GB).  Rather than calculate memory slots or otherwise approximate results, we just deal with them individually and give very accurate plans.
• for greater than 3 host failures, we make conservative decisions by approximating every VM to be as large as the largest, and considering each host to be the same as the most densely packed host.  We do not approximate the host memory, and so having pools with uneven amounts of host memory will be fine.  However, in approximate planning mode having a single very large VM will result in a low nhtol value.  If this is a problem, then try to reduce the nhtol or try to have a more even spread of VM memory sizes.

Since planning algorithms are designed for unexpected host failures, we only consider absolutely essential resource reservations which would prevent the VM from starting on the alternative host (e.g. storage is visible, and enough memory is present).  We do not perform CPU reservation on the basis that it can be optimised at a later stage via live relocation once the VM is back up and running.

Overcommit protection

We now have HA armed and a failover plan for our VMs.  But what if you want to make changes to your configuration after HA is enabled?  This is dealt with via overcommit protection.

The XenServer pool dynamically calculates a new failover plan in response to every XenAPI call which would affect it (e.g. starting a new VM).  If a new plan cannot be calculated due to insufficient resources across the pool, the XenServer will return an overcommitment error message to the client which blocks the operation.

The "What if?" Machine

This overcommit protection would be quite irritating if you have to keep trying things and seeing if a plan exists or not, and so we built in a "What If?" machine into XenServer to facilitate counter-factual reasoning.

When reconfiguring HA via XenCenter, you can supply a hypothetical series of VM priorities, and XenServer will return a number of host failures which would be tolerated under this scheme.  This lets you try various combinations of VM protections depending on your business needs, and see if the number of host failures is appropriate to the level of paranoia you desire.

This can even be done via the CLI, using the snappily named "xe pool-ha-compute-max-host-failures-to-tolerate" when HA is enabled.

The nice thing about XenServer HA is that it is done at the XenAPI level, and so  any of the standard clients (such as the xe CLI or XenCenter) or any third-party clients which use the XenAPI will all interoperate just fine.  The XenServer pool dynamically recalculates plans in response to the client requests, and so no special "oracle" is required outside of the pool to figure out HA plans.

Finally, HA makes master election completely invisible.  Any host in a pool can be a master host, and the pool database is constantly replicated across all nodes and also backed up to shared storage on the heartbeat SR for additional safety.  Any XenAPI client can connect to any host, and a redirect is issued to the current master host.

Protection Levels

Each VM in an HA pool can be either fully protected, best-effort or unprotected. VMs which are protected are all included in the failover planning, and if no plan exists for which they can all be reliably restarted then the pool is considered to be overcommitted.

Hugh Warrington (who implemented the XenCenter HA support) explained what use protection levels are:

"Best-effort VMs are not considered when calculating a failover plan, but the pool will still try to start them as a one-off if a host that is running them fails.  This restart is attempted after all protected VMs are restarted, and if the attempt to start them fails then it will not be retried.  This is a useful setting for test/dev VMs which aren't critical to keep running, but would be nice to do so in a pool which also has some important VMs which absolutely must run."

There are some advanced features which are only available via the CLI.   Each protected VM in an HA pool can be assigned a numeric ha-restart-priority.  If a pool is well-resourced with a high nhtol, then these restart priorities are not relevant: the VMs are all guaranteed to be started.

If more hosts fail than have been planned for, then the priorities are used to determine the order in which VMs are restarted.  This ensures that in over-committed pools, the most important VMs are restarted first.  Although the pool will start priority 1 VMs first, they might not finish booting before the priority 2 VMs, and so this should not be used as the basis for service ordering.

Note that it's very important to ensure that a VM is agile when protecting it by HA.  If the VM is not agile (e.g has a physical CD drive mapped in from a host), then it can only be assigned Best Effort restart since it is tied to one host!

XenCenter support for HA

The best practice for HA is not to make configuration changes while it is enabled.  Instead, it is intended to be the "2am safeguard" which will restart hosts in the event of a problem when there isn't a human administrator nearby.  If you are actively making configuration changes such as applying patches, then HA should be disabled for the duration of these changes.

XenCenter makes some common changes under HA much more user-friendly, which I asked Ewan Mellor (the principal GUI engineer) about:

• Normally a protected VM cannot be shut down via the CLI or from within the guest (a shutdown from within the guest will automatically restart it).  If you try to shutdown from XenCenter, it will give you the option of unprotecting the VM and then shutting it down first.  Thus, accidental in-guest shutdowns wont result in downtime, but administrators can still stop a protected guest if they really want to.
• If you want to reboot a host when HA is enabled, XenCenter automatically uses the hypothetical planning calculation to determine if this would invalidate the failover plan.  If it doesn't affect it, then the host is shut down normally.  If the plan would be violated, but the nhtol is greater than 1, XenCenter will give the administrator the option of lowering the nhtol value by 1.  This reduces the overall resilience of the pool, but always ensures that at least one host failure will be tolerated.  When the host comes back up, the plan is automatically recalculated and the original nhtol value restored if appropriate.
• If you try to apply a hotfix, then XenCenter will disable HA for the duration of the pool patching wizard.  It is important to manually keep an eye on hotfix application to ensure that host failures do not disrupt the operation of the pool.

So, I hope this short article has given you a taster... just kidding! This post is almost as long as my PhD thesis, but then, HA is a complex topic. Please do feel free to get back to me with comments and feedback about how we can improve it in the future releases, or if you just love it the way it is.  Many thanks to Dave Scott, Richard Sharp, Ewan Mellor and Hugh Warrington for their input to this article.

You won't be surprised to hear that we spend a lot of time improving XenApp performance when running on XenServer. Although there are some good benchmark comparisons available (such as the Tolly Group report), I still get a lot of customers asking about what the "secret sauce" is. I sat down with George Dunlap, the lead XenServer performance engineer to chat about the very first optimisation we did back in XenServer 4.0 last year.

Before we dive in, we first need to explain how a normal operating system handles memory. George explains:

"Modern desktop and server processors don't access memory directly using its physical address. They use 'virtual memory' to separate the addresses that processes use to read and write memory from the actual memory itself. This allows operating systems to hide from processes all the dirty details of how much memory there is, where in physical memory the process needs to write to, and so on."

"However, the actual processor still needs to translate from a virtual address to the physical memory address in order to actually read and write any memory. This translation is done with something called page tables."

"Page tables are used to implement virtual memory by mapping virtual addresses to physical addresses. The operating system constructs page tables using physical memory addresses, and then puts the physical address of the "top-level" page table into a hardware register called the 'base pointer'. Then the processor will read these page tables to translate virtual addresses to physical addresses as needed, before reading and writing to physical memory."

Most modern processor types have some sort of paging mechanism, although XenServer is specifically tuned for x86-64 CPUs. An excellent book on the general topic is Modern Operating Systems by Andrew Tanenbaum. When XenServer creates Windows VMs, it takes advantage of the virtualization extensions in modern CPUs, which requires special memory handling in Xen. George explains this further:

"When we create a virtual machine, we virtualize the memory as well; that means that the guest operating system's idea of physical memory does not match up to real physical memory on the host. Traditionally, what the guest thinks of as physical memory is called "physical memory", and what the hypervisor thinks of as physical memory is called "machine memory". Since this terminology is a bit confusing, Xen tends to call what the guest thinks of as physical memory as "guest physical" memory, just to help make things more clear."

"This means that any fully-virtualized operating system, like Windows, will create page tables using guest physical memory, and will point the base pointer at the guest physical address of the top-level page table. Unfortunately, the hardware still needs to map from virtual memory address to machine addresses, not guest physical addresses."

"In order to allow this to happen, the hypervisor sets up shadow page tables. These page tables are generated by the hypervisor are copies of the guest page tables, but with the guest physical addresses converted into machine physical addresses. The guest cannot access them directly, and they don't reside in the guest's physical memory; they're generated out of a pool of memory that the hypervisor allocates when a VM is created, called "shadow page table memory"."

"What this means is that whenever the guest operating system wants to map some new memory, after it writes the data into the page table but before it can actually use it, the hypervisor needs to translate the change to the guest page table into changes to the shadow page table. So any workload that involves a lot of this will necessarily involve the hypervisor a lot, which causes overhead."

So shadow page tables are our mechanism of giving a guest an interface which is identical to real hardware (so it doesn't need to be modified), but still intercepting changes before they reach the real hardware. You can find more details from the XenSummit 2006 talk or from the 2005 NSDI paper. So how is this all relevant to XenApp performance? Back to George...

"The hypervisor allocates a certain amount of memory for each VM to use for shadow page tables; this is called shadow page table memory. As new page tables are created and old ones aren't used anymore, the hypervisor cycles through this shadow page table memory. When it needs a new page and there isn't enough, it will 'unshadow' the guest page tables that haven't been used for the longest time to reclaim shadow memory, so that it can use more."

"We don't know ahead of time how much shadow memory a given workload will use, but we can estimate based on the amount of memory that the VM has. We allocate enough shadow memory for each page to be mapped once, more or less, then add an extra 50% to have some slack. For all the workloads we've tested, that's been enough – except XenApp."

"XenApp is the one workload we've found that requires more shadow page table memory than our standard default. Because XenApp generally starts hundreds of copies of the same process, the same memory ends up mapped in hundreds of different processes. What happens when all of those processes are active is that XenServer is continually unshadowing one process' page tables in order to shadow another process' pagetables; only to have to re-shadow the original ones a second or two later when it runs again! This is called thrashing, when there's not enough of a limited resource."

Once the bottleneck was discovered, the solution was simple. In XenServer 4.1, we created a special XenServer application template called "Citrix XenApp", which has an increased shadow multiplier that reserves more shadow memory for the guest when it starts. This is also a good example of how templates hide the complexities of performance tuning from the user, but still permitting custom modifications if they are required. For example, on your XenServer host with a VM called "XenApp", you could view the shadow multiplier by using the CLI:

# xe vm-list name-label=XenApp params=HVM-shadow-multiplier
HVM-shadow-multiplier ( RW) : 4.000

The same value is also available from XenCenter in the Optimization pane, but of course do remember that the default value was chosen through extensive testing and doesn't need to be changed. Most of the other templates in XenServer also have carefully tuned settings (e.g. the hardware platform flags) to ensure smooth running, or in the case of Linux templates, to support para-virtual installation. This is why it's so important that you not use the "Other Install Media" template in preference of a more specialised one!

I mentioned at the beginning of this post that this was the first of many XenApp optimisations. We've just released the public beta of the latest XenServer ("Orlando") which is even faster. The story of what those improvements are, and the tools which George and his team uses to analyze the inner workings of Xen, are a topic for a future post. For now, get downloading XenServer and start virtualizing your XenApp installations! Or if you're feeling inspired, go over to xen.org, check out the source, and get coding...

I thought I'd kick off my Citrix blog with a question I get pretty often from Linux enthusiasts: how to install unsupported Linux distributions on XenServer 4.1.

The most common solution people find is to use the "Other Install Media" template, insert the distribution installation CD, and find that the mouse cursor doesn't work when they boot into X11. The reason for this is that they are using the hardware-assisted emulation mode of installing Linux. In this mode (dubbed "HVM"), all input and output is emulated, and in particular the mouse interface uses the USB tablet interface. If the distribution doesn't include a driver for USB tablets, then no mouse will appear.

Windows guests run at high-speed in HVM mode due to the installation of the XenServer tools which install high-speed drivers, but these are not necessary for Linux distributions since they can be run in para-virtualized mode (dubbed "PV"). This involves obtaining a Xen-enabled PV kernel from the distribution, and modifying the VM record in XenServer to boot into this kernel instead of HVM mode. The XenServer built-in templates for popular distributions such as RHEL, CentOS or SUSE Linux already automate all this and are in PV mode from the installer onwards.

In the remainder of this post, I'll explain how to take a distribution without direct support (Ubuntu 8.04), get it installed in HVM mode on XenServer 4.1, and convert it to PV mode with a XenCenter graphical console.

• Download the "Alternative Installation CD". The main installation CD uses graphical mode, which won't install as well in HVM mode due to the use of esoteric 16-bit mode instructions for the graphics operations. The 16-bit emulation mechanisms vary between processors (with better support on AMD chips, and a software instruction emulator required on Intel VT chips). However, the Ubuntu alternate CD uses a text-based installer which works fine.

• Create a new VM on the XenServer 4.1 host using the "Windows Server 2003" template. This template is set up with a sensible set of hardware emulation flags and default disks, and so is a good base for the HVM installation of Ubuntu as well. Attach the Ubuntu ISO you just downloaded to the VM, and proceed to install Ubuntu as normal. You should install it onto the first disk, to make the subsequent steps in this guide easier.

• When the installation is finished, reboot the VM (don't forget to detach the installation ISO first). It should boot up in HVM mode into the graphical login screen. The XenCenter display will show it as not being optimized, which is fine. At this stage, I prefer to work via a remote command-line using SSH. Open up a Terminal from Ubuntu, and run "sudo apt-get install openssh-server". Then find out the VM's IP address with "ifconfig eth0", and then connect to it remotely. Alternatively, you can continue to type in the commands directly into the terminal as well.

• On the Ubuntu guest, you now need to install the latest Xen version of the Ubuntu kernel:
  • Install the Linux kernel virtual package with "sudo apt-get install linux-image-xen". This is a virtual package which pulls in the latest Xen kernel and modules, in my case 2.6.24.19.21.
  • You now need to workaround a bug in grub. Due to the switch in recent versions of Linux to work with the hypervisor-independent paravirt_ops interface, update-grub doesn't update the grub configuration with your newly installed Xen kernel. To fix this:
    • Open /boot/grub/menu.lst in your favourite editor.
    • Scroll to the bottom to the kernel list, and find the entry which looks like:

      title           Ubuntu 8.04, kernel 2.6.24-16-generic
      root            (hd0,0)
      kernel          /boot/vmlinuz-2.6.24-16-generic root=UUID=<uuid> ro quiet splash
      initrd          /boot/initrd.img-2.6.24-16-generic
      quiet
      

    • Add a new entry which is similar to this, but change all references to the 2.6.24-16-generic to the Xen kernel. In /boot I have vmlinuz-2.6.24-19-xen, so my new entry looks like:

      title           Ubuntu 8.04, kernel 2.6.24-19-xen
      root            (hd0,0)
      kernel          /boot/vmlinuz-2.6.24-19-xen root=UUID=<uuid> ro quiet splash
      initrd          /boot/initrd.img-2.6.24-19-xen
      quiet
      

    • Also edit the default entry in the menu.lst to match the number of the kernel you just added. I set mine to 3, since it is the fourth entry in the list and the indexing starts from 0.

• When this is done, shut down the guest but do not reboot it just yet. You first need to edit the VM record for your Ubuntu VM to convert it to PV boot mode. From the control domain console of your XenServer:
  • Determine the UUID of the Ubuntu VM by using the xe CLI:
    • xe vm-list name-label=Ubuntu params=uuid --minimal : this will print out the UUID of the VM named "Ubuntu". If you are logged into the control domain, pressing the <tab> key will perform auto-completion of UUIDs in subsequent XE commands, so you don't need to keep typing it in every time!
    • xe vm-param-set uuid=<uuid> HVM-boot-policy= : this will clear the HVM boot mode from the VM.
    • xe vm-param-set uuid=<uuid> PV-bootloader=pygrub : this will switch the VM to using to the pygrub bootloader which starts the guest in PV mode by examining its filesystem for kernel.
    • vm vm-param-set uuid=<uuid> PV-args="console=tty0 xencons=tty" : this configures the kernel boot arguments to display the login console on the correct TTY, so that it shows up in the XenCenter console.
  • Next, you need to flag the root disk of the VM as bootable so that pygrub knows where to look for the PV kernel:
    • xe vm-disk-list uuid=<uuid> and look for the UUID of the VBD for the disk. VBD stands for "Virtual Block Device" and represents how to map the virtual disk into the virtual machine.
    • xe vbd-param-set uuid=<vbd uuid> bootable=true will set the root disk VBD to be bootable.

• You should be all set now! If you boot up the Ubuntu VM, it should start up in text-mode with the high-speed PV kernel. If it doesn't work due to an incorrect grub configuration, you can use the xe-edit-bootloader script in the XenServer control domain to edit the grub.conf until it works.

• The next step is to install the XenServer tools within the guest, so that metrics such as the network interface IP addresses are recorded and reported from XenCenter. To do this:
  • Due to a portability issues with the default shell in Ubuntu (dash), you will need to replace it by: sudo apt-get -y install bash && sudo dpkg-reconfigure dash. We've actually fixed this issue in future releases of XenServer, but for XenServer 4.1 you will need to use bash.
  • Attach the XenServer Tools ISO into the VM, and mount it into the guest with sudo mount /dev/xvdd /mnt
  • Install the tools with sudo dpkg -i /mnt/Linux/xe-guest-utilities_4.1.0-257_i386.deb.
  • The warnings about the VM being unoptimized should disappear, and additional information such as the IP address of the guest should appear in XenCenter.

• In order to access the Ubuntu installation via the graphical console, you need to configure it to run VNC on the external network interface. XenCenter polls the guest to see if it is listening on the VNC port 5900, and offers the option to switch to the graphical console if it finds it. I followed the excellent instructions on this forum post. To summarise them:
  • sudo apt-get install vnc4server xinetd : to install the required packages
  • Edit /etc/gdm/gdm.conf and uncomment the RemoteGreeter=/usr/lib/gdm/gdmlogin line, set the key Enable=true in the [xdcmp] section.
  • Install a new service file for xinetd into /etc/xinetd.d/Xvnc with the following contents:

    service Xvnc
    {
      type = UNLISTED
      disable = no
      socket_type = stream
      protocol = tcp
      wait = no
      user = nobody
      server = /usr/bin/Xvnc
      server_args = -inetd -query localhost -geometry 1024x768  -depth 16 -cc 3 -once -SecurityTypes=none -extension XFIXES
      port = 5900
    }
    

  • The major difference from the forum poster is to run it on port 5900, and not to restrict it to just localhost (since XenCenter also needs to connect to it).
  • Finally, restart the xinetd service by running sudo /etc/init.d/xinetd restart.

Once you're done with this installation, you can shut down the VM and convert it to a template. Any exports or clones will continue to run in PV mode, since the XenServer XVA export format records all of the metadata required to re-create the VM records.

Enjoy the Ubuntu on XenServer experience! Remember to report any issues you have with the in-guest packages on the Ubuntu support forums, or just give them positive feedback.

PS: many thanks to Andrew Peace and Ian Campbell for assistance. May their Linux beards remain long and uncut.