02 Jul 2008 10:54 AM EDT
[ Tags: xenserver ]

I thought I'd kick off my Citrix blog with a question I get pretty often from Linux enthusiasts: how to install unsupported Linux distributions on XenServer 4.1.

The most common solution people find is to use the "Other Install Media" template, insert the distribution installation CD, and find that the mouse cursor doesn't work when they boot into X11. The reason for this is that they are using the hardware-assisted emulation mode of installing Linux. In this mode (dubbed "HVM"), all input and output is emulated, and in particular the mouse interface uses the USB tablet interface. If the distribution doesn't include a driver for USB tablets, then no mouse will appear.

Windows guests run at high-speed in HVM mode due to the installation of the XenServer tools which install high-speed drivers, but these are not necessary for Linux distributions since they can be run in para-virtualized mode (dubbed "PV"). This involves obtaining a Xen-enabled PV kernel from the distribution, and modifying the VM record in XenServer to boot into this kernel instead of HVM mode. The XenServer built-in templates for popular distributions such as RHEL, CentOS or SUSE Linux already automate all this and are in PV mode from the installer onwards.

In the remainder of this post, I'll explain how to take a distribution without direct support (Ubuntu 8.04), get it installed in HVM mode on XenServer 4.1, and convert it to PV mode with a XenCenter graphical console.

  • Download the "Alternative Installation CD". The main installation CD uses graphical mode, which won't install as well in HVM mode due to the use of esoteric 16-bit mode instructions for the graphics operations. The 16-bit emulation mechanisms vary between processors (with better support on AMD chips, and a software instruction emulator required on Intel VT chips). However, the Ubuntu alternate CD uses a text-based installer which works fine.
  • Create a new VM on the XenServer 4.1 host using the "Windows Server 2003" template. This template is set up with a sensible set of hardware emulation flags and default disks, and so is a good base for the HVM installation of Ubuntu as well. Attach the Ubuntu ISO you just downloaded to the VM, and proceed to install Ubuntu as normal. You should install it onto the first disk, to make the subsequent steps in this guide easier.
  • When the installation is finished, reboot the VM (don't forget to detach the installation ISO first). It should boot up in HVM mode into the graphical login screen. The XenCenter display will show it as not being optimized, which is fine. At this stage, I prefer to work via a remote command-line using SSH. Open up a Terminal from Ubuntu, and run "sudo apt-get install openssh-server". Then find out the VM's IP address with "ifconfig eth0", and then connect to it remotely. Alternatively, you can continue to type in the commands directly into the terminal as well.
  • On the Ubuntu guest, you now need to install the latest Xen version of the Ubuntu kernel:
    • Install the Linux kernel virtual package with "sudo apt-get install linux-image-xen". This is a virtual package which pulls in the latest Xen kernel and modules, in my case 2.6.24.19.21.
    • You now need to workaround a bug in grub. Due to the switch in recent versions of Linux to work with the hypervisor-independent paravirt_ops interface, update-grub doesn't update the grub configuration with your newly installed Xen kernel. To fix this:
      • Open /boot/grub/menu.lst in your favourite editor.
      • Scroll to the bottom to the kernel list, and find the entry which looks like: 
        title           Ubuntu 8.04, kernel 2.6.24-16-generic
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.24-16-generic root=UUID=<uuid> ro quiet splash
initrd          /boot/initrd.img-2.6.24-16-generic
quiet
      • Add a new entry which is similar to this, but change all references to the 2.6.24-16-generic to the Xen kernel. In /boot I have vmlinuz-2.6.24-19-xen, so my new entry looks like: 
        title           Ubuntu 8.04, kernel 2.6.24-19-xen
root            (hd0,0)
kernel          /boot/vmlinuz-2.6.24-19-xen root=UUID=<uuid> ro quiet splash
initrd          /boot/initrd.img-2.6.24-19-xen
quiet
      • Also edit the default entry in the menu.lst to match the number of the kernel you just added. I set mine to 3, since it is the fourth entry in the list and the indexing starts from 0.
  • When this is done, shut down the guest but do not reboot it just yet. You first need to edit the VM record for your Ubuntu VM to convert it to PV boot mode. From the control domain console of your XenServer:
    • Determine the UUID of the Ubuntu VM by using the xe CLI:
      • xe vm-list name-label=Ubuntu params=uuid --minimal : this will print out the UUID of the VM named "Ubuntu". If you are logged into the control domain, pressing the <tab> key will perform auto-completion of UUIDs in subsequent XE commands, so you don't need to keep typing it in every time!
      • xe vm-param-set uuid=<uuid> HVM-boot-policy= : this will clear the HVM boot mode from the VM.
      • xe vm-param-set uuid=<uuid> PV-bootloader=pygrub : this will switch the VM to using to the pygrub bootloader which starts the guest in PV mode by examining its filesystem for kernel.
      • vm vm-param-set uuid=<uuid> PV-args="console=tty0 xencons=tty" : this configures the kernel boot arguments to display the login console on the correct TTY, so that it shows up in the XenCenter console.
    • Next, you need to flag the root disk of the VM as bootable so that pygrub knows where to look for the PV kernel:
      • xe vm-disk-list uuid=<uuid> and look for the UUID of the VBD for the disk. VBD stands for "Virtual Block Device" and represents how to map the virtual disk into the virtual machine.
      • xe vbd-param-set uuid=<vbd uuid> bootable=true will set the root disk VBD to be bootable.
  • You should be all set now! If you boot up the Ubuntu VM, it should start up in text-mode with the high-speed PV kernel. If it doesn't work due to an incorrect grub configuration, you can use the xe-edit-bootloader script in the XenServer control domain to edit the grub.conf until it works.
  • The next step is to install the XenServer tools within the guest, so that metrics such as the network interface IP addresses are recorded and reported from XenCenter. To do this:
    • Due to a portability issues with the default shell in Ubuntu (dash), you will need to replace it by: sudo apt-get -y install bash && sudo dpkg-reconfigure dash. We've actually fixed this issue in future releases of XenServer, but for XenServer 4.1 you will need to use bash.
    • Attach the XenServer Tools ISO into the VM, and mount it into the guest with sudo mount /dev/xvdd /mnt
    • Install the tools with sudo dpkg -i /mnt/Linux/xe-guest-utilities_4.1.0-257_i386.deb.
    • The warnings about the VM being unoptimized should disappear, and additional information such as the IP address of the guest should appear in XenCenter.
  • In order to access the Ubuntu installation via the graphical console, you need to configure it to run VNC on the external network interface. XenCenter polls the guest to see if it is listening on the VNC port 5900, and offers the option to switch to the graphical console if it finds it. I followed the excellent instructions on this forum post. To summarise them:
    • sudo apt-get install vnc4server xinetd : to install the required packages
    • Edit /etc/gdm/gdm.conf and uncomment the RemoteGreeter=/usr/lib/gdm/gdmlogin line, set the key Enable=true in the [xdcmp] section.
    • Install a new service file for xinetd into /etc/xinetd.d/Xvnc with the following contents: 
      service Xvnc
{
  type = UNLISTED
  disable = no
  socket_type = stream
  protocol = tcp
  wait = no
  user = nobody
  server = /usr/bin/Xvnc
  server_args = -inetd -query localhost -geometry 1024x768  -depth 16 -cc 3 -once -SecurityTypes=none -extension XFIXES
  port = 5900
}
    • The major difference from the forum poster is to run it on port 5900, and not to restrict it to just localhost (since XenCenter also needs to connect to it).
    • Finally, restart the xinetd service by running sudo /etc/init.d/xinetd restart.

Once you're done with this installation, you can shut down the VM and convert it to a template. Any exports or clones will continue to run in PV mode, since the XenServer XVA export format records all of the metadata required to re-create the VM records.

Enjoy the Ubuntu on XenServer experience! Remember to report any issues you have with the in-guest packages on the Ubuntu support forums, or just give them positive feedback.

PS: many thanks to Andrew Peace and Ian Campbell for assistance. May their Linux beards remain long and uncut.

Permalink | Comments (40) |

Comments  (Hide Comments)

This is really useful, Anil - thanks for posting.

Posted by Anonymous at Jul 03, 2008 12:40 | Reply To This

Awesome post Anil...Very helpful
Vishal

Posted by Vishal Ganeriwala at Jul 03, 2008 15:31 | Reply To This

Thanks for the positive comments. Would love to hear about any other PV distributions folks play with and get working... drop me a line or comment here.

Posted by Anil Madhavapeddy at Jul 10, 2008 10:15 | Reply To This

I tryed to install Ubuntu 8.04 server amd64 in a Xen Server 4.1, but the console say :

Kernel alive

kernel direct mapping tables up to 20000000 @ 8000-a000

I tryed also with the w2003 template but it was the same

 Fabio

Posted by Anonymous at Jul 10, 2008 11:03 | Reply To This

So x64 HVM installs of Ubuntu appear to be problematic for other reasons (and varies in behaviour depending on Intel or AMD hardware). I've not had a chance to look into why yet, but sticking to 32-bit installs for now will definitely work (Fabio confirmed this in e-mail)

-anil

Posted by Anonymous at Jul 14, 2008 05:11Updated by Anil Madhavapeddy | Reply To This

I hear that it's easier to use vmware?

 --rip

Posted by Anonymous at Jul 11, 2008 20:11 | Reply To This

You need to stop drinking from that Google water tap Rip

-anil

Posted by Anonymous at Jul 14, 2008 05:12Updated by Anil Madhavapeddy | Reply To This

I'd like to install Fedora 9 in similar fashion, any advice?

Posted by Anonymous at Jul 15, 2008 18:49 | Reply To This

Never tried it myself...a quick glance at our guest agent script shows that it should be able to resolve the distribution tag. Post here and let me know how it goes.

-anil

Posted by Anonymous at Jul 15, 2008 18:52 | Reply To This

I was able to follow your advice, though with "yum install kernel-xen" and everything seemed ok, but now when I boot up I get "The bootloader for this VM returned an error -- did the VM installation succeed?  Error from bootloader: Failed to parse the output of bootloader: ()" on the XenCenter logs tab.

 Suggestions?

Posted by Anonymous at Jul 15, 2008 21:02 | Reply To This

The reason for this is that 'pygrub' is the bootloader for PV VMs, and it block attaches the bootable VDI of the VM in order to find the correct kernel to boot. It looks for a grub config file in /boot and then selects the default kernel to boot.

What's happening to your install is that the grub configuration is incorrect for some reason. You can get more details (including the pygrub command line) by examining /var/log/xensource.log just after you do the vm-start. Sorry I can't provide more details; I'm sitting in an airport lounge at the moment quite far away from my test machines.

Posted by Anil Madhavapeddy at Jul 16, 2008 05:39 | Reply To This

Hope your flight went well!

 There is no /var/log/xensource.log, nor any log by that name anywhere that I can see via the control domain console.  I'm running the embedded version (Dell Express Edition).

I noticed that in the grub menu file, it doesn't list /boot in front of the kernel because I have a boot partition.  If I add /boot, I get:

 IOError: [Errno 2] No such file or directory

On line 669 of pygrub. And when I edit the menu file, I have to add -p 1 to the xe-edit-bootloader command.

If it's helpful to converse via email, you can reach me at eric at intellovations, with a dot com at the end.

Posted by Eric Floehr at Jul 16, 2008 19:27 | Reply To This

BTW, do you know the group within Citrix that is responsible for XenServer Embedded (specifically the Dell Express Edition).  That's what I'm playing with and one suggestion for them (I also opened a case with Dell), is that default xe-edit-bootloader errors with:

/opt/xensource/bin/xe-edit-bootloader: line 244: nano: command not found

Nano isn't installed on the CentOS host. The other option (which I did) is edit .bash_profile and add EDITOR=vi, export EDITOR lines, which allows it to work.

-ace

Posted by Anonymous at Jul 15, 2008 21:38 | Reply To This

You've done the right thing... opening up a Dell support case will be propagated through to our issue tracking systems by Dell. I've got an engineering ticket open about the EDITOR issue anyway.

It's pretty cosmetic since only low-level "get out of jail" scripts like xe-edit-bootloader depend on the EDITOR variable, and your workaround is good. You can also just run it with env EDITOR=vi xe=edit-bootloader if you dont want to edit the bash profile for whatever reason

Posted by Anil Madhavapeddy at Jul 16, 2008 05:41 | Reply To This

So I'm not having much luck with paravirtualizing unsupported OSes (when will more "modern" linux OSes be supported?)...

I'm the one trying to get Fedora 9 to work.  I gave up on that for now, and am trying my luck with Ubuntu 8.04 to see if I can at least get it to work.  No luck.

Install goes fine, appears that everything is installed ok, ran the xe commands, but on reboot get errors like:

Couldnt get a file descriptor referring to the console

and

/dev/mem: mmap: Bad address

among others.  The XenCenter console also won't take any input.

Thoughts?

-Ace

Posted by Anonymous at Jul 19, 2008 20:30 | Reply To This

I still get the "Couldnt get a file descriptor" but can type into the console by removing console=tty0 from the PV-args property.

-Ace

Posted by Anonymous at Jul 19, 2008 20:44 | Reply To This

Yeah FC9 has its console setup differently from Ubuntu. The file descriptor error is harmless; sounds like you have it all working fine now!

Posted by Anil Madhavapeddy at Aug 04, 2008 09:17 | Reply To This

I had this same problem with the keyboard input when installing Debian Lenny AMD64 based loosely on the instructions above.  Restarting XenCenter solved the issue for me.

Posted by Anonymous at Aug 06, 2008 16:43 | Reply To This

yes, a great post Anil - thanks heaps.

I did encounter one issue when implementing this... upon reboot I can see the console output in XenCenter however no keyboard input is recognised so you cannot login via the console itself.  Login via SSH terminal is fine however.

This is displayed in XenCenter..

Ubuntu 8.04.1 xen1 tty1
xen1 login:

From dmesg

[110362.401236] Kernel command line: root=UUID=e4ea27ec-2292-4fea-97e8-c94174398050 ro quiet splash console=tty0 xencons=tty
[    0.213521] console [tty0] enabled
...
[    0.213536] Console: colour dummy device 80x25
[    0.213538] console [tty0] enabled
[    0.091141] Xen virtual console successfully installed as tty1

Any ideas?

Cheers
Darren

Posted by Anonymous at Aug 01, 2008 05:44 | Reply To This

Seeing the console and then not being able to type into it is pretty odd. Are you sure you have focus in XenCenter (click on the console window itself and then start typing).

If not, you could run vncclient directly to the VNC port and see if that helps. Best way is to determine which VNC port it's using. You can log onto dom0 and do "xenstore-read /local/domain/$(xe vm-list name-label="thenameofthevm" params=dom-id --minimal)/serial/0/vnc-port" to retrieve the localhost VNC port, and then SSH tunnel that to your local machine (e.g. ssh -L5901:localhost:5901 thexenserver.host) and then "vncclient localhost:1".

Posted by Anil Madhavapeddy at Aug 04, 2008 09:16 | Reply To This

I'm also able to see the console through XenCenter, but I can't type into it from XenCenter. I'm experiencing this with the Ubuntu 8.04.1 Server Edition, following the instructions you posted. Console I/O works fine when I use VNC, and the console continues to update (including things I type that are echoed back to the console) both in the VNC window and in the XenCenter window.

Posted by Robert Sheets at Sep 07, 2008 23:10 | Reply To This

I have the very same situation:  I have created a virtual machine without using this guide, but performing the same steps. I can see the console from XenCenter, but I cannot type in the console. If I access directly via VNC, the console is working. 
I'm using XenServer 4.1.0 and ubuntu 8.04.1, with the xen kernel provided by the ubuntu repositories. Any ideaa ? Something I can try ?

Posted by Simone Lazzaris at Sep 15, 2008 10:54 | Reply To This
View the rest of this thread. Most recent comment: 40 days ago
2 more comments by: Anil Madhavapeddy , Simone Lazzaris

Hi Anil,

 I'm really interested in any information on enabling vnc access to the vm guests on the XenServer host itself.

 E.g. if you open a shell session as root and run 'xenstore-ls' you'll see output that includes the vnc port for each vm.  In theory then it should be possible to connect to these VMs using a VNC viewer. 

The /opt/xensource/libexec/vncterm-wrapper and /opt/xensource/libexec/qemu-dm-wrapper scripts only let vnc listen on 127.0.0.1.  So again throetically I should be able to ssh to the XenServer host and then access the VM guest consoles from a remote vnc viewer.  I can't.  I imagine this is becuase vnc is not enabled - I know this because if I go 'ps aux | grep qemu' I find that any instance of qemu that is running has the option -vncunused

 I'd really love some assistance on where to look for this, or a definitive 'no it's not possible' and an explanation as to why.

Thanks,

Nathan.

Posted by Anonymous at Aug 26, 2008 09:07 | Reply To This

Ignore me.  It does work.  And how ironic, the captcha for this post is 'thickest'

Thanks,

Nathan.

Posted by Anonymous at Aug 27, 2008 02:43 | Reply To This

Anyone find how to resolve the "The bootloader for this VM returned an error -- did the VM installation succeed?  Error from bootloader: Failed to parse the output of bootloader: ()" issue? I've ran through the installation using the XenServer beta, and Ubuntu 8.04.1 with no success.

Posted by Anonymous at Aug 26, 2008 12:11 | Reply To This

You need to use the xe-edit-bootloader script to edit the grub.conf in the guest to fix it... for some reason, the bootloader (pygrub) cannot find a default kernel. Could be a parse error in the grub.conf causing this.

Posted by Anil Madhavapeddy at Sep 17, 2008 06:23 | Reply To This

I miss a rollback plan/howto. I tried to convert my Ubuntu 8.04.1 x64 and I get this error:

<pre>

Error: Starting VM 'ubuntu64' - Internal error: Failure("Error from xenguesthelper: caught exception: Failure(\\\"Subprocess failure: Failure(\\\\\\\"xc_dom_linux_build: [2] xc_dom_find_loader: no loader found\\\\\\\\n\\\\\\\")\\\")")

</pre>

BTW, this is a script to do the conversion described in the article:

NAME=ubuntu64
DUUID=$(xe vm-list name-label=$NAME params=uuid --minimal)

xe vm-param-set uuid=$DUUID HVM-boot-policy=
xe vm-param-set uuid=$DUUID PV-bootloader=pygrub
xe vm-param-set uuid=$DUUID PV-args="console=tty0 xencons=tty"

VBDUUID=$(xe vm-disk-list uuid=$DUUID --minimal | sed -e 's/^.*,//')
xe vbd-param-set uuid=$VBDUUID bootable=true

Posted by Anonymous at Sep 19, 2008 15:12 | Reply To This

If you get it wrong, you can always try booting in "recovery mode" from XenCenter, which forces it back to HVM. The above issue could well be because the Xen kernel image is bzipped, which would be an upstream change since I installed.

Am travelling to ICFP 2008 this week, but can give it a shot when I'm back the week after. If someone else can try 32-bit Ubuntu to see if it still works that would be great!

Posted by Anil Madhavapeddy at Sep 22, 2008 03:14 | Reply To This

Instrcutions worked like a charm for Ubuntu-server-8.04 32-bit !

 Only you have to restart XenCenter after the changes, because the console doesn't take keyboard input. If you close and restart XenCenter keyboard input is fine. Probably something to do with the changing from HVM to PV and XenCenter not yet up-to-date on these changes.

Posted by Anonymous at Sep 24, 2008 16:26 | Reply To This

Do you see the console restart problem on the 5.0 XenCenter as well? Say it aint so!

Posted by Anil Madhavapeddy at Sep 26, 2008 13:16 | Reply To This

I just installed with Ubunut 8.04 amd64 in Xen5 and everything went completely smooth up until installing the XenTools.  XenTools throws the error :

 sudo dpkg -i /mnt/Linux/xe-guest-utilities_5.0.0-361_amd64.deb
(Reading database ... 17486 files and directories currently installed.)
Preparing to replace xe-guest-utilities 5.0.0-361 (using .../xe-guest-utilities_5.0.0-361_amd64.deb) ...
[: 5: ==: unexpected operator
Stopping xe daemon:  OK
Unpacking replacement xe-guest-utilities ...
Setting up xe-guest-utilities (5.0.0-361) ...
Detecting Linux distribution version: local: 1: 8.04: bad variable name
OK
Starting xe daemon:  OK
[: 30: ==: unexpected operator

 It is booting great in PV mode and I can access everything from XenCenter, just no stats.  Should I just switch to i386 or can I work around this?

Posted by Anonymous at Sep 26, 2008 10:20 | Reply To This

I get the same error on i386, so that wont help you.  If you do/did find a resolution, please post it - I will do the same.

Posted by Anonymous at Sep 26, 2008 13:02 | Reply To This

Looks to me like you didnt replace dash as the default shell; you need to use bash instead. If that doesnt work, try running the bash -x xe-linux-distribution and see where it goes wrong.

Posted by Anil Madhavapeddy at Sep 26, 2008 13:15 | Reply To This

I assumed from what you wrote above that in Xen5 we wouldn't need that tweak: "We've actually fixed this issue in future releases of XenServer, but for XenServer 4.1 you will need to use bash."  So I skipped it, but that fixed it and the install works properly.  Even with the install error, though, it did seem to be working properly before I switched to bash and reinstalled it.

Posted by Anonymous at Sep 26, 2008 14:41 | Reply To This

After a restart the system is able to retrieve all the stats and the IP, so I assume the tools actually installed succesfully on Ubuntu-server-amd64 inspite of the error messages. 

Posted by Anonymous at Sep 26, 2008 14:21 | Reply To This

Thanks Anil,

you mentioned in July: "So x64 HVM installs of Ubuntu appear to be problematic for other reasons (and varies in behaviour depending on Intel or AMD hardware). I've not had a chance to look into why yet, but sticking to 32-bit installs for now will definitely work
(Fabio confirmed this in e-mail)"

 I am trying to set up Ubuntu x64 on a Dell server with 128GB of RAM, and it crashes after kernel comes alive to start the installer.
I think you'd agree that using 32 bit would not make sense. The only logical question is when will Ubuntu reach the supported OS status? When the majority of people start using Ubuntu (and hasn't that happened already) or when XENServer is able to run Ubuntu correctly  - and how is that going to happen unless Citrix says it's a priority and dedicates time to fixing XENServer to run Ubuntu?

Thanks!

Posted by Anonymous at Sep 26, 2008 14:54 | Reply To This

I've managed to get the xentools installed and PV mode working.  I can vnc to the machine but all I see at the xencenter is the console for ubuntu.  Is there a way to get the ubuntu gui desktop on the xenserver console rather than just the terminal?

Thanks

Posted by Anonymous at Oct 16, 2008 11:50 | Reply To This

Did this on XenServer 5.  I did not replace dash as it says the fix is in later builds.  One error in the article

 vm vm-param-set uuid=<uuid> PV-args="console=tty0 xencons=tty"

should read

xe vm-param-set uuid=<uuid> PV-args="console=tty0 xencons=tty"

correct?  Keyboard came back after restarting XenCenter.  I can connect via VNC, but nothing ever comes up.  No gui in XenConsole and I would like to get the gui back there.

Posted by Anonymous at Nov 14, 2008 12:11 | Reply To This