When using Ubuntu 12.04 – Precise as your base for a XEN host you need to do a number of things to make it work.  The reason I write this is in the first place for myself as a mental post-it.   In the second place I want others to know about a few problems I came across.  In brief here are the demands for my installation and the issues that I had. I recommend using this guide as a cross reference while doing the actual install.

Required:

• Running VM’s VHD’s on LVM ( This gives us more features, explained later).
• Ability to snapshot VM’s
• Use stock xcp-xapi (but it needs some fixing to make it work).
• Use simple bridging.
• Internet gateway for package download

Issues:

• Kernel ‘3.2.0-40-generic’ acts weird, failing to fully boot on the machines this was tried on.   3.2.0-39-generic seemed to be stable, this is important as when you enable security updates, it would have updated to that kernel.
• Booting with 16 cores with the Xen Kernel will fail, you need to configure grub : openvswitch doesn’t work for me, bridging will when you have a bridge configured manually in advance it will not be touched by xen, but a bug in openvswitch install will not honour your choice for bridging when asked by the install script.  You’ll have to correct manually.
• blktap.  If you don’t have this module after installing (it can happen depending on when you do apt-get update / upgrade right after the initial install and after you install xcp-xapi.   You will lack a kernel module, and when doing xe host-list you’ll see that the dom0 isn’t running.  The error in the logs is about as cryptic as it gets , but manually starting xend will show you the error.  So when installing a new kernel, you have to verify that the xen blktap modules are being generated.

install guide

The short guide using the precise netboot iso:
• Boot from the CD/DVD
• Install LVM (On whole disk).   When it asks how much to use, don’t use it all, you need to create logical volumes for the clients later on, I usually use about 80/90 Gigs for dom0, enough to contain a few VM snaphot dumps.  It’s annoying that the swap space is about the same size as your physical memory, when you have 24Gigs, that is taken from the 90 you selected earlier, so you’ll want to compensate.  It’s possible to shrink the swap partition later, and even grow the root partition (online).   I did it a once without initial LVM, with it it’s even easier.   But that’s outside the scope here.  But you can reclaim later, but if done remotely, It’s not for the faint of heart, but it can be done.
• Don’t install any of the server type selections (like ‘lamp’ or ‘virtualisation’ and so on) except SSH if you need to.  Just keep it base, clean and finish it off with apt-get package picking.
• Enter your network details, you’ll rework them later for bridge use.  But now you’ll be able to do apt-get update & upgrade
• Continue the installation, it will ask some more questions.   Once you’ve booted from the disk, you’ll need to get updated

Update to latest version

• sudo apt-get update
• sudo apt-get upgrade
• sudo apt-get dist-upgrade

Make sure you boot to the latest kernel version right now before installing xcp-xapi. At this point we’ll be modding grub to boot a XEN aware kernel. If you are in my situation, you’ll need to remove the ‘bad’ first, if it happens to be the latest, it will be the one booted. So for me I do:

If you happen to be missing the blktap drivers, chances are you don’t have the linux headers / source for your installation, so you’ll have to do something like this for your kernel (substitute version number!) :

and then rebuild when xcp-xapi is already installed

Selecting the right kernel

Merge/replace the content of  /etc/default/grub with this:

The last line is the most interesting, it will tell XEN to assign and pin 2 VCPU’s to dom0 together with 1G fixed Ram.  This is enough to run quite some VM’s as I’ve researched, some say it’s enought for 70 of them, but for me, with just 5 clients, I have no issues or lack of resources.  If you have 8 cores or more, chances are that 12.04 will not boot decently, or hang.  So make sure you limit before trying a XEN kernel boot.

After setting grub options, apply with

You will see some action indicating your grub config is being built. Before rebooting into xen, setup the bridged network, doing it now will save you some headscratching later. Create a simple bridge setup, assuming your primary interface now is named eth0, here’s a interfaces example of what works for me. You might have to install the bridge utils first if it’s not already on there. So optionally:

Modify the interfaces file into a bridged setup.

Also, adapt your sysctl.conf file with these

and apply with sysctl -p. This will do a few things, mainly make sure your iptables aren’t picking up and firewalling bridged traffic, you exclude this traffic from being filtered. I also enable forwarding so every host can be the default gateway of the VM’s, mainly for being able to do apt-get update and so forth. It might be possible you will not need proxy arp, I have found the need for this when building loadbalancer nodes on VM’s (haproxy / keepalived). You could just not enable that proxy_arp line. Now can now set up the forwarding on the XEN host and use that ip as your gateway, very handy for VM installations.

You’re about ready now to get xcp-xapi installed on the machine.

Once xcp-xapi has been installed, verify your package list, these look like to be about as minimal as you can get

Now it’s time to fix a few things, You probably had a question on what to use for networking, either openvswitch or bridge, even when selecting bridge it will not end up in the config file, therefor you’ll need to manually fix this. before rebooting, modify the following:

Adjust “/etc/default/xen”

Verify “bridge” is set in “/etc/xcp/network.conf”, chances are, it’s wrong. Add workaround for XAPI/XEND-conflict:

Now, reboot to XEN-kernel. To see how far you got so far, you can check to see if xcp-xapi works by verifying for dom0.

If you get this, it means your kernel booted fine and that the xcp-xapi seems to have correctly started and responding on this simple request. It’s possible that xcp-xapi fails to start with some cryptic error, usually after a fresh install it means the blktap module isn’t getting loaded, so install the missing headers / rebuild the module / update-grub.

At this point we could start deploying virtual machine but there is still a problem with the xcp-xapi drivers vs the lvm version and capabilities in xen if you need to deploy using VHD format. The xcp-xapi package maintainer for ubuntu decided to keep out those drivers since they weren’t fully tested. Therefor, you’ll need to download/modify the xcp drivers. It’s using options the lvm with 12.04 LTS doesn’t support, so failing miserably. If you don’t deploy on some shared filesystem (SAN/NAS) and don’t cluster xen host, you’re ok doing so.

So let’s demonstrate what we will accomplish by doing this. Suppose at this point you start creating virtual machine, it will work but at one point you’ll want to copy or clone a running VM from a snapshot. Dang! This is not possible since the functionality for this has been (kind of) stripped from the official xcp-xapi packages. Why?

Because your SR with LVM backing (xcp driver) has a limit. You should use a modded one, we’ll explain later how. For comparison, see the capabilities of both SR’s.

A SR that with more capabilities by using LVM + VHD’s:

You notice that for machine X you don’t have VDI.snapshot which is kind of essential if you have a backup in mind and/or fast deployment. To achieve this, you’ll need to dive into the xcp-xapi driver code, luckily this is python, which is quite readable most of the times.

Navigate to /usr/lib/xcp/sm. The driver we want to make available is : LVHDSR
You can either get that from the source package. ( download xcp-storage-managers-0.1.1 and extract this file from drivers directory ). You also need to take the lvutil.py file, we’ll be modding this one. The LVHDSR should be copied to the current directory. Put a symbolic link in place too. Suppose you unpacked this in /usr/local/src. You might have to remove LVMSR if that file already exists.

Lets adapt the lvutil.py file, essentially, our version of LVM that comes with Precise doesn’t like the –master option used in it. We need to remove this. We also need to adapt the path when copied from the source package, essentially ‘debianify‘ it.

change lines:

This will fix any errors while creating SR’s on the LVM backend, give us the ability to snapshot a running VM, and take a file backup (.xva) from that snapshot, all while keep the original machine running.

So at this point, you should have a running dom0 with the appropriate kernel. In case you still have trouble starting xend or xcp-xapi, go check out the log files in /var/log called SMlog and xcp-xapi.log. They will give you clues as to what is holding you back. You also modified and introduced the LVHD driver, in xe you can use tab completion on options, and this will confirm or not if your mods have been done right. Try to create an SR, to do this, we’ll have to create some logical volumes first.

Go ahead and create a logical volume:

Then type this command but don’t enter at the end, try tab-completion:

You should see a choice if types there:

if lvm is probably not on there, you haven’t restarted xcp-xapi yet, now restart old school style and try again:

Now it should show you the lvm option, select lvm and hit enter:

It will respond with a UUID, check the capabilities of that SR like this, with the most interesting options:

To doublecheck, see what xcp did to the LVM layout by issuing lvs, you should see a new volume group with a logical volume with name MGT. Relevant entries are:

I guess that’s management. This is a good sign it’s working as expected, you can dive deeper and find out how it trickles down in the chain.

No files are on there since we haven’t created any VM’s on this SR. Lets create a custom template for installing 12.04 LTS as guest.

Creating a 12.04 install template

Preparing the VM with Network interface (vif)

Right about now is the time to set some limits according to your wishes. The standard disk size is 8GB, if you want a bigger machine, you’ll have to mod the VDI before starting the VM.

Resize the standard sized VDI in halted state

You better do this before starting the new installed vm. Otherwise you end up having to resize filesystems/volumes etc. Finding back what VDI belongs to with VM and SR gets complicated when you have lots of machines. But these commands should help:

Substitute the uuid’s with yours of course. Pay attention to the way units are written, use the built in help for more info.

Assigning system resources to a VM

And so forth. Plenty of options there. When you do make an error there, you can clear the parameter like this:

Performing an initial VM start and installation

Before we going to start a vm, we want to have console access to it, we can use the old school xl, utility for that, set it up as an alias

This way you can attach to a console of a vm by using it’s name. If you want to bail out of the console use the old school terminal shortcuts (CTRL + ] ). It might be possible that VNC gets started, you might have to sort that out since taking over the console of a VM while it’s running reacts akward in a way you don’t immediately realise this fact.

If all went well you should see a Ubuntu 12.04 install dialog, asking for the language. From there on it’s like an other Ubuntu OS installation.

Some hints when installing VM’s

• Write down your disk UUID’s inside your VM and their function, these will come in handy later, to solve some issues booting your VM’s after redeploying backups, this is very important!
• When updating VM kernel

Restoring a exported vm using our additional features

Right now, this system is capable of some very interesting things. For instance, you can now snapshot a running VM (if in fact it’s also Ubuntu 12.04 ). You can then dump the VM from that snapshot to a file and import. Let’s start with importing a previously exported machine.

Let’s try one in the SR we created earlier:

Nasty error by the looks of it, but as always, go check the xcp-xapi logs, they usually give out a more detailed message.

So apparently our SR is too small to contain the VM we are about to import. We could resize the SR but lets just remove it, remove the logical volume and create it all again

So we need to identify the correct one and get rid of the pbd.

If the last one fails, do it twice, that seems to work all the time for me, it complains it can’t detach the SR. Now destroy it. I guess there are other ways to do this, but this one works for me.

Everything is now gone, lets remove the logical volume

And create a larger one

And do all steps we did before again to the point we can try the import again, which is not a lot…

The last command is using a different sr-uuid than before of course. Press enter and go make some coffee or go to the bathroom or both, depending on how fast your disks are. It will take some time.

When this is done, you probably want to rename the vm as it will contain the old name. It’s always wise to disable networking inside the vm (put the network config in comments), because once it boots, it will try use the IP of the base, which is probably in use. We’ll cover snapshotting later.

When renaming a virtual machine, the console alias from earlier will not work anymore, but the old name will still work. When doing this when the vm is in a halted state, the new name will work right away. The machine should be booting now. Note that you didn’t have to create a vif manually for this VM, it’s all taken care of, importing only requires an SR (if you don’t have a default SR setup).

Now we have this machine imported, let’s explain the steps needed to create such a xva export. Let’s try one from the currently running VM.

Taking a snapshot from VM’s

This will NOT be a consistent snapshot, it will be more like you would freeze the machine in time and copy all files, open and closed over at once. In that state your snapshot will be created. Now a smart person would shut about every service down on the machine to make it as if the power failed. Your VM will have to filecheck the system but will probably startup again. A consistent snapshot would require the machine to be halted.

Also note that the VM’ operating system matters while doing this on a running machine, there is some kernel support needed for this to work as expected.