Use Xen to Host Virtual Machines

Use Xen to Host Virtual Machines

Use Ubuntu as a deployment platform for multiple virtual machines.

A virtual machine (VM) is a simulated computer-inside-a-computer, allowing you to boot an entire extra operating system inside your primary environment. You may already be familiar with the concept of emulation: booting Microsoft Windows within your Linux machine using VMware [Hack #92], running an arcade-game emulator such as MAME on your computer so you can play old-style console games on your PC, or using Virtual PC on a Macintosh to allow it to run Windows programs.

These systems generally work by using both software that pretends to be hardware (emulation) and software that encapsulates and grants access to physical devices (virtualization). So, in practice, what we commonly think of as an emulator is actually a combination of emulation and virtualization. For example, if you're running an x86 emulator on an x86 system, the CPU certainly doesn't need to be emulated (so it's virtualized), but other devices, such as network adapters, may be emulated. For performance reasons, a virtualization environment such as VMware will emulate as little as possible and virtualize everything it can.

This scheme allows an application designed for the target hardware to run unmodified. The application itself probably won't be able to tell the difference: it just thinks it's running on whatever hardware is being emulated and virtualized by the host system.

While this approach can be applied to single applications, it can even be applied to entire operating systems: it's possible to boot an entire extra copy of Ubuntu, for example, right inside the copy you already have running. The extra copy is a complete, self-contained virtual computer with its own IP address, kernel, users, and applications. Mainframe systems have been doing this for decades, with many virtual machines running simultaneously under the management of a hypervisor. By taking this approach, the primary operating system running directly on the hardware can be relatively simple since it doesn't need to provide any end-user services or applications itself. All it needs to do is run the hypervisor to manage the child operating systems and provide a stable environment on which they can run.

The benefit of this approach is that the core system itself can be extremely stable, secure, and reliable, allowing mainframes to run for years or decades without any downtime at all. Once the parent OS has been installed and configured, it can be left running indefinitely and be almost immune to hackers and other external threats by being completely segregated from the network. Then any applications or services that need to be provided to end users can be installed within a virtual machine that can be backed up easily and "rebooted" at any time without affecting the underlying hardware or any other virtual machines that happen to be running alongside it. Separating services into different virtual machines is a great way to limit the impact of security problems, since a compromised web server running on one virtual machine does not give the attackers any access to mail services running in a different virtual machine, even if they're on the same physical server.

Other advantages of this setup include rapid backups, versioning, and rollbacks; live migration of running services to another host; rapid provisioning of new services; segregated system administration; and support for multiple runtime environments running simultaneously on one server.

This technology has now started to filter down to Linux in a number of forms, including UML (User Mode Linux), VMware, and Xen. This hack focuses on Xen, a project to implement a mainframe-style hypervisor within a Linux environment, giving Linux much of the power and flexibility of the older Unix systems that inspired it. Xen's approach to virtualization is to provide a virtual architecture to which the Linux kernel can be ported. In just the same way that Linux is supported on architectures such as ia32 (386), ia64, PowerPC, AMD64, and many others, Xen defines an architecture for which the kernel can be compiled. This virtual architecture is provided by a special kernel that runs on the host machine, and virtual machines that execute on it run an otherwise unmodified kernel that has been compiled for the Xen architecture.

The host machine is generally referred to as dom0, and guest virtual machines that run on it are referred to as domU machines.

Initial Host Installation

The whole idea of virtualization is to run the dom0 base system in a totally cut-down form with as little installed as possible. The less you have in the base system, the less there is that can go wrong. With an extremely stable base system, you can then run all your services in virtual machines.

Start by doing a clean install of Ubuntu in server mode [Hack #93] so you have as little cruft on your system as possible.

Base Packages Required by Xen

To run Xen and provide networking services to virtual machines, you will need to install a number of packages:

$ sudo apt-get install grub bridge-utils iproute python \\
                 python-twisted gcc libcurl3 libcurl3-dev zlib1g zlib1g-dev
            

Obtain Xen3 Binary Distribution

While you can apply the Xen patches and build a kernel [Hack #78], the simpler approach for now is to just grab the latest prebuilt binaries from the Xen project and install them using the provided installer. To reduce load on the Xen project infrastructure, the developers request that P2P systems such as BitTorrent be used to fetch the binaries where possible, but if you submit your email address, they will also provide you with a direct download link if you prefer. Visit http://www.xensource.com/xen/downloads for more information. For Ubuntu, the item you need to download is the Xen 3.0 Tarball. (Even if you're on a single-processor system, it's OK to download the one marked 32-bit SMP.)

Extract Binary Distribution

Once you've downloaded the binary distribution, just unpack it and run the provided install script:

$ tar zxf xen-3.0.1-install-[arch].tgz
$ cd xen-3.0.1-install
$ sudo ./install.sh
            

The install script places a number of special kernels and initrd images in /boot, so assuming all went well you should now have quite a few extra items in there.

Configure GRUB Bootloader

GRUB is designed to automatically detect new kernels and autoconfigure them, but unfortunately it doesn't manage the Xen kernels in quite the right way, so you will need to make some manual adjustments. Start by updating the autogenerated kernel list:

$ sudo update-grub
            

You should see GRUB report your new Xen kernels in addition to any that were previously installed. Now open the GRUB configuration file (/boot/grub/menu.lst) in your favorite editor and look down near the bottom for all the autogenerated kernel definitions. You should find one that looks something like this:

title           Ubuntu, kernel 2.6-xen0
root            (hd0,0)
kernel          /boot/vmlinuz-2.6-xen0 root=/dev/hda1 ro quiet splash
boot

To allow the Xen kernel to boot properly, the configuration stanza needs to be restructured. It's also a good idea to put the new stanza above the list of automagically detected kernels, so look for a line that reads:

### BEGIN AUTOMAGIC KERNELS LIST

and just before it, put in an entry using the same values as the autogenerated stanza but restructured to use a module line like the one shown here, and with the kernel line modified as shown:

title           Ubuntu, kernel 2.6-xen0 static
root            (hd0,0)
kernel          /boot/xen-3.0.1.gz console=vga
               module          /boot/vmlinuz-2.6-xen0 root=/dev/hda1 ro console=tty0
boot

The static enTRy in the title is just a label so that when GRUB displays a list of available kernels, you can tell which ones were added automagically and which one is your statically defined kernel.

Now tell GRUB to update its kernel list again, just to make sure you haven't made any mistakes:

$ sudo update-grub
            

Xen Services

Xen needs to start up a number of services at bootup, so set appropriate runlevel symlinks:

$ sudo update-rc.d xend defaults 20 21
$ sudo update-rc.d xendomains defaults 21 20
            

Enable Networking

Virtual machines you run on your host will most likely need some way to make network connections. Obviously, they don't have actual Ethernet cards installed in their virtual motherboards, so the usual solution is to have the host machine provide a virtual "bridge" from its own networking stack to the guest system. Each guest virtual machine can then be assigned its own IP address and even a virtual MAC address, with packets passing through the virtual bridge, into the host networking stack, and finally out the host's Ethernet card and onto the real network.

To keep everything as secure as possible, it's also a good idea to implement firewalling within the dom0 host to prevent external machines attacking the domU guests.

Install the packages required to provide network connectivity and network-management services to the guest operating systems:

$ sudo apt-get install iproute bridge-utils screen ssh
            

Disable TLS Libraries

The standard Thread Local Storage (TLS) library is incompatible with the Xen kernel, so it's necessary to either disable the library or replace it with a special Xen-friendly version before attempting to reboot with the Xen kernel. Otherwise, an emulation mode is used within Xen that reduces performance considerably.

The simplest solution for now is to move the library aside so it's not available when the host reboots:

$ sudo mv /lib/tls /lib/tls.disabled
            

If necessary, you can always move it back later if you need it for a non-Xen kernel.

For running heavily threaded code, you may prefer to install a nosegneg (no segmentation negotiation) version of the library.

Reboot

You should now be ready to reboot your dom0 host with the Xen kernel, so sudo reboot and make sure GRUB loads the correct kernel when it restarts. You can also double-check the kernel version once the machine has finished booting:

$ uname -r
2.6.12.6-xen0

Your dom0 machine is now ready to host guest domU virtual machines, so either download a pre-prepared image or create your own [Hack #91].