Xen: Debian Lenny Dom0 with Ubuntu Lucid Guest

~ Matthias Marschall

Creative Commons License ChrisDag

It sounds like a simple thing to do: As Ubuntu does not have support for Xen by default, I wanted to run a Debian Lenny Server as Xen host (Dom0) with Ubuntu Lucid (10.04) LTS as guest (DomU). But there were some obstacles:

  • debootstrap does not support Ubuntu Lucid by default
  • Ubuntu Lucid cannot boot using the Debian Lenny kernel in para-virtualized mode

In this post I want to show you all the steps to come up with a nice and simple setup to be able to run Ubuntu 10.04 LTS as Xen DomU on Debian Lenny as Xen Dom0 using xen-tools to create new VMs.

Install Xen Kernel on Debian Lenny

# apt-get install xen-hypervisor-3.2-1-amd64 xen-linux-system-2.6.26-1-xen-amd64 xen-utils-3.2-1 xenstore-utils xenwatch xen-shell xen-tools
# reboot

Now you should have a running Xen kernel. To validate just type

# xm list

and you should see something like this

Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0 20080     8     r-----    482.1

Enable enough loop devices for disk images

We will use disk images for our guests as we run a root server where it is difficult to get a partition for LVM (which would be the preferred way). To make sure we can mount enough disk images as loop devices, we have to edit the modules config to give us up to 64 loop devices:

# vi /etc/modules

loop max_loop=64

Configure xend for bridged networking

(see: Xen Networking)
As we’ll use public IP addresses for our guests, we enable bridged networking.

# vi /etc/xen/xend-config.sxp

(network-script network-bridge)
...
# (network-script network-dummy)
...
(vif-script vif-bridge)
...

And restart the Xen Daemon

# /etc/init.d/xend restart

Configure xen-tools for booting Ubuntu with pygrub

(see: How to boot via pygrub on lenny)

Ubuntu Lucid (or newer) uses some features, which are not available in the debian lenny kernel (bootall, etc). If you try to debootstrap Ubuntu Lucid and run the VM on Debian Lenny, Ubuntu will not boot. Therefore, we need to install a separate kernel into our Ubuntu Guests and use pygrub to boot it. This gives us paravirtualized VMs using their own (new enough) kernels. We avoid all the hassle (e.g. no xm console access) and performance penalties of using full virtualization (HVM with qemu).

Fix pygrub location

ln -s /usr/lib/xen-3.2-1/bin/pygrub /usr/bin/

Enable installation of newer Ubuntu versions (e.g 10.04 aka Lucid)

(see: Update debootstrap)

There might be an easier way to do this (just install the fitting debootstrap package somehow) but I already had this running before I read about it. So I never tried something else… 😉

# cd /usr/lib/xen-tools/
# ln -s edgy.d/ lucid.d
# vi /etc/apt/sources.list.d/tmp.list
deb http://archive.ubuntu.com/ubuntu/ lucid main restricted
deb-src http://archive.ubuntu.com/ubuntu/ lucid main restricted

# apt-get update
# apt-get install -y --force-yes debootstrap
# rm /etc/apt/sources.list.d/tmp.list
# apt-get update

Configure xen-tools

Absolutely crucial is the fact “kernel” and “initrd” are undefined as we will boot with pygrub (and not with the kernel of the Debian Lenny Dom0)

# vi /etc/xen-tools/xen-tools.conf

dir = /home/xen

install-method = debootstrap

size   = 40Gb      # Disk image size.
memory = 4Gb    # Memory size
swap   = 1Gb    # Swap size
fs     = ext3     # use the EXT3 filesystem for the disk image.
dist   = lucid	  # Default distribution to install.
image  = sparse   # Specify sparse vs. full disk images.

broadcast = YOUR_BROADCAST
netmask   = YOUR_NETMASK
gateway   = YOUR_GATEWAY

arch=amd64
passwd=1 # let xen-tools ask for a root password for the DomU

# Use the mirror most local to you (we use "de"):
mirror_lucid=http://de.archive.ubuntu.com/ubuntu

ext3_options   = noatime,nodiratime,errors=remount-ro
ext2_options   = noatime,nodiratime,errors=remount-ro
xfs_options    = defaults
reiser_options = defaults
serial_device = hvc0
disk_device = xvda

Now create a new role script, which will install the required kernel into the guest on creation. xen-tools will execute it after debootstrapping the disk image.

# wget http://wejn.org/stuff/pygrub-role
# mv pygrub-role /etc/xen-tools/role.d/pygrub
# chmod a+x /etc/xen-tools/role.d/pygrub

Now, change the kernel package to be loaded, as the one from the original script does not exist in current versions of ubuntu.

# vi /etc/xen-tools/role.d/pygrub

Replace
installDebianPackage ${prefix} linux-image-2.6-xen-amd64
with
installDebianPackage ${prefix} linux-image-ec2
(see: Stacklet Ubuntu 10.04 Image – expand package list)

Create a guest VM

Choose a HOSTNAME and use one of your public IP_ADDRESSes

# xen-create-image --hostname=HOSTNAME --ip=IP_ADDRESS --role=udev,pygrub

Edit Guest config

Switch the disk entries in /etc/xen/hostname.cfg from

disk        = [
                  'file:/home/xen/domains/dev/swap.img,sda1,w',
                  'file:/home/xen/domains/dev/disk.img,sda2,w',
              ]

to

disk        = [
                  'file:/home/xen/domains/dev/disk.img,sda2,w',
                  'file:/home/xen/domains/dev/swap.img,sda1,w',
              ]

otherwise pygrub will fail with Error: Boot loader didn't return any data!
 (it needs the root disk first…)

Launch VM and connect to it’s console

# xm create -c hostname.cfg

Published by Matthias Marschall

The opinions stated here are my own, not those of my company.

46 thoughts on “Xen: Debian Lenny Dom0 with Ubuntu Lucid Guest”

  1. Pingback: Tweets that mention Xen: Debian Lenny Dom0 with Ubuntu Lucid Guest — Agile Web Operations -- Topsy.com
  2. Pingback: Xen: Debian Lenny Dom0 with Ubuntu Lucid Guest | Debian-News.net - Your one stop for news about Debian

  3. Hi,

    great tutorial. Just one note: I had to add bootloader=’/usr/bin/pygrub’ to the domU’s config file manually. Otherwise the domU would boot the wrong kernel.

    Don’t now if I forgot something.

    Like

    Reply

  9. I’m sorry Matthias, I’m just very confused. I’m currently discussing with some people on the Xen mailing list, and they are telling me that I can’t install Ubuntu 10.04 in Xen 3.2.x as apperently because Ubuntu uses grub2 and pygrub doesn’t support grub2.

    Can you please explain to me how you are working around this?

    I’m sure your instructions do work, I’m just trying to figure out *how* it works. I’m coming from a VMWare background, so I’m very new to Xen..

    Thanks

    Like

    Reply

  10. I know that it is a common mis-perception that you cannot run Ubuntu 10.04 on debian Lenny. But using pygrub and preparing the image with the correct kernel (the ec2 one) makes it happen 🙂

    Like

    Reply

  12. Hi Matthias!

    Thanks for the link!

    So I take it amazon have modified the Ubuntu 10.04 kernel to run on a normal Xen install?

    It’s still as secure as the normal 10.04 kernel, right? 🙂

    Thanks

    Like

    Reply

  14. Fantastic tutorial, thanks. Just a note: I used my previously created xen-tools.conf (in order to be able to create domUs as used before) which specified kernel and initrd. This, of course, results in a domU cfg file including those entries which will cause the domU to fail booting (mountall errors) because the configured kernel is used instead of pygrub. After deleting those 2 lines from the domU cfg it’s working fine 🙂

    Cheers,
    Mathias

    Like

    Reply

  15. I just wanted to chime in the rest of these guys and THANK YOU for drafting up this great howto for Lucid. My install went off without a hitch after following this guide. And after converting the DomU to a libvirt usable one, via `xen new HOSTNAME` , Im again having fun with all of the libvirt tools; virt-*

    Like

    Reply
  16. Pingback: iFolder on Debian Lenny (as xen DomU) | Jonathan Arbib

  17. Tks for the tutorial I have my Ubuntu Lucid server running. I have two questions.

    1] Is it possible to use ext4 or btrfs?
    2] Can the installation of the Desktop version be used?

    Tks in Advance
    Eric

    Like

    Reply

  18. congrats cougarmaster.

    I’ve not tried to run Ubuntu with any other file system but I don’t see any reason why it should not work.

    It should be no problem to install Ubuntu Lucid desktop on top of Debian Lenny with Xen.

    Like

    Reply

  19. Hi,
    very great howto (for me its bad enough that I have to struggle with old Lenny as 10.04 lucid does not run well as a xen host, so its great to get this here), but…

    I still get “Error: Boot loader didn’t return any data!” although I’ve switched the disks in the hostname.cfg so root is first…any idea?
    I think it may has something to do with the fact I am using LVM not local loopback files for disks…

    Thanks,

    Matthias

    Like

    Reply

  21. Hi Matthias,
    yep quite sure…unfortunately 😦
    I think I give it a try to get Ubu lucid running as a Xen Dom0…;)
    Matthias

    Like

    Reply

  22. Thanks for the reply. So to be sure when you say I can use ext4 do I just replace ext3 with ext4 in xen-tools? Or do I need to have debian lenny to support it as well?

    Tks in advance
    Eric

    Like

    Reply

  24. The ext4 does not work at all. You need to have XEN 4.0 at least. So I guess I have to get XEN 4.0. Do you have any good guides or tutorial to install XEN 4 on Debian lenny?

    Tks in Advance
    Eric

    Like

    Reply

  26. Hi Matthias,

    Thanks for this HowTO! Really helpfull.

    I tried to make it work with HARDY (8.04 ubuntu version), but it stops with:
    # xm create /etc/xen/vm1.cfg
    Using config file “/etc/xen/vm1.cfg”.
    Error: Boot loader didn’t return any data!

    Then I
    Yes, I remembered to switch disk and swap in the cfg.
    Any ideas?

    Like

    Reply

  27. # cat /etc/xen/vm1.cfg
    memory = ‘2048’
    root = ‘/dev/xvda2 ro’
    disk = [ ‘phy:/dev/afrodite/vm1-disk,xvda2,w’, ‘phy:/dev/afrodite/vm1-swap,xvda1,w’ ]
    name = ‘vm1’
    vif = [ ‘ip=10.254.0.240,mac=00:16:3E:A1:C8:36’ ]
    on_poweroff = ‘destroy’
    on_reboot = ‘restart’
    on_crash = ‘restart’

    # cat /etc/xen-tools/xen-tools.conf
    lvm = afrodite

    install-method = debootstrap

    size = 50Gb # Disk image size.
    memory = 2Gb # Memory size
    swap = 4Gb # Swap size
    fs = xfs # use the EXT3 filesystem for the disk image.
    dist = hardy # Default distribution to install.
    image = sparse # Specify sparse vs. full disk images.

    gateway = 10.254.0.1
    netmask = 255.255.255.0
    broadcast = 10.254.0.255

    arch = amd64
    mirror_hardy = http://archive.ubuntu.com/ubuntu/
    passwd = 1

    ext3_options = noatime,nodiratime,errors=remount-ro
    ext2_options = noatime,nodiratime,errors=remount-ro
    xfs_options = defaults
    reiser_options = defaults
    serial_device = hvc0
    disk_device = xvda

    Maybe something differente I have to do with the “installDebianPackage” directive?

    Thanks

    Like

    Reply

  28. In the part where you enable debootstrap to use your Ubuntu, have you changed “lucid” to “hardy” there:

    # cd /usr/lib/xen-tools/
# ln -s edgy.d/ hardy.d
# vi /etc/apt/sources.list.d/tmp.list
deb http://archive.ubuntu.com/ubuntu/ hardy main restricted
deb-src http://archive.ubuntu.com/ubuntu/ hardy main restricted

    Like

    Reply

  31. Hi,
    let me thank you for your post. It was really helpful for me.

    I successfully build a 10.04 domU on a 8.04 dom0.
    It’s a little different to get this working. First of all it’s not needed to update debootstrap. Just create a symbolic link lucid.d -> edgy.d/ in /usr/lib/xen-tools/ and another one lucid -> hardy in /usr/share/debootstrap/scripts/
    xen-create-image should now successfully create a lucid-domU.
    After that I changed change my /etc/xen/domu.cfg as suggested in http://blog.webangel.ie/2010/04/ubuntu-upgrade-to-9-1010-04-for-xen-domu/

    This is my domU.cfg:

    bootloader=”/usr/bin/pygrub”
    #kernel = ‘/boot/vmlinuz-2.6.24-28-xen’
    #ramdisk = ‘/boot/initrd.img-2.6.24-28-xen’
    memory = ‘4096’

    #
    # Disk device(s).
    #
    root = ‘/dev/xvda2 ro’
    disk = [
    ‘tap:aio:/data/xen/domains/domU1/disk.img,xvda2,w’,
    ‘tap:aio:/data/xen/domains/domU1/swap.img,xvda1,w’,
    ]

    Hope that helps.
    Axel

    Like

    Reply

  32. Great article! Thanks for sharing this with the rest of us.

    Two suggestions if I may:

    Instead of editing the original pygrub role script from wejn.org
    I suggest you make a copy of it and name it pygrub-lucid.
    That way you can have xen-tools ready to install both lenny as well as lucid domUs with pygrub. It would be nicer to handle

    The pygrub role script should handle “noswap = 1”. In that case the root partition goes to xvda1 and the boot/grub/menu.lst generated by the pygrub role script will point to a wrong device. (This should probably be fixed upstream at wejn.org)

    Like

    Reply
  33. Pingback: Xen: Debian Lenny Dom0 with Ubuntu Lucid Guest | l i n u x f r e a k . n e t | 3.0

  34. Thanks for the tips.

    An easier way to install an updated debootstrap (>= 1.0.21) is to download the .deb for Debian Squeeze from a mirror and install that with dpkg -i

    Ben: I was using xfs (my “default” fs) and couldn’t get pygrub to work with that. Haven’t dug into that but repeated the process with ext3 and all is good.

    Like

    Reply

  35. Hey Matthius!

    It worked! It Worked! Thank you very much! For your instructions!

    I made some modifications to this slightly though. I wanted to install Xen 4.0.

    I had to install Debian Squeeze (Testing) and made just a few odd adjustments here and there to your how to and know it works perfectly.

    I will post Instructions in a few days.

    Is there a place I can post it or would you like me to post it here in the comment box?

    Cheers,

    Zubin Parihar

    Like

    Reply

  37. Hi,

    Awesome tutorial…I just wanted to ask whether this tutorial will help me to run lucid as DomU on my intrepid Dom0…..?

    Like

    Reply

  38. Hi Matthias,

    I tried the above tutorial with ubuntu intrepid (8.10) as Dom0 and tried to install 10.4 as DomU, but i got the following error:

    root@chetan-laptop:/etc/xen# xen create-image –hostname=lucid –ip=192.168.1.5 –role=udev,pygrub
    bash: xen: command not found

    What is missing..? Please help me. Also if you know any tutorial which shows how to install ubuntu 10.4 as DomU on ubuntu 8.10 dom0, please let me know. I am new to the Xen environment.

    Like

    Reply

  39. Hi chetan,

    xen-create-image is one command (mind the ‘-‘ between xen and create).

    I’ve not tried to install on Ubuntu Intrepid myself. Pls let us know here in the comments if you are successful.

    Like

    Reply

  40. Hi guyz,
    I have gotten XEN 4.0.1 on Debian Squeeze. It has deb packages already and uses the newest kernel. Below is how I got it to work except changing ubuntu to pv .

    The HVM is very stright forward no problems there at all. Just the PV part that is killing me. Also when installing xen 4.0.1 remember to install qemu-4.0 otherwise HVM will not work and there is already a deb package available on Debian Squeeze. It is stable and good to use, haven’t had any troubles on stability on that. Here is a short summary of what I did to install on squeeze :-

    1] Install Debian Squeeze with minimal
    2] Install SSH
    3] Install Xen 4.0.1 with :-
    apt-get install xen-linux-system-2.6.32-5-xen-amd64
    apt-get install xen-qemu-dm-4.0
    apt-get install xen-tools (if you need it)
    4] edit the boot options in grub2 here is my setup /etc/default/grub
    # If you change this file, run ‘update-grub’ afterwards to update
    # /boot/grub/grub.cfg.

    GRUB_DEFAULT=4 /dev/null || echo Debian`
    # GRUB_CMDLINE_LINUX_DEFAULT=”quiet”
    # GRUB_CMDLINE_LINUX=””
    GRUB_CMDLINE_XEN_DEFAULT=”dom0_mem=512M loglvl=all guest_loglvl=all dom0_max_vc$

    # Uncomment to enable BadRAM filtering, modify to suit your needs
    # This works with Linux (no patch required) and with any kernel that obtains
    # the memory map information from GRUB (GNU Mach, kernel of FreeBSD …)
    #GRUB_BADRAM=”0x01234567,0xfefefefe,0x89abcdef,0xefefefef”

    # Uncomment to disable graphical terminal (grub-pc only)
    #GRUB_TERMINAL=console

    # The resolution used on graphical terminal
    # note that you can use only modes which your graphic card supports via VBE
    # you can see them in real GRUB with the command `vbeinfo’
    #GRUB_GFXMODE=640×480

    # Uncomment if you don’t want GRUB to pass “root=UUID=xxx” parameter to Linux
    #GRUB_DISABLE_LINUX_UUID=true

    # Uncomment to disable generation of recovery mode menu entries
    #GRUB_DISABLE_LINUX_RECOVERY=”true”

    # Uncomment to get a beep at grub start
    #GRUB_INIT_TUNE=”480 440 1″

    5] update-grub
    5a] if you need other options on the module line the way I know is only to edit the /boot/grub/grub.cfg which is not the proper way as I don’t know if you know please let me know. Here is the part of the grub.cfg :-

    ### BEGIN /etc/grub.d/20_linux_xen ###
    menuentry ‘Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN 4.0-amd64′ –class debian –class gnu-linux –class gnu –class os –class xen {
    insmod part_msdos
    insmod ext2
    set root='(hd1,msdos1)’
    search –no-floppy –fs-uuid –set 08ea09a3-b0d2-4b00-802c-48fdc8e0885d
    echo ‘Loading Linux 2.6.32-5-xen-amd64 …’
    multiboot /xen-4.0-amd64.gz placeholder dom0_mem=512M loglvl=all guest_loglvl=all dom0_max_vcpus=1 dom0_vcpus_pin
    module /vmlinuz-2.6.32-5-xen-amd64 placeholder root=/dev/mapper/cougarlvm-root ro nomodeset console=hvc0 xen-pciback.permissive xen-pciback.hide=(01:0a)
    echo ‘Loading initial ramdisk …’
    module /initrd.img-2.6.32-5-xen-amd64
    }

    6] This took me about 1 hr from start to finnish to get up. Everything else is same as before.

    Hope this help and tks in Advance
    Eric

    Like

    Reply
  42. Pingback: Too Much Information » Blog Archive » Xen: Lenny dom0 & Lucid domU - Blogging instead of receiving that glazed-over look from my wife…

  43. I followed this guide carefully, but still hit the “Error: Boot loader didn’t return any data!” issue. After a few attempts (each a bit time-consuming) I decided to throw down the $10 that getting images via Stacklet would cost me.

    Their pre-packaged Ubuntu 10-4 worked fine, and I’m happy (and no longer spending coding time on admin).

    Thanks for the guide (first steps with xen-create-image so still a valuable lesson for me) and for the pointer to Stacklet too!

    Like

    Reply

  44. You, Sir, are a genius!! :D. Thanks a lot!! I was trying to do the exact same thing since 5 nights with no results!! Your instructions worked flawlessly!! :D.

    Can you also post how to upgrade the xen-3.2.1 to xen 4.0.1 on this system which is set through your instructions?

    Like

    Reply

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.