ChrisDagIt sounds like a simple thing to do: As Ubuntu does not have support for Xen by default, I wanted to run a Debian Lenny Server as Xen host (Dom0) with Ubuntu Lucid (10.04) LTS as guest (DomU). But there were some obstacles:
- debootstrap does not support Ubuntu Lucid by default
- Ubuntu Lucid cannot boot using the Debian Lenny kernel in para-virtualized mode
In this post I want to show you all the steps to come up with a nice and simple setup to be able to run Ubuntu 10.04 LTS as Xen DomU on Debian Lenny as Xen Dom0 using xen-tools to create new VMs.
Install Xen Kernel on Debian Lenny
# apt-get install xen-hypervisor-3.2-1-amd64 xen-linux-system-2.6.26-1-xen-amd64 xen-utils-3.2-1 xenstore-utils xenwatch xen-shell xen-tools # reboot
Now you should have a running Xen kernel. To validate just type
# xm list
and you should see something like this
Name ID Mem VCPUs State Time(s) Domain-0 0 20080 8 r----- 482.1
Enable enough loop devices for disk images
We will use disk images for our guests as we run a root server where it is difficult to get a partition for LVM (which would be the preferred way). To make sure we can mount enough disk images as loop devices, we have to edit the modules config to give us up to 64 loop devices:
# vi /etc/modules
loop max_loop=64
Configure xend for bridged networking
(see: Xen Networking)
As we’ll use public IP addresses for our guests, we enable bridged networking.
# vi /etc/xen/xend-config.sxp
(network-script network-bridge) ... # (network-script network-dummy) ... (vif-script vif-bridge) ...
And restart the Xen Daemon
# /etc/init.d/xend restart
Configure xen-tools for booting Ubuntu with pygrub
(see: How to boot via pygrub on lenny)
Ubuntu Lucid (or newer) uses some features, which are not available in the debian lenny kernel (bootall, etc). If you try to debootstrap Ubuntu Lucid and run the VM on Debian Lenny, Ubuntu will not boot. Therefore, we need to install a separate kernel into our Ubuntu Guests and use pygrub to boot it. This gives us paravirtualized VMs using their own (new enough) kernels. We avoid all the hassle (e.g. no xm console access) and performance penalties of using full virtualization (HVM with qemu).
Fix pygrub location
ln -s /usr/lib/xen-3.2-1/bin/pygrub /usr/bin/
Enable installation of newer Ubuntu versions (e.g 10.04 aka Lucid)
(see: Update debootstrap)
There might be an easier way to do this (just install the fitting debootstrap package somehow) but I already had this running before I read about it. So I never tried something else… 😉
# cd /usr/lib/xen-tools/ # ln -s edgy.d/ lucid.d # vi /etc/apt/sources.list.d/tmp.list deb http://archive.ubuntu.com/ubuntu/ lucid main restricted deb-src http://archive.ubuntu.com/ubuntu/ lucid main restricted # apt-get update # apt-get install -y --force-yes debootstrap # rm /etc/apt/sources.list.d/tmp.list # apt-get update
Configure xen-tools
Absolutely crucial is the fact “kernel” and “initrd” are undefined as we will boot with pygrub (and not with the kernel of the Debian Lenny Dom0)
# vi /etc/xen-tools/xen-tools.conf
dir = /home/xen install-method = debootstrap size = 40Gb # Disk image size. memory = 4Gb # Memory size swap = 1Gb # Swap size fs = ext3 # use the EXT3 filesystem for the disk image. dist = lucid # Default distribution to install. image = sparse # Specify sparse vs. full disk images. broadcast = YOUR_BROADCAST netmask = YOUR_NETMASK gateway = YOUR_GATEWAY arch=amd64 passwd=1 # let xen-tools ask for a root password for the DomU # Use the mirror most local to you (we use "de"): mirror_lucid=http://de.archive.ubuntu.com/ubuntu ext3_options = noatime,nodiratime,errors=remount-ro ext2_options = noatime,nodiratime,errors=remount-ro xfs_options = defaults reiser_options = defaults serial_device = hvc0 disk_device = xvda
Now create a new role script, which will install the required kernel into the guest on creation. xen-tools will execute it after debootstrapping the disk image.
# wget http://wejn.org/stuff/pygrub-role # mv pygrub-role /etc/xen-tools/role.d/pygrub # chmod a+x /etc/xen-tools/role.d/pygrub
Now, change the kernel package to be loaded, as the one from the original script does not exist in current versions of ubuntu.
# vi /etc/xen-tools/role.d/pygrub
Replace
installDebianPackage ${prefix} linux-image-2.6-xen-amd64
with
installDebianPackage ${prefix} linux-image-ec2
(see: Stacklet Ubuntu 10.04 Image – expand package list)
Create a guest VM
Choose a HOSTNAME and use one of your public IP_ADDRESSes
# xen-create-image --hostname=HOSTNAME --ip=IP_ADDRESS --role=udev,pygrub
Edit Guest config
Switch the disk entries in /etc/xen/hostname.cfg from
disk = [
'file:/home/xen/domains/dev/swap.img,sda1,w',
'file:/home/xen/domains/dev/disk.img,sda2,w',
]
to
disk = [
'file:/home/xen/domains/dev/disk.img,sda2,w',
'file:/home/xen/domains/dev/swap.img,sda1,w',
]
otherwise pygrub will fail with Error: Boot loader didn't return any data! (it needs the root disk first…)
Launch VM and connect to it’s console
# xm create -c hostname.cfg
Agile Web Operations 
Hi,
great tutorial. Just one note: I had to add bootloader=’/usr/bin/pygrub’ to the domU’s config file manually. Otherwise the domU would boot the wrong kernel.
Don’t now if I forgot something.
LikeLike
Hi snod, that’s interesting. I read about that it is necessary, but I did not need to do it. Strange…
Thanks for sharing!
LikeLike
Which version of Xen are you using?
LikeLike
Hi Jonathan, I’m using Xen 3.2-1-amd64 / Debian GNU/Linux, kernel 2.6.26-1-xen-amd64
LikeLike
Thanks.
Do you think the above will work with a CentOS Dom0?
LikeLike
Yes, why not.
LikeLike
I’m sorry Matthias, I’m just very confused. I’m currently discussing with some people on the Xen mailing list, and they are telling me that I can’t install Ubuntu 10.04 in Xen 3.2.x as apperently because Ubuntu uses grub2 and pygrub doesn’t support grub2.
Can you please explain to me how you are working around this?
I’m sure your instructions do work, I’m just trying to figure out *how* it works. I’m coming from a VMWare background, so I’m very new to Xen..
Thanks
LikeLike
I know that it is a common mis-perception that you cannot run Ubuntu 10.04 on debian Lenny. But using pygrub and preparing the image with the correct kernel (the ec2 one) makes it happen 🙂
LikeLike
If you need a ready made Ubuntu 10.04 disk image to run on your Xen server (complete with configuration script) go to stacklet.com. Dave from stacklet.com is a real master! His stuff helped me to get it running!
LikeLike
Hi Matthias!
Thanks for the link!
So I take it amazon have modified the Ubuntu 10.04 kernel to run on a normal Xen install?
It’s still as secure as the normal 10.04 kernel, right? 🙂
Thanks
LikeLike
It better should be as they run tons of virtual servers on EC2.
LikeLike
Fantastic tutorial, thanks. Just a note: I used my previously created xen-tools.conf (in order to be able to create domUs as used before) which specified kernel and initrd. This, of course, results in a domU cfg file including those entries which will cause the domU to fail booting (mountall errors) because the configured kernel is used instead of pygrub. After deleting those 2 lines from the domU cfg it’s working fine 🙂
Cheers,
Mathias
LikeLike
I just wanted to chime in the rest of these guys and THANK YOU for drafting up this great howto for Lucid. My install went off without a hitch after following this guide. And after converting the DomU to a libvirt usable one, via `xen new HOSTNAME` , Im again having fun with all of the libvirt tools; virt-*
LikeLike
Tks for the tutorial I have my Ubuntu Lucid server running. I have two questions.
1] Is it possible to use ext4 or btrfs?
2] Can the installation of the Desktop version be used?
Tks in Advance
Eric
LikeLike
congrats cougarmaster.
I’ve not tried to run Ubuntu with any other file system but I don’t see any reason why it should not work.
It should be no problem to install Ubuntu Lucid desktop on top of Debian Lenny with Xen.
LikeLike
Hi,
very great howto (for me its bad enough that I have to struggle with old Lenny as 10.04 lucid does not run well as a xen host, so its great to get this here), but…
I still get “Error: Boot loader didn’t return any data!” although I’ve switched the disks in the hostname.cfg so root is first…any idea?
I think it may has something to do with the fact I am using LVM not local loopback files for disks…
Thanks,
Matthias
LikeLike
Are you sure you removed “kernel” and “initrd” settings from your cfg?
LikeLike
Hi Matthias,
yep quite sure…unfortunately 😦
I think I give it a try to get Ubu lucid running as a Xen Dom0…;)
Matthias
LikeLike
Thanks for the reply. So to be sure when you say I can use ext4 do I just replace ext3 with ext4 in xen-tools? Or do I need to have debian lenny to support it as well?
Tks in advance
Eric
LikeLike
Should be enough to tell xen-tools to use ext4. Good luck!
LikeLike
The ext4 does not work at all. You need to have XEN 4.0 at least. So I guess I have to get XEN 4.0. Do you have any good guides or tutorial to install XEN 4 on Debian lenny?
Tks in Advance
Eric
LikeLike
Unfortunately I’ve not worked with Xen 4.0 so far. But if you find out how to do it we would love to publish your how to here as a guest post, if you like.
LikeLike
Hi Matthias,
Thanks for this HowTO! Really helpfull.
I tried to make it work with HARDY (8.04 ubuntu version), but it stops with:
# xm create /etc/xen/vm1.cfg
Using config file “/etc/xen/vm1.cfg”.
Error: Boot loader didn’t return any data!
Then I
Yes, I remembered to switch disk and swap in the cfg.
Any ideas?
LikeLike
# cat /etc/xen/vm1.cfg
memory = ‘2048’
root = ‘/dev/xvda2 ro’
disk = [ ‘phy:/dev/afrodite/vm1-disk,xvda2,w’, ‘phy:/dev/afrodite/vm1-swap,xvda1,w’ ]
name = ‘vm1’
vif = [ ‘ip=10.254.0.240,mac=00:16:3E:A1:C8:36’ ]
on_poweroff = ‘destroy’
on_reboot = ‘restart’
on_crash = ‘restart’
# cat /etc/xen-tools/xen-tools.conf
lvm = afrodite
install-method = debootstrap
size = 50Gb # Disk image size.
memory = 2Gb # Memory size
swap = 4Gb # Swap size
fs = xfs # use the EXT3 filesystem for the disk image.
dist = hardy # Default distribution to install.
image = sparse # Specify sparse vs. full disk images.
gateway = 10.254.0.1
netmask = 255.255.255.0
broadcast = 10.254.0.255
arch = amd64
mirror_hardy = http://archive.ubuntu.com/ubuntu/
passwd = 1
ext3_options = noatime,nodiratime,errors=remount-ro
ext2_options = noatime,nodiratime,errors=remount-ro
xfs_options = defaults
reiser_options = defaults
serial_device = hvc0
disk_device = xvda
Maybe something differente I have to do with the “installDebianPackage” directive?
Thanks
LikeLike
In the part where you enable debootstrap to use your Ubuntu, have you changed “lucid” to “hardy” there:
LikeLike
I did that. Still the same result.
LikeLike
Hm, maybe for hardy it’s not necessary to use pygrub at all as its kernel is way older. Other than that I fear I cannot help, sorry 😦
LikeLike
Hi,
let me thank you for your post. It was really helpful for me.
I successfully build a 10.04 domU on a 8.04 dom0.
It’s a little different to get this working. First of all it’s not needed to update debootstrap. Just create a symbolic link lucid.d -> edgy.d/ in /usr/lib/xen-tools/ and another one lucid -> hardy in /usr/share/debootstrap/scripts/
xen-create-image should now successfully create a lucid-domU.
After that I changed change my /etc/xen/domu.cfg as suggested in http://blog.webangel.ie/2010/04/ubuntu-upgrade-to-9-1010-04-for-xen-domu/
This is my domU.cfg:
bootloader=”/usr/bin/pygrub”
#kernel = ‘/boot/vmlinuz-2.6.24-28-xen’
#ramdisk = ‘/boot/initrd.img-2.6.24-28-xen’
memory = ‘4096’
#
# Disk device(s).
#
root = ‘/dev/xvda2 ro’
disk = [
‘tap:aio:/data/xen/domains/domU1/disk.img,xvda2,w’,
‘tap:aio:/data/xen/domains/domU1/swap.img,xvda1,w’,
]
Hope that helps.
Axel
LikeLike
Great article! Thanks for sharing this with the rest of us.
Two suggestions if I may:
Instead of editing the original pygrub role script from wejn.org
I suggest you make a copy of it and name it pygrub-lucid.
That way you can have xen-tools ready to install both lenny as well as lucid domUs with pygrub. It would be nicer to handle
The pygrub role script should handle “noswap = 1”. In that case the root partition goes to xvda1 and the boot/grub/menu.lst generated by the pygrub role script will point to a wrong device. (This should probably be fixed upstream at wejn.org)
LikeLike
Thanks for the tips.
An easier way to install an updated debootstrap (>= 1.0.21) is to download the .deb for Debian Squeeze from a mirror and install that with dpkg -i
Ben: I was using xfs (my “default” fs) and couldn’t get pygrub to work with that. Haven’t dug into that but repeated the process with ext3 and all is good.
LikeLike
Hey Matthius!
It worked! It Worked! Thank you very much! For your instructions!
I made some modifications to this slightly though. I wanted to install Xen 4.0.
I had to install Debian Squeeze (Testing) and made just a few odd adjustments here and there to your how to and know it works perfectly.
I will post Instructions in a few days.
Is there a place I can post it or would you like me to post it here in the comment box?
Cheers,
Zubin Parihar
LikeLike
Hi Zubin Parihar, great that it helped you.
Pls feel free to post your modifications here at the comments. I can then update the post as it fits.
LikeLike
Hi,
Awesome tutorial…I just wanted to ask whether this tutorial will help me to run lucid as DomU on my intrepid Dom0…..?
LikeLike
Hi Matthias,
I tried the above tutorial with ubuntu intrepid (8.10) as Dom0 and tried to install 10.4 as DomU, but i got the following error:
root@chetan-laptop:/etc/xen# xen create-image –hostname=lucid –ip=192.168.1.5 –role=udev,pygrub
bash: xen: command not found
What is missing..? Please help me. Also if you know any tutorial which shows how to install ubuntu 10.4 as DomU on ubuntu 8.10 dom0, please let me know. I am new to the Xen environment.
LikeLike
Hi chetan,
xen-create-imageis one command (mind the ‘-‘ between xen and create).I’ve not tried to install on Ubuntu Intrepid myself. Pls let us know here in the comments if you are successful.
LikeLike
Hi guyz,
I have gotten XEN 4.0.1 on Debian Squeeze. It has deb packages already and uses the newest kernel. Below is how I got it to work except changing ubuntu to pv .
The HVM is very stright forward no problems there at all. Just the PV part that is killing me. Also when installing xen 4.0.1 remember to install qemu-4.0 otherwise HVM will not work and there is already a deb package available on Debian Squeeze. It is stable and good to use, haven’t had any troubles on stability on that. Here is a short summary of what I did to install on squeeze :-
1] Install Debian Squeeze with minimal
2] Install SSH
3] Install Xen 4.0.1 with :-
apt-get install xen-linux-system-2.6.32-5-xen-amd64
apt-get install xen-qemu-dm-4.0
apt-get install xen-tools (if you need it)
4] edit the boot options in grub2 here is my setup /etc/default/grub
# If you change this file, run ‘update-grub’ afterwards to update
# /boot/grub/grub.cfg.
GRUB_DEFAULT=4 /dev/null || echo Debian`
# GRUB_CMDLINE_LINUX_DEFAULT=”quiet”
# GRUB_CMDLINE_LINUX=””
GRUB_CMDLINE_XEN_DEFAULT=”dom0_mem=512M loglvl=all guest_loglvl=all dom0_max_vc$
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD …)
#GRUB_BADRAM=”0x01234567,0xfefefefe,0x89abcdef,0xefefefef”
# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo’
#GRUB_GFXMODE=640×480
# Uncomment if you don’t want GRUB to pass “root=UUID=xxx” parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true
# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_LINUX_RECOVERY=”true”
# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE=”480 440 1″
5] update-grub
5a] if you need other options on the module line the way I know is only to edit the /boot/grub/grub.cfg which is not the proper way as I don’t know if you know please let me know. Here is the part of the grub.cfg :-
### BEGIN /etc/grub.d/20_linux_xen ###
menuentry ‘Debian GNU/Linux, with Linux 2.6.32-5-xen-amd64 and XEN 4.0-amd64′ –class debian –class gnu-linux –class gnu –class os –class xen {
insmod part_msdos
insmod ext2
set root='(hd1,msdos1)’
search –no-floppy –fs-uuid –set 08ea09a3-b0d2-4b00-802c-48fdc8e0885d
echo ‘Loading Linux 2.6.32-5-xen-amd64 …’
multiboot /xen-4.0-amd64.gz placeholder dom0_mem=512M loglvl=all guest_loglvl=all dom0_max_vcpus=1 dom0_vcpus_pin
module /vmlinuz-2.6.32-5-xen-amd64 placeholder root=/dev/mapper/cougarlvm-root ro nomodeset console=hvc0 xen-pciback.permissive xen-pciback.hide=(01:0a)
echo ‘Loading initial ramdisk …’
module /initrd.img-2.6.32-5-xen-amd64
}
6] This took me about 1 hr from start to finnish to get up. Everything else is same as before.
Hope this help and tks in Advance
Eric
LikeLike
Sorry something is wrong I will stop posting. But most will get the idea.
Sorry for the mess here
Eric
LikeLike
I followed this guide carefully, but still hit the “Error: Boot loader didn’t return any data!” issue. After a few attempts (each a bit time-consuming) I decided to throw down the $10 that getting images via Stacklet would cost me.
Their pre-packaged Ubuntu 10-4 worked fine, and I’m happy (and no longer spending coding time on admin).
Thanks for the guide (first steps with xen-create-image so still a valuable lesson for me) and for the pointer to Stacklet too!
LikeLike
You, Sir, are a genius!! :D. Thanks a lot!! I was trying to do the exact same thing since 5 nights with no results!! Your instructions worked flawlessly!! :D.
Can you also post how to upgrade the xen-3.2.1 to xen 4.0.1 on this system which is set through your instructions?
LikeLike
Great that it worked for you. Currently I’m not planning to upgrade to Xen 4.0.x but I might do later this year…
LikeLike
Great howto !
Only think i needed to add was the bootloader = /usr/bin/pygrub .
LikeLike