How to set up a guest OS using xen3
From NetBSD Wiki
Contents |
Requirements
Xen3 is supported from NetBSD-4.0 onward. If you plan on using NetBSD-CURRENT, please read the article How to build NetBSD-current to do so. Also keep a spare partition for each guest operating system you intend to install.
This tutorial describes how to:
- install and configure NetBSD as a DOM0
- install and run a Debian system as a DOMU
Installing Xen tools and kernels
Xen tools
To run and administer xen domains, we need the xentools3 packages.
You can get them directly from pkgsrc, under sysutils/xentools3 for traditional xentools, and sysutils/xentools3-hvm for HVM goodies used to run un-modified OSes, through HVM support. The benefit of hardware virtualization is that you can run OSes that don't know they are being virutalized like Windows XP, for example. However, you must have a CPU which supports this. Intel CPUs must have the 'VT' instruction. AMD CPUs will have the 'VMX' instruction. You can find out if your CPU supports HVM by taking a look at this page:
http://wiki.xensource.com/xenwiki/HVM_Compatible_Processors
# cd /usr/pkgsrc/sysutils/xentools3 # make install
Xen kernel
Next, we will get the xen hypervisor kernel itself (sysutils/xenkernel3 in pkgsrc):
# cd /usr/pkgsrc/sysutils/xenkernel3 # make install
And copy it into / directory, like this:
# cp /usr/pkgsrc/sysutils/xenkernel3/work/xen.gz /
Finally, decompress it so that all versions of GRUB can read it:
# gunzip /xen.gz
Xen DOM0 kernel
Lastly, we need a XEN-enabled kernel for our DOM0 domain. There are two possibilities: downloading the binary version, or building it from source.
Downloading the binary version
From NetBSD-4.0 onward, NetBSD supports Xen, and provides some XEN-enabled kernel directly from [1], in each binary/kernel directory associated with a particular release.
For example, with NetBSD-4.0, You can grab one from (ftp.NetBSD.org/pub/NetBSD/NetBSD-4.0/i386/binary/kernel/netbsd-XEN3_DOM0.gz [2]):
# ftp -a ftp://ftp.netbsd.org/pub/NetBSD/NetBSD-4.0/i386/binary/kernel/netbsd-XEN3_DOM0.gz
The netbsd-XEN3_DOM0.gz file contains a gzipped version of the kernel. Just copy and move it into the root directory like this:
# cp netbsd-XEN3_DOM0.gz /
Now decompress it to avoid any issues GRUB or Xen might have with gzipped kernels
# gunzip /netbsd-XEN3_DOM0.gz
Building it from source
Building a kernel from source is out of the scope of this section. Please consult How to build a kernel from NetBSD's FAQ for more details.
Once building is done, you can find them in /usr/src/obj/_releasedir_/i386/binary/kernel/. Identically to binary version, copy or move netbsd-XEN3_DOM0 in the root directory /.
Selecting a bootloader
Starting with NetBSD 5.0 the native boot loader, /boot, can load Xen directly. This means if you're using NetBSD 5.0 or -current you do not need Grub, unless you're using it with another operating system.
Using NetBSD's native Xen support in /boot is a quick and easy way to use NetBSD + Xen. It is also worth noting that the new /boot can used on a NetBSD 4.0 system as well if you copy it into /boot and run installboot(8) properly.
Setting up /boot
As mentioned above, NetBSD -current (of roughly mid October 2008) added support for booting Xen directly from the native NetBSD boot loader. If you're using NetBSD 5.0 or -current from after October 2008 you should have everything needed to setup NetBSD + Xen. Note that you may need to install an updated copy of /boot if you're tracking -current or you upgraded to 5.0 via source.
Updating /boot
For full details refer to installboot(8), but for a standard configuration with VGA console and an IDE or SATA drive with an FFSv1 root file system use the following:
# cp /usr/mdec/boot /boot # installboot -v -o timeout=5 /dev/rwd0a /usr/mdec/bootxx_ffsv1
Updating /boot.cfg
If you installed NetBSD 5.0 or a -current snapshot directly, you should already have a /boot.cfg file with a basic configuration. Enabling Xen support only requires one additional line in this case. If you're upgrading from an earlier version or do not have an existing /boot.cfg use the following example:
banner=Welcome to NetBSD banner================== banner= banner=Please choose an option from the following menu: menu=Boot normally:boot menu=Boot single-user:boot -s menu=Boot backup kernel:boot onetbsd menu=Go to command line (advanced users only):prompt menu=Boot Xen:modules enabled;load /netbsd.dom0;multiboot xen.gz dom0_mem=512M console=vga -- bootdev=wd0a ro console=pc timeout=5 default=1
Make sure you update the "load /netbsd.dom0", the "dom0_mem=512M" and the "bootdev=wd0a" arguments to match your setup. Once you have updated /boot and created /boot.cfg you should be able to reboot and will be presented with a boot menu. Option 5 should be "Boot Xen", select 5 and make sure you see Xen kernel messages and the normal NetBSD kernel messages. Once you're satisfied it is working you can change the "default=1" line to "default=5" to automatically boot Xen on reboot.
Setting up Grub
Before installing and configuring XEN3, we need to install grub, the bootloader required to boot our XEN-enabled system.
To install it, use its package provided in pkgsrc (sysutils/grub):
# cd /usr/pkgsrc/sysutils/grub # make install
Building grub may take a while.
When installation is finished, you can proceed to grub's configuration in the /grub directory.
GRUB configuration
First, create /grub (if it is not present), and copy paste the menu.lst file below used by grub to read its configuration:
# mkdir /grub # vi /grub/menu.lst
This typical /grub/menu.lst file should have following content:
#Grub config file for NetBSD/xen. Copy as /grub/menu.lst and run # grub-install /dev/rwd0d (assuming your boot device is wd0). # # The default entry to load will be the first one default=0 # boot after 10s the default entry if the user didn't hit keyboard timeout=10 # Same as above, but using VGA console # We can use console=tty0 (Linux syntax) or console=pc (NetBSD syntax) title Xen 3.0 / NetBSD (hda0, vga) root(hd0,0) kernel (hd0,a)/xen dom0_mem=514624 module (hd0,a)/netbsd-XEN3_DOM0 bootdev=/dev/wd0a ro console=tty0 #Load a regular NetBSD/i386 kernel. Can be useful if you end up with a #nonworking /xen.gz title NetBSD 4.0 root (hd0,a) kernel --type=netbsd /netbsd #Load the NetBSD bootloader, letting it load the NetBSD/i386 kernel. #May be better than the above, as grub can't pass all required infos #to the NetBSD/i386 kernel (e.g. console, root device, ...) title NetBSD via chain root (hd0,a) chainloader +1 ## end of grub config file.
Notice that the statement dom0_mem is allocating half a gig to Xen. If you have a box with 4Gb of RAM you probably want to change that to something larger. A smaller number might be wise on systems with less RAM, etc...
Setting up DOM0
Creating xen devices
To create all xen devices, change to /dev and run ./MAKEDEV xen
cd /dev ./MAKEDEV xen
This should create the devices xencons, xenevt and xsd_kva. If any of these are missing you may not have updated to CURRENT using the latest sources and you will have to create the missing device files.
Configuring the bridge interface
The bridge(4) interface is used to provide network access to DOMUs.
To use one, edit (or create) the file /etc/ifconfig.bridge0 and insert following lines to:
create !brconfig $int add bge0 stp bge0 up
Where 'bge0' should be changed to the name of the interface you want to use with your guest operating systems. use ifconfig(8) to get more details about your actual interfaces.
Rebooting into DOM0
Time to reboot:
# shutdown -r now
If all has gone well, you should have booted into the XEN3_DOM0 kernel. Check this with uname(1):
# uname -v NetBSD 4.0 (XEN3_DOM0) #0: Sun Dec 16 01:20:31 PST 2007 builds@wb34:/home/builds/ab/netbsd-4-0-RELEASE/i386/200712160005Z-obj/home/builds/ab/netbsd-4-0-RELEASE/src/sys/arch/i386/compile/XEN3_DOM0
You should have XEN3 DOM0 mentioned somewhere.
Configuring rc scripts
Copy or symlink xend, xenbackendd and xendomains from /usr/pkg/share/examples/rc.d to /etc/rc.d.
# cp /usr/pkg/share/examples/rc.d/xend /etc/rc.d/ # cp /usr/pkg/share/examples/rc.d/xenbackendd /etc/rc.d/ # cp /usr/pkg/share/examples/rc.d/xendomains /etc/rc.d/
Edit /etc/rc.conf and add the following lines:
xend=YES xenbackendd=YES xendomains="dom1"
Later on, when you have created a configuration file for 'dom1', the xendomains variable specified above will trigger 'dom1' to be started when the system is booted. At this point no configuration exists for dom1, therefore it does nothing at this point. If you choose to name your configuration file something else, adapt the name accordingly.
To avoid rebooting a second, start all three services:
# /etc/rc.d/xend start # /etc/rc.d/xenbackendd start # /etc/rc.d/xendomains start
Run ifconfig -a to ensure the bridge interface is present and issue a ps ax | grep xen to ensure you have a similar output.
12 ? DK 0:00.00 [xenwatch] 13 ? DK 0:00.00 [xenbus] 411 ? I 0:00.24 xenstored --pid-file=/var/run/xenstore.pid 594 ? IWa 0:00.26 xenconsoled 629 ? IW 0:00.00 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start 631 ? IWsa 0:00.02 /usr/pkg/sbin/xenbackendd 639 ? IWa 0:00.52 /usr/pkg/bin/python2.3 /usr/pkg/sbin/xend start
The DOM0 configuration is now done. We will proceed to configuring DOMU domains.
Configuring DOMU
Configuring and installing a NetBSD DOMU
Create (or modify) /usr/pkg/etc/xen/dom1 and include this:
kernel = "/usr/src/obj/releasedir/i386/binary/kernel/netbsd-INSTALL_XEN3_DOMU.gz" #kernel = "/netbsd-XEN3_DOMU" memory = 64 name = "dom1" #vcpus = 1 disk = [ 'phy:/dev/wd0g,0x03,w','file:/usr/src/obj/releasedir/i386/installation/cdrom/netbsd-i386.iso,0x04,r' ] vif = [ 'bridge=bridge0' ] root = "/dev/wd0d"
This configuration boots into the NetBSD sysinst program and allows you to install a NetBSD DOMU using the normal sysinst method. This configuration uses a DOMU_INSTALL kernel and an ISO image provided by a successful 'build release' and 'build iso-image'. You may be able to locate a valid Xen3 DOMU_INSTALL kernel from ftp://ftp.netbsd.org/pub/NetBSD-daily/ but if not, building a release is your best bet.
In this configuration file, /dev/wd0g is the reserved partition for the guest operating system. This should be changed to the partition you reserved prior to following the instructions within this document.
If you would like to use a physical CDROM instead of an ISO image, change the disk line to:
disk = [ 'phy:/dev/wd0g,0x03,w','phy:/dev/cd0a,0x04,r' ]
Now boot into sysinst using the command:
xm create dom1 -c
The reserved partition will appear as /dev/xbd0. Proceed as you would with a normal NetBSD installation using xbd0 as the target drive and xbd1 as the CDROM.
When you have finished, run shutdown -hp now to dom1.
Now edit /usr/pkg/etc/xen/dom1. Comment the INSTALL kernel and uncomment the DOMU kernel.
You should now have a working NetBSD DOMU (dom1). Boot into dom1 again with the command:
xm create dom1 -c
and ensure the file, /etc/ttys contains only this line or has all other lines commented:
console "/usr/libexec/getty Pc" vt100 on secure
and the file /etc/wscons.conf is completely empty or has all lines commented out. These last two steps ensure no errors should be present on boot. Setting wscons=NO in /etc/rc.conf may effectively do the same thing.
From here, configure /etc/rc.conf and all runtime configuration files as you would normally. The network interface name should be xennet0. Use this name when configuring an IP address.
More information can be obtained by referencing the Xen user guide and the official NetBSD Xen Howto. Questions can be addressed to the port-xen@NetBSD.org mailling list.
Configuring and installing a GNU/Linux DOMU
We will do this in two steps:
- install a GNU/Linux system, from a livecd or any installation media
- configure the DOM0 so that it can create and start the Linux DOMU.
Installing a Linux distribution (soon-to-be DOMU)
Before proceeding with DOMU configuration, we will install our favorite GNU/Linux distribution on the computer.
In order to do it, we need at least two partitions (only one, if you do not consider the swap). These partitions must reside outside of the NetBSD slice, and may be either of primary or extended type. Of course, you can use more than two, but adapt your labels and partitions accordingly.
We do not cover the partition/slices manipulations through fdisk(8) and disklabel(8), as it depends strongly on how you manage your hard drive's space.
For this tutorial, we will use this partitioning:
# fdisk /dev/wd0d
fdisk: removing corrupt bootsel information
fdisk: Cannot determine the number of heads
Disk: /dev/wd0d
NetBSD disklabel disk geometry:
cylinders: 486344, heads: 16, sectors/track: 63 (1008 sectors/cylinder)
total sectors: 490234752
BIOS disk geometry:
cylinders: 1023, heads: 255, sectors/track: 63 (16065 sectors/cylinder)
total sectors: 490234752
Partition table:
0: Linux native (sysid 131)
start 63, size 20482812 (10001 MB, Cyls 0-1274)
PBR is not bootable: All bytes are identical (0x00)
1: Linux swap or Prime or Solaris (sysid 130)
start 20482875, size 1959930 (957 MB, Cyls 1275-1396)
PBR is not bootable: All bytes are identical (0x00)
2: NetBSD (sysid 169)
start 61464690, size 428770062 (209360 MB, Cyls 3826-30515/178/63), Active
3: <UNUSED>
Drive serial number: -286527765 (0xeeebeeeb)
Here, you notice that we decide to use two primary partitions for our future Linux DOMU:
- partition 0 (for the root directory /)
- partition 1 (for the swap)
Labels:
16 partitions: # size offset fstype [fsize bsize cpg/sgs] a: 30720816 61464690 4.2BSD 2048 16384 0 # (Cyl. 60976*- 91453*) b: 1049328 92185506 swap # (Cyl. 91453*- 92494*) c: 428770062 61464690 unused 0 0 # (Cyl. 60976*- 486343) d: 490234752 0 unused 0 0 # (Cyl. 0 - 486343) e: 20480000 93234834 4.2BSD 0 0 0 # (Cyl. 92494*- 112812*) f: 20480000 113714834 4.2BSD 0 0 0 # (Cyl. 112812*- 133129*) g: 20480000 134194834 4.2BSD 0 0 0 # (Cyl. 133129*- 153447*) h: 335559918 154674834 4.2BSD 0 0 0 # (Cyl. 153447*- 486343) i: 20482812 63 Linux Ext2 0 0 # (Cyl. 0*- 20320*) j: 1959930 20482875 swap # (Cyl. 20320*- 22264*)
Bear in mind that we added two labels here, namely i and j, which maps respectively to partition 0 and partition 1 of the disk. We will use these labels later for DOMU configuration.
Now that we have partitioned the disk, proceed with installing your Linux distribution. We will not cover that part in this tutorial. You can either install it from an installation media (a cdrom from example), or copy files from an already installed distribution on your computer.
Tip: to manipulate ext2/3 filesystems (the traditional fs under Linux) from NetBSD, you can use sysutils/e2fsprogs from pkgsrc:
# cd /usr/pkgsrc/sysutils/e2fsprogs # make install
And then use e2fsck, mke2fs and mount_ext2fs(8) directly from NetBSD.
Getting XEN aware Linux kernels
Once installation is done, reboot your computer and return to our Xen-NetBSD system.
To boot our Linux DOMU, we will need a Linux kernel supporting the XENU virtualisation. Depending on your Linux distribution, you can grab one from its repository (it is up to you to find it through aptitude, yum or whatever package manager you use), or get one from the Xen binary distribution.
To get a XENU Linux kernel from Xen binary distribution, get it directly from Xen website download page. Download the tarball and extract the vmlinuz-*-xen from it. In our case, with a 2.6.18 Linux kernel:
# ftp -a http://bits.xensource.com/oss-xen/release/3.1.0/bin.tgz/xen-3.1.0-install-x86_32.tgz # cd /tmp # tar -xzf xen-3.1.0-install-x86_32.tgz dist/install/boot/vmlinuz-2.6.18-xen
vmlinuz-2.6.18-xen is the kernel that Xen will use to start the DOMU. Move it to any directory you like (just remember it when configuring the kernel entry in the DOMU configuration file):
# mv dist/install/boot/vmlinuz-2.6.18-xen /vmlinuz-XEN3-DOMU
Configuring DOMU
Configuring the Linux DOMU is a bit different than a NetBSD one; some options tend to differ.
Edit (or create) the configuration file domu-linux, in /usr/pkg/etc/xen/:
# vi /usr/pkg/etc/xen/domu-linux
Here's a typical config file for a Linux DOMU:
#---------------------------------------------------------------------------- # Kernel image file. This kernel will be loaded in the new domain. kernel = "/vmlinuz-XEN3-DOMU" # Memory allocation (in megabytes) for the new domain. memory = 256 # A handy name for your new domain. This will appear in 'xm list', # and you can use this as parameters for xm in place of the domain # number. All domains must have different names. # name = "domu-linux" # Which CPU to start domain on (only relevant for SMP hardware). CPUs # numbered starting from ``0''. # cpu = "^1" # leave to Xen to pick #---------------------------------------------------------------------------- # Define network interfaces for the new domain. # Number of network interfaces (must be at least 1). Default is 1. vif = [ '' ] # Define MAC and/or bridge for the network interfaces. # # The MAC address specified in ``mac'' is the one used for the interface # in the new domain. The interface in domain0 will use this address XOR'd # with 00:00:00:01:00:00 (i.e. aa:00:00:51:02:f0 in our example). Random # MACs are assigned if not given. # # ``bridge'' is a required parameter, which will be passed to the # vif-script called by xend(8) when a new domain is created to configure # the new xvif interface in domain0. # # In this example, the xvif is added to bridge0, which should have been # set up prior to the new domain being created -- either in the # ``network'' script or using a /etc/ifconfig.bridge0 file. # vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ] #---------------------------------------------------------------------------- # Define the disk devices you want the domain to have access to, and # what you want them accessible as. # # Each disk entry is of the form: # # phy:DEV,VDEV,MODE # # where DEV is the device, VDEV is the device name the domain will see, # and MODE is r for read-only, w for read-write. You can also create # file-backed domains using disk entries of the form: # # file:PATH,VDEV,MODE # # where PATH is the path to the file used as the virtual disk, and VDEV # and MODE have the same meaning as for ``phy'' devices. # # /dev/wd0i will be seen as "hda1" under DOMU (the root partition) # /dev/wd0j will be seen as "hda2" under DOMU (the swap) # disk = [ 'phy:/dev/wd0i,hda1,w','phy:/dev/wd0j,hda2,w' ] #---------------------------------------------------------------------------- # Set the kernel command line for the new domain. # Set root device. root = "/dev/hda1"
Now, you should be able to start your first Linux DOMU!
# xm create -c /usr/pkg/etc/xen/domu-linux
Possible caveats
If you intend to have more than one box configured with the above configuration on the same network, you will most likely have to specify a unique MAC address per guest OS, otherwise it is likely you will have a conflict. I'm not sure if the MAC assignment is random, incremental or if Xen is able to check for the existence of the proposed MAC address, so specifying the MAC address is recommended.
Here is a method to assign a MAC address to a newly created Xen Domu. First as described before, use the following vif parameter in your config file :
vif = [ 'bridge=bridge0' ]
Then, run the Xen DomU and, once logged run the following command :
# ifconfig xennet0
Output Sample :
xennet0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500
capabilities=2800<TCP4CSUM_Tx,UDP4CSUM_Tx>
enabled=0
address: 00:16:3e:2e:32:5f
inet 192.168.4.81 netmask 0xffffff00 broadcast 192.168.4.255
inet6 fe80::216:3eff:fe2e:325f%xennet0 prefixlen 64 scopeid 0x2
A MAC address is automaticaly generated, use it in your config file with the following syntax :
vif = [ 'mac=00:16:3e:2e:32:5f, bridge=bridge0' ]
And now you can restart the Xen DomU.
Please, note that the MAC Address MUST start with "00:16:3e".
