phaqphaq

This week I started on evaluating an iSCSI storage system which should get connected to a Xen-enabled host. An article covering this particular topic is in preparation.

Since the iSCSI device offered management capabilities over serial console I though to attach it to COM1 (/dev/ttyS0) of my host system.
This is well the point when I noticed that Xen had already grabbed /dev/ttyS0. Since my host system offered only one serial port and I didn’t have a spare serial server at hands I needed to change this behaviour to get it working.

By means of ‘dmesg’ it can be easily verified if Xen grabbed the serial port:

# dmesg|grep ttyS0
Xen virtual console successfully installed as ttyS0

Then make sure you know which kernel version you’re running:

# uname -r
Linux gic-ftp-lin-001.genotec.ch 2.6.18-8.el5xen #1 SMP Fri Jan 26 14:29:35 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

Look for the corresponding kernel section in your grub.conf (or whatever boot loader you use).

title Red Hat Enterprise Linux Server (2.6.18-8.el5xen)
        root (hd0,0)
        kernel /boot/xen.gz-2.6.18-8.el5
        module /boot/vmlinuz-2.6.18-8.el5xen ro root=LABEL=/
        module /boot/initrd-2.6.18-8.el5xen.img

Then add the ‘xencons’ option to the module line stating the kernel arguments.

title Red Hat Enterprise Linux Server (2.6.18-8.el5xen)
        root (hd0,0)
        kernel /boot/xen.gz-2.6.18-8.el5
        module /boot/vmlinuz-2.6.18-8.el5xen ro root=LABEL=/ xencons=tty6
        module /boot/initrd-2.6.18-8.el5xen.img

In this case I added ‘xencons=tty6′ to have Xen grab /dev/tty6 for it’s console.

Other possibilities are:

After changing this line and rebooting the system I could finally connect my iSCSI’s serial terminal.

It seems impossible at first to run the NVIDIA binary driver on Linux x86_64 when Xen is enabled.

That’s because NVIDIA does not support Xen in the first place albeit there’s some demand for it for sure.

Luckily the only things to be changed are within the kernel interface contained in the driver package. There exist already a lot of patches on the net, I took mine from nvnews.

You can grab a copy of the patch from my site, too.

The following instructions are derived from my steps in getting NVIDIAS’s binary driver to work on Fedora Core 6 running Xen on the x86_64 platform (Intel Core 2 Duo 6420 CPU).
It was verified to work with driver version 1.0.9755 (march 29, 2007) and 1.0-9639 (may 29, 2007), both of which you can get from the NVidia website.

First you need to have some prerequisites installed on your system:

• toolchain (gcc, make, etc.)
• applicable kernel headers (I had kernel 2.6.20-1.2948.fc6xen on my system, so I installed kernel-xen-devel-2.6.20-1.2948.fc6 and kernel-xen-2.6.20-1.2948.fc6 packages)

Then the appropriate driver packages needs to be downloaded and extracted to a temporary directory:

mkdir /tmp/nv_xen
cd /tmp/nv_xen
wget http://us.download.nvidia.com/XFree86/Linux-x86_64/1.0-9639/NVIDIA-Linux-x86_64-1.0-9639-pkg2.run
/bin/sh NVIDIA-Linux-x86_64-1.0-9639-pkg2.run -x
Creating directory NVIDIA-Linux-x86_64-1.0-9639-pkg2
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 1.0-9639.

Also get a copy of the NVidia Patch for XEN:

wget http://phaq.phunsites.net/wp-content/uploads/2007/06/nvidia_xenpatch.gz

Then change to the source directory, apply the patch and compile the driver.
The kernel source is required. If installed properly it should be located at /usr/src/kernels. If more than one exists choose the right one according to your ‘uname -r’ output.

cd NVIDIA-Linux-x86_64-1.0-9639-pkg2/usr/src/nv
zcat ../../../../nvidia_xenpatch.gz | patch -p4
[output omitted]
make SYSSRC=/usr/src/kernels/2.6.20-1.2948.fc6xen-x86_64/ module
[output omitted]
NVIDIA: left KBUILD.

If the compile run succeeds you should see the message NVIDIA: left KBUILD by the end of the output.
If anything weird happens, e.g. build process aborts, either the patch did not apply cleanly, the headers are not up to date or your sources are not configured.
In the last case a simple ‘make oldconfig’ from the kernel source directory may be enough to get it fixed.
There’s also the chance that you try this patch on a non-supported kernel source version or NVIDIA driver package which may prevent successful compilations.
Since I did not implement the patch by myself I cannot be of much help in either case.

Upon successful compilation you can install and load the driver. Verify that the driver loaded by means of ‘lsmod’.

install -D -o root -g root -m 0644 nvidia.ko /lib/modules/`uname -r`/video/nvidia.ko
depmod -a
modprobe nvidia
lsmod|grep nvidia
nvidia               7771096  22

At this point run the NVIDIA installer to get the remaining utilities.
Use the ‘no-kernel-module’ option as you installed the kernel module already.

cd ../../../
/bin/sh nvidia-installer --no-kernel-module

This is it! You are now ready to configure your X server to use the NVIDIA driver.
Stick to the docs included with the driver on how to achieve this.

This is my Xorg sample configuration with twin-view enabled accross two displays at 1280×1024 resolution.

# sample Xorg configuration file for NVIDIA graphics driver

Section "ServerLayout"
	Identifier	"Default Layout"
	Screen		0  "Screen0" 0 0
	InputDevice	"Keyboard0" "CoreKeyboard"
EndSection

Section "Module"
	Load	"dbe"
	Load	"extmod"
	Load	"record"
	Load	"xtrap"
	Load	"freetype"
	Load	"type1"
	Load	"glx"
EndSection

Section "InputDevice"
	Identifier	"Keyboard0"
	Driver		"kbd"
	Option		"XkbModel" "pc105"
	Option		"XkbLayout" "ch"
	Option		"XkbVariant" "de_nodeadkeys"
EndSection

Section "Device"
	Identifier	"Videocard0"
	Driver		"nvidia"
	Option		"TwinView" "1"
	Option		"CursorShadow" "1"
	Option		"TwinViewOrientation" "RightOf"
	Option		"MetaModes" "1280x1024,1280x1024"
EndSection

Section "Screen"
	Identifier	"Screen0"
	Device		"Videocard0"
	DefaultDepth	24
	SubSection "Display"
		Depth	24
		Modes	"1280x1024"
	EndSubSection
EndSection

I finally took the time getting Vista to work in Xen.

Unfortunately there is a bug which prevents the emulated RTL8139 NIC from working properly in Vista (32-bit version concerned only, 64-bit version worked properly).
This seems a known issue after all and there exist several workarounds on the net concerning this case. So far they basically cover how alternate drivers may be patched together to go along with Vista.

The main reason for this is simple: the RTL8139 runs at 100 mbit/s and is required to get decent speeds.

However there is also the possibility to run the guest domain with NE2000 NIC emulation. Albeit this will only give you a network connection speed of 10 mbit/s full-duplex it is easier to get working right away without fiddling around with INF files and such.

Even though Microsoft dropped NE2000 driver support in Vista the NIC may still be used along with the NE2000 Drivers for Windows 2000.
To enable the use of NE2000 NIC emulation I changed my HVM configuration file accordingly:

vif = [ 'type=ioemu, bridge=xenbr0, model=ne2k_pci0 ]

Since my Vista lacked networking support I placed my driver image into an ISO image and changed the HVM to assign a virtual CD-ROM drive:

disk = [ 'file:/mnt/vm/Xen/Vista/disk1_vol1.img,hda,w',
'file:/mnt/vm/Xen/Vista/drivercd.iso,hdc:cdrom,r' ]

The ISO image was created as follows:

mkisofs -o drivercd.iso -J -r /path/to/w2k_8029.zip

The NE2000 Driver ISO is also available as download from my site.

After starting up Vista skip the ‘Add New Hardware Wizard’ in the first. Instead extract the ZIP file from the (virtual) CD-ROM drive to a temporary location.
Afterwards go to Device Manager and select the (yet) unknown pci device and choose the ‘Update Driver’ option from the Drivers tab.
From there go along with ‘Browse My Computer for Software’ and ‘List of Drivers on My Computer’ and clich the ‘Have Disk’ button.
Open the folder containing the extracted ZIP file’s contents and locate the NETRTP5.INF file. You should be given the choice of the ‘Realtek RTL8029 PCI Ethernet NIC’ which you are to accept.
After completing driver installation you should go device properties’ Extended tab and set ‘Duplex Mode’ to be ‘Full Duplex’ for best performance.

Using NE2000 NIC emulation Windows Vista will get basic network connectivity. It is easy enough to install for everybody and provides modest network speed.
If you need FastEthernet performance there is no way around patching drivers, using Vista64, waiting for an updated official driver or some guru to fix it in Xen itself (even if they didn’t screw it up in the first place…).

[Update 2007/06/30]

Here’s one thing I forget to note when I wrote the article in the first place.

Xen (at least 3.0.3 which I run) seems to have a bug which resets the NIC emulation from NE2000 to RTL8139 upon reset of the HVM.

Because of this I changed my HVM config file to destroy the HVM so I won’t end up with a non-working NIC emulation.

on_reboot = ‘destroy’
on_crash = ‘destroy’