Chaining FreeBSD’s pxeboot with pxelinux

Posted by: admin  :  Category: Networking

Recently I invested some development time on my company’s PXE-based network boot system. While pxelinux serves as a general purpose network boot loader at our site, current demands required further extension beyond it’s capabilities. The main reason for this was the inability of pxelinux to be used for certain bootstrap scenarios. As an example to this we may note FreeBSD. While it can be booted from floppy images or an hd-converted ISO-image via pxelinux’s memdisk loader, this actually has some serious limitations.

  1. You’re always limited to the size of the floppy image
  2. Converting ISO’s to hd-like images tends to crash on some buggy BIOS versions

So I decided to conquer FreeBSD to boot directly from pxe. Because I had to retain compatibility with pxelinux as our primary PXE bootloader, FreeBSD’s own loader had to be “chained” to pxelinux. Now this is the easy part as you will only add your pxelinux default file like this:

label fbsdpxe
             KERNEL pxeboot

You see the error!? Well, I didn’t either at first and stumbled accross an ever lasting error message while trying to boot.

     NetInstall :: Main Menu
     F1 :: F2 :: F3 :: F4 :: F5 :: F6 :: F9 :: exit

     local boot is default after 10 seconds.

     Press F1 - F10 to cycle menu pages, enter "exit" or option name to
boot: fbsdpxe
Invalid or corrupt kernel image

At first I though my pxeboot image got corrupted so I fetched another copy still receving the same error.
However I saw the tftp download request in the logs so the config by itself could not be in error.

Jul 19 20:21:39 setup tftpd[22265]: read request for /pxelinux.cfg/default: success
Jul 19 20:21:39 setup tftpd[22267]: read request for /pxelinux.cfg/screens/f1.txt: success
Jul 19 20:21:48 setup tftpd[22272]: read request for /pxeboot: success

So what had happened?

When copying the FreeBSD pxeboot loader over to the tftpd boot directory I must have forgotten to type the filename correctly, so pxeboot.0 suddenly became pxeboot.

Without noticing this I added it like above to the pxelinux config file and ended up with the exact error message as previously shown.

After changing the filename to read pxeboot.0 it actually chain-booted via pxelinux.

During further examination I made the same mistake once before when I iniatlly configured pxelinux years ago, omitting the .0 by the end of the filename.
Interestingly enough pxelinux was served without any problems and could be booted successfully despite it’s possibly wrong filename.
Even more amazing was the fact that also the FreeBSD pxeboot loader would work if served up as primary loader as specified from dhcpd.conf using the “wrong” filename.

Now I’m not sure if there’s a real reason or a naming convention which actually defines the boot loader’s filename to end in .0.
The official specs mention REMOTE.0 to be used as NBP (Network Boot Program) and an optional REMOTE.1 to be fetched additionally in case the NBP exceeds 32k in size and needs to be splitted.

However there seems to be no real, fixed naming convention so to my understanding the file could actually be called anything.

So the mystical .0 at the end of the filename might haven been choosen initially to comply with the original specs and show that the file in question is the first part of the NBP (even if a second part would not exist).

Finally remains the question if pxelinux’s behaviour is by error or by design.

The true answer to this is: by design.

H. Peter Anvin explains this on the common problems page as follows:

[..]It is unfortunate that there isn’t a standard extension used for Linux kernels, and that none of the commonly loaded data formats (except perhaps COM32) have reliable magic numbers.[..]

To conclude from this and the information on reserved filename extensions pxelinux loader routines will decide what to do with a given kernel only upon its filename extension.

7 Responses to “Chaining FreeBSD’s pxeboot with pxelinux”

  1. swygue Says:

    Are you fetching the loader via NFS or TFTP ? I am unable to get it to install using tftp only its expecting a nfs root. I tried building TFTP with LOADER_TFTP_SUPPORT=YES and still no luck.

  2. chrisb Says:

    This is very interesting.

    After being very successful with debian + dhcp3 + tftpd-hpa + pxelinux I have been having a difficult time with FreeBSD pxeboot. No success!

    Thanks for working on this.

    I and my tftpserver are confused about pxeboot’s idea where root is, i think. My boot stops after identifying server, root, and gateway.

    Where does pxeboot want /pxeroot to be on the tftpserver?

  3. chrisb Says:

    Here’s some more work on this problem, in this case with OpenBSD.

  4. Gianpaolo Del Matto Says:

    @ chrisb:

    If you’re going to bootstrap FreeBSD via PXE with
    it’s root device located on an NFS server, you will
    need to specify it’s location via a statement like this
    in dhcpd.conf:

    option root-path “serverip:/pxeroot”

    So you’re virtually free to choose any directory on the
    nfs host as long as it’s nfs exported.

  5. Gianpaolo Del Matto Says:


    Actually I’m fetching it via tftp only, which means loader was recompiled
    with the LOADER_TFTP_SUPPORT=YES flag enabled.

    But there is some code in loader which will still try on NFS and
    therefore cause some delays and sometimes unpredictable results.
    So if you want to do pure TFTP only, it’s better to patch out some
    sections in /usr/src/sys/boot/i386/libi386/pxe.c.

    Then to have loader download the files properly via TFTP, some
    prerequisites must be fulfilled.

    1. A root disk image (e.g. a floppy disk image) must exist on the TFTP server-
    As loader runs in real mode, the size of the kernel, any pre-loaded kernel modules
    and the disk image (aka mfs root) must no exceed 16 Mbytes.
    2. loader must be scripted to download kernel, modules and the mfs root image
    to boot from.

    So we may end up with a directory structure like this on the TFTP host:

    /boot/i386/6.2-RELASE/* (kernel and modules)

    The script loader.rc might be looking like this:

    \ load kernel and mfs root
    set kernel=6.2-RELEASE
    set module_path=/boot;/boot/kernel;/boot/i386/6.2-RELEASE
    load kernel
    load -t mfs_root mfsroot

    \ initialize kernel environment
    include /boot/device.hints
    set vfs.root.mountfrom=ufs:/dev/md0c
    set console=vidconsole
    unset boot.nfsroot.path
    unset boot.nfsroot.server

    \ boot kernel
    echo booting …

    This is of course only an excerpt from the actual work.

    I look forward to writing an article on the whole process, as
    besides patching there’s also the possibility to implement a dynamic
    boot loader with a menu system from FreeBSDs loader.

    Maybe you want to check back on this at a later time.

  6. Lev Serebryakov Says:

    @Gianpaolo Del Matto:

    Is it possible to re-define “option root-path” in loader.conf? I want to make install-server with four versions of FreeBSD avail (6.3/7.0 i386/amd64) and I need different “root-path”s in this case. I can prepare menu in forth, but I don’t understand what variable should I set!

    It seems, that “boot.nfsroot.path” is set but “common/dev_net.c” and “i386/libi386/pxe.c”, but it is never used again…

    PXE will load common pxeboot (loader(8)), it will load common loader.conf and show menu. What’s next?

    Thank you.

  7. Johan Says:

    This info is deeply hidden away in a text file in the syslinux documentation, but you can force sys-/pxe-/isolinux to load a file of which type you specify manually, ignoring the extension.
    In your case, simply replace “KERNEL” with “PXE”.
    Here is the full list of “KERNEL”-replacements:
    LINUX image – Linux kernel image (default)
    BOOT image – Bootstrap program (.bs, .bin)
    BSS image – BSS image (.bss)
    PXE image – PXE Network Bootstrap Program (.0)
    FDIMAGE image – Floppy disk image (.img)
    COMBOOT image – COMBOOT program (.com, .cbt)
    COM32 image – COM32 program (.c32)
    CONFIG image – New configuration file

    (I can not see any way to change the behavior of DEFAULT tho..)