Overview
Last summer, I made a post about creating a virtual FreeBSD kernel development environment. Since then, my interest in the FreeBSD kernel has grown to the point where I now have a bare-metal development environment. The purpose of this post is to describe that environment. Hopefully, it might help people looking to play around with the FreeBSD kernel.
Instead of diving into a lot of the nitty-gritty of setting up such an environment, I’m releasing a series of scripts to serve as a demonstration of how one could automate this process. I use these scripts every time I want to build and debug a custom kernel, so they should work for others with some minor tweaks. These scripts are available here
builder.py
- automates the building of a custom kernel
runner.py
- creates a virtual environment for running your custom kernel (supports both QEMU and bhyve)
src.conf
- a sample FreeBSD build configuration file
If anyone reading this has suggestions for improving the process I’m about to describe, please let me know. I’m always looking to improve my own processes.
Host Environment
Setting up the host environment is relatively straightforward: Install FreeBSD. I’ve been using FreeBSD 12.1 for a while now and haven’t had any problems. The bigger choice I’ve had to make is whether to develop locally or remotely. Personally, I run Emacs on a MacBook Air and use a combination of Tramp and Projectile to work with codebases that live on beefier remote servers. Working with the FreeBSD source presented a bit of a problem for this workflow, though. The size of the repository made Projectile over Tramp unbearably slow. My current solution is to use SSHFS to create a local mount-point to the remote server, then point Projectile at the mount-point. Initialization is a bit slow, but I haven’t experienced any other significant drawbacks.
Building the FreeBSD Kernel
For the purpose of brevity, let’s assume that you have already pulled down the FreeBSD source code. Building and installing the kernel consists of running two commands: make buildkernel and make installkernel. You’ll probably want to create your own kernel configuration file. This page is a good resource to follow. A very basic configuration is sufficient and becomes necessary if you want to debug your kernel.
As you would expect, building the FreeBSD kernel can take a bit of time. However, depending on what you actually need to build, you can dramatically speed things up by reducing what gets built. Here are a few variables that you should consider utilizing:
Non-FreeBSD Specific
- make -j
- Not a FreeBSD-specific variable, but if you have spare cores you should consider using them
FreeBSD Specific
- NO_KERNELCLEAN
- the build process does not run
make clean
- NO_KERNELCONFIG
- the build process does not run
config
- NO_KERNELOBJ
- the build process does not run
make obj
- KERNFAST
- Sets NO_KERNELCLEAN, NO_KERNELCONF, and NO_KERNELOBJ
- NO_MODULES
- the build process does not build modules with the kernel
There are a few different ways to set these variables. You can find examples of how to set these variables in both builder.py (environment variables) and src.conf.
Running your New Kernel
So you’ve successfully built and installed your custom kernel. Now let’s run it. There are a couple different ways to do this.
Baremetal
Let’s say you built the 12.1 kernel for your host’s architecture. You could try rebooting your host and selecting your new kernel in the loader menu. The downside to this method is that if you want to rebuild, you have to boot back into your build environment and set everything up again. Luckily, we can use virtualization to avoid this overhead.
QEMU
To use QEMU, you’ll need to install it on your host machine. Once installed, you’ll need to create an image into which you’ll install FreeBSD:
qemu-img create -f qcow2 kernel_dev.img 8G
Next, you have to actually install FreeBSD in the newly created image:
qemu -m 512 -hda kernel_dev.img -cdrom <path_to_FreeBSD_iso> -boot -d
Now you have a FreeBSD QEMU VM. This next step can probably be improved, but I lack the QEMU experience to do so. We need to move your new kernel into the VM. One way to do this is to mount the VM image file locally, then copy the kernel into it. There is some code in runner.py that does this. The end result is a FreeBSD VM with your custom kernel inside of it. To run your new kernel, boot the VM with QEMU and select the new kernel at the boot menu.
bhyve
A native alternative to QEMU is bhyve. The overall process for testing your new kernel with bhyve is the same as that for QEMU (e.g. create the VM, give it access to your new kernel, boot the VM, load the new kernel). However, bhyve’s -H option allows us to provide the VM with direct access to your custom kernel. Most of this is performed by runner.py, so I won’t detail the steps here. You will still have to manually create the VM image, which can be done by using the truncate command and using bhyve to create the image.
The only thing the build script does not do is modify the host to support networking in the guest. You can do this by adding the following lines to /etc/rc.conf on your host:
autobridge_interfaces="bridge0"
autobridge_bridge0="re0 tap*"
cloned_interfaces="bridge0 tap0 tap1 tap2"
ifconfig_bridge0="up"
After making these changes, be sure to restart your network interface:
/etc/rc/d/netif restart
When you start the bhyve VM, you’ll still need to load your new kernel. Enter the loader prompt from the boot menu and run the following commands:
unloadload host0:/<path_to_kernel>
- this path should be relative to the path you passed to bhyve via the -H argument
boot
Debugging your New Kernel
If you’re building your own kernel, you’ll probably want to be able to debug it. I’ve been able to use both KGDB and DDB to accomplish this. In both cases, you’ll need to perform some additional configuration of the kernel. Earlier I mentioned a configuration file for your custom kernel. You will need to add the following lines to that file to enable kernel debugging:
options KDB
options DDB
options GDB
If you want to use bhyve as a hypervisor, you’ll want to add this line as well:
devices bvmdebug
You will also probably want to generate a debug build of the kernel. You can do this with the following line:
makeoptions DEBUG=-g
As of the writing of this blog post, this option is enabled in the GENERIC configuration file. You’ll also want to track the debug file generated for your kernel. Those files are stored in /usr/lib/debug/ by default.
KGDB
There is some code in runner.py for automating most of the KGDB setup for both QEMU and bhyve. For QEMU, you’ll need to pass in the -s -S command line arguments when starting the VM. This will cause the VM to break on start and listen on port :1234 for a remote debugger. Start KGDB in another console and run the following commands:
kgdb <path_to_kernel>symbol-file <path_to_symbol_file> [optional]
- By default, your kernel debug files will be stored at
/usr/lib/debug/. KGDB seems smart enough to look there first. If your debug file is elsewhere, use the above command to specify the location.
target remote :1234
If done correctly, your KGDB instance should connect to the VM and break.
Bhyve requires a similar process. My recommended order of operations is to create the VM with bhyveload, set up KGDB with the same arguments used for QEMU, then start the VM with bhyve. The runner.py script does this, and will pause between running bhyveload and bhyve to allow you to set up KGDB. If everything was done correctly, KGDB will break shortly after you start the VM.
DDB
DDB is FreeBSD’s online debugger. As such, you can use it within the VM in which you are loading your custom kernel. To break into DDB, use the -d flag from the loader prompt: boot -d. The runner.py script is not configured for DDB. If you want to use that code as a template, remove the remote debugger options (-s -S for QEMU and -G for bhyve).
