Creating a FreeBSD Kernel Development Environment with bhyve and Nested Virtualization

Posted 06/10/19

Overview

For the past few weeks, I’ve found myself spending a lot of time doing FreeBSD kernel hacking. The biggest issue I had starting out was establishing an efficient development and testing workflow. I wasn’t willing to blow away the OS on my laptop and install FreeBSD on bare metal, so virtualization seemed like the best route to take. Most of the references that I found involved a two virtual machine (VM) setup. In these setups, development occurs on the first VM while testing of the modified kernel occurs in a second, diskless PXE-booted VM¹. However, during that initial background reading phase, I also came across an article describing a process for loading custom kernels with bhyve². After reading that article, I decided to try going a similar route but instead leveraging nested virtualization to create the bhyve VM inside of a FreeBSD VM running under KVM. The rest of this post details the steps I had to take to get the entire setup working.

Environment Structure

The following diagram outlines the target environment structure:

L0 : Baremetal (Ubuntu 16.04 LTS)
— L1 : Guest OS (FreeBSD 12.0)
—— L2 : Nested Guest OS (FreeBSD 12.0)

The terms L0, L1, and L2 will be used throughout this article. For clarification, all of the development work was done in L1 and the testing of the modified kernel was done in L2. Both L1 and L2 ran FreeBSD 12. L0 ran Ubuntu 16.04 LTS. L0 was an Acer Aspire F15 with an Intel i5-7200U⁴. This is an important detail since your underlying processor needs to support both EPT (or equivalent) and VT-X⁴.

Setup and Configuration.

The following steps give a general outline of the process I followed to create the nested virtual development environment.

Install the necessary software and ISOs on L0
- KVM
- ISOs for whatever FreeBSD versions you want to run in L1 and L2 (I used FreeBSD 12.0)
Configure L0 to enable KVM’s⁵
Create the L1 guest with QEMU/KVM and install FreeBSD⁶
- qemu-img create -f raw image_file 4G
- qemu-system-x86_64 -name image_name -cdrom iso_image -boot order=d -drive file=__disk_image__,format=raw
Modify the L1 VM config to pass through VT-Xhttps://hacking-on.systems/i
- run “virsh edit image_name”
  - Change the “cpu” entry to match this:
    <cpu mode= ‘host passthrough’>
    <feature policy=’require’ name=’vmx’/>
    </cpu>
Start L1 (restart if the VM was running when the configuration change was made)
- run “virsh start vm_name”
Log into L1 and check to make sure that VT-X is enabled
- One way to test is to run “sudo dmesg | grep vmx”
  - If you see something similar to vmx_init: processor does not support VMX operation, then VMX is not enabled
Configure L1 to your desired development environment. Make sure to download the ISO for the version of FreeBSD to which you will be deploying your modified kernel.
Ensure that bhyve is installed.
- Try to add it with “pkg install bhyve”
Build your modified kernel.
1. This post assumes that you already know what to do doing this step. If you don’t the FreeBSD documentation⁷ is a good place to start.
From here, follow the steps outlined in one of the previously mentioned articles². I suggest reading the entire article, but following the directions in the “Configuring Guests,” and “Using a bhyve Guest as a Target” sections should suffice. At a high level, the idea is to create a bhyve VM (L2) with a base FreeBSD ISO, but override the version of the kernel in the ISO with the modified kernel residing on L1.
If everything worked, L2 should run a base version of FreeBSD with your modified kernel. Depending on your kernel build process, you may be able to verify this by checking the output of the “uname -a” command.

Development

Once everything is working, your development process should look something like this:

Checkout your kernel source code in L1
Build your modified kernel in L1
Deploy a new VM (L2) from within L1 that runs your modified kernel using bhyve -H
Boot L2 and perform your kernel testing
Rinse and repeat as necessary