Hacking On Systems

A Systems Hacking Blog

Posted

Overview

Last summer, I made a post about creating a virtual FreeBSD kernel development environment. Since then, my interest in the FreeBSD kernel has grown to the point where I now have a bare-metal development environment. The purpose of this post is to describe that environment. Hopefully, it might help people looking to play around with the FreeBSD kernel.

Instead of diving into a lot of the nitty-gritty of setting up such an environment, I’m releasing a series of scripts to serve as a demonstration of how one could automate this process. I use these scripts every time I want to build and debug a custom kernel, so they should work for others with some minor tweaks. These scripts are available here

  • builder.py
    • automates the building of a custom kernel
  • runner.py
    • creates a virtual environment for running your custom kernel (supports both QEMU and bhyve)
  • src.conf
    • a sample FreeBSD build configuration file

If anyone reading this has suggestions for improving the process I’m about to describe, please let me know. I’m always looking to improve my own processes.

Host Environment

Setting up the host environment is relatively straightforward: Install FreeBSD. I’ve been using FreeBSD 12.1 for a while now and haven’t had any problems. The bigger choice I’ve had to make is whether to develop locally or remotely. Personally, I run Emacs on a MacBook Air and use a combination of Tramp and Projectile to work with codebases that live on beefier remote servers. Working with the FreeBSD source presented a bit of a problem for this workflow, though. The size of the repository made Projectile over Tramp unbearably slow. My current solution is to use SSHFS to create a local mount-point to the remote server, then point Projectile at the mount-point1. Initialization is a bit slow, but I haven’t experienced any other significant drawbacks.

Building the FreeBSD Kernel

For the purpose of brevity, let’s assume that you have already pulled down the FreeBSD source code. Building and installing the kernel2 consists of running two commands: make buildkernel and make installkernel. You’ll probably want to create your own kernel configuration file. This page3 is a good resource to follow. A very basic configuration is sufficient and becomes necessary if you want to debug your kernel.

As you would expect, building the FreeBSD kernel can take a bit of time. However, depending on what you actually need to build, you can dramatically speed things up by reducing what gets built. Here are a few variables that you should consider utilizing:

Non-FreeBSD Specific

  • make -j4
    • Not a FreeBSD-specific variable, but if you have spare cores you should consider using them

FreeBSD Specific

  • NO_KERNELCLEAN5
    • the build process does not run make clean
  • NO_KERNELCONFIG5
    • the build process does not run config
  • NO_KERNELOBJ5
    • the build process does not run make obj
  • KERNFAST5
    • Sets NO_KERNELCLEAN, NO_KERNELCONF, and NO_KERNELOBJ
  • NO_MODULES6
    • the build process does not build modules with the kernel

There are a few different ways to set these variables. You can find examples of how to set these variables in both builder.py (environment variables) and src.conf.

Running your New Kernel

So you’ve successfully built and installed your custom kernel. Now let’s run it. There are a couple different ways to do this.

Baremetal

Let’s say you built the 12.1 kernel for your host’s architecture. You could try rebooting your host and selecting your new kernel in the loader menu. The downside to this method is that if you want to rebuild, you have to boot back into your build environment and set everything up again. Luckily, we can use virtualization to avoid this overhead.

QEMU7

To use QEMU, you’ll need to install it on your host machine. Once installed, you’ll need to create an image into which you’ll install FreeBSD:
qemu-img create -f qcow2 kernel_dev.img 8G
Next, you have to actually install FreeBSD in the newly created image:
qemu -m 512 -hda kernel_dev.img -cdrom <path_to_FreeBSD_iso> -boot -d
Now you have a FreeBSD QEMU VM. This next step can probably be improved, but I lack the QEMU experience to do so. We need to move your new kernel into the VM. One way to do this is to mount the VM image file locally, then copy the kernel into it. There is some code in runner.py that does this. The end result is a FreeBSD VM with your custom kernel inside of it. To run your new kernel, boot the VM with QEMU and select the new kernel at the boot menu.

bhyve8

A native alternative to QEMU is bhyve. The overall process for testing your new kernel with bhyve is the same as that for QEMU (e.g. create the VM, give it access to your new kernel, boot the VM, load the new kernel). However, bhyve’s -H option allows us to provide the VM with direct access to your custom kernel. Most of this is performed by runner.py, so I won’t detail the steps here. You will still have to manually create the VM image, which can be done by using the truncate command and using bhyve to create the image9.

The only thing the build script does not do is modify the host to support networking in the guest. You can do this by adding the following lines to /etc/rc.conf on your host:

autobridge_interfaces="bridge0"
autobridge_bridge0="re0 tap*"
cloned_interfaces="bridge0 tap0 tap1 tap2"
ifconfig_bridge0="up"

After making these changes, be sure to restart your network interface:
/etc/rc/d/netif restart

When you start the bhyve VM, you’ll still need to load your new kernel. Enter the loader prompt from the boot menu and run the following commands:

  1. unload
  2. load host0:/<path_to_kernel>
    1. this path should be relative to the path you passed to bhyve via the -H argument
  3. boot

Debugging your New Kernel

If you’re building your own kernel, you’ll probably want to be able to debug it. I’ve been able to use both KGDB and DDB to accomplish this. In both cases, you’ll need to perform some additional configuration of the kernel. Earlier I mentioned a configuration file for your custom kernel3. You will need to add the following lines to that file to enable kernel debugging:

options KDB
options DDB
options GDB

If you want to use bhyve as a hypervisor, you’ll want to add this line as well:
devices bvmdebug

You will also probably want to generate a debug build of the kernel. You can do this with the following line:
makeoptions DEBUG=-g

As of the writing of this blog post, this option is enabled in the GENERIC configuration file. You’ll also want to track the debug file generated for your kernel. Those files are stored in /usr/lib/debug/ by default.

KGDB10

There is some code in runner.py for automating most of the KGDB setup for both QEMU and bhyve. For QEMU, you’ll need to pass in the -s -S command line arguments when starting the VM. This will cause the VM to break on start and listen on port :1234 for a remote debugger. Start KGDB in another console and run the following commands:

  1. kgdb <path_to_kernel>
  2. symbol-file <path_to_symbol_file> [optional]
    • By default, your kernel debug files will be stored at /usr/lib/debug/. KGDB seems smart enough to look there first. If your debug file is elsewhere, use the above command to specify the location.
  3. target remote :1234

If done correctly, your KGDB instance should connect to the VM and break.

Bhyve requires a similar process. My recommended order of operations is to create the VM with bhyveload, set up KGDB with the same arguments used for QEMU, then start the VM with bhyve. The runner.py script does this, and will pause between running bhyveload and bhyve to allow you to set up KGDB. If everything was done correctly, KGDB will break shortly after you start the VM.

DDB11

DDB is FreeBSD’s online debugger. As such, you can use it within the VM in which you are loading your custom kernel. To break into DDB, use the -d flag from the loader prompt: boot -d. The runner.py script is not configured for DDB. If you want to use that code as a template, remove the remote debugger options (-s -S for QEMU and -G for bhyve).

1 Editing Remote Code With Emacs

2 Building and Installing a Custom Kernel

3 The Configuration File

4 5.4 Parallel Execution

5 FreeBSD Manual Pages: build

6 FreeBSD Manual Pages: make.conf

7 QEMU

8 bhyve, the BSD Hypervisor

9 FreeBSD as a Host with bhyve

10 FreeBSD Manual Pages: kgdb

11 On-Line Kernel Debugging Using DDB

Author
Categories FreeBSD, Kernel

Posted

Overview

Over the past few months, I’ve been slowly streamlining my day-to-day routine. One of the first things I do when I start my work day is try to catch up on current events in my research area. For awhile now, my process has been to use my phone to check for new articles or posts on a set of websites. I realized that I could reduce both the amount of time I spent doing this by using an RSS feed reader. I ended up choosing Tiny Tiny RSS1 because it’s lightweight and can be self-hosted.

Self-Hosting Tiny Tiny RSS (tt-rss)

I haven’t done much self-hosting in the past, so I figured this would be a great place to start. My home network lacks a static IP, so I decided to host tt-rss via a VPS provider. I ended up choosing Vultr2, primarily based on the reviews provided by joshtronic3. Creating a Linux instance was quick and easy and took less than five minutes after the initial signup.

Once the Linux instance was configured, the installation process for tt-rss was relatively straightforward. For the sake of convenience, I deployed using the provided, static docker images4. The tt-rss docker install FAQ provides additional details on how to install with TLS enabled5. TLS support requires an externally reachable domain name. I decided to put in the extra effort to set up TLS since I already owned a domain name. I was able to link the IP of my server to the domain name via Vultr by following these steps:

1. Log into your domain name registrar
2. Change your domain’s name servers to Vultrs’ (you may need to wait up to a day for the name server changes to propagate)

  • ns1.vultr.com
  • ns2.vultr.com

3. Log into the Vultr management portal
4. Select your server (make a copy of its IP address) and use the “…” side menu to open the “Server Details” page
5. Click the plus sign and “Add Domain”
6. Enter your domain name and the IP of the server

Once your domain is configured, you can continue following the TLS steps. When you are ready to run docker compose, I suggest doing so in a tmux or screen window. You can check to see if your install succeeded by visiting the tt-rss frontend via your domain (e.g. https://mydomainname/tt-rss/). I made the mistake of not properly setting SELF_URL_PATH in my .env file. Luckily, the resultant error message supplied the expected value. Changing that required terminating the running docker containers and making the required change to my .env file.

Emacs Integration with elfeed

Following the process in the section above was sufficient for creating a functional, self-hosted tt-rss instance. However, I wanted to take it a step further and integrate tt-rss with my emacs environment. This was made possible by the elfeed package6.

To setup elfeed to receive data from a tt-rss instance requires both the elfeed and elfeed-protocol packages, both of which are available on melpa. Once you install both packages, you’ll need to configure them. Added the following lines to your emacs config file:

;; elfeed
(require 'elfeed)
(require 'elfeed-protocol)
;; debugging
(setq elfeed-use-curl t)
(setq elfeed-log-level 'debug)
(toggle-debug-on-error)
(setq elfeed-protocol-log-trace t)
;; feeds
(setq elfeed-feeds
  '(("ttrss+https://<username>@<domain_name>/tt-rss"
      :password "<password>")))
(elfeed-protocol-enable)

The debugging lines are included in case something goes wrong during the setup. Once you have everything working, feel free to remove them. To incorporate your tt-rss feed, replace <username> and <domain_name> with the user name used to log into tt-rss at the specified domain. Similarly, replace <password> with the password for the specified user name.

Once your emacs environment is configured, you’ll need to log into your tt-rss web management page, and enabled API usage:

1. Log into tt-rss as the user specified in <username>
2. Select “Preferences”
3. Under “General,” check the “Enable API” box
4. Save your changes

Now, test your setup:

1. Start emacs
2. M-x elfeed
3. M-x elfeed-update
4. Wait for a minute or two

If everything worked, you should see your feed updates in your emacs window. If not, now is the time to use those extra debugging flags. Navigate to the *elfeed-log* buffer (space – b – B in doom emacs) and check for error messages.

Conclusion

I’ve only been using the setup for a few days, but I have already noticed a difference in my productivity. Since all of my information sources are now concentrated in a single location, I spend less time mindlessly perusing the internet. Additionally, this setup has helped me spend less time on my phone. Before, I found myself constantly getting distracted by checking sites on my phone. By forcing myself to go through my desktop, it has been easier to resist this temptation, especially if I’m not working at my computer.

1 Tiny Tiny RSS

2 VULTR

3 VPS Showdown

4 Install tt-rss

5 TLS tt-rss Configuration

6 elfeed

Author
Categories emacs

Posted

Overview

For the past few weeks, I’ve found myself spending a lot of time doing FreeBSD kernel hacking. The biggest issue I had starting out was establishing an efficient development and testing workflow. I wasn’t willing to blow away the OS on my laptop and install FreeBSD on bare metal, so virtualization seemed like the best route to take. Most of the references that I found involved a two virtual machine (VM) setup. In these setups, development occurs on the first VM while testing of the modified kernel occurs in a second, diskless PXE-booted VM1. However, during that initial background reading phase, I also came across an article describing a process for loading custom kernels with bhyve2. After reading that article, I decided to try going a similar route but instead leveraging nested virtualization to create the bhyve VM inside of a FreeBSD VM running under KVM. The rest of this post details the steps I had to take to get the entire setup working.

Environment Structure

The following diagram outlines the target environment structure:

L0 : Baremetal (Ubuntu 16.04 LTS)
— L1 : Guest OS (FreeBSD 12.0)
—— L2 : Nested Guest OS (FreeBSD 12.0)

The terms L0, L1, and L2 will be used throughout this article. For clarification, all of the development work was done in L1 and the testing of the modified kernel was done in L2. Both L1 and L2 ran FreeBSD 12. L0 ran Ubuntu 16.04 LTS. L0 was an Acer Aspire F15 with an Intel i5-7200U4. This is an important detail since your underlying processor needs to support both EPT (or equivalent) and VT-X4.

Setup and Configuration.

The following steps give a general outline of the process I followed to create the nested virtual development environment.

  1. Install the necessary software and ISOs on L0
    • KVM
    • ISOs for whatever FreeBSD versions you want to run in L1 and L2 (I used FreeBSD 12.0)
  2. Configure L0 to enable KVM’s5
  3. Create the L1 guest with QEMU/KVM and install FreeBSD6
    • qemu-img create -f raw image_file 4G
    • qemu-system-x86_64 -name image_name -cdrom iso_image -boot order=d -drive file=__disk_image__,format=raw
  4. Modify the L1 VM config to pass through VT-Xhttps://hacking-on.systems/i
    • run “virsh edit image_name”
      • Change the “cpu” entry to match this:
        <cpu mode= ‘host passthrough’>
        <feature policy=’require’ name=’vmx’/>
        </cpu>
  5. Start L1 (restart if the VM was running when the configuration change was made)
    • run “virsh start vm_name”
  6. Log into L1 and check to make sure that VT-X is enabled
    • One way to test is to run “sudo dmesg | grep vmx”
      • If you see something similar to vmx_init: processor does not support VMX operation, then VMX is not enabled
  7. Configure L1 to your desired development environment. Make sure to download the ISO for the version of FreeBSD to which you will be deploying your modified kernel.
  8. Ensure that bhyve is installed.
    • Try to add it with “pkg install bhyve”
  9. Build your modified kernel.
    1. This post assumes that you already know what to do doing this step. If you don’t the FreeBSD documentation7 is a good place to start.
  10. From here, follow the steps outlined in one of the previously mentioned articles2. I suggest reading the entire article, but following the directions in the “Configuring Guests,” and “Using a bhyve Guest as a Target” sections should suffice. At a high level, the idea is to create a bhyve VM (L2) with a base FreeBSD ISO, but override the version of the kernel in the ISO with the modified kernel residing on L1.
  11. If everything worked, L2 should run a base version of FreeBSD with your modified kernel. Depending on your kernel build process, you may be able to verify this by checking the output of the “uname -a” command.

Development

Once everything is working, your development process should look something like this:
  1. Checkout your kernel source code in L1
  2. Build your modified kernel in L1
  3. Deploy a new VM (L2) from within L1 that runs your modified kernel using bhyve -H
  4. Boot L2 and perform your kernel testing
  5. Rinse and repeat as necessary

Footnotes

1 FreeBSD diskless on VirtualBox

2 Using bhyve for FreeBSD Development

3 Intel i5-7200U

4 FreeBSD as a Host with bhyve

5 Nested Guests

6 QEMU

7 Chapter 9. Building and Installing a FreeBSD Kernel

Author
Categories FreeBSD, Virtualization