Intro

The goal here is to setup a RISC-V kernel development environment on Nix. I’ll show how this can be done by cross-compiling the kernel and running an out-of-tree module against it in QEMU. The cross-compiling infrastructure in Nixpkgs helps nicely to simplify this process.

Cross compiling in Nix

Cross compilation in Nix is described in detail in the manual. Over-simplified summary incoming.

Platforms

There are 3 platforms to consider:

  1. The build platform - where the package is built.
  2. The host platform - where the package will be ran.
  3. The target platform - if the package emits binaries at run time, where they’re to be ran.

In what follows we’ll specify the latter two using the crossSystem attribute and leave Nix to infer the build platform.

Dependencies

Dependencies are specified in lists which follow the naming convention deps<p1><p2>, where <p1> is the platform on which the dependency is to be ran and <p2> the platform on which the binaries it emits (if any) are to be ran. These labels will be one of Build, Host and Target.

For example, in what follows I use depsBuildBuild to specify dependencies that are ran on the build platform.

Note, depsBuildHost and depsHostTarget don’t exist - they are named nativeBuildInputs and buildInputs respectively.

Steps

Prepare the RAMFS

Lets start by building a minimal ramfs [1, 2, 3]. First create some directories. Not all of these are needed but why not:

sudo mkdir --parents ./RAMFS/{dev,etc,lib,lib64,mnt/root,proc,root,sbin,sys}
sudo cp --archive /dev/{null,console,tty} RAMFS/dev

Next we want to place a RISC-V copy of busybox (statically linked) in ./RAMFS/bin. In Nix this is dead simple using the pkgsStatic attribute (along with the crossSystem attribute mentioned above):

buildbox_dir=$(nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/lib>).systems.examples.riscv64' -A pkgsStatic.busybox)
sudo cp -r $buildbox_dir/bin RAMFS

Finally we need an init script in the ramfs, located at RAMFS/init:

#!/bin/sh

mount -t proc none /proc
mount -t sysfs none /sys

cat <<'EOF'
████████╗░█████╗░██╗░░██╗███████╗  ██████╗░██╗░██████╗░█████╗░░██████╗
╚══██╔══╝██╔══██╗██║░██╔╝██╔════╝  ██╔══██╗██║██╔════╝██╔══██╗██╔════╝
░░░██║░░░███████║█████═╝░█████╗░░  ██████╔╝██║╚█████╗░██║░░╚═╝╚█████╗░
░░░██║░░░██╔══██║██╔═██╗░██╔══╝░░  ██╔══██╗██║░╚═══██╗██║░░██╗░╚═══██╗
░░░██║░░░██║░░██║██║░╚██╗███████╗  ██║░░██║██║██████╔╝╚█████╔╝██████╔╝
░░░╚═╝░░░╚═╝░░╚═╝╚═╝░░╚═╝╚══════╝  ╚═╝░░╚═╝╚═╝╚═════╝░░╚════╝░╚═════╝░
EOF

exec /bin/sh

Don’t forget to make it executable:

chmod +x RAMFS/init

Build the kernel and modules

In order to cross-compile the kernel we need both the RISC-V toolchain and some build-platform dependencies. The following Nix shell provides both (with the toolchain prefixed by riscv64-unknown-linux-gnu-):

let
  pkgs = import <nixpkgs>
    {
      crossSystem = (import <nixpkgs/lib>).systems.examples.riscv64;
    };
in
pkgs.mkShell {
  name = "kernel-qemu";
  depsBuildBuild = with pkgs; [
    # Kernel
    gcc
    gnumake
    flex
    bison
    bc
    ncurses
    pkg-config
    perl
    # Modules
    kmod
    # ramfs
    cpio
    #justqemuthings
    qemu
  ];
}

Here we’ve exploited the depsBuildBuild attribute (see above) to insure the dependencies are compiled for our local system.

Next, grab yourself a copy of the kernel, e.g.:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git linux-stable

Build it and install the in-tree modules:

cd linux-stable
make ARCH=riscv defconfig
make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -j $(nproc)
make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -j $(nproc) modules
sudo make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -j $(nproc) modules_install INSTALL_MOD_PATH=../RAMFS/
cd ../

Next let’s prepare a “hello world” out-of-tree module - just to show how we can interact with the kernel once in qemu. Add a file called ./helloworld/hello.c containing:

#include <linux/module.h>
#include <linux/kernel.h>

int init_module(void) {
    printk(KERN_INFO "Hello world.\n");
    return 0;
}

void cleanup_module(void) {
    printk(KERN_INFO "Goodbye world.\n");
}

MODULE_DESCRIPTION("hello module"); 
MODULE_LICENSE("GPL");

and a Makefile ./helloworld/Makefile containing:

obj-m += hello.o

Build and install the module:

cd linux-stable/
sudo make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -j $(nproc) M=../src/modules/helloworld/
sudo make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- -j $(nproc) M=../src/modules/helloworld/ modules_install INSTALL_MOD_PATH=../RAMFS/
cd ../

Actually run it all

Build the initramfs:

cd RAMFS
sudo find . | cpio -oHnewc | gzip > ../initramfs.gz
cd ../

and now we’re at last ready to run it all in qemu:

qemu-system-riscv64 -nographic -machine virt -kernel linux-stable/arch/riscv/boot/Image -append "root=/dev/ram init=/init" -initrd initramfs.gz

Once in the shell we can load and unload the hello module with modprobe:

> modprobe hello
hello: loading out-of-tree module taints kernel.
Hello world.
> modprobe -r hello
Goodbye world.

And there you have it! We’ve cross compiled the kernel and ran an out-of-tree kernel module against it all in Nix.

Thanks to Sean Borg for reviewing the Nix herein and Ben Dooks for the general kernel principles.