Bare metal builds require a specific toolchain and compile options to insure that code normally generated for processes running in a standard OS environment is not generated and various standard libraries are not included.

Cross Compiling

I chose to use a cross-compiling approach for development. It should absolutely be possible to build the OS on a RPI itself with the correct toolchain but there are a lot of advantages that accrue from building on a larger x86_64 system with NFS, etc.

A variety of toolchains are available on the ARM developer website. At present they are all GCC based. Clang builds ought to work as well, though the make files will have to be modified. The toolchain I have been using is: AArch64 bare-metal target (aarch64-none-elf).

Libraries and Header Files

I have avoided using any libraries and the absolute bare minimum of header files to build the OS. This includes creating very minimal replacements for the C standard library, the standard IO library and the C++ standard library. There are a number of platform specific libraries that are required.

Catch2 for Unit Testing

Unit Tests for the project are built using Catch2. At present, unit tests using Catch2 are compiled for the host machine – not cross-compiled for the target platform. Running Catch2 also requires access to some headers and libraries for standard compilation on the host. This is messy – no doubt about it – and this approach will miss issues with data structure alignment, which matters a lot on AArch64 platforms without memory management enabled. Eventually, it will be possible to host the tests on the target platform but for now my focus is to get the code tested in the most straightforward fashion.

Source code coverage metrics are also available for the unit tests using a coverage target in the makefile.

Directory Structure

The make files (yep good old fashioned make) expect the following structure:

~/dev
        |--- RPIBareMetalOS
                |--- project cloned from github
        |--- gcc-cross
                |--- aarch64-none-elf
                         |--- cross compiling files
~/dev_tools
        |--- arm-gcc-toolchain
        |--- Catch2

There is a script in the root directory of the project named setup_dev_env.sh which will setup the development environment. If you have a vanilla Ubuntu (or probably any reasonable Debian based instance) you can simply execute that script and it will install the cross compiling toolchain, Catch2, QEMU for AArch64 and then create the correct directories and copy files to the right places.

Quick Build

There will be another post with greater detail on the build process, however the steps to build are straightforward. To get a running OS, simply do the following after cloning the project from Github:

cd RPIBareMetalOS/
cd minimalclib/
make all
cd ../minimalstdio/
make all
cd ../rpibaremetalos/
cd resources
unzip sd_compressed.zip
cd ..
make armstub_all
make all

The to run the OS in QEMU:

cd image/
qemu-system-aarch64 -M raspi3b -kernel kernel8.img -serial stdio -sd sd.img

Repeatable Scripted Setup

My development pattern is to create an Ubuntu VM customized for various projects and develop within it using the Remote Development Extension in Visual Studio Code. I run a Proxmox server at home but also use VirtualBox.

I have a separate project with Packer scripts to create Ubuntu development VMs in either of the two hypervisors listed above. Modifying the script for other hypervisors should be straightforward. In addition to the build process, there are a number of install scripts that are available to customize the VM after creation, one of which sets up an AArch64 bare metal toolchain and the directory structure I use for development. After building the VM with the right options, one should be able to simply clone the bare metal OS repository, build and run on QEMU.

The github repository for the Packer scripts is: https://github.com/stephanfr/Packer

An example command line to build a development VM template in Proxmox is:

packer build -var "dev_username=????" -var "dev_password=password" -var "proxmox_host=????" -var "proxmox_node_name=????" -var "proxmox_api_user=packer@pve" -var "proxmox_api_password=ubuntu" -var "ssh_username=packer" -var "ssh_password=password" -var "vmid=????" -var "http_interface=????" -var "install_aarch64_cross=true" -var-file="./22.04/ubuntu-22-04-version.pkrvars.hcl" -var-file="./proxmox/proxmox-config.pkrvars.hcl" -var-file="vm_personalization.pkrvars.hcl" ./proxmox/ubuntu-proxmox.pkr.hcl

After the template is created, clone it and then you will be good to go.

An example command line to build a development VM in VirtualBox appears below:

packer build -var "dev_username=????" -var "dev_password=password" -var "ssh_username=packer" -var "ssh_password=ubuntu" -var "install_aarch64_cross=true" -var-file="./22.04/ubuntu-22-04-version.pkrvars.hcl" -var-file="./virtualbox/virtualbox-config.pkrvars.hcl" -var-file="vm_personalization.pkrvars.hcl" ./virtualbox/ubuntu-virtualbox.pkr.hcl

Replace the ‘????’ sequences with appropriate values. At first login as the development user, you will be forced to change the user’s password.

More documentation can be found in the README file in the Packer script repository.