Monthly Archives: January 2024

Bare Metal Build System

Leave a reply

Normally, I prefer using CMake for C/C++ projects but for bare metal development, there are a number of requirements that are difficult to meet with CMake. Put simply, I could probably find a way to manage a bare metal build with CMake but at present, using good old Make seems preferable. One example of a bare metal challenge for CMake is specifying a Linker Script. There are ways to add a linker script as a dependency using CMake but they are generally pretty cryptic.

This post describes features of the GNU Make tool.

Primary Requirements for Bare Metal Build System

There are a handful of requirements I have for the build system:

  1. DRY – define compile and link settings once in one location
  2. Build subdirectories without a separate makefile in each subdirectory
  3. Support linker scripts naturally
  4. Simple debugging capability

Make is really just the combination of a rule execution engine with a text substitution and expansion engine. This combination is powerful but it is sometimes difficult to wrap one’s head around what is happening when make processes a makefile. In short, when making a build, make starts with the top level goal and then derives all the dependencies required to complete that goal and then satisfies those dependencies. For the case of an executable, the dependencies are compiling the source files and then linking the executable. Along the way, there is the potential for a lot of test expansion to generate the dependency names.

Make ‘include’

To satisfy the DRY (Do not Repeat Yourself) requirement, make has the ability to include other make files using the ‘include’ command. I have extracted the parameters needed across all the different projects in my bare metal system and put them into a ‘makefile.mk‘ file which is included at the top of each project’s makefile. For example, the shared symbols for the AArch64 bare metal cross compilation build appear below:

TOOLS := ${HOME}/dev_tools
GCC_CROSS_DIRECTORY := ${HOME}/dev/gcc-cross

GCC_VERSION := 12.3.1

GCC_CROSS_TOOLS_PATH := $(TOOLS)/arm-gnu-toolchain-12.3.rel1-x86_64-aarch64-none-elf/bin/
GCC_CROSS_INCLUDE := $(GCC_CROSS_DIRECTORY)/aarch64-none-elf

CC := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-gcc

LD := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-ld

AR := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-ar

OBJCOPY := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-objcopy

CPREPROCESSOR := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-cpp

ASM_FLAGS := -Wall -O2 -ffreestanding -mcpu=cortex-a53 -mstrict-align

C_FLAGS := -Wall -O2 -ffreestanding -fno-stack-protector -nostdinc -nostdlib -nostartfiles -fno-exceptions -fno-unwind-tables -mcpu=cortex-a53 -mstrict-align

CPP_FLAGS := $(C_FLAGS) -std=c++20 -fno-rtti

LD_FLAGS := -nostartfiles -nodefaultlibs -nostdlib -static

INCLUDE_DIRS := -I$(GCC_CROSS_INCLUDE)/lib/gcc/aarch64-none-elf/$(GCC_VERSION)/include -I$(GCC_CROSS_INCLUDE)/lib/gcc/aarch64-none-elf/$(GCC_VERSION)/include-fixed -I$(GCC_CROSS_INCLUDE)/aarch64-none-elf/include

CATCH2_PATH := $(TOOLS)/Catch2

TEST_CC := gcc

TEST_LD := g++

TEST_CFLAGS := -Wall -O2

TEST_CPP_FLAGS := $(TEST_CFLAGS) -std=c++20

COVERAGE_CC := gcc

COVERAGE_LD := g++

COVERAGE_CFLAGS := -Wall -O0 -fprofile-arcs -ftest-coverage

COVERAGE_CPP_FLAGS := $(COVERAGE_CFLAGS) -std=c++20

This file is included in subsequent makefiles with:

include ../Makefile.mk

Build Subdirectories with Make Functions

Make has a powerful set of functions which operate on symbols and lists specified in the makefile. I have used six of these functions to allow my makefile to process subdirectories in a project. The functions and snippets of the GNU Make documentation for each appears below :

Key Functions

  • $(addprefix prefix,names…) – prepends each ‘name‘ with the specified ‘prefix‘
  • $(patsubst pattern,replacement,text) – Finds whitespace-separated words in ‘text’ that match ‘pattern’ and replaces them with ‘replacement‘. Here ‘pattern’ may contain a ‘%’ which acts as a wildcard, matching any number of any characters within a word. If ‘replacement’ also contains a ‘%’, the ‘%’ is replaced by the text that matched the ‘%’ in ‘pattern‘. Words that do not match the ‘pattern’ are kept without change in the output. Only the first ‘%’ in the ‘pattern’ and ‘replacement’ is treated this way; any subsequent ‘%’ is unchanged.
  • $(wildcard pattern…) – used anywhere in a makefile, is replaced by a space-separated list of names of existing files that match one of the given file name ‘patterns‘. If no existing file name matches a ‘pattern‘, then that ‘pattern’ is omitted from the output of the wildcard function. Note that this is different from how unmatched wildcards behave in rules, where they are used verbatim rather than ignored (see Pitfalls of Using Wildcards).
  • $(foreach var,list,text) – The first two arguments, ‘var’ and ‘list‘, are expanded before anything else is done; note that the last argument, ‘text‘, is not expanded at the same time. Then for each word of the expanded value of ‘list‘, the variable named by the expanded value of ‘var’ is set to that word, and ‘text’ is expanded. Presumably ‘text’ contains references to that variable, so its expansion will be different each time. The result is that ‘text’ is expanded as many times as there are whitespace-separated words in ‘list‘. The multiple expansions of ‘text’ are concatenated, with spaces between them, to make the result of ‘foreach‘.
  • $(call variable,param,param,…) – The ‘call’ function is unique in that it can be used to create new parameterized functions. You can write a complex expression as the value of a ‘variable‘, then use call to expand it with different values. When make expands this function, it assigns each ‘param’ to temporary variables $(1), $(2), etc. The variable $(0) will contain ‘variable‘. There is no maximum number of parameter arguments. There is no minimum, either, but it doesn’t make sense to use ‘call’ with no parameters.
  • $(eval param) – The ‘eval’ function is very special: it allows you to define new makefile constructs that are not constant; which are the result of evaluating other variables and functions. The argument to the ‘eval’ function is expanded, then the results of that expansion are parsed as makefile syntax. The expanded results can define new make variables, targets, implicit or explicit rules, etc. The result of the ‘eval’ function is always the empty string; thus, it can be placed virtually anywhere in a makefile without causing syntax errors. It’s important to realize that the ‘eval’ argument is expanded twice; first by the ‘eval’ function, then the results of that expansion are expanded again when they are parsed as makefile syntax. This means you may need to provide extra levels of escaping for “$” characters when using ‘eval’. The ‘value’ function (see The value Function) can sometimes be useful in these situations, to circumvent unwanted expansions.

With the exception of ‘addprefix‘ the other functions can be a bit complex. GNU Make documentation has more descriptive detail and examples. The key to the makefile operation is the use of these functions together with the text expansion engine to automatically generate dependencies from subdirectories.

Example Makefile

include ../Makefile.mk

SRC_ROOT := src
BUILD_ROOT := build
IMAGE_DIR   := image

BUILD_DIRS := $(IMAGE_DIR) $(BUILD_ROOT) $(BUILD_ROOT)/asm $(BUILD_ROOT)/c $(BUILD_ROOT)/c/utility $(BUILD_ROOT)/c/platform $(BUILD_ROOT)/c/platform/rpi3 $(BUILD_ROOT)/c/platform/rpi4 $(BUILD_ROOT)/c/devices $(BUILD_ROOT)/c/devices/rpi3 $(BUILD_ROOT)/c/devices/rpi4 $(BUILD_ROOT)/c/isr $(BUILD_ROOT)/c/filesystem $(BUILD_ROOT)/c/services

ASM_DIRS   := asm
C_DIRS     := c c/utility c/platform c/platform/rpi3 c/platform/rpi4 c/devices c/devices/rpi3 c/devices/rpi4 c/isr c/filesystem c/services
CPP_DIRS   := c c/utility c/platform c/platform/rpi3 c/platform/rpi4 c/devices c/devices/rpi3 c/devices/rpi4 c/isr c/filesystem c/services

ASM_SRC_DIRS := $(addprefix $(SRC_ROOT)/,$(ASM_DIRS))
C_SRC_DIRS   := $(addprefix $(SRC_ROOT)/,$(C_DIRS))
CPP_SRC_DIRS := $(addprefix $(SRC_ROOT)/,$(CPP_DIRS))

ELF := $(BUILD_ROOT)/kernel8.elf
IMG := $(IMAGE_DIR)/kernel8.img

ASM_SRC := $(foreach sdir,$(ASM_SRC_DIRS),$(wildcard $(sdir)/*.S))
C_SRC   := $(foreach sdir,$(C_SRC_DIRS),$(wildcard $(sdir)/*.c))
CPP_SRC := $(foreach sdir,$(CPP_SRC_DIRS),$(wildcard $(sdir)/*.cpp))

OBJ := $(patsubst src/asm/%.S,build/asm/%.o,$(ASM_SRC)) $(patsubst src/c/%.c,build/c/%.o,$(C_SRC)) $(patsubst src/c/%.cpp,build/c/%.o,$(CPP_SRC))

INCLUDE_DIRS += -Iinclude -I../minimalstdio/include -I../minimalclib/include -I../minimalstdlib/include
LDFLAGS += -L../minimalstdio/lib -L../minimalclib/lib
LDLIBS = -lminimalstdio -lminimalclib

LINKER_SCRIPT_TEMPLATE=link.template.ld
LINKER_SCRIPT=$(BUILD_ROOT)/link.ld


all: clean checkdirs $(IMG)


$(IMG): $(ELF)
	$(OBJCOPY) -O binary $(ELF) $(IMG)
	/bin/cp redistrib/*.* image/.
	/bin/cp armstub/image/armstub_minimal.bin image/.
	/bin/cp resources/*.txt image/.
	/bin/cp resources/sd.img image/.

$(ELF): $(OBJ) $(LINKER_SCRIPT)
	$(LD) $(LDFLAGS) $(OBJ) $(LDLIBS) -T $(LINKER_SCRIPT) -o $(ELF)


$(LINKER_SCRIPT): 
	$(CPREPROCESSOR) -Iinclude  $(LINKER_SCRIPT_TEMPLATE) -o $(LINKER_SCRIPT)

define make-asm-goal
$(BUILD_ROOT)/$1/%.o: $(SRC_ROOT)/$1/%.S
	$(CC) $(INCLUDE_DIRS) $(ASM_FLAGS) -c $$< -o $$@
endef

define make-c-goal
$(BUILD_ROOT)/$1/%.o: $(SRC_ROOT)/$1/%.c
	$(CC) $(INCLUDE_DIRS) $(C_FLAGS) -c $$< -o $$@
endef

define make-cpp-goal
$(BUILD_ROOT)/$1/%.o: $(SRC_ROOT)/$1/%.cpp
	$(CC) $(INCLUDE_DIRS) $(CPP_FLAGS) -c $$< -o $$@
endef


$(foreach bdir,$(ASM_DIRS), $(eval $(call make-asm-goal,$(bdir))))
$(foreach bdir,$(C_DIRS), $(eval $(call make-c-goal,$(bdir))))
$(foreach bdir,$(CPP_DIRS), $(eval $(call make-cpp-goal,$(bdir))))

The BUILD_DIRS symbol contains the list of all the directories into which source code will be compiled. It is critical (though unsurprising) that the subdirectory structure of the build directory match the subdirectory structure of the source code.

ASM_DIRS, C_DIRS and CPP_DIRS specifies the list of directories with assembly, C and C++ source code. There must be agreement between these directories and those listed in BUILD_DIRS.

In the first bit of Make function magic, the ASM_SRC_DIRS, C_SRC_DIRS and CPP_SRC_DIRS are generated by adding the SRC_ROOT prefix to ASM_DIRS, C_DIRS and CPP_DIRS respectively.

ASM_SRC, C_SRC and CPP_SRC are then generated by looping over each entry in ASM_SRC_DIRS, C_SRC_DIRS and CPP_SRC_DIRS with ‘foreach‘ and then using the ‘wildcard‘ function to find all assembly, C and C++ source code in each of the directories in each of the source directories list respectively.

After all the source files are assembled above, all of the object files are generated in the OBJ list by using the ‘patsubst‘ function on each of the three source lists to replace the source code extensions ‘.S’, ‘.c’ and ‘.cpp’ with ‘.o’.

In the top section of the makefile, we have used a set of Make functions to transform a list of directories into a fully enumerated list of source files and object files. If a new subdirectory is added to the project, it simply needs to be added to the list of directories in the BUILD_DIRS list and the correct ASM_DIRS, C_DIRS and/or CPP_DIRS list depending on the compilation requirements.

Skipping ahead a bit, the variables make-asm-goal, make-c-goal and make-cpp-goal are defined using the ‘define‘ syntax which permits the variable to contain newlines. This is helpful for makefile snippets which will be passed to ‘eval‘.

Below those definitions, the is another set of ‘foreach‘ functions which then use ‘eval‘ and ‘call‘ to take the goals just defined and generate a fully enumerated set of goals for the ASM_DIRS, C_DIRS and CPP_DIRS directory lists.

What finally ties everything together is the $(ELF) target which has a dependency on all of the object files in the $(OBJ) list. This also pulls in the linker script as a separate target which has been fed to the C preprocessor. I had a handful of symbols that needed to be shared across the C/C++ source code and the linker script, so I chose to use the C preprocessor to include those symbols and expand them in the linker script. This is a pretty clean, elegant way to insure the symbols can be defined outside of the linker script and shared with other source code.

Double-Checking the Build

The echo target lists all of the directories, source files an object files to be built and demonstrates the expansion engine in action with the operation of the different functions:

$ make echo

Build Directories:       image build build/asm build/c build/c/utility build/c/platform build/c/platform/rpi3 build/c/platform/rpi4 build/c/devices build/c/devices/rpi3 build/c/devices/rpi4 build/c/isr build/c/filesystem build/c/services

ASM Source Directories:  src/asm

C Source Directories:    src/c src/c/utility src/c/platform src/c/platform/rpi3 src/c/platform/rpi4 src/c/devices src/c/devices/rpi3 src/c/devices/rpi4 src/c/isr src/c/filesystem src/c/services

CPP Source Directories:  src/c src/c/utility src/c/platform src/c/platform/rpi3 src/c/platform/rpi4 src/c/devices src/c/devices/rpi3 src/c/devices/rpi4 src/c/isr src/c/filesystem src/c/services

ASM Files:               src/asm/configure_gic.S src/asm/get_exception_level.S src/asm/global_variables.S src/asm/identify_board_type.S src/asm/isr_kernel_entry.S src/asm/park_core.S src/asm/setup_physical_timer.S src/asm/start.S src/asm/utility.S

C Files:  
              
CPP Files:               src/c/main.cpp src/c/utility/cppsupport.cpp src/c/utility/dump_diagnostics.cpp src/c/utility/memory.cpp src/c/utility/regex.cpp src/c/platform/exception_manager.cpp src/c/platform/kernel_command_line.cpp src/c/platform/platform.cpp src/c/platform/platform_info.cpp src/c/platform/rpi3/rpi3_platform_info.cpp src/c/platform/rpi4/rpi4_platform_info.cpp src/c/devices/block_io.cpp src/c/devices/emmc.cpp src/c/devices/log.cpp src/c/devices/mailbox.cpp src/c/devices/physical_timer.cpp src/c/devices/power_manager.cpp src/c/devices/sd_card.cpp src/c/devices/std_streams.cpp src/c/devices/system_timer.cpp src/c/devices/uart0.cpp src/c/devices/uart1.cpp src/c/devices/rpi3/rpi3_hw_rng.cpp src/c/devices/rpi4/rpi4_hw_rng.cpp src/c/isr/system_timer_reschedule_isr.cpp src/c/isr/task_switch_isr.cpp src/c/filesystem/fat32_blockio_adapter.cpp src/c/filesystem/fat32_directory_cluster.cpp src/c/filesystem/fat32_filesystem.cpp src/c/filesystem/filesystem_errors.cpp src/c/filesystem/filesystems.cpp src/c/filesystem/master_boot_record.cpp src/c/services/murmur_hash.cpp src/c/services/os_entity_registry.cpp src/c/services/random_number_generator.cpp src/c/services/uuid.cpp src/c/services/xoroshiro128plusplus.cpp

Object Files:            build/asm/configure_gic.o build/asm/get_exception_level.o build/asm/global_variables.o build/asm/identify_board_type.o build/asm/isr_kernel_entry.o build/asm/park_core.o build/asm/setup_physical_timer.o build/asm/start.o build/asm/utility.o build/c/main.o build/c/utility/cppsupport.o build/c/utility/dump_diagnostics.o build/c/utility/memory.o build/c/utility/regex.o build/c/platform/exception_manager.o build/c/platform/kernel_command_line.o build/c/platform/platform.o build/c/platform/platform_info.o build/c/platform/rpi3/rpi3_platform_info.o build/c/platform/rpi4/rpi4_platform_info.o build/c/devices/block_io.o build/c/devices/emmc.o build/c/devices/log.o build/c/devices/mailbox.o build/c/devices/physical_timer.o build/c/devices/power_manager.o build/c/devices/sd_card.o build/c/devices/std_streams.o build/c/devices/system_timer.o build/c/devices/uart0.o build/c/devices/uart1.o build/c/devices/rpi3/rpi3_hw_rng.o build/c/devices/rpi4/rpi4_hw_rng.o build/c/isr/system_timer_reschedule_isr.o build/c/isr/task_switch_isr.o build/c/filesystem/fat32_blockio_adapter.o build/c/filesystem/fat32_directory_cluster.o build/c/filesystem/fat32_filesystem.o build/c/filesystem/filesystem_errors.o build/c/filesystem/filesystems.o build/c/filesystem/master_boot_record.o build/c/services/murmur_hash.o build/c/services/os_entity_registry.o build/c/services/random_number_generator.o build/c/services/uuid.o build/c/services/xoroshiro128plusplus.o

This is where one can see all of the pieces of the makefile come together and meets my requirement for a simple debugging capability. 

Extending the build

As mentioned above, adding a new subdirectory is straightforward – all one must do is add the subdirectory to the list of directories in the BUILD_DIRS list and the correct ASM_DIRS, C_DIRS and/or CPP_DIRS list depending on the compilation requirements. Yes, it has to be put in two places – but that is because the BUILD_DIRS list specifies where the object file will be written whereas the ASM_DIRS, C_DIRS and CPP_DIRS lists specify which tool is used to assemble or compile the source code.

Beyond that, really you don’t need to understand the mechanics of the makefile. If you have a C++ only project, you can strip out the C and Assembly code processing, though for C++ only, CMake is probably a better choice. If you don’t want to use CMake, then the makefile skeleton above *should* meet just about any need.

RPI Bare Metal Project

The makefile above is part of the Raspberry Pi Bare Metal OS project I have been pursuing. This project can be found in Github at: https://github.com/stephanfr/RPIBareMetalOS.git

Using Packer To Build Development VMs

Leave a reply

One fundamental development practice is to have a bullet-proof, repeatable process for building, upgrading and maintaining development environments. My current development practice relies on developing inside a VM using Visual Studio Code’s Remote Development plugin. I treat the development VMs as ‘disposable’, i.e. at any moment in time I ought to be able to commit my work in progress to Github, destroy the VM, build a new one and pick up right where I left off. I should also be able to move seamlessly from one virtualized environment to another – for example, I should be able to develop in the Proxmox VM at home and a VirtualBox VM on my laptop when I travel without any friction between the two.

I use Hashicorp’s Packer tool to automate the VM build and configuration process. I usually maintain two target platforms: 1) a Proxmox server with an NFS backend I maintain at home and 2) VirtualBox which is installed on my laptops. Packer is declarative and supports multiple ‘builders’ for different backends. Both Proxmox and VirtualBox are supported and there is minimal difference in the builder specifications between the two.

Practical Packer

Packer and its builders and provisioners do most of the heavy lifting for you in terms of getting a ‘vanilla’ VM built. Perhaps the most complicated bit is figuring out the correct ‘boot command prefix’ to get past the bootloader. Honestly, getting a prefix that works involves a bit of trial and error and is somewhat cryptic. For the projects I have in Github, the prefix ‘works for me’ but if you are running on either a very fast or very slow machine, then your mileage may vary.

With a ‘vanilla’ VM in hand, the next step is to tailor it to your development needs. Packer is not inherently modular but I have managed to introduce some modularity by providing a collection of scripts that will be run inside the VM after it boots to customize the environment. The Packer specification will invoke these scripts which will either execute or return immediately based on environment variables set from Packer variables which can be set in a HCL file or set on the Packer command line.

The main challenge when executing the scripts is determining if you want the scripted commands to run as root, which is how the provisioner executes shell scripts, or as the ‘development user’ created early in the provisioning process. Essentially, the ‘development user’ is the username you will want to use when logging into the VM for development. The scripts will automatically create this user and assign a single-use password that will have to be changed on first login. The ‘change on login’ feature was not straightforward – so if you want a similar capability, just lift it from my code.

Github Projects

In the https://github.com/stephanfr/Packer repository you will find the Packer specifications for building Ubuntu development VMs in either a Proxmox or VirtualBox environment as well as a project which will allow you to build a VM which can then be used to build bootable, customized RPi images in QEMU. This is a very nice capability and is all due to the work of Mateusz Kaczanowski’s in his PackerBuilderArm Project.

One VM build option sets up an AArch64 bare-metal build environment inside the VM with a directory structure for my RPi Bare Metal OS project. In general though, if you are looking for an easy ARM toolchain setup – you can lift that code as well and modify it for your purposes.

Maintenance

Since I use these specs myself, I will track Ubuntu releases and tooling updates – but the timing is likely to be a bit erratic. I do not generally update tools immediately, I tend to value a stable environment over ‘latest and greatest’ but once a year or so I will update. Mostly updates *should* be limited to tweaking config values but sometimes, stuff just breaks. For example, I have not had success automating deployment of the very most recent Ubuntu VMs.

RPI Bare Metal Cross Compiling Toolchain

Leave a reply

Bare metal builds require a specific toolchain and compile options to insure that code normally generated for processes running in a standard OS environment is not generated and various standard libraries are not included.

Cross Compiling

I chose to use a cross-compiling approach for development. It should absolutely be possible to build the OS on a RPI itself with the correct toolchain but there are a lot of advantages that accrue from building on a larger x86_64 system with NFS, etc.

A variety of toolchains are available on the ARM developer website. At present they are all GCC based. Clang builds ought to work as well, though the make files will have to be modified. The toolchain I have been using is: AArch64 bare-metal target (aarch64-none-elf).

Libraries and Header Files

I have avoided using any libraries and the absolute bare minimum of header files to build the OS. This includes creating very minimal replacements for the C standard library, the standard IO library and the C++ standard library. There are a number of platform specific libraries that are required.

Catch2 for Unit Testing

Unit Tests for the project are built using Catch2. At present, unit tests using Catch2 are compiled for the host machine – not cross-compiled for the target platform. Running Catch2 also requires access to some headers and libraries for standard compilation on the host. This is messy – no doubt about it – and this approach will miss issues with data structure alignment, which matters a lot on AArch64 platforms without memory management enabled. Eventually, it will be possible to host the tests on the target platform but for now my focus is to get the code tested in the most straightforward fashion.

Source code coverage metrics are also available for the unit tests using a coverage target in the makefile.

Directory Structure

The make files (yep good old fashioned make) expect the following structure:

~/dev
        |--- RPIBareMetalOS
                |--- project cloned from github
        |--- gcc-cross
                |--- aarch64-none-elf
                         |--- cross compiling files
~/dev_tools
        |--- arm-gcc-toolchain
        |--- Catch2

There is a script in the root directory of the project named setup_dev_env.sh which will setup the development environment. If you have a vanilla Ubuntu (or probably any reasonable Debian based instance) you can simply execute that script and it will install the cross compiling toolchain, Catch2, QEMU for AArch64 and then create the correct directories and copy files to the right places.

Quick Build

There will be another post with greater detail on the build process, however the steps to build are straightforward. To get a running OS, simply do the following after cloning the project from Github:

cd RPIBareMetalOS/
cd minimalclib/
make all
cd ../minimalstdio/
make all
cd ../rpibaremetalos/
cd resources
unzip sd_compressed.zip
cd ..
make armstub_all
make all

The to run the OS in QEMU:

cd image/
qemu-system-aarch64 -M raspi3b -kernel kernel8.img -serial stdio -sd sd.img

Repeatable Scripted Setup

My development pattern is to create an Ubuntu VM customized for various projects and develop within it using the Remote Development Extension in Visual Studio Code. I run a Proxmox server at home but also use VirtualBox.

I have a separate project with Packer scripts to create Ubuntu development VMs in either of the two hypervisors listed above. Modifying the script for other hypervisors should be straightforward. In addition to the build process, there are a number of install scripts that are available to customize the VM after creation, one of which sets up an AArch64 bare metal toolchain and the directory structure I use for development. After building the VM with the right options, one should be able to simply clone the bare metal OS repository, build and run on QEMU.

The github repository for the Packer scripts is: https://github.com/stephanfr/Packer

An example command line to build a development VM template in Proxmox is:

packer build -var "dev_username=????" -var "dev_password=password" -var "proxmox_host=????" -var "proxmox_node_name=????" -var "proxmox_api_user=packer@pve" -var "proxmox_api_password=ubuntu" -var "ssh_username=packer" -var "ssh_password=password" -var "vmid=????" -var "http_interface=????" -var "install_aarch64_cross=true" -var-file="./22.04/ubuntu-22-04-version.pkrvars.hcl" -var-file="./proxmox/proxmox-config.pkrvars.hcl" -var-file="vm_personalization.pkrvars.hcl" ./proxmox/ubuntu-proxmox.pkr.hcl

After the template is created, clone it and then you will be good to go.

An example command line to build a development VM in VirtualBox appears below:

packer build -var "dev_username=????" -var "dev_password=password" -var "ssh_username=packer" -var "ssh_password=ubuntu" -var "install_aarch64_cross=true" -var-file="./22.04/ubuntu-22-04-version.pkrvars.hcl" -var-file="./virtualbox/virtualbox-config.pkrvars.hcl" -var-file="vm_personalization.pkrvars.hcl" ./virtualbox/ubuntu-virtualbox.pkr.hcl

Replace the ‘????’ sequences with appropriate values. At first login as the development user, you will be forced to change the user’s password.

More documentation can be found in the README file in the Packer script repository.

Building a Raspberry Pi 64 Bit Operating System with C++

Leave a reply

I have undertaken many different projects through the years, one area which I have not really explored is Operating System development. When I started developing software on 8 bit computers, the closest you came to an OS was a ‘monitor’ or perhaps a ‘Master Control Program’ for those old enough to remember Tron.

I have started tinkering with a 64 bit operating system for Raspberry Pi based computers. Given how powerful those small single board computers have become, they make a great platform for OS experimentation.

My goals for this project are four-fold:

  1. Get back to ‘bare metal basics’ for a while
  2. Provide a platform for experimentation with different approaches to OS architecture
  3. Explore the advantages and disadvantages of C++ for OS development
  4. Provide a collection of tutorials and working code for others to learn from

C++ for OS Development

There is a definite bias against C++ for bare metal programming, though increasingly there are bare metal projects utilizing C++. In the Raspberry Pi ecosystem, the Circle – C++ Bare Metal Environment for Raspberry Pi is perhaps the best and most useful example. It is a remarkable system.

Prior to C++ 11, I probably would not have considered this but now at C++ 20 and beyond the language system is both rich enough and flexible enough to span from bare metal up to the highest application layer development. At the time of writing, this project is built with C++ 20.

Part of my goal is to create a single image which runs across multiple RPI versions and makes obvious the points at which board specific code is required. My approach is to create interfaces using classes and abstract virtual functions which, yes adds a bit of overhead but anymore it is minimal. The optimizations available in modern compilers and increased clarity associated with C++ code may help close the performance gap between C and C++.

I am not particularly concerned about size at present. On systems with gigabytes of RAM, the difference between a 64k kernel and a 128k kernel is negligible. Honestly the kernel size is going to be much more tightly correlated to the OS architecture than the implementation language or optimizations. A monolithic kernel containing lots of services will be big whereas a microkernel with most services running in user space will be much smaller. These days though, I tend to favor speed over size.