Bare Metal Build System

Leave a reply

Normally, I prefer using CMake for C/C++ projects but for bare metal development, there are a number of requirements that are difficult to meet with CMake. Put simply, I could probably find a way to manage a bare metal build with CMake but at present, using good old Make seems preferable. One example of a bare metal challenge for CMake is specifying a Linker Script. There are ways to add a linker script as a dependency using CMake but they are generally pretty cryptic.

This post describes features of the GNU Make tool.

Primary Requirements for Bare Metal Build System

There are a handful of requirements I have for the build system:

  1. DRY – define compile and link settings once in one location
  2. Build subdirectories without a separate makefile in each subdirectory
  3. Support linker scripts naturally
  4. Simple debugging capability

Make is really just the combination of a rule execution engine with a text substitution and expansion engine. This combination is powerful but it is sometimes difficult to wrap one’s head around what is happening when make processes a makefile. In short, when making a build, make starts with the top level goal and then derives all the dependencies required to complete that goal and then satisfies those dependencies. For the case of an executable, the dependencies are compiling the source files and then linking the executable. Along the way, there is the potential for a lot of test expansion to generate the dependency names.

Make ‘include’

To satisfy the DRY (Do not Repeat Yourself) requirement, make has the ability to include other make files using the ‘include’ command. I have extracted the parameters needed across all the different projects in my bare metal system and put them into a ‘makefile.mk‘ file which is included at the top of each project’s makefile. For example, the shared symbols for the AArch64 bare metal cross compilation build appear below:

TOOLS := ${HOME}/dev_tools
GCC_CROSS_DIRECTORY := ${HOME}/dev/gcc-cross

GCC_VERSION := 12.3.1

GCC_CROSS_TOOLS_PATH := $(TOOLS)/arm-gnu-toolchain-12.3.rel1-x86_64-aarch64-none-elf/bin/
GCC_CROSS_INCLUDE := $(GCC_CROSS_DIRECTORY)/aarch64-none-elf

CC := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-gcc

LD := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-ld

AR := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-ar

OBJCOPY := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-objcopy

CPREPROCESSOR := $(GCC_CROSS_TOOLS_PATH)aarch64-none-elf-cpp

ASM_FLAGS := -Wall -O2 -ffreestanding -mcpu=cortex-a53 -mstrict-align

C_FLAGS := -Wall -O2 -ffreestanding -fno-stack-protector -nostdinc -nostdlib -nostartfiles -fno-exceptions -fno-unwind-tables -mcpu=cortex-a53 -mstrict-align

CPP_FLAGS := $(C_FLAGS) -std=c++20 -fno-rtti

LD_FLAGS := -nostartfiles -nodefaultlibs -nostdlib -static

INCLUDE_DIRS := -I$(GCC_CROSS_INCLUDE)/lib/gcc/aarch64-none-elf/$(GCC_VERSION)/include -I$(GCC_CROSS_INCLUDE)/lib/gcc/aarch64-none-elf/$(GCC_VERSION)/include-fixed -I$(GCC_CROSS_INCLUDE)/aarch64-none-elf/include

CATCH2_PATH := $(TOOLS)/Catch2

TEST_CC := gcc

TEST_LD := g++

TEST_CFLAGS := -Wall -O2

TEST_CPP_FLAGS := $(TEST_CFLAGS) -std=c++20

COVERAGE_CC := gcc

COVERAGE_LD := g++

COVERAGE_CFLAGS := -Wall -O0 -fprofile-arcs -ftest-coverage

COVERAGE_CPP_FLAGS := $(COVERAGE_CFLAGS) -std=c++20

This file is included in subsequent makefiles with:

include ../Makefile.mk

Build Subdirectories with Make Functions

Make has a powerful set of functions which operate on symbols and lists specified in the makefile. I have used six of these functions to allow my makefile to process subdirectories in a project. The functions and snippets of the GNU Make documentation for each appears below :

Key Functions

  • $(addprefix prefix,names…) – prepends each ‘name‘ with the specified ‘prefix‘
  • $(patsubst pattern,replacement,text) – Finds whitespace-separated words in ‘text’ that match ‘pattern’ and replaces them with ‘replacement‘. Here ‘pattern’ may contain a ‘%’ which acts as a wildcard, matching any number of any characters within a word. If ‘replacement’ also contains a ‘%’, the ‘%’ is replaced by the text that matched the ‘%’ in ‘pattern‘. Words that do not match the ‘pattern’ are kept without change in the output. Only the first ‘%’ in the ‘pattern’ and ‘replacement’ is treated this way; any subsequent ‘%’ is unchanged.
  • $(wildcard pattern…) – used anywhere in a makefile, is replaced by a space-separated list of names of existing files that match one of the given file name ‘patterns‘. If no existing file name matches a ‘pattern‘, then that ‘pattern’ is omitted from the output of the wildcard function. Note that this is different from how unmatched wildcards behave in rules, where they are used verbatim rather than ignored (see Pitfalls of Using Wildcards).
  • $(foreach var,list,text) – The first two arguments, ‘var’ and ‘list‘, are expanded before anything else is done; note that the last argument, ‘text‘, is not expanded at the same time. Then for each word of the expanded value of ‘list‘, the variable named by the expanded value of ‘var’ is set to that word, and ‘text’ is expanded. Presumably ‘text’ contains references to that variable, so its expansion will be different each time. The result is that ‘text’ is expanded as many times as there are whitespace-separated words in ‘list‘. The multiple expansions of ‘text’ are concatenated, with spaces between them, to make the result of ‘foreach‘.
  • $(call variable,param,param,…) – The ‘call’ function is unique in that it can be used to create new parameterized functions. You can write a complex expression as the value of a ‘variable‘, then use call to expand it with different values. When make expands this function, it assigns each ‘param’ to temporary variables $(1), $(2), etc. The variable $(0) will contain ‘variable‘. There is no maximum number of parameter arguments. There is no minimum, either, but it doesn’t make sense to use ‘call’ with no parameters.
  • $(eval param) – The ‘eval’ function is very special: it allows you to define new makefile constructs that are not constant; which are the result of evaluating other variables and functions. The argument to the ‘eval’ function is expanded, then the results of that expansion are parsed as makefile syntax. The expanded results can define new make variables, targets, implicit or explicit rules, etc. The result of the ‘eval’ function is always the empty string; thus, it can be placed virtually anywhere in a makefile without causing syntax errors. It’s important to realize that the ‘eval’ argument is expanded twice; first by the ‘eval’ function, then the results of that expansion are expanded again when they are parsed as makefile syntax. This means you may need to provide extra levels of escaping for “$” characters when using ‘eval’. The ‘value’ function (see The value Function) can sometimes be useful in these situations, to circumvent unwanted expansions.

With the exception of ‘addprefix‘ the other functions can be a bit complex. GNU Make documentation has more descriptive detail and examples. The key to the makefile operation is the use of these functions together with the text expansion engine to automatically generate dependencies from subdirectories.

Example Makefile

include ../Makefile.mk

SRC_ROOT := src
BUILD_ROOT := build
IMAGE_DIR   := image

BUILD_DIRS := $(IMAGE_DIR) $(BUILD_ROOT) $(BUILD_ROOT)/asm $(BUILD_ROOT)/c $(BUILD_ROOT)/c/utility $(BUILD_ROOT)/c/platform $(BUILD_ROOT)/c/platform/rpi3 $(BUILD_ROOT)/c/platform/rpi4 $(BUILD_ROOT)/c/devices $(BUILD_ROOT)/c/devices/rpi3 $(BUILD_ROOT)/c/devices/rpi4 $(BUILD_ROOT)/c/isr $(BUILD_ROOT)/c/filesystem $(BUILD_ROOT)/c/services

ASM_DIRS   := asm
C_DIRS     := c c/utility c/platform c/platform/rpi3 c/platform/rpi4 c/devices c/devices/rpi3 c/devices/rpi4 c/isr c/filesystem c/services
CPP_DIRS   := c c/utility c/platform c/platform/rpi3 c/platform/rpi4 c/devices c/devices/rpi3 c/devices/rpi4 c/isr c/filesystem c/services

ASM_SRC_DIRS := $(addprefix $(SRC_ROOT)/,$(ASM_DIRS))
C_SRC_DIRS   := $(addprefix $(SRC_ROOT)/,$(C_DIRS))
CPP_SRC_DIRS := $(addprefix $(SRC_ROOT)/,$(CPP_DIRS))

ELF := $(BUILD_ROOT)/kernel8.elf
IMG := $(IMAGE_DIR)/kernel8.img

ASM_SRC := $(foreach sdir,$(ASM_SRC_DIRS),$(wildcard $(sdir)/*.S))
C_SRC   := $(foreach sdir,$(C_SRC_DIRS),$(wildcard $(sdir)/*.c))
CPP_SRC := $(foreach sdir,$(CPP_SRC_DIRS),$(wildcard $(sdir)/*.cpp))

OBJ := $(patsubst src/asm/%.S,build/asm/%.o,$(ASM_SRC)) $(patsubst src/c/%.c,build/c/%.o,$(C_SRC)) $(patsubst src/c/%.cpp,build/c/%.o,$(CPP_SRC))

INCLUDE_DIRS += -Iinclude -I../minimalstdio/include -I../minimalclib/include -I../minimalstdlib/include
LDFLAGS += -L../minimalstdio/lib -L../minimalclib/lib
LDLIBS = -lminimalstdio -lminimalclib

LINKER_SCRIPT_TEMPLATE=link.template.ld
LINKER_SCRIPT=$(BUILD_ROOT)/link.ld


all: clean checkdirs $(IMG)


$(IMG): $(ELF)
	$(OBJCOPY) -O binary $(ELF) $(IMG)
	/bin/cp redistrib/*.* image/.
	/bin/cp armstub/image/armstub_minimal.bin image/.
	/bin/cp resources/*.txt image/.
	/bin/cp resources/sd.img image/.

$(ELF): $(OBJ) $(LINKER_SCRIPT)
	$(LD) $(LDFLAGS) $(OBJ) $(LDLIBS) -T $(LINKER_SCRIPT) -o $(ELF)


$(LINKER_SCRIPT): 
	$(CPREPROCESSOR) -Iinclude  $(LINKER_SCRIPT_TEMPLATE) -o $(LINKER_SCRIPT)

define make-asm-goal
$(BUILD_ROOT)/$1/%.o: $(SRC_ROOT)/$1/%.S
	$(CC) $(INCLUDE_DIRS) $(ASM_FLAGS) -c $$< -o $$@
endef

define make-c-goal
$(BUILD_ROOT)/$1/%.o: $(SRC_ROOT)/$1/%.c
	$(CC) $(INCLUDE_DIRS) $(C_FLAGS) -c $$< -o $$@
endef

define make-cpp-goal
$(BUILD_ROOT)/$1/%.o: $(SRC_ROOT)/$1/%.cpp
	$(CC) $(INCLUDE_DIRS) $(CPP_FLAGS) -c $$< -o $$@
endef


$(foreach bdir,$(ASM_DIRS), $(eval $(call make-asm-goal,$(bdir))))
$(foreach bdir,$(C_DIRS), $(eval $(call make-c-goal,$(bdir))))
$(foreach bdir,$(CPP_DIRS), $(eval $(call make-cpp-goal,$(bdir))))

The BUILD_DIRS symbol contains the list of all the directories into which source code will be compiled. It is critical (though unsurprising) that the subdirectory structure of the build directory match the subdirectory structure of the source code.

ASM_DIRS, C_DIRS and CPP_DIRS specifies the list of directories with assembly, C and C++ source code. There must be agreement between these directories and those listed in BUILD_DIRS.

In the first bit of Make function magic, the ASM_SRC_DIRS, C_SRC_DIRS and CPP_SRC_DIRS are generated by adding the SRC_ROOT prefix to ASM_DIRS, C_DIRS and CPP_DIRS respectively.

ASM_SRC, C_SRC and CPP_SRC are then generated by looping over each entry in ASM_SRC_DIRS, C_SRC_DIRS and CPP_SRC_DIRS with ‘foreach‘ and then using the ‘wildcard‘ function to find all assembly, C and C++ source code in each of the directories in each of the source directories list respectively.

After all the source files are assembled above, all of the object files are generated in the OBJ list by using the ‘patsubst‘ function on each of the three source lists to replace the source code extensions ‘.S’, ‘.c’ and ‘.cpp’ with ‘.o’.

In the top section of the makefile, we have used a set of Make functions to transform a list of directories into a fully enumerated list of source files and object files. If a new subdirectory is added to the project, it simply needs to be added to the list of directories in the BUILD_DIRS list and the correct ASM_DIRS, C_DIRS and/or CPP_DIRS list depending on the compilation requirements.

Skipping ahead a bit, the variables make-asm-goal, make-c-goal and make-cpp-goal are defined using the ‘define‘ syntax which permits the variable to contain newlines. This is helpful for makefile snippets which will be passed to ‘eval‘.

Below those definitions, the is another set of ‘foreach‘ functions which then use ‘eval‘ and ‘call‘ to take the goals just defined and generate a fully enumerated set of goals for the ASM_DIRS, C_DIRS and CPP_DIRS directory lists.

What finally ties everything together is the $(ELF) target which has a dependency on all of the object files in the $(OBJ) list. This also pulls in the linker script as a separate target which has been fed to the C preprocessor. I had a handful of symbols that needed to be shared across the C/C++ source code and the linker script, so I chose to use the C preprocessor to include those symbols and expand them in the linker script. This is a pretty clean, elegant way to insure the symbols can be defined outside of the linker script and shared with other source code.

Double-Checking the Build

The echo target lists all of the directories, source files an object files to be built and demonstrates the expansion engine in action with the operation of the different functions:

$ make echo

Build Directories:       image build build/asm build/c build/c/utility build/c/platform build/c/platform/rpi3 build/c/platform/rpi4 build/c/devices build/c/devices/rpi3 build/c/devices/rpi4 build/c/isr build/c/filesystem build/c/services

ASM Source Directories:  src/asm

C Source Directories:    src/c src/c/utility src/c/platform src/c/platform/rpi3 src/c/platform/rpi4 src/c/devices src/c/devices/rpi3 src/c/devices/rpi4 src/c/isr src/c/filesystem src/c/services

CPP Source Directories:  src/c src/c/utility src/c/platform src/c/platform/rpi3 src/c/platform/rpi4 src/c/devices src/c/devices/rpi3 src/c/devices/rpi4 src/c/isr src/c/filesystem src/c/services

ASM Files:               src/asm/configure_gic.S src/asm/get_exception_level.S src/asm/global_variables.S src/asm/identify_board_type.S src/asm/isr_kernel_entry.S src/asm/park_core.S src/asm/setup_physical_timer.S src/asm/start.S src/asm/utility.S

C Files:  
              
CPP Files:               src/c/main.cpp src/c/utility/cppsupport.cpp src/c/utility/dump_diagnostics.cpp src/c/utility/memory.cpp src/c/utility/regex.cpp src/c/platform/exception_manager.cpp src/c/platform/kernel_command_line.cpp src/c/platform/platform.cpp src/c/platform/platform_info.cpp src/c/platform/rpi3/rpi3_platform_info.cpp src/c/platform/rpi4/rpi4_platform_info.cpp src/c/devices/block_io.cpp src/c/devices/emmc.cpp src/c/devices/log.cpp src/c/devices/mailbox.cpp src/c/devices/physical_timer.cpp src/c/devices/power_manager.cpp src/c/devices/sd_card.cpp src/c/devices/std_streams.cpp src/c/devices/system_timer.cpp src/c/devices/uart0.cpp src/c/devices/uart1.cpp src/c/devices/rpi3/rpi3_hw_rng.cpp src/c/devices/rpi4/rpi4_hw_rng.cpp src/c/isr/system_timer_reschedule_isr.cpp src/c/isr/task_switch_isr.cpp src/c/filesystem/fat32_blockio_adapter.cpp src/c/filesystem/fat32_directory_cluster.cpp src/c/filesystem/fat32_filesystem.cpp src/c/filesystem/filesystem_errors.cpp src/c/filesystem/filesystems.cpp src/c/filesystem/master_boot_record.cpp src/c/services/murmur_hash.cpp src/c/services/os_entity_registry.cpp src/c/services/random_number_generator.cpp src/c/services/uuid.cpp src/c/services/xoroshiro128plusplus.cpp

Object Files:            build/asm/configure_gic.o build/asm/get_exception_level.o build/asm/global_variables.o build/asm/identify_board_type.o build/asm/isr_kernel_entry.o build/asm/park_core.o build/asm/setup_physical_timer.o build/asm/start.o build/asm/utility.o build/c/main.o build/c/utility/cppsupport.o build/c/utility/dump_diagnostics.o build/c/utility/memory.o build/c/utility/regex.o build/c/platform/exception_manager.o build/c/platform/kernel_command_line.o build/c/platform/platform.o build/c/platform/platform_info.o build/c/platform/rpi3/rpi3_platform_info.o build/c/platform/rpi4/rpi4_platform_info.o build/c/devices/block_io.o build/c/devices/emmc.o build/c/devices/log.o build/c/devices/mailbox.o build/c/devices/physical_timer.o build/c/devices/power_manager.o build/c/devices/sd_card.o build/c/devices/std_streams.o build/c/devices/system_timer.o build/c/devices/uart0.o build/c/devices/uart1.o build/c/devices/rpi3/rpi3_hw_rng.o build/c/devices/rpi4/rpi4_hw_rng.o build/c/isr/system_timer_reschedule_isr.o build/c/isr/task_switch_isr.o build/c/filesystem/fat32_blockio_adapter.o build/c/filesystem/fat32_directory_cluster.o build/c/filesystem/fat32_filesystem.o build/c/filesystem/filesystem_errors.o build/c/filesystem/filesystems.o build/c/filesystem/master_boot_record.o build/c/services/murmur_hash.o build/c/services/os_entity_registry.o build/c/services/random_number_generator.o build/c/services/uuid.o build/c/services/xoroshiro128plusplus.o

This is where one can see all of the pieces of the makefile come together and meets my requirement for a simple debugging capability. 

Extending the build

As mentioned above, adding a new subdirectory is straightforward – all one must do is add the subdirectory to the list of directories in the BUILD_DIRS list and the correct ASM_DIRS, C_DIRS and/or CPP_DIRS list depending on the compilation requirements. Yes, it has to be put in two places – but that is because the BUILD_DIRS list specifies where the object file will be written whereas the ASM_DIRS, C_DIRS and CPP_DIRS lists specify which tool is used to assemble or compile the source code.

Beyond that, really you don’t need to understand the mechanics of the makefile. If you have a C++ only project, you can strip out the C and Assembly code processing, though for C++ only, CMake is probably a better choice. If you don’t want to use CMake, then the makefile skeleton above *should* meet just about any need.

RPI Bare Metal Project

The makefile above is part of the Raspberry Pi Bare Metal OS project I have been pursuing. This project can be found in Github at: https://github.com/stephanfr/RPIBareMetalOS.git

Leave a Reply