Building Operating System from Assembly with Tumble Forth

Added internal links to article.

Modern operating system development is buried under toolchain bloat. If you want to write a simple kernel today, the standard advice is to configure a cross-compiler, write a linker script, set up GRUB, and compile hundreds of lines of C boilerplate just to get a screen cleared.

You can bypass this entire stack. By building a minimal Forth runtime directly in assembly-which we will call Tumble Forth-you get a self-hosting environment that runs on bare metal. From there, you can write a tiny, single-pass C compiler directly in Forth words. No GCC, no GNU binutils, and no linker scripts required. Just raw x86_64 assembly and a interactive terminal running over a serial port.

Here is how to build this system from the boot sector up.

The Boot Sequence

The PC boot process starts in 16-bit real mode. The BIOS loads the first 512-byte sector of our boot medium into physical memory address 0x7C00 and jumps to it. Our bootloader must transition the processor from 16-bit real mode, through 32-bit protected mode, and finally into 64-bit long mode.

First, we define the boot sector structure and initialize the segment registers.

[org 0x7c00]
[bits 16]
 
start:
    cli
    xor ax, ax
    mov ds, ax
    mov es, ax
    mov ss, ax
    mov sp, 0x7c00
 
    ; Clear screen
    mov ax, 0x0003
    int 0x10
 
    ; Check for long mode compatibility
    call check_cpuid
    call check_long_mode
 
    ; Set up page tables
    call setup_page_tables
 
    ; Enter long mode
    call switch_to_long_mode
 
    jmp $ ; Hang if we return

To enter 64-bit long mode, we must configure page tables. We will map the first 1 gigabyte of physical memory using 2-megabyte pages. This identity mapping allows our kernel to access physical memory addresses directly. We allocate space for our page tables at address 0x1000.

setup_page_tables:
    ; Zero out the page table space (4 pages = 16KB)
    mov edi, 0x1000
    mov cr3, edi
    xor eax, eax
    mov ecx, 4096
    rep stosd
 
    ; Re-initialize EDI to start of PML4
    mov edi, 0x1000
 
    ; PML4 points to PDPT
    ; Set present and writable flags (0x3)
    mov dword [edi], 0x2003
    
    ; PDPT points to Page Directory
    mov dword [edi + 0x1000], 0x3003
 
    ; Page Directory points to 2MB pages
    ; Set present, writable, and page size flags (0x83)
    mov edi, 0x3000
    mov eax, 0x00000083
    mov ecx, 512
 
.map_pages:
    mov [edi], eax
    add eax, 0x200000 ; Move to next 2MB page
    add edi, 8
    loop .map_pages
    ret

Now we define a minimal Global Descriptor Table (GDT) to describe the 64-bit code and data segments.

align 8
gdt64:
    dq 0 ; Null descriptor
.code: equ $ - gdt64
    dq (1<<43) | (1<<44) | (1<<47) | (1<<53) ; 64-bit Code segment
.data: equ $ - gdt64
    dq (1<<41) | (1<<44) | (1<<47) ; 64-bit Data segment
.pointer:
    dw $ - gdt64 - 1
    dq gdt64

With paging configured and the GDT defined, we enable Physical Address Extension (PAE), load the GDT, enable long mode in the Extended Feature Enable Register (EFER), and toggle paging on.

switch_to_long_mode:
    ; Enable PAE
    mov eax, cr4
    or eax, 1 << 5
    mov cr4, eax
 
    ; Load GDT
    lgdt [gdt64.pointer]
 
    ; Enable Long Mode in EFER MSR
    mov ecx, 0xC0000080
    rdmsr
    or eax, 1 << 8
    wrmsr
 
    ; Enable Paging and Protected Mode
    mov eax, cr0
    or eax, 0x80000001
    mov cr0, eax
 
    ; Jump to 64-bit code segment
    jmp gdt64.code:long_mode_entry
 
[bits 64]
long_mode_entry:
    mov ax, gdt64.data
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax
 
    ; Set stack pointers
    mov rsp, 0x90000  ; Return stack pointer
    mov r12, 0x80000  ; Data stack pointer
 
    ; Jump to Forth entry point
    jmp forth_cold

The Tumble Forth Runtime

Forth uses two stacks: a parameter (data) stack and a return stack. We map the hardware stack pointer RSP to the return stack, which handles function call return addresses. We map register R12 to the data stack pointer.

To optimize performance, we keep the top value of the data stack in register RAX. This register allocation scheme avoids memory access overhead for simple operations.

We use Direct Threaded Code (DTC). In DTC, a compiled word contains a list of addresses pointing directly to machine code or other compiled words. The instruction pointer RSI tracks our position in the compiled code.

The inner interpreter is a single assembly instruction that fetches the next address and jumps to it.

macro NEXT {
    lodsq
    jmp [rax]
}

Every primitive word ends by executing this macro. Let's write the core data stack manipulation words in x86_64 assembly.

; DUP ( x - x x )
forth_dup:
    dq .code
.code:
    sub r12, 8
    mov [r12], rax
    NEXT
 
; DROP ( x - )
forth_drop:
    dq .code
.code:
    mov rax, [r12]
    add r12, 8
    NEXT
 
; SWAP ( x y - y x )
forth_swap:
    dq .code
.code:
    mov rdx, [r12]
    mov [r12], rax
    mov rax, rdx
    NEXT
 
; OVER ( x y - x y x )
forth_over:
    dq .code
.code:
    mov rdx, [r12]
    sub r12, 8
    mov [r12], rax
    mov rax, rdx
    NEXT
 
; + ( n1 n2 - sum )
forth_add:
    dq .code
.code:
    add rax, [r12]
    add r12, 8
    NEXT

We also need memory access primitives to read and write physical addresses. In Forth, these are @ (fetch) and ! (store).

; @ ( addr - val )
forth_fetch:
    dq .code
.code:
    mov rax, [rax]
    NEXT
 
; ! ( val addr - )
forth_store:
    dq .code
.code:
    mov rdx, [r12]
    add r12, 8
    mov [rax], rdx
    mov rax, [r12]
    add r12, 8
    NEXT

Hardware Communication

To make the system interactive, we need serial port communication. We write primitives to read and write bytes using x86 in and out instructions. The standard serial port COM1 maps to port 0x3F8.

; KEY ( - char )
forth_key:
    dq .code
.code:
    sub r12, 8
    mov [r12], rax
.wait_rx:
    mov dx, 0x3FD ; Line Status Register
    in al, dx
    test al, 1
    jz .wait_rx
    mov dx, 0x3F8 ; Data Register
    in al, dx
    movzx rax, al
    NEXT
 
; EMIT ( char - )
forth_emit:
    dq .code
.code:
    mov rbx, rax
.wait_tx:
    mov dx, 0x3FD
    in al, dx
    test al, 0x20
    jz .wait_tx
    mov dx, 0x3F8
    mov al, bl
    out dx, al
    mov rax, [r12]
    add r12, 8
    NEXT

The Dictionary and Interpreter

A Forth word header consists of:

1. A link pointer to the previous word in the dictionary.
2. A length byte plus the name characters.
3. The code field pointer.

We define a macro in assembly to construct these headers.

%define link 0
 
%macro defword 3
    align 8
    %%link: dq link
    %define link %%link
    db %2
    db %1
    forth_%3:
%endmacro

We can now define our dictionary words in assembly, building up to the outer interpreter loop. The outer interpreter reads whitespace-delimited words from the serial port, searches the dictionary, and either executes them or parses them as numbers.

defword "KEY", 3, key
defword "EMIT", 4, emit
defword "+", 1, add
defword "DUP", 3, dup
defword "DROP", 4, drop

The lookup routine compares the input string against the dictionary links. If found, it returns the execution token (XT). If not, it parses the string as a base-10 or base-16 number.

; FIND ( addr len - xt | 0 )
forth_find:
    dq .code
.code:
    mov rcx, rax ; length
    mov rsi, [r12] ; string address
    mov rdx, link ; start at latest word
.loop:
    test rdx, rdx
    jz .not_found
    
    ; Compare length
    movzx rbx, byte [rdx + 8]
    cmp bl, cl
    jne .next_word
 
    ; Compare string bytes
    lea rdi, [rdx + 9]
    push rsi
    push rcx
    repe cmpsb
    pop rcx
    pop rsi
    je .found
 
.next_word:
    mov rdx, [rdx]
    jmp .loop
 
.found:
    pop rbx ; clean stack
    ; Calculate XT address (aligned to 8 bytes after name)
    add rdx, 9
    add rdx, rcx
    add rdx, 7
    and rdx, ~7
    mov rax, rdx
    NEXT
 
.not_found:
    add r12, 8
    xor rax, rax
    NEXT

With FIND and basic text parsing, we can write the main outer interpreter loop in Forth itself. We compile the loop into our dictionary.

Compiling C from Forth

Writing a full ANSI C compiler on bare metal is a massive task. However, we can write a compiler for a subset of C (let's call it "Sub-C") in about 200 lines of Forth.

Our C compiler will compile code on the fly. It reads C tokens and emits x86_64 machine code directly into the dictionary memory space. We will support:

• Local and global variables.
• Basic arithmetic (+, -, *, /).
• Control flow (if, else, while).
• Function declarations and calls.

Since Forth has direct access to the compiling pointer HERE (which tracks the current free memory location), we can write machine code bytes directly to memory. We define a helper word C, to write a byte and increment HERE.

: C, ( char - ) HERE C! 1 HERE +! ;
: , ( val - ) HERE ! 8 HERE +! ;

Parsing C Expressions

We parse expressions using a simple top-down parser. Because we want to emit code immediately, we use a single-pass design.

When the compiler encounters a number, it emits an x86 push instruction. When it encounters a variable (ideally named using variable naming best practices), it calculates the offset and emits a load instruction.

Let's define the machine code patterns we need:

\ Push constant to CPU stack: push imm32 (0x68 followed by 4 bytes)
: emit-push-imm ( n - )
    104 C, \ 0x68
    dup C, 8 rshift dup C, 8 rshift dup C, 8 rshift C, ;
 
\ Add top two elements on CPU stack: pop rax; add [rsp], rax
: emit-add ( - )
    90 C, \ pop rax (0x58)
    72 C, 1 C, 36 C, \ add [rsp], rax (0x01 0x04 0x24)
    ;

We can build a parser loop that reads tokens and matches operators.

: parse-expression ( - )
    next-token
    is-number? IF
        token-value emit-push-imm
    ELSE
        find-variable IF
            emit-load-variable
        THEN
    THEN
    next-token
    dup [char] + = IF
        parse-expression emit-add
    THEN
    ;

Control Flow and Patches

Control flow requires backpatching jump destinations. While we use standard conditional jumps here, modern processors often benefit from eliminating them entirely, a concept explored in branchless Rust optimization. When we compile a while loop, we record the start address. When we parse the closing brace }, we emit a jump instruction back to the start, and patch the conditional exit jump to point to the address after the loop.

We define a structure to track these patch locations on the Forth parameter stack.

: c-if ( - patch-addr )
    \ Emit comparison and jump-if-zero (JZ) with dummy offset
    131 C, 248 C, 0 C, \ cmp eax, 0
    15 C, 132 C, \ JZ (0x0F 0x84)
    HERE 0 , \ Save address for patch and emit 8-byte dummy
    ;
 
: c-else ( patch-addr1 - patch-addr2 )
    \ Emit unconditional jump (JMP)
    233 C, HERE 0 , \ JMP with dummy offset
    swap
    HERE swap ! \ Patch first jump to point here
    ;
 
: c-then ( patch-addr - )
    HERE swap ! \ Patch jump to point to current location
    ;

Parsing Function Declarations

When the compiler sees void name() { ... }, it adds name to the Forth dictionary. The definition of name will point to the current code generation address (HERE).

: parse-function ( - )
    expect-token void
    get-name-token
    create-dictionary-entry
    expect-token {
    BEGIN
        parse-statement
        current-token [char] } =
    UNTIL
    \ Emit ret instruction (0xC3)
    195 C,
    ;

Running the OS

To build the bootloader and kernel, we assemble the boot sector and the assembly primitives using nasm, then write them to a raw disk image.

nasm -f bin boot.asm -o boot.bin
nasm -f bin kernel.asm -o kernel.bin
cat boot.bin kernel.bin > os.img

We run the OS using QEMU, redirecting the virtual serial port to our host terminal.

qemu-system-x86_64 -drive format=raw,file=os.img -serial stdio

When the system boots, you receive a prompt over the serial line. You are now inside the Tumble Forth environment.

You can interactively define new hardware drivers:

: clear-screen ( - )
    0xB8000 ( VGA video memory )
    80 25 * 0 DO
        32 over I 2 * + C! ( Space character )
        15 over I 2 * + 1+ C! ( White text on black )
    LOOP
    drop ;

You can then load your C compiler code and compile C functions directly into the same memory space:

void hello() {
    print_string("Hello from compiled C code on bare metal!");
}

The transition from assembly to Forth, and from Forth to C, happens entirely in memory without a linker or loader. The system is live, customizable, and completely self-contained.