Fuzzing the Gleam Compiler to Find Type System Edge Cases

It features a strict, fast type system with no null values, no implicit conversions, and a compiler written in Rust. If you are transitioning from C++ or Java, understanding the Rust ownership and borrow checker is crucial for working with the compiler's codebase.

But compilers are just software. They contain bugs. A bug in a type checker can allow invalid programs to compile, causing runtime crashes on the BEAM, or crash the compiler itself with a panic.

To find these flaws, we can write unit tests. But humans write unit tests based on what they expect to go wrong. To find the weird, deeply nested edge cases, we need automated fuzz testing.

Why Traditional Fuzzing Fails on Compilers

If you run a standard byte-level fuzzer like AFL++ or libFuzzer on the Gleam compiler, you will mostly test the parser's error handling. The fuzzer will generate random bytes like this:

fn main() { @#$! }

The parser will immediately reject this with a clean error message. While testing parser resilience is useful, it does not reach the type checker. The type checker only runs after the parser successfully generates an Abstract Syntax Tree (AST).

To fuzz a type checker, we need to generate syntactically valid code that exercises type inference, generic unification, and pattern matching. We need grammar-based fuzzing or AST-level mutation.

The Gleam Compiler Pipeline

Understanding where to inject our fuzzer requires looking at how Gleam processes code. The compiler runs through several distinct stages:

1. Parsing: The parser takes Gleam source code and builds an AST.
2. Analysis: The type checker traverses the AST, infers types, and performs unification.
3. Code Generation: The compiler translates the typed AST into Erlang or JavaScript.

We can target two areas: the parser (via structured byte fuzzing) and the type checker (via AST generation).

Setting Up a Parser Fuzzer in Rust

Let's write a simple Rust fuzz target using cargo-fuzz. Since the Gleam compiler is written in Rust, we can import its internal crates directly. Writing efficient tooling in Rust is a common goal; for instance, see how Rust Glancer cuts language server memory overhead to run fast features on low-spec hardware. First, we add the dependency to our Cargo.toml:

[dependencies]
gleam-core = { git = "https://github.com/gleam-lang/gleam.git" }
arbitrary = { version = "1.0", features = ["derive"] }

Now we write the fuzz target to test the parser. We want to ensure that no input string can cause the parser to panic. It should either return a valid AST or a clean error.

#![no_main]
use libfuzzer_sys::fuzz_target;
use gleam_core::parser;
 
fuzz_target!(|data: &[u8]| {
    if let Ok(s) = std::str::from_utf8(data) {
        // We parse the string as a module
        let _ = parser::parse_module(s);
    }
});

This simple setup is great for finding parser crashes, such as stack overflows caused by deeply nested parentheses or integer overflow panics when parsing large numbers. These kinds of low-level failures are common in compiled languages; for example, debugging why musl binaries segfault on large searches reveals similar stack and memory limits.

Grammar-Based Fuzzing for the Type Checker

To reach the type checker, we need to generate valid ASTs. We can use the arbitrary crate to derive random structures that map to Gleam expressions. Let's define a simplified AST generator in Rust:

use arbitrary::Arbitrary;
 
#[derive(Arbitrary, Debug)]
enum GleamExpr {
    Int(i64),
    Float(f64),
    String(String),
    Var(String),
    Fn {
        args: Vec<String>,
        body: Box<GleamExpr>,
    },
    Call {
        fun: Box<GleamExpr>,
        args: Vec<GleamExpr>,
    },
    Let {
        name: String,
        value: Box<GleamExpr>,
        then: Box<GleamExpr>,
    },
}

We then write a printer that converts this AST back into a Gleam source code string.

fn to_source(expr: &GleamExpr) -> String {
    match expr {
        GleamExpr::Int(n) => n.to_string(),
        GleamExpr::Float(f) => f.to_string(),
        GleamExpr::String(s) => format!("\"{}\"", s.replace("\"", "\\\"")),
        GleamExpr::Var(v) => sanitize_identifier(v),
        GleamExpr::Fn { args, body } => {
            let args_str = args.iter()
                .map(|a| sanitize_identifier(a))
                .collect::<Vec<_>>()
                .join(", ");
            format!("fn({}) {{ {} }}", args_str, to_source(body))
        }
        GleamExpr::Call { fun, args } => {
            let args_str = args.iter()
                .map(|a| to_source(a))
                .collect::<Vec<_>>()
                .join(", ");
            format!("{}({})", to_source(fun), args_str)
        }
        GleamExpr::Let { name, value, then } => {
            format!(
                "let {} = {}\n{}",
                sanitize_identifier(name),
                to_source(value),
                to_source(then)
            )
        }
    }
}
 
fn sanitize_identifier(s: &str) -> String {
    let mut clean: String = s.chars()
        .filter(|c| c.is_ascii_lowercase())
        .collect();
    if clean.is_empty() {
        clean = "x".to_string();
    }
    clean
}

This generator guarantees that the output code is syntactically valid. When we feed this code to the compiler, we bypass the parser error checks and hit the type checker directly.

Tracking Down Type Inference Loops

One common class of bugs in type checkers is infinite recursion during type inference. This happens when the type checker tries to resolve a self-referential generic constraint and runs out of stack space.

Consider this Gleam code generated by a fuzzer:

pub fn loop(x) {
  loop(fn() { x })
}

What is the type of x?

The type checker infers that x is passed to a function that expects a function returning x. This creates an infinite type: a = fn() -> a.

If the type checker does not implement occurs-checking properly, it will loop forever trying to expand this type. An occurs-check detects if a type variable appears within the type it is being unified with, preventing recursive types of infinite size.

By running our fuzzer, we can monitor the compiler process. If a compilation run takes longer than 2 seconds, we flag it as a potential hang and save the input.

Differential Testing and Code Generation Discrepancies

A compiler might not crash, but it might generate incorrect code. This is where differential testing comes in.

For Gleam, we can compile the same code to both Erlang and JavaScript, run both outputs, and compare the results. If the Erlang program returns 1 but the JavaScript program returns undefined, we have found a compiler bug.

Let's look at how we can automate this pipeline:

1. Generate a random Gleam function that returns a primitive value.
2. Compile the function to Erlang.
3. Compile the function to JavaScript.
4. Run the Erlang code using escript.
5. Run the JavaScript code using node.
6. Assert that the outputs are identical.

Here is a bash script that handles this execution loop:

#!/usr/bin/env bash
set -euo pipefail
 
# Run the generator to create target.gleam
cargo run -bin generator > src/target.gleam
 
# Compile to Erlang
gleam build -target=erlang
 
# Compile to JS
gleam build -target=javascript
 
# Run Erlang and capture output
ERL_OUT=$(escript ./build/dev/erlang/target.escript)
 
# Run JS and capture output
JS_OUT=$(node ./build/dev/javascript/target.js)
 
if [ "`ERL_OUT" != "`JS_OUT" ]; then
    echo "Mismatch found!"
    echo "Erlang: $ERL_OUT"
    echo "JS: $JS_OUT"
    exit 1
fi

This approach is highly effective at finding bugs in how pattern matching is compiled. Pattern matching in functional languages is compiled down to nested decision trees. If the compiler simplifies these trees incorrectly, it might execute the wrong branch at runtime.

Triage and Minimization (Shrinking)

Fuzzers are great at finding bugs, but the inputs they generate are usually messy. If the fuzzer finds a crash, it might output a file with 500 lines of random nested functions. Debugging this is tedious.

We need to shrink the input to the smallest possible snippet that still reproduces the bug.

We can write a simple test runner script that returns 0 if the compiler runs successfully, and 1 if it crashes. We then use a tool like creduce or a custom AST shrinker to minimize the code.

For example, if the fuzzer generated this:

pub fn main() {
  let a = 1
  let b = fn(x) { x }
  let c = b(a)
  let d = case True {
    True -> fn(y) { y }
    False -> fn(y) { todo }
  }
  d(c)
}

A smart shrinker will try removing variables one by one. It will verify if the compiler still crashes after each removal. Eventually, it will reduce the code to the absolute minimum:

pub fn main() {
  let d = case True {
    True -> fn(y) { y }
    False -> fn(y) { todo }
  }
  d(1)
}

This minimal example makes it obvious to the compiler maintainers where the type unification logic broke down.

Setting Up Continuous Fuzzing Pipelines

Fuzzing should not be a one-off task before a release. It works best when integrated into your continuous integration (CI) pipeline.

Because fuzzing is resource-heavy, running it on every pull request can be expensive. A better pattern is to run a fuzzing job nightly. You can set up a GitHub Action that runs the fuzzer for a few hours every night and files an issue if it finds a crash.

Here is an example GitHub Actions configuration:

name: Nightly Fuzzing
 
on:
  schedule:
    - cron: '0 0 * * *'
 
jobs:
  fuzz:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Install Rust
        uses: dtolnay/rust-toolchain@stable
        
      - name: Install cargo-fuzz
        run: cargo install cargo-fuzz
        
      - name: Run Fuzzer
        run: cargo fuzz run fuzz_parser - -max_total_time=10800
        
      - name: Archive Crashes
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: crashes
          path: fuzz/artifacts/

This configuration runs the parser fuzzer for three hours (10,800 seconds). If it encounters a panic, the step fails, and the crash artifacts are uploaded for inspection the next morning.

Building a compiler that developers can trust requires aggressive testing. Unit tests cover the paths we know; fuzzing covers the paths we cannot predict. By generating random ASTs and verifying the compiler's behavior across different compilation targets, we can eliminate type system bugs before they ever reach production.