Most machine learning papers are marketing documents. You read a paper claiming a five percent improvement in training efficiency or a breakthrough in sparse attention. You clone the repository, try to run the code, and immediately hit a wall of compiler errors, broken CUDA bindings, or hardcoded paths pointing to a directory that only exists on the author's local workstation.
It often takes a systems engineer days or weeks to get a single training script running. By the time they do, they often discover the claimed results only hold under highly specific, undocumented conditions. This is the replication crisis in artificial intelligence. As the volume of published research grows, the gap between what is claimed in PDFs and what runs in code is widening.
Inherent recently released Faraday, an AI agent designed to automate research replication. Faraday takes a paper PDF, finds the source code, builds the environment, executes the training or evaluation loop, and verifies the metrics. It does this by combining language models with sandboxed runtimes and developer tools.
Replicating a paper is rarely a conceptual problem. It is a systems engineering problem. The barrier is not understanding the math behind a new loss function; it is getting a legacy version of PyTorch to compile alongside a specific driver version on modern GPU hardware.
Faraday treats replication as a debugging loop. It begins in an isolated sandbox, a containerized environment with access to GPU resources, package managers, and system-level compilers.
When Faraday starts a job, it parses the paper to extract the core claims. These claims might be a specific accuracy score on a benchmark or a reduction in training time. Faraday translates these claims into target metrics.
Next, it clones the repository. If no repository is linked, it attempts to write the implementation from the pseudocode and descriptions in the paper, though its success rate drops without a starting codebase.
The first test is building the environment. Faraday uses a resolver that scans files like requirements.txt, pyproject.toml, or setup.py.
Python's packaging ecosystem is notoriously fragile. A library might compile on a developer's machine because of a pre-existing system library, but fail on a clean container. Faraday addresses this by building a dependency graph from source files when metadata files like setup.py are missing. It parses import statements across the entire codebase to build a list of required packages.
When a package installation fails, Faraday reads the compiler errors. If it detects a missing header file, it searches its database of system packages to find which library provides that header. It then installs the dependency using the system package manager before retrying the Python installation.
If dependencies conflict, Faraday does not stop. It runs an iterative resolver. It attempts to install the packages, parses the error logs, and modifies the dependency tree. If a package requires a specific C++ compiler version, Faraday writes a Nix expression to provision the correct system libraries.
For example, if a paper from three years ago relies on a version of deepspeed that fails to compile on modern compiler versions, Faraday identifies the compiler mismatch from the build logs, downgrades the compiler in its sandbox, and retries the build. This loop runs until the package compiles.
Once the environment compiles, Faraday needs to run the code. Research repositories often have undocumented command-line arguments, missing preprocessing scripts, and hardcoded data paths.
Faraday runs static analysis on the codebase to build an execution graph. It identifies the entry points, usually scripts named train.py or main.py, and maps their arguments. It inspects the code to find where datasets are loaded. If it finds a path pointing to a local directory, it attempts to resolve the dataset.
If the dataset is public, Faraday downloads it using APIs from Hugging Face or direct URLs found in the paper. If the dataset is private, Faraday generates synthetic data matching the expected schema. It does this by analyzing the data loading pipeline in the code to see what tensor shapes and data types the model expects.
To track metrics without breaking the code, Faraday uses Python's ast module. Instead of manually editing strings, which can introduce syntax errors, the agent parses the source code into an Abstract Syntax Tree. It locates the training loop by searching for loops containing optimizer steps. Once found, it inserts new AST nodes that capture the loss tensor, detach it from the computation graph, convert it to a CPU float, and log it to a local database. This structural modification ensures that the logging code does not interfere with the model's backpropagation or memory allocation.
When execution starts, it usually crashes. Out-of-memory errors, shape mismatches, runtime type errors, and missing configuration files are common.
Faraday runs a continuous feedback loop. When a run fails, the agent captures the output and the stack trace. It feeds this context, along with the relevant code snippets, to its reasoning model. The agent then writes a patch, applies it using git, and restarts the run.
A PyTorch error might originate in a custom CUDA kernel but surface as a generic runtime error in a high-level training script. Faraday uses a backtrace parser to isolate the exact line of code that triggered the failure. It then inspects the variable state at that frame if a debugger is active, or inserts print statements to log the shapes of all tensors immediately preceding the crash. Once it identifies the mismatch, such as a batch size mismatch during a linear projection, it writes a patch. The patch is tested in isolation before being merged into the main execution branch.
If Faraday encounters a CUDA out-of-memory error, reducing the batch size might alter the optimization dynamics. Instead, Faraday checks if it can use gradient accumulation. It inspects the training loop, locates the optimizer step, and modifies the code to accumulate gradients over multiple steps while dividing the batch size. This preserves the effective batch size while fitting the model into the available GPU memory.
If the error is a tensor shape mismatch during a forward pass, Faraday uses its static analysis tool to trace the tensor shapes from the input layer to the point of failure. It then inserts a reshape operation or fixes the underlying layer initialization.
Once the code runs to completion, Faraday compares the output metrics against the targets extracted from the paper.
Faraday evaluates the replication on a spectrum. A full replication means the metrics match the paper's claims within an acceptable statistical margin. A partial replication means the code runs, but the metrics fall short of the claims. A failed replication means the code runs but produces random results, or the environment cannot be resolved.
Faraday generates a replication report. This report includes the Nix or Docker configuration, the git patch containing all modifications made to the codebase, the execution logs, and a comparison of the original claims versus the replicated results.
Faraday is a multi-agent system built on top of a shared workspace. Coordinating these systems requires managing agent swarms and model economics to control context overhead.
A Lead Researcher agent plans the replication strategy and extracts claims. An Environment Engineer agent manages the sandbox, installs dependencies, and resolves compiler errors. A Code Debugger agent writes patches and alters the code. A Verification agent analyzes the output logs and drafts the final report.
These agents communicate through a state machine. The state machine prevents the system from getting stuck in loops. If the Environment Engineer fails three times with the same error, the Lead Researcher intervenes to change the strategy, perhaps by switching from a package manager to building the dependency from source.
The agents share a workspace containing the codebase, the execution logs, and the patch history. The Lead Researcher coordinates the process by writing a plan to a shared markdown file. The Environment Engineer reads the plan, attempts to build the environment, and updates the workspace state with the build log. If the build fails, the Environment Engineer logs the error and yields control back to the Lead Researcher. This structured turn-taking prevents agents from overwriting each other's changes. The memory model uses a local database to index execution logs, drawing on techniques used to evaluate vector stores and context systems for agent recall, allowing the Code Debugger to retrieve historical error resolutions that match the current failure pattern.
The underlying models are fine-tuned specifically for code generation, systems debugging, and scientific reading. Inherent trained these models on GitHub commits, compiler error logs, and open-access scientific papers.
In benchmark tests conducted by Inherent, Faraday was tasked with replicating 100 machine learning papers published between 2022 and 2025. These papers covered architectures including large language models, diffusion models, and reinforcement learning systems.
Faraday replicated the primary claims of 43 papers without human intervention. For 31 papers, it achieved partial replication, identifying discrepancies in the claimed training speeds or minor drops in accuracy. For the remaining 26 papers, replication failed. The failures were due to unavailable datasets, missing code components, or hardware requirements exceeding the benchmark allocation.
Faraday identified several papers where the claimed results were due to data leakage or specific, non-generalizable hyperparameter tuning. In one case, a paper claiming high performance on a tabular dataset had hardcoded the test set indices into the preprocessing pipeline. Faraday uncovered this by analyzing the git repository's history and the data loader code.
The current peer review process is broken. Reviewers rarely run the code of the papers they evaluate. They do not have the time or the hardware to debug a stranger's codebase. They rely on the text of the paper and the reputation of the authors.
Faraday changes this. A future where submitting a paper to a conference requires submitting a Faraday replication token is possible. The conference runs the agent on its own hardware, verifies the claims, and attaches a replication badge to the paper before it reaches human reviewers.
This would reduce the amount of unreplicable science published every year. It would also force researchers to write cleaner code and document their dependencies. If they know an AI agent will try to run their code in a clean sandbox, they will think twice before pushing a broken repository.
There is a question of trust here. Can we trust an AI agent to verify the work of other AI systems? Studies show humans miss security threats when approving AI commands, suggesting manual oversight is also flawed.
If Faraday modifies the code to make it run, does it alter the original experiment? The line between debugging and altering the methodology is thin. If Faraday changes a learning rate or switches an optimizer to get a script to compile, it might be evaluating a different system than the one the authors intended.
Inherent addresses this by keeping the patches minimal and transparent. Every change is logged, and the agent must justify why a change was necessary. If a patch alters the core model architecture or the optimization algorithm, the agent flags the replication as modified rather than clean.
Automated replication is not about replacing human verification. It handles the tedious task of environment setup and basic debugging, allowing human researchers to focus on the conceptual validity of the work. Faraday is a step toward a more rigorous scientific process. In an industry dominated by rapid releases, tools that enforce empirical truth are necessary.
Tags: ai-agents, llm, ai-engineering, machine-learning
