We spent years building web applications that are essentially thin wrappers around remote servers. If you wanted to run a language model in your app, you spun up a backend server, hid your API key behind an environment variable, and paid a provider for every single token generated. That model works, but it has clear limits. Your latency is tied to network speed, offline mode is impossible, and your server costs scale directly with your user base.
The browser is changing. We now have two distinct paths to run language models directly on user hardware: Chrome's built-in AI API (window.ai) and WebGPU-driven engines like WebLLM or Transformers.js.
Both approaches bypass the cloud. They run entirely on local silicon. But they do it in fundamentally different ways, with completely different performance characteristics and memory layouts.
The Native Contender: Chrome's Built-in AI
Chrome is trying to make AI a browser primitive. Instead of shipping a multi-gigabyte model down the wire to your users, Chrome downloads a version of Gemini Nano directly to the user's operating system. Once it is there, any web application running in Chrome can access it through a simple JavaScript API.
The code is minimal:
const session = await window.ai.createTextSession();
const result = await session.prompt("Write a short email subject line.");
console.log(result);You do not have to worry about downloading weights, managing GPU memory, or writing shader code. The browser handles execution, using the local GPU or NPU where available.
But this simplicity comes with constraints. The API is experimental and requires setting flags in Chrome. Google controls the model version, the system prompt, and the hardware acceleration backend. You cannot swap Gemini Nano for Llama 3 or Phi-3. You get what Google ships, and you have to accept how they quantize and run it.
The WebGPU Path: Raw Hardware Control
If you need control, you use WebGPU. WebGPU is the successor to WebGL, providing direct, low-level access to the GPU. Unlike WebGL, which forced developers to pretend their compute tasks were graphics rendering operations, WebGPU supports compute shaders natively.
Libraries like WebLLM and ONNX Runtime Web compile model weights into formats that run directly on WebGPU. They load models like Llama-3-8B-Instruct or Phi-3-mini into browser memory, compile WebGPU shaders written in WGSL, and run the matrix multiplications on the GPU.
This gives you total control over the model, the temperature, the system prompt, and the context window. But you pay a heavy price in initial download size and memory footprint.
The Performance Test Setup
To see how these two approaches stack up, we set up a benchmark. We tested two configurations:
- Chrome Native AI running Gemini Nano (approx 1.8B parameters, 4-bit quantized).
- WebLLM running Phi-3-mini (3.8B parameters, 4-bit quantized) and Llama-3-8B-Instruct (4-bit quantized) via WebGPU.
We ran these tests on two machines:
- A MacBook Pro with an M3 Max chip and 128GB of unified memory.
- A Windows 11 desktop with an Intel Core i9-13900K and an NVIDIA RTX 4080 (16GB VRAM).
We measured three metrics: Time to First Token (TTFT), Generation Speed (tokens per second), and Memory Overhead.
Time to First Token (TTFT)
TTFT measures how long the model takes to ingest the prompt and output the very first token. High TTFT makes an application feel sluggish.
Chrome's Native AI is fast here. On the M3 Max, the TTFT was consistently under 60 milliseconds. The model is already resident in system memory or quickly loaded by the OS daemon, so there is almost no initialization overhead.
WebGPU tells a different story. The first run requires compiling the WGSL shaders. On the RTX 4080, WebLLM took about 1.5 seconds to compile the shaders and process the initial prompt for Phi-3. Once cached, subsequent runs dropped to around 150 milliseconds. That is still slower than Chrome's native API, but it is acceptable for real-time applications.
If you run a larger model like Llama-3-8B over WebGPU, the TTFT climbs. On the M3 Max, Llama-3-8B averaged 320 milliseconds for the first token. The hardware has to move a massive amount of data into the GPU buffers before it can start processing the prompt.
Token Throughput
Once generation starts, raw throughput determines the user experience.
On the M3 Max, Chrome's Native AI (Gemini Nano) clocked in at roughly 22 tokens per second. On the Windows desktop with the RTX 4080, it managed 28 tokens per second. This is fast enough to read comfortably, but it is not blazing fast.
WebGPU with Phi-3-mini on the RTX 4080 reached 65 tokens per second. The unified memory architecture of the M3 Max allowed Phi-3-mini to run at 52 tokens per second.
Even Llama-3-8B, which is more than twice the size of Phi-3-mini, ran at 28 tokens per second on the M3 Max via WebGPU. On the RTX 4080, it hit 35 tokens per second.
When you target the GPU directly with optimized WGSL shaders, you can run larger, more capable models at speeds that match or beat the native browser API.
Memory Layouts and Bottlenecks
Running LLMs in a browser sandbox introduces unique memory challenges.
Chrome's Native AI runs in a separate process outside the utility sandbox of the specific tab. This means if the model crashes or runs out of memory, it does not take down your web page. The memory footprint of Gemini Nano is kept low (around 1.2GB to 1.5GB of system RAM) because Chrome shares the model instance across tabs.
WebGPU, however, allocates memory directly inside the GPU process of the browser. When you load Phi-3-mini, the browser allocates a continuous GPU buffer of about 2.2GB. For Llama-3-8B, that buffer jumps to 4.8GB.
Browsers have strict limits on how much memory a single WebGPU context can allocate. Chrome limits the maximum buffer size to a fraction of the system's VRAM. On an 8GB GPU, you might hit limit errors when trying to load a 4-bit 8B model. You have to request specific device limits during adapter initialization:
const adapter = await navigator.gpu.requestAdapter();
const device = await adapter.requestDevice({
requiredLimits: {
maxBufferSize: 6 * 1024 * 1024 * 1024, // 6GB
maxStorageBufferBindingSize: 6 * 1024 * 1024 * 1024
}
});If the user's hardware cannot meet these limits, the promise rejects, and your app fails to initialize.
Another bottleneck is garbage collection. JavaScript is single-threaded, and WebGPU requires constant communication between the CPU (managing the execution loop) and the GPU (running the matrix math). If your code creates temporary arrays for token IDs or attention masks in the render loop, the JavaScript garbage collector will trigger mid-generation. This causes micro-stutters where token delivery pauses for 50 to 100 milliseconds.
To avoid this, WebGPU engines use pre-allocated ring buffers. Instead of creating new arrays for every token, they overwrite existing memory locations.
The Storage Barrier: OPFS vs. OS-Level
The biggest hurdle for WebGPU is the initial download. Downloading a 2.2GB model file over a standard home connection takes time.
We use the Origin Private File System (OPFS) to cache these weights. OPFS is a private, highly optimized filesystem accessible only to your origin. It bypasses the standard Cache API limits, allowing you to store gigabytes of binary data without the browser silently deleting it when disk space runs low.
Once the weights are in OPFS, loading the model on subsequent visits takes less than a second. But the first-visit barrier is high. Most users will not wait for a 2GB download just to use a chat interface.
Chrome's Native AI solves this. The browser downloads the model in the background, independent of your website. When the user visits your page, the model is already there.
Development and Debugging
Debugging WebGPU is difficult. The browser GPU process runs in a sandbox, and errors often manifest as generic "Device Lost" exceptions. If you write a bad shader or misalign a memory buffer, the GPU driver might reset, causing Chrome to display a blank screen or a crash page.
Chrome's native API is simpler to debug, but it offers almost no visibility. You cannot inspect the attention weights, you cannot modify the token sampling logic, and you cannot view the raw log probabilities. You get a text input and a text output.
How to Choose
Use Chrome Native AI if:
- You need instant startup with zero download wait times.
- You are building simple text helpers like summarization, smart replies, or translation.
- You do not want to manage GPU memory limits or write fallback code.
- You are comfortable targeting only Chrome users for now.
Use WebGPU if:
- You need cross-browser support, as Firefox and Safari are actively shipping WebGPU.
- You need specific models like Llama-3 for coding or specialized fine-tunes.
- You require precise control over parameters, system prompts, or token generation settings.
- Your application is a dedicated tool where users expect a startup loading screen.
Client-side AI is moving fast. WebGPU is already capable of running highly optimized models at speeds that rival cloud APIs, without the hosting costs. As Chrome's native API stabilizes and other browsers adopt similar built-in models, the web will split. Small, utility tasks will run via native browser APIs, while complex, specialized tasks will pull down custom weights via WebGPU. The era of shipping every single token over a WebSocket is ending.
