Cloudflare doesn't usually come up when people talk about AI inference. They're the CDN company, right? The one that sits between your server and the internet, caching images and blocking DDoS attacks. But over the past year, they've quietly built out a GPU fleet that runs open-weight models like Moonshot AI's Kimi and Zhipu AI's GLM families across their global edge network. And the way they've done it reveals a lot about what production inference actually demands once you stop optimizing for benchmark leaderboards.
Why Run These Models at the Edge?
Most AI inference today happens in a handful of hyperscale data centers. You send a request to US-East-1 or some region in West Virginia, it hits a cluster of A100s or H100s, and 300 milliseconds later you get a response. That's fine for a chatbot where you're staring at a streaming response. It's not fine for autocomplete, content moderation, RAG pipelines embedded in page loads, or any feature where a model call is one step inside a larger latency budget of 200 milliseconds total.
Cloudflare's bet is straightforward: if you can run capable smaller models on GPUs distributed across 300+ cities, you cut the network hop from 80-150ms down to single digits. The actual compute time might be identical, but the round-trip savings add up fast. For an autocomplete feature hitting a model on every keystroke, that difference compounds across millions of requests per second.
Kimi and GLM models fit this use case well. They're not trying to be GPT-4 competitors. Moonshot's Kimi series focuses on long-context processing with relatively compact parameter counts. GLM, coming out of Tsinghua University's lab through Zhipu AI, has a strong track record in bilingual (Chinese-English) tasks and efficient architectures. These are models where the value isn't raw capability on MMLU benchmarks - it's the ability to run fast enough, cheap enough, close enough to the user.
The Inference Stack Nobody Talks About
Here's where things get interesting. Cloudflare didn't just rent some GPU boxes and call it a day. They've built a custom inference stack - internally called Workers AI - that handles model loading, batching, quantization, and request routing across their edge fleet.
The models themselves are quantized. Kimi and GLM checkpoints get compressed using INT4 and INT8 quantization before deployment. This trades off a small amount of accuracy for dramatically lower memory footprint and faster matrix operations. For production workloads where you're not running evals against academic benchmarks, that trade usually makes sense. A model that's 2% worse on a math benchmark but 3x faster in inference is the right call when you're serving autocomplete suggestions.
And batching is where the real throughput gains come from. Cloudflare's inference engine groups incoming requests into dynamic batches - multiple user queries processed simultaneously through the same model forward pass. The trick is doing this without adding latency to individual requests. If you wait too long to fill a batch, you've defeated the purpose. Their system uses a hybrid approach: short micro-batching windows of a few milliseconds combined with continuous batching that keeps the GPU busy as sequences finish at different rates.
KV cache management matters here too. These transformer models build up key-value caches as they generate tokens, and managing that cache across hundreds of concurrent requests on a single GPU is a non-trivial engineering problem. Cloudflare's stack handles cache eviction and memory pooling so that one long-running generation doesn't starve shorter requests from getting their turn on the GPU.
What "Production" Actually Means
There's a gap between "we can run this model" and "we can run this model for paying customers at 2 AM on a Tuesday during a traffic spike." Cloudflare's infrastructure closes that gap in a few ways.
First, cold start times. Loading a multi-billion parameter model into GPU memory isn't instant. Workers AI keeps warm model pools across their edge locations, so the first request doesn't pay a multi-second penalty while weights load from storage. Models are pre-loaded on GPU instances and stay resident as long as traffic warrants.
Second, routing. Not every edge location needs every model. Cloudflare routes inference requests to the nearest location that has the requested model loaded, balancing between latency and utilization. A Kimi-7B instance in Frankfurt might be saturated while Amsterdam has capacity, so requests get redirected. This sounds obvious, but it's the kind of infrastructure that only works when you already have global traffic management baked into your DNA - which Cloudflare does from their CDN business.
Third, observability. Production inference needs per-request latency breakdowns, queue depth monitoring, GPU utilization tracking, and error rate alerting. Cloudflare built this into their Workers AI platform so they can catch when a specific model version starts producing garbage output or when a particular GPU cluster develops thermal throttling issues.
The Bigger Picture
What Cloudflare's doing with Kimi and GLM points to a broader shift in how the industry thinks about model serving. The "throw a giant model at every problem" era is ending - or at least, it's not the only playbook anymore.
Running a 7B or 13B parameter model at the edge, quantized and batched, costs a fraction of what serving a 70B+ model from a centralized data center does. And for many production workloads - classification, extraction, short-form generation, semantic search - the smaller model does the job. Cloudflare's pricing for Workers AI reflects this: they charge per request at rates that make high-volume inference economically viable for mid-size applications.
The constraint has shifted from "can this model do the task?" to "can we serve this model fast enough and cheap enough to make the product work?" That's a systems engineering problem more than a machine learning problem, and it's exactly the kind of problem Cloudflare is built to solve.
There's also a less obvious play here. By hosting these models on their edge network, Cloudflare positions themselves as the inference layer between application developers and model providers. You don't need to set up your own GPU cluster or manage a SageMaker endpoint. You call an API, Cloudflare handles the rest. It's the same "let us sit in the middle" playbook that made their CDN business dominant, now applied to AI compute.
Whether this model wins long-term depends on how fast edge hardware evolves and whether the model families they're investing in (like Kimi and GLM) keep improving. But right now, the approach is working. The latency numbers are real, the cost structure makes sense for production workloads, and the engineering is solid. That's more than most AI infrastructure plays can claim.
