Building AI features usually starts with a simple SDK call to OpenAI. You write a few lines of code, pass a prompt, and get a response. It works fine for a prototype. But when you scale to millions of requests a day, relying on a single model provider is a production risk. Outages happen. Rate limits hit without warning. Model performance drifts, and new, cheaper options launch weekly.
To survive at scale, software teams end up building their own abstraction layers. They write custom middleware to catch rate-limit errors, translate OpenAI's payload schema to Anthropic's format, and calculate token costs on the fly.
Ramp, a company that manages corporate spend and financial workflows, recently open-sourced their internal solution to this problem: a unified API router designed for dynamic LLM switching. This infrastructure handles provider fallbacks, cost-based routing, and schema translation at the network edge.
The API Abstraction Problem
Every LLM provider has its own API quirks. OpenAI uses messages with specific roles like system, user, and assistant. Anthropic's Claude historically preferred different structures, especially around system prompts and prefilling assistant responses. Google Gemini and Cohere have their own SDKs and payload shapes.
If you want to switch from gpt-4o to claude-3-5-sonnet because Claude handles a specific reasoning task better, you have to rewrite your integration code. If you want to run local AI models for simple classification tasks to save money, you need yet another code path.
This code drift creates technical debt. The unified API router solves this by acting as a reverse proxy. It exposes a single, OpenAI-compatible endpoint. Your application sends a standard request, and the router decides where to send it based on real-time rules.
Architecture of the Unified Router
The router sits between your application backend and the LLM providers. It performs three main tasks:
- Payload Translation: It accepts a standard request (usually following the OpenAI Chat Completions schema) and translates it into the target provider's format. When the provider responds, the router translates the response back to the standard format.
- Routing Decision Engine: It evaluates routing rules in real-time. These rules can be static (like "always send classification tasks to Llama-3") or active (like "if OpenAI latency exceeds 1500ms, route to Anthropic").
- Telemetry and Logging: It measures latency, token usage, and error rates across all providers, giving teams a single dashboard to monitor model performance.
Let's look at how the payload translation works. If your app sends a tool-calling request, the router must translate the JSON schema definitions. OpenAI defines tools under the tools parameter, using standard JSON Schema. Anthropic expects them in a slightly different format. The router parses the incoming JSON, restructures the tool declarations, executes the request against the Anthropic API, and reformats Anthropic's tool-use response into an OpenAI-compatible tool-call object.
The Mechanics of Streaming Translation
Streaming responses are a major pain point when using multiple LLM providers. When a user interacts with a chat interface, they expect to see tokens stream in real-time. The router must handle this streaming connection while translating the response on the fly.
LLM providers use Server-Sent Events (SSE) to stream tokens. However, the structure of these events varies. OpenAI sends chunks with a choices array containing a delta object. Anthropic sends events like content_block_delta and message_delta.
The router solves this by implementing a streaming parser for each provider. When a streaming request comes in, the router opens an SSE connection to the chosen provider. As chunks arrive, the router's parser extracts the text content or tool calls, wraps them in the standard OpenAI-compatible SSE format, and forwards them to the client. This translation happens with sub-millisecond overhead, ensuring the user experience remains fast.
Here is a simplified example of how the router maps an incoming stream from Anthropic into an OpenAI-compatible stream:
// Pseudocode for streaming translation middleware
function handleAnthropicStream(anthropicEventStream, clientResponseStream) {
for await (const chunk of anthropicEventStream) {
if (chunk.type === 'content_block_delta') {
const openAiChunk = {
id: `chatcmpl-${chunk.message_id}`,
object: 'chat.completion.chunk',
created: Math.floor(Date.now() / 1000),
model: 'claude-3-5-sonnet',
choices: [{
index: 0,
delta: { content: chunk.delta.text },
finish_reason: null
}]
};
clientResponseStream.write(`data: ${JSON.stringify(openAiChunk)}\n\n`);
}
}
}This translation logic runs inside the router's event loop, keeping memory usage low because it does not need to buffer the entire response before sending it to the client.
Fallback Logic and Error Budgets
Active routing based on performance metrics requires a stateful system. The router uses a configuration file to establish routing policies. Here is an example configuration that defines a fallback chain with latency thresholds:
routing_policies:
- path: "/v1/chat/completions"
default_model: "gpt-4o"
fallbacks:
- model: "claude-3-5-sonnet"
on_errors: [429, 500, 503]
latency_threshold_ms: 1200
- model: "llama-3.1-70b-instruct"
on_errors: [429, 500, 503]In this setup, the router first attempts to call gpt-4o. If the provider returns a rate limit error (429) or a server error (500 or 503), it immediately retries the request using claude-3-5-sonnet.
The router also tracks the rolling average latency of gpt-4o. If the latency exceeds 1200 milliseconds over the last ten requests, the router temporarily de-prioritizes OpenAI and routes incoming traffic to Claude, even if OpenAI is not throwing explicit errors. This prevents slow responses from degrading the user experience.
To prevent cascading failures, the router implements a circuit breaker pattern. If a provider fails repeatedly within a short window, the router trips the breaker and stops sending requests to that provider entirely for a cool-down period. This protects your application from hanging on dead connections while waiting for timeouts.
The Challenge of State and Context Windows
Switching models dynamically is not just a matter of changing the API endpoint. Different models have different context windows and token limits.
If your application sends a long chat history to gpt-4o (which supports a large context window), and the router falls back to a model with a smaller context window, the request will fail.
To handle this, the router performs active context management. It tokenizes the incoming messages list using the target model's tokenizer (like Tiktoken for OpenAI or LlamaTokenizer for open-source models). If the token count exceeds the target model's limit, the router applies a truncation strategy:
- System Prompt Preservation: The system prompt is always kept at the top of the message list.
- Sliding Window: The oldest messages in the chat history are discarded first until the payload fits within the limit.
- Summarization: Some configurations allow the router to call a cheap model to summarize the older conversation history before passing it to the fallback model.
Another issue is system prompt formatting. Some models perform poorly if the system prompt is too long or contains complex formatting. The router allows developers to define model-specific system prompts. If a request is routed to Claude, the router can swap the default system prompt for one optimized for Anthropic's XML-style prompting guidelines.
Tool Calling and Structured Outputs
Structured output (forcing the model to return JSON matching a specific schema) is essential for building reliable software agents. OpenAI supports response_format: { type: "json_schema", json_schema: ... } which guarantees schema compliance.
When routing dynamically, maintaining this guarantee is difficult. If the router switches to a provider that does not support JSON Schema enforcement natively, the output might be malformed JSON.
Ramp's router addresses this by injecting validation middleware. If the target provider does not support structured outputs natively, the router:
- Appends instructions to the prompt, demanding the JSON format.
- Validates the incoming response against the requested JSON Schema.
- If validation fails, it automatically retries the request (up to a configurable limit) or routes it to a secondary provider that does support native JSON schemas.
This validation step ensures that your application code can always assume the returned data is valid, regardless of which model actually generated the response.
Latency Overhead of the Router
Adding a proxy between your application and the LLM provider introduces network latency. If the router is hosted in a single AWS region (like us-east-1) and your application servers are in Europe, every LLM call incurs transatlantic latency twice.
To minimize this overhead, the unified router is designed to run on edge computing platforms like Cloudflare Workers or AWS CloudFront Functions.
Edge workers execute code in data centers close to the user. Because the router does not perform heavy computation (it only parses JSON, evaluates simple routing rules, and forwards HTTP requests), it can run within the strict CPU and memory limits of edge runtimes.
Running at the edge keeps routing latency under 10 milliseconds. The time saved by routing around slow or degraded providers far outweighs this minimal overhead.
Cost and Rate-Limit Management
LLM costs can quickly get out of hand. A single feature using a frontier model can cost thousands of dollars a day under heavy load. Often, you do not need a large model for every request.
For example, if a user asks a simple question like "What is my current balance?", a small model like gpt-4o-mini or claude-3-haiku can handle it just as well as a larger model, at a fraction of the cost.
The router allows for cost-based routing. You can configure the router to inspect the incoming prompt. If the prompt length is short and does not require complex reasoning, the router sends it to a cheaper model.
You can also set daily or monthly spend limits per API key. Once a limit is reached, the router automatically downgrades requests to cheaper open-source models or returns a rate-limit error to prevent cost overruns. Implementing a custom token ledger for LLM quota management helps track these consumption metrics.
Comparison with Existing Solutions
The idea of an LLM router is not new. The market is shifting rapidly, highlighted by how Stripe bought OpenRouter to control LLM API costs. Tools like LiteLLM, Portkey, and LangChain's routing integrations exist. Why did Ramp build their own?
Most existing tools are designed as software libraries (SDKs) rather than network infrastructure. If you use a Python library for routing, you are locked into Python. If you have services written in Go, Node.js, and Rust, you have to maintain duplicate routing logic in each language.
By building the router as a standalone network proxy, Ramp decoupled routing logic from the application stack. Any service that can make an HTTP request can use the router.
Furthermore, Ramp's router focuses on financial-grade reliability. It includes built-in rate-limiting queues, request queuing, and detailed audit logs to comply with strict data privacy regulations.
Getting Started with the Router
Deploying the router is straightforward. It can be run as a Docker container in your own VPC or deployed directly to Cloudflare Workers.
Once deployed, you update your application's SDK configuration to point to the router's URL and pass your routing API key.
Here is a basic example using the standard OpenAI Node.js SDK:
import OpenAI from 'openai';
const openai = new OpenAI({
apiKey: process.env.ROUTER_API_KEY,
baseURL: 'https://your-router-domain.com/v1',
});
async function main() {
const response = await openai.chat.completions.create({
model: 'auto-route-balanced', // The router maps this to the best provider
messages: [{ role: 'user', content: 'Extract invoice details from this text...' }],
});
console.log(response.choices[0].message.content);
}
main();By changing the baseURL and using a virtual model name like auto-route-balanced, the application code remains clean and independent of the underlying provider. The engineering team can update routing rules, add new models, or change fallback behaviors in the router's configuration file without redeploying the application code.
