Most search engines are built for eyes. They assume a human will read a title, scan a short snippet, and click a link. The entire web ecosystem, from search engine optimization to ad placement, serves this loop. But when an autonomous AI agent searches the web, this model breaks down. An agent does not click links, browse pages, or tolerate ads. It calls an API, expects structured data, and feeds the output directly into a context window.
This shift changes what we need from search infrastructure. Relevance is no longer about matching keywords to generate a clickable list. It is about extracting precise, structured information from a massive index with sub-second latency. Building a search API capable of serving agents across a 100-billion-page index requires rethinking the entire retrieval stack.
The 100-Billion-Page Index
Scaling a search index to 100 billion pages is a known infrastructure challenge. For human search, you build a massive inverted index, rank pages using link analysis, and serve cached results. For agentic search, the constraints change.
Agents need raw, clean text or structured data. Storing 100 billion pages of raw HTML is financially impractical and technically inefficient. A typical web page is 80% boilerplate-navigation bars, footer links, tracking scripts, and styling. Keenable strips this clutter at the ingestion phase. The raw HTML passes through a fast parser that outputs clean markdown and extracts structured metadata like tables, lists, and schema markup.
This reduces the storage footprint from petabytes to terabytes. But indexing it is still difficult. Standard vector databases cannot handle 100 billion vectors with low latency and reasonable costs. Running approximate nearest neighbor searches across a dense vector index of this size requires massive GPU clusters.
Instead, the architecture relies on a hybrid retrieval model. We use a distributed sparse index for the first pass, narrowing down billions of documents to a few thousand candidate pages. Only then do we apply dense vector reranking on the retrieved chunks. This hybrid approach keeps infrastructure costs manageable while maintaining semantic accuracy.
Structured Extraction Over Snippets
A human reads a search snippet to decide if a page is worth visiting. An agent uses search results to answer a question directly. If the search API returns a generic 150-character snippet, the agent has to make follow-up HTTP requests to fetch the full page content, parse it, and extract the answer. This multi-step process adds seconds of latency and consumes thousands of tokens.
Keenable replaces snippets with structured extraction. The API returns the actual content chunks containing the answer, formatted in markdown or JSON. If the query asks for a company's quarterly earnings table, the API does not return a link to the investor relations page with a text snippet. It returns the markdown representation of the table itself.
Achieving this requires pre-computing document structure during ingestion. The parser identifies document layouts, headers, and tabular data, saving them as distinct logical blocks. When a query hits the index, the retrieval engine returns these specific blocks rather than arbitrary text snippets.
For complex queries where pre-computed blocks fall short, the system uses a fast, local extraction model. This model runs at the edge, scanning the top retrieved documents to pull out key-value pairs or specific paragraphs before returning the payload to the agent.
Handling JavaScript-Heavy and Dynamic Pages
Crawling the modern web requires dealing with client-side rendering. A simple GET request often returns an empty shell of HTML with a bundle of JavaScript. For human search engines, rendering everything using headless browsers is expensive but necessary. For an agent-focused index, we have to be smarter.
Running headless browser instances for 100 billion pages is a massive waste of compute. Keenable uses a tiered rendering pipeline. The crawler first attempts to parse the raw HTML. If the page contains schema markup or pre-rendered text, we skip rendering. If the HTML is empty or contains signs of client-side frameworks without server-side rendering, the page is flagged.
Flagged pages are queued for a headless rendering worker. To keep costs down, these workers run a customized, headless browser engine optimized for speed rather than visual fidelity. We disable image loading, CSS rendering, and web fonts. We only execute the JavaScript necessary to populate the DOM text content. Once the DOM stabilizes, we extract the text and discard the browser state. This approach reduces the memory footprint of each render job, allowing us to handle dynamic pages at scale.
Index Sharding Strategy
A 100-billion-page index cannot fit on a single machine. The index must be split across thousands of servers. The way you shard this data directly impacts query latency.
In traditional search, you can shard by term or by document. Term sharding means one server holds all documents containing a specific word, while another holds all documents containing a different word. This makes single-word queries fast but multi-word queries slow, as servers must coordinate to intersect document lists.
Keenable uses document sharding. Each node holds a complete index of a subset of the 100 billion documents. When a query comes in, it is broadcast to all shards. Each shard searches its local index and returns its top matches. The coordinator node then merges and ranks these results.
Document sharding is highly parallel. As you add more shards, the search time per shard remains constant, allowing the system to scale horizontally. The challenge is the network overhead of broadcasting queries to thousands of nodes. We mitigate this by grouping shards into clusters based on page authority and topic. Queries are routed only to the clusters most likely to contain the answer, reducing the number of active nodes per query.
Latency: The Agent's Hard Constraint
When a human searches, a 500-millisecond delay is barely noticeable. For an agent executing a complex loop-where search is just one tool call among many-latency compounds. If an agent needs to make five sequential search queries to solve a problem, a 1-second delay per search means 5 seconds of idle time. The agent's overall execution budget is tight, and slow APIs cause timeouts.
To keep latency under 200 milliseconds across a 100-billion-page index, Keenable optimizes every step of the pipeline.
First, we use semantic caching. Agents often ask different variations of the same question. A semantic cache maps these queries to the same underlying search results, bypassing the index lookup entirely for cached hits.
Second, we parallelize retrieval and extraction. The query engine splits the search request across multiple index shards simultaneously. As soon as the first shard returns candidate documents, the extraction engine begins processing them, streaming the structured results back to the agent.
Third, the API supports token-budgeting. The agent can specify the maximum number of tokens it wants to receive. The search engine then optimizes the output, using a cross-encoder to select only the most relevant chunks that fit within that budget, cutting out fluff and saving downstream LLM costs.
Token-Budgeting and Context Compression
LLMs are sensitive to context length. Feeding irrelevant text into an agent's context window increases latency, raises API costs, and degrades the quality of the model's output. The search API must act as a filter, delivering only the high-value tokens.
When an agent calls the search API, it specifies a token limit, say T_limit = 1000 tokens. The search engine retrieves the top candidate chunks. These chunks are of varying lengths and relevance scores.
Instead of returning the top chunks until we hit the limit, we use a scoring algorithm to select the best combination. Each chunk has a relevance score S and a token length L. We want to maximize the total relevance score of the returned chunks while keeping the sum of their lengths under T_limit.
We use a greedy approximation. We sort the chunks by their density score-relevance divided by token length-and select the highest-scoring chunks until the token budget is full. We then apply a deduplication step to ensure we do not return redundant information from different sources, maximizing the diversity of the context provided to the agent.
The Ingestion Pipeline and Parser
The data ingestion pipeline must process millions of pages per second to keep the index fresh. The pipeline starts with a distributed crawler that prioritizes high-value domains based on update frequency and authority.
Once a page is fetched, it enters the processing queue. The parsing engine is written in Rust to ensure high throughput and low memory usage. It strips out JS, CSS, and interactive elements, leaving only the semantic content. The parser identifies semantic regions: headers, paragraphs, lists, and tables.
These regions are converted into clean markdown. Any structured data embedded in the page, such as microdata or JSON-LD, is extracted and stored in a metadata store. The cleaned markdown is then chunked using a semantic chunking algorithm. Unlike simple character-count chunking, semantic chunking splits documents at logical boundaries like headers or paragraph breaks to keep context intact.
Each chunk is indexed in the sparse search engine. A subset of metadata is generated for each chunk, including document authority, publication date, and language. This metadata allows agents to filter search results by date or source reliability directly in the API call.
Hybrid Retrieval Mechanics
Let us look at how the hybrid retrieval engine processes a query. When the agent sends a query, the API first normalizes the text and generates two representations: a set of keyword tokens and a dense vector embedding.
The keyword tokens query the distributed sparse index. This index is sharded across a cluster of memory-optimized nodes. The sparse search returns the top 1,000 document chunks based on BM25 scores.
Next, these 1,000 candidates are passed to a lightweight vector reranking model. This model compares the dense vector embedding of the query with the pre-computed embeddings of the candidate chunks. The reranker reorders the chunks, pushing the most semantically relevant ones to the top.
Finally, a cross-encoder model performs a final pass on the top 50 chunks. The cross-encoder evaluates the exact relationship between the query and the chunk text, scoring them for factual alignment. The top-scoring chunks are formatted into the requested output structure and sent back to the agent.
This multi-stage retrieval pipeline balances speed and accuracy. The sparse index handles the massive scale, the vector reranker adds semantic understanding, and the cross-encoder ensures precision.
API Design for Machines
Designing an API for AI agents requires a different philosophy than designing for human developers. Human APIs focus on readability and simple pagination. Agent APIs must focus on token efficiency, predictability, and machine readability.
The Keenable API allows agents to request specific schemas. For instance, an agent can pass a JSON schema in the request, and the API will attempt to map the search results to that schema before returning the response. This saves the agent from having to write its own parsing logic.
The API also includes detailed metadata about the source documents. It returns the crawl date, author information, domain authority, and a confidence score for the extraction. This allows the agent to make decisions about the reliability of the information, ignoring low-confidence sources or outdated pages.
Pagination is replaced by streaming. Instead of requesting page 1, page 2, and so on, the agent opens a stream. The search engine pushes results as they are retrieved and processed, allowing the agent to start generating its response before the search is fully complete.
Scalability and Maintenance
Maintaining a 100-billion-page index requires continuous updates. The web is dynamic; pages are added, modified, and deleted every second. We use a priority queue for crawling, where the recrawl rate is determined by the historical change frequency of the domain. News sites are recrawled every few minutes, while static documentation pages might be recrawled once a month.
To update the index without causing query downtime, we use a blue-green index deployment strategy. We build and update a shadow index in the background. Once the update is complete, we swap the query traffic to the new index. This swap happens at the routing layer, ensuring zero downtime for active agents.
The infrastructure is designed to fail gracefully. If a shard node goes down, the coordinator node routes the query to a replica shard. If the vector reranker experiences high load, the system automatically downgrades to pure sparse retrieval to maintain the latency SLA. The agent receives a slightly less accurate result but gets it within the required time window, preventing execution timeouts.
