Anyone who has run a large language model in production knows the feeling. You launch a new feature, your users love it, and then the first hosting bill arrives. The numbers can make you dizzy.
Your immediate instinct might be to look for smaller models, or perhaps run a local model like Meta Muse Glimmer on your own hardware. But training models takes time, clean data, and hardware you might not have right now.
There is a different way to tackle this. Instead of changing the model weights, you change how you handle the tokens. This is what we call post-training token harnessing. It is a set of techniques designed to prune, compress, and route tokens before, during, and after inference. You keep your high-quality model while slashing your compute bill.
The Economics of the Inference Wall
To understand why token harnessing works, we have to look at how LLMs process data. LLM billing is split into input tokens and output tokens. Input tokens are cheap to process because the GPU can handle them in parallel. Output tokens are expensive because the model must generate them one by one.
Every time the model generates a token, it must read the entire model weight matrix from High Bandwidth Memory to the GPU SRAM. This process is memory-bandwidth bound. The GPU spends most of its time waiting for data to arrive from memory, not actually performing calculations.
When you send a prompt with ten thousand tokens of context, you force the model to load a massive amount of data for every single word it generates. This overhead is a primary driver of rising context window costs. If you can reduce the number of tokens the model has to look at, you directly reduce the time the GPU spends waiting on memory.
The KV Cache Memory Tax
During generation, the model calculates key-value states for every token in the sequence. It stores these states in the KV cache so it does not have to recalculate them for every new token.
The size of this cache grows with your sequence length and batch size. We can calculate the memory footprint of the KV cache using a simple formula:
Memory_bytes = 2 * layers * heads * head_dim * precision * sequence_length * batch_size
For a model like LLaMA-3-70B running at 16-bit precision, a single user session with a 4096-token context window can easily consume several gigabytes of GPU memory just for the KV cache. While post-training quantization can reduce the memory footprint of the weights themselves, the KV cache remains a major bottleneck.
When your GPU runs out of memory, you cannot run large batches. Your throughput drops, and your cost per request goes up because you are running your hardware under capacity. Reducing the active tokens in the KV cache is the key to unlocking higher batch sizes and lower costs.
Token Pruning at the Gate
We often write prompts filled with boilerplate instructions. We add system prompts, few-shot examples, and strict formatting rules. The model needs some of this context, but it rarely needs all of it.
Prompt compression looks at the informational density of your prompt. A smaller, faster model like GPT-2 or a tiny LLaMA variant evaluates the prompt first. It calculates the perplexity of each token in the context of the prompt.
Tokens with low perplexity are highly predictable. They do not add much information to the prompt. If you remove the words that the model can easily guess, you can shrink a prompt by 30% to 50% without hurting the final output quality.
Here is a simplified Python representation of how you might implement a prompt compressor:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
class PromptCompressor:
def __init__(self, model_name="gpt2"):
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModelForCausalLM.from_pretrained(model_name)
def compress(self, prompt, target_ratio=0.6):
inputs = self.tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
outputs = self.model(**inputs, labels=inputs["input_ids"])
loss = outputs.loss
logits = outputs.logits
# Calculate perplexity per token
shift_logits = logits[..., :-1, :].contiguous()
shift_labels = inputs["input_ids"][..., 1:].contiguous()
loss_fct = torch.nn.CrossEntropyLoss(reduction="none")
token_losses = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
# Sort tokens by their contribution to information density
threshold = torch.quantile(token_losses, 1 - target_ratio)
keep_indices = torch.where(token_losses > threshold)[0]
# Reconstruct the compressed prompt
input_ids = inputs["input_ids"][0]
compressed_ids = [input_ids[0].item()] # Keep the first token
for idx in keep_indices:
compressed_ids.append(input_ids[idx + 1].item())
return self.tokenizer.decode(compressed_ids)To a human, the compressed prompt might look like broken English. To the LLM, the semantic meaning remains clear. You just saved hundreds of input tokens before the request even reached your main model.
Dynamic Cache Eviction
What happens when you cannot compress the prompt any further, but you still have a long, ongoing conversation? This is where KV cache eviction comes in.
During generation, not all tokens in the history are equally important. Some tokens, like the first few tokens in the prompt (attention sinks) and a few key nouns or verbs, receive most of the attention. Other tokens, like punctuation marks or transition words, are ignored after a few steps.
Instead of keeping the entire history in memory, we can use an eviction policy. The Heavy Hitter Oracle (H2O) algorithm tracks which tokens receive the most attention during generation. It keeps the top-performing tokens and the very first tokens in the sequence, then drops the rest from the KV cache.
By capping the KV cache size to a fixed budget, say 1024 tokens, you keep memory usage flat. The model can run indefinitely without running out of GPU memory. The generation speed remains high because the GPU is not swapping data back and forth to system memory.
Speculative Decoding
Another way to harness tokens is to change how they are generated. Normally, an LLM generates tokens one by one. Each token requires a full forward pass through the massive model.
Speculative decoding changes this. You pair your large target model with a small draft model. The small model is cheap and fast. It generates a draft of five tokens.
The large model then looks at all five tokens at once in a single parallel pass. It decides which tokens it agrees with. If it agrees with four of them, you generated four tokens for the computational cost of one large model pass and a few cheap small model passes. If it rejects the third token, you keep the first two, discard the rest, and let the large model generate the correct third token.
This approach does not change the mathematical output of the large model. You get the exact same quality as if the large model wrote the whole thing, but you get it much faster. It works best when the draft model is highly aligned with the target model.
Model Cascades and Routing Logic
Not every user query needs a 70-billion parameter model. If a user asks "What is the capital of France?", using a massive model is a waste of compute.
A model cascade uses a routing layer to inspect incoming queries. The router is usually a small classifier or a cheap model. It estimates the difficulty of the prompt.
If the prompt is simple, the router sends it to a small, cheap model. If the small model returns an answer with high confidence, the system returns it to the user. If the confidence score is low, the system escalates the query to the large model.
[User Query] ──> [Router Model]
│
┌─────────┴─────────┐
(Low Complexity) (High Complexity)
▼ ▼
[Small Cheap Model] [Large Complex Model]
You can also do this at the layer level inside a single model. This is called early exiting. The model processes the input through its first few layers. If the intermediate representations are highly confident about the next token, the model skips the remaining layers and outputs the token. You save the compute of the upper layers.
Engineering the Pipeline
If you want to build this, start with prompt compression. It is the easiest to implement because it does not require modifying the model architecture or hosting custom inference engines. You can run a small compression script on your application server before sending the payload to your LLM provider.
If you host your own models, look at engines like vLLM or TensorRT-LLM. They have built-in support for chunked prefill, speculative decoding, and optimized KV cache management.
You will need to run evaluations to find the right balance. Every time you drop tokens or evict cache items, you risk losing accuracy. Build a test suite of your typical user queries. Measure the output similarity and task accuracy as you dial up the compression ratio.
The goal is to find the sweet spot where your costs drop significantly, but your users do not notice any change in performance. Start small, measure everything, and prune aggressively.
