Enterprise AI runs on a simple, painful equation. Every single character your model processes or generates costs money. When you scale a system to serve thousands of customers querying millions of documents, those fractions of a cent turn into massive monthly bills, particularly as context overhead is reshaping infrastructure costs for complex workflows.
Writer, a platform that builds generative AI applications for businesses, recently ran into this scaling wall. They wanted to deploy Z.ai's new GLM-5.2 model. The model is highly capable at reasoning tasks, but it has a problem common to large foundation models. It is incredibly chatty. It uses a high number of tokens to explain things that could be said in a few words.
To fix this, Writer did not just change their system prompts. They took the model through a post-training optimization pipeline. To make sure their optimizations did not break the model's intelligence, they built a new validation harness. Much like the standard practice to test model upgrades with saved prompts and latency baselines, this harness specifically tracks token efficiency alongside accuracy metrics.
The Problem with Raw Foundation Models
Out-of-the-box models like GLM-5.2 are trained to be helpful assistants. In a consumer chat app, a long, detailed explanation is great. In an enterprise workflow, like extracting data from a PDF, it is waste.
If a model generates fifty words of explanation before giving the JSON payload you actually asked for, you pay for those fifty words. You also pay in latency, because generating those extra tokens takes time.
Writer's team realized they needed to train the model to stop explaining itself when it was not necessary. They used supervised fine-tuning (SFT) to teach the model brevity. They also applied FP8 quantization to reduce the memory footprint.
But optimization is risky. If you push a model too hard to be brief, it starts leaving out important details. If you quantize too aggressively, the model might start repeating itself, which actually increases token usage.
Inside Writer's Upgraded Validation Harness
The new validation harness solves this by running automated checks during the post-training process. Instead of just checking if the model gets the right answer, the harness evaluates the cost of getting that answer.
The harness measures a specific metric they call the token efficiency index, or E_idx. This is calculated by comparing the number of generated tokens to a target length baseline for each task.
E_idx = (Target Token Count) / (Actual Generated Token Count)
If a model version achieves a high accuracy score but has a low E_idx, the harness flags it. This prevents overly wordy models from making it to production.
The harness also tests for repetition loops. Quantized models sometimes get stuck in patterns, repeating the same phrase over and over until they hit the maximum token limit. The harness uses n-gram repetition detection to catch these loops early in the training run.
Fine-Tuning for Brevity Without Losing Accuracy
During the fine-tuning phase, Writer used pairs of responses to train GLM-5.2. The preferred response was always the shorter one, provided it contained all the correct facts.
The validation harness played a key role here. It ran evaluation sets after every training epoch. These evaluation sets were not generic academic benchmarks. They were real enterprise tasks, like summarizing financial reports and extracting entities from legal contracts.
For legal document analysis, the base GLM-5.2 model averaged 340 tokens per response. After the SFT pass, the optimized model dropped to 180 tokens. The validation harness verified that the accuracy of the extracted entities remained at 99.2%, matching the base model.
The harness also evaluated system prompt pruning. Often, developers write long system prompts to force a model to behave. These prompts are sent with every single request, inflating input token costs. The harness automatically tests shorter versions of the system prompt to find the minimum instruction set required to keep the model on track.
Quantization and the Risk of Token Inflation
Quantization is a common way to lower hardware costs. By converting model weights from FP16 to FP8, you can run the model on cheaper GPUs. While consumer-grade workarounds like AirLLM run 70B models on consumer GPUs with minimal VRAM, enterprise environments require a more delicate balance. But quantization can introduce noise.
This noise often shows up as hesitation in the model's output. A quantized model might generate extra filler words like "actually" or "basically" because its probability distributions have been flattened.
Writer's validation harness includes a filler-word detector. It flags model versions that show a spike in these low-information tokens. If the FP8 version of GLM-5.2 starts using 15% more tokens to say the same thing as the FP16 version, the hardware savings are wiped out by the token inflation.
By using the harness to guide the quantization calibration process, Writer's engineers adjusted the scale factors for specific layers in GLM-5.2. This kept the filler-word rate close to zero.
Calibration Datasets and Domain Specifics
A major challenge in post-training is selecting the calibration dataset. If you calibrate GLM-5.2 using generic web text, the model loses its grasp of domain-specific terminology.
When processing medical records, a poorly calibrated model might struggle to find the right clinical term. It starts generating long-winded descriptions instead of the precise medical code. This increases both latency and token cost.
To prevent this, Writer's validation harness tests the model against domain-specific calibration sets. The harness evaluates how the model performs on specialized vocabularies in finance and medicine.
If the model's token efficiency drops on these specialized tasks, the engineers know they need to adjust the calibration mix. This level of testing ensures the model remains efficient even when handling complex, industry-specific data.
Writer's engineers also found that the order of training data matters. If the model is exposed to too many short-form examples early in the SFT process, it begins to lose its reasoning depth. It starts generating short answers that are factually incorrect. The validation harness catches this by running a battery of logic tests alongside the token count checks. If the model's logic score drops below a certain threshold, the training run is aborted automatically.
The Financial Impact
The results of this post-training work are clear. In production tests, the optimized GLM-5.2 model achieved a 32% reduction in overall token usage compared to the base model.
For an enterprise customer processing ten million API calls a month, that reduction translates directly to thousands of dollars saved. It also cut latency by nearly a third, since the model spent less time writing useless text.
This feedback loop between training and validation allows Writer to iterate quickly. Instead of waiting days to see if a fine-tuning run was successful, they get feedback within hours. They can run dozens of small experiments, tweaking hyperparameters to squeeze out every bit of token efficiency.
Writer's approach shows that model optimization is not just about raw speed. It is about controlling the output behavior. A model that says exactly what is needed and nothing more is the most cost-effective model you can run.
The custom validation harness is now a permanent part of Writer's deployment pipeline. As they look to integrate newer models, the focus will remain on keeping those models quiet and accurate.
