Moving Cron Curation Jobs from Cloud APIs to Local Small Language Models

A few months ago, I looked at our API bill and felt a dull ache in my chest. We were spending hundreds of dollars a month on OpenAI API calls just to summarize RSS feeds, categorize blog posts, and rank developer news for an internal newsletter. These jobs ran on a quiet system cron every hour. Nobody was waiting for the results in real-time. Yet, we were paying premium rates for GPT-4o to do basic text processing while we slept.

For interactive chat features, you need fast response times and smart models. While you might use a state-of-the-art model like Claude Sonnet 5 for user-facing applications, users get annoyed if a bot takes ten seconds to reply. But cron jobs don't care about latency. If a background job takes three minutes to process a batch of twenty articles instead of three seconds, nobody notices. The database doesn't cry, and the server doesn't crash. This makes background curation the perfect candidate for local AI models and self-hosted Small Language Models (SLMs).

You don't need a 70-billion parameter model to extract three tags and write a two-sentence summary. Models like Llama 3 8B, Mistral 7B, or Microsoft's Phi-3 are capable enough. They run comfortably on consumer hardware or cheap cloud instances. If you configure them right, they do the job just as well as the big API providers, but for a flat monthly server cost of zero extra dollars.

Here is how we migrated our background pipelines from the cloud to local models, the code changes required, and how we handled the drop in reasoning power.

The Curation Pipeline

Our curation setup follows a simple loop. Every hour, a script fetches new posts from a list of 150 blogs. It strips the HTML, extracts the main text, and passes it to an LLM.

The LLM has two jobs:

1. Summarize the article in two sentences.
2. Assign up to three tags from a predefined list.
3. Score the article's relevance to our team from 1 to 5.

With OpenAI, we used a standard Node.js script pointing to gpt-4o-mini. It worked, but the costs scaled with the number of feeds we added. If a blog published a long-form essay, we paid for thousands of tokens of input just to get a fifty-word summary back.

Setting Up the Local Inference Engine

To run models locally, you need an inference engine. We chose Ollama because it is simple to set up and provides an API that mimics OpenAI's endpoint structure.

You can install Ollama on a local machine or a remote VPS with a single command:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, you pull the model you want to use. For our curation tasks, we found llama3:8b offered the best balance between speed and accuracy.

ollama run llama3:8b

This starts the background service and downloads the model weights. The service runs on port 11434 by default. You can test it with a quick curl request:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3:8b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

The Migration Code

The best part about Ollama is its compatibility layer. You do not need to rewrite your entire codebase or swap out your SDKs. You can keep using the official OpenAI client library and simply point it to your local instance.

Here is what our script looked like before the migration:

import OpenAI from 'openai';
 
const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});
 
async function summarizeArticle(title, content) {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [
      { role: 'system', content: 'You are a helpful assistant.' },
      { role: 'user', content: `Summarize this: ${content}` }
    ],
  });
  return response.choices[0].message.content;
}

Here is the modified version running against our local Ollama instance:

import OpenAI from 'openai';
 
const localClient = new OpenAI({
  baseURL: 'http://localhost:11434/v1',
  apiKey: 'ollama', // Ollama does not validate keys, but the SDK requires a string
});
 
async function summarizeArticleLocal(title, content) {
  try {
    const response = await localClient.chat.completions.create({
      model: 'llama3:8b',
      messages: [
        { role: 'system', content: 'You are a helpful assistant.' },
        { role: 'user', content: `Summarize this: ${content}` }
      ],
      options: {
        temperature: 0.2, // Keep it low for consistent output
      }
    });
    return response.choices[0].message.content;
  } catch (error) {
    console.error('Failed to run local summary:', error);
    // Fallback logic goes here
  }
}

We only changed the baseURL, the apiKey placeholder, and the model name. Keeping our integration logic isolated in small, readable functions meant the rest of our application logic remained untouched.

Handling the Quality Gap

Small models are not drop-in replacements for GPT-4o. If you give them a loose prompt, they will wander off. They might add conversational filler like "Here is the summary you requested:" or ignore your formatting rules completely.

To make llama3:8b reliable enough for production cron jobs, we had to adjust our prompting strategy.

1. Force Structured JSON

If you need to parse the output of an SLM, do not rely on markdown parsing. Force the model to return raw JSON. Ollama has built-in support for this. You can pass a format parameter in your request.

const response = await localClient.chat.completions.create({
  model: 'llama3:8b',
  messages: [
    { 
      role: 'system', 
      content: 'You output raw JSON only. Format: {"summary": "string", "score": number, "tags": []}' 
    },
    { role: 'user', content: `Analyze: ${content}` }
  ],
  response_format: { type: 'json_object' }
});

When you set the format to json_object, the engine constrains the model's token generation to output valid JSON. It prevents the model from writing intro or outro text outside the JSON braces.

2. Few-Shot Prompting

An 8B model needs examples to understand the desired tone and style. If you want two-sentence summaries that get straight to the point, show the model what a good summary looks like.

We updated our system prompt to include a concrete example:

const systemPrompt = `You are a data extraction script. You analyze articles and output JSON. 
Example Input: "Today Apple announced the new M4 Mac Mini, featuring a smaller footprint and faster processing speeds starting at $599."
Example Output: {"summary": "Apple launched a redesigned, smaller Mac Mini powered by the M4 chip.", "score": 4, "tags": ["hardware", "apple"]}
 
Analyze the provided text and output matching JSON. Do not write explanations.`;

Adding this single example dropped our parsing error rate from 12% to zero.

Hardware and Cost Math

We run our cron jobs on a self-hosted server. Let's look at the numbers to see if this setup makes sense financially.

Suppose you process 10,000 articles per month.

• Average article length: 1,500 words (roughly 2,000 tokens).
• Total input tokens per month: 20,000,000.
• Average output tokens per summary: 100 tokens.
• Total output tokens per month: 1,000,000.

Using OpenAI's gpt-4o-mini pricing:

• Input: $0.150 per million tokens.
• Output: $0.600 per million tokens.
• Monthly cost: (20 * 0.150) + (1 * 0.600) = $3.60.

At this scale, the cloud API is incredibly cheap. It is not worth managing your own hardware for $3.60 a month.

But what if you scale up? What if you run a monitoring tool that processes 500,000 social media posts, GitHub commits, and news articles every month to track trends?

• Total input tokens: 1,000,000,000.
• Total output tokens: 50,000,000.
• Monthly cost with gpt-4o-mini: (1000 * 0.150) + (50 * 0.600) = $180.
• Monthly cost with gpt-4o (standard): (1000 * 2.50) + (50 * 10.00) = $3,000.

If you need the reasoning level of a larger model, or if your volume increases, the API bill grows quickly.

We set up a dedicated Hetzner Cloud instance to handle our local models. We chose a CCX22 instance (4 dedicated vCPUs, 16GB RAM) for around $25 a month.

Because it is a cron job, we do not need a GPU. The CPU runs the quantized 8B model at about 12 tokens per second. A single summary takes about 10 seconds to generate.

A single thread can process 360 articles per hour. By running the script sequentially, we can process up to 8,640 articles a day on a $25 VPS. That is over 250,000 articles a month for a flat fee. If we need more speed, we run the cron job during off-peak hours when the CPU is idle.

If you have a spare Mac Mini with an M-series chip in your office, the cost drops to zero. Apple Silicon handles unified memory operations fast. An M2 Mac Mini can run the same model at 35 tokens per second, pulling under 30 watts of power.

The Trade-Offs

You should not move everything to local models. There are real trade-offs to consider before ripping out your API keys.

• Maintenance overhead: You are responsible for keeping the service running. If Ollama crashes or the server runs out of memory, your cron jobs fail. You need to write health checks and auto-restart scripts.
• Context window limits: Small models struggle with very long articles. If you feed a 10,000-word academic paper into an 8B model on a CPU, it will take a long time to process, and the model might lose track of the prompt instructions.
• No intelligence upgrades for free: When OpenAI updates their models, your script gets smarter automatically. With local models, you have to manually pull new weights and test them to ensure they do not break your existing prompts.

For background curation, these trade-offs are manageable. The cost savings and data privacy of running everything on your own network make it a smart choice for long-term projects.