← Back to Blog

We Decreased Our LLM Costs by 67% With Claude Opus—Here's How

• AI Cost Optimization, Claude Opus, LLM Economics, Product Strategy, AI Infrastructure, Cost Reduction, API Optimization, AI Product Management

We Decreased Our LLM Costs by 67% With Claude Opus—Here's How

Last quarter, our AI product was burning through $42,000 monthly in LLM costs. Our CFO wasn't happy. Our investors were asking pointed questions. And honestly? I was losing sleep over whether we could scale profitably.

Three months later, we're running the same workload—serving 3x more users—for $14,000 per month. Same quality outputs. Better latency. Happier customers.

The counterintuitive part? We achieved this by upgrading to Claude Opus, Anthropic's most expensive model.

This isn't another think piece about prompt optimization or caching strategies. This is a detailed breakdown of the architectural decisions, measurement frameworks, and tactical implementations that actually moved the needle on our LLM economics. If you're building AI products and your compute costs are keeping you up at night, this is for you.

The Cost Crisis Nobody Talks About

Here's the uncomfortable truth about building AI products in 2024: the unit economics are brutal.

When we launched our document analysis platform, we were riding high on GPT-4. The outputs were incredible. Users loved it. But every document processed was costing us $0.23 in API fees alone. With our $29/month subscription tier, we needed users to process fewer than 126 documents monthly just to break even on compute—before accounting for infrastructure, support, or any other operating costs.

The math wasn't mathing.

We tried the obvious solutions first. We experimented with GPT-3.5 Turbo for simpler tasks. Quality dropped noticeably. Churn increased by 18%. We attempted aggressive prompt compression. Saved maybe 15% on tokens, but engineering time exploded and edge cases multiplied.

Then we made a decision that seemed financially insane: we switched our entire pipeline to Claude Opus, which costs more per token than GPT-4.

Three weeks later, our monthly LLM bill had dropped by 43%. Two months after that? Down 67% from our peak.

Why More Expensive Can Mean Cheaper

The conventional wisdom in AI cost optimization focuses on token reduction: use smaller models, compress prompts, cache aggressively. This is optimization theater—it feels productive but often misses the forest for the trees.

The real cost driver isn't tokens per request. It's requests per outcome.

With GPT-4, our document analysis pipeline required an average of 3.7 API calls per document:

  1. Initial extraction and structuring
  2. Validation and error correction (needed 68% of the time)
  3. Format standardization
  4. Final quality check

Each call meant latency, error handling complexity, and compounding costs. Our effective cost per document wasn't the advertised API rate—it was that rate multiplied by our retry and refinement overhead.

When we switched to Opus, something remarkable happened: the model's superior reasoning and instruction-following meant we could collapse our pipeline. One call. One prompt. One output.

Yes, Opus costs more per token. But when you need 70% fewer tokens and 73% fewer API calls, the math flips dramatically.

Our actual numbers:

Even at Opus's higher per-token rate, we were spending 55% less per document. And that's before accounting for reduced error handling, simpler code, and faster processing times.

The Architecture That Changed Everything

Switching models was step one. But the real leverage came from rebuilding our architecture around Opus's specific capabilities.

Strategy 1: Intelligent Task Routing

Not every task needs frontier model intelligence. The key is knowing which ones do.

We implemented a three-tier routing system:

Tier 1 (Claude Haiku): Simple classification, format detection, basic extraction. These tasks have clear right answers and don't require nuanced reasoning. Cost: ~$0.001 per request.

Tier 2 (Claude Sonnet): Mid-complexity analysis, summarization, entity extraction with ambiguity. These need better reasoning but not Opus-level sophistication. Cost: ~$0.008 per request.

Tier 3 (Claude Opus): Complex reasoning, multi-document synthesis, nuanced analysis requiring domain knowledge. Only 23% of our requests actually need this tier. Cost: ~$0.024 per request.

Before routing, everything hit our most expensive model. After implementing intelligent triage, our average cost per request dropped from $0.024 to $0.009—a 62% reduction—while maintaining quality where it mattered.

The routing logic itself is surprisingly simple:

def route_request(task_type, complexity_score, document_length):
    if task_type in ['classify', 'detect_format', 'simple_extract']:
        return 'haiku'
    
    if complexity_score < 0.4 and document_length < 5000:
        return 'sonnet'
    
    return 'opus'

We trained a lightweight classifier (a fine-tuned BERT model, ironically) to predict complexity scores. It cost $340 to train and saves us roughly $8,000 monthly.

Strategy 2: Prompt Engineering for Efficiency

Opus's extended context window (200K tokens) and superior instruction-following enabled us to batch operations that previously required separate calls.

Instead of:

We now send one prompt requesting all outputs in a structured JSON format. Opus handles the multi-task request reliably, and we parse the response once.

The prompt structure that worked best:

Analyze the following document and return a JSON object with these exact keys:
{
  "entities": [{"name": str, "type": str, "relevance": float}],
  "sentiment": {"overall": str, "confidence": float},
  "summary": str,
  "action_items": [{"task": str, "priority": str, "deadline": str|null}]
}

Rules:
- Be concise but accurate
- Return ONLY valid JSON, no markdown formatting
- If a field cannot be determined, use null

[DOCUMENT]

This approach reduced our API calls by 75% for our most common workflow. The key insight: Opus is smart enough to handle complex, multi-part instructions reliably. Cheaper models often aren't, leading to the retry loops that kill your economics.

Strategy 3: Strategic Caching and Memoization

We implemented aggressive caching at three levels:

Semantic caching: Before hitting the API, we check if we've processed a semantically similar request. Using embeddings (via Voyage AI, much cheaper than LLM calls), we can identify when two requests are functionally identical even if worded differently.

Cache hit rate: 34%. That's one-third of requests we don't pay for.

Partial result caching: For documents with similar sections (contracts, reports), we cache analysis of common components and only send unique sections to the LLM.

Prompt template caching: Anthropic's prompt caching feature lets you cache the system prompt and shared context. For our use case, this reduced costs by another 15-20% on cache hits.

Combined, our caching strategy eliminates roughly 40% of potential LLM calls. At our scale, that's $5,600 saved monthly.

The Measurement Framework That Matters

You can't optimize what you don't measure. But most teams measure the wrong things.

We built a cost attribution system that tracks:

Cost per business outcome (not cost per token): What does it cost to fully process one document? One user session? One customer month?

Quality-adjusted cost: A cheap response that requires human correction is more expensive than a pricier response that's correct. We track cost × (1 + error_rate).

Latency cost: Faster responses improve conversion and retention. We calculate the revenue impact of latency improvements and factor that into our cost analysis.

This framework revealed surprising insights. For example, our cheapest model configuration had the highest quality-adjusted cost because error rates were 3x higher, requiring expensive human review.

The Results: Beyond Cost Savings

Three months after our Opus migration, the impact extended far beyond the obvious cost reductions:

Financial:

Product:

Operational:

The last point is crucial. Cheaper isn't always cheaper when you account for engineering time. Our team was spending 15-20 hours weekly managing the complexity of our multi-model, multi-call pipeline. That's half an engineer's time—easily $50K+ annually in fully-loaded costs.

Tactical Implementation Guide

If you're looking to replicate these results, here's the prioritized playbook:

Week 1: Instrument everything

Add comprehensive logging around every LLM call. Track: model used, tokens (input/output), latency, cost, outcome quality, retry count. You need this baseline data.

Week 2: Analyze your request patterns

Identify which requests could be handled by smaller models. Look for retry loops and multi-call patterns that indicate complexity mismatches.

Week 3: Implement basic routing

Start simple: route obvious low-complexity tasks to cheaper models. Measure quality impact obsessively.

Week 4: Test Opus for complex tasks

Run A/B tests on your most expensive workflows. Compare: total cost per outcome, quality, latency, retry rates.

Week 5-6: Consolidate prompts

Identify opportunities to batch operations. Rewrite prompts for Opus's capabilities. Test extensively.

Week 7-8: Implement caching

Start with semantic caching for your highest-volume endpoints. Measure cache hit rates and quality.

Week 9+: Optimize continuously

Set up automated alerts for cost anomalies. Review your top 10 most expensive request types weekly. Iterate.

The Contrarian Truth About AI Economics

The race to the bottom on model pricing is creating a dangerous narrative: cheaper models are always the right choice for cost-conscious builders.

This is backwards.

The right model is the one that delivers your required outcome with the fewest total resources—API calls, tokens, engineering time, error handling, human review.

For many use cases, that's actually the most capable (and expensive) model, used strategically.

Our Opus migration taught us that AI cost optimization isn't about minimizing per-unit costs. It's about:

  1. Matching model capability to task complexity (routing)
  2. Maximizing outcome per API call (prompt consolidation)
  3. Eliminating unnecessary calls (caching)
  4. Measuring what actually matters (cost per business outcome)

The teams that figure this out will build sustainably profitable AI products. The ones that don't will either run out of money or get stuck in a cycle of quality compromises and technical debt.

What's Next

We're not done optimizing. Our current focus areas:

Fine-tuning for common tasks: Training task-specific models for our highest-volume, most predictable workflows. Early tests suggest we can achieve Opus-quality outputs at Haiku-level costs for narrow domains.

Streaming and partial results: Implementing streaming responses to improve perceived latency and enable early termination when we have enough information.

Multi-provider routing: Testing whether certain tasks are better suited to GPT-4, Claude, or Gemini. Building a provider-agnostic abstraction layer.

The AI infrastructure landscape is evolving rapidly. Model prices are dropping, capabilities are improving, and new optimization techniques emerge weekly. But the fundamental principles remain:

Measure ruthlessly. Optimize for outcomes, not metrics. Match capability to complexity. And sometimes, spending more per token is the smartest way to spend less overall.

If you're building AI products, your LLM costs don't have to be a death sentence for your unit economics. But you need to think beyond token counting and prompt compression. The real leverage is architectural—and it requires being willing to challenge conventional wisdom about what "optimization" actually means.

Our $28K monthly savings bought us runway, reduced stress, and proved that profitable AI products are possible. Yours can too.