AI's Affordability Crisis: What Product Builders Need to Know About Rising Costs

TL;DR

• AI costs are escalating faster than Moore's Law can compensate: Training frontier models now costs hundreds of millions, and inference expenses are growing exponentially as usage scales, creating a structural affordability problem that won't resolve through hardware improvements alone.
• The economics favor incumbents with deep pockets: Companies like Google, Microsoft, and Amazon can subsidize AI losses indefinitely, while startups and mid-market product teams face brutal unit economics that make sustainable AI products increasingly difficult to build.
• Product builders must architect for cost from day one: The era of "build first, optimize later" is over—successful AI products now require sophisticated cost management strategies including model selection, caching, prompt optimization, and ruthless prioritization of high-value use cases.
• A bifurcated market is emerging: Premium AI capabilities will concentrate among well-funded players, while a parallel ecosystem of smaller, specialized models and cost-effective alternatives will serve price-sensitive segments, forcing product teams to choose their lane early.

The Sticker Shock No One Wants to Talk About

We're in the middle of an AI gold rush, but here's the uncomfortable truth: most of us can't afford the shovels.

While headlines celebrate GPT-5, Claude Opus 4, and whatever Google is calling their latest model this week, a quieter crisis is unfolding in finance departments and startup burn rate spreadsheets. The cost of building with AI isn't just high—it's accelerating in ways that fundamentally challenge the business models we've spent the past two years constructing.

David Rosenthal's analysis in "AI's Affordability Crisis" crystallizes what many of us have been feeling but haven't wanted to say out loud: the economics of AI are broken for most builders, and they're getting worse. His examination of training costs, inference expenses, and the structural advantages of hyperscalers paints a sobering picture for anyone trying to build sustainable AI products outside the protective moat of Big Tech.

As someone who's spent the last few years helping teams integrate AI into products, I've watched this affordability crisis move from theoretical concern to existential threat. The question isn't whether AI costs are too high—it's whether the current trajectory is compatible with the diverse, innovative ecosystem we need for AI to reach its potential.

The Three-Headed Cost Monster

The affordability crisis manifests in three distinct but interconnected ways, each creating its own set of constraints for product builders.

Training Costs: The Entry Barrier

Training frontier models has become an exercise in burning money at a scale that would make even venture capitalists wince. We're talking hundreds of millions of dollars for a single training run, with costs doubling roughly every six months as models scale. The compute requirements alone demand thousands of specialized GPUs running for months.

For most product teams, this means one thing: you're not training your own foundation model. That ship has sailed, burned its fuel, and sunk to the bottom of the ocean. The capital requirements have effectively locked foundation model development to a handful of companies with either massive cash reserves or an unusual tolerance for lighting money on fire in pursuit of strategic positioning.

But here's where it gets interesting for builders: this consolidation at the foundation layer doesn't necessarily doom innovation at the application layer. The real question is whether the foundation model providers will price their APIs in ways that allow sustainable businesses to exist downstream.

Inference Costs: The Silent Killer

Training costs grab headlines, but inference costs kill products.

Every API call, every generated token, every user interaction—they all cost money. And unlike training, which is a one-time expense (per model version), inference costs scale linearly with usage. Success literally costs more.

Rosenthal's piece highlights how inference costs have become the dominant expense for deployed AI systems, and this creates a vicious paradox: the more users love your product, the faster you burn through your budget. Traditional software economics promised that marginal costs approached zero as you scaled. AI products flip this on its head—your marginal costs might actually increase as you scale if you're not extraordinarily careful about optimization.

I've seen teams celebrate hitting 100K monthly active users only to realize their inference bill just made their unit economics completely unsustainable. The champagne turns to tears real fast when you're paying $0.50 per user per month in inference costs and charging $9.99 for a subscription.

The Hardware Bottleneck

Underlying both training and inference costs is a fundamental hardware constraint: we're running up against the limits of what current chip architectures can deliver at reasonable prices. Moore's Law hasn't died, but it's definitely wheezing, and the specialized compute required for AI workloads commands premium pricing.

GPU supply remains tight, and even when you can get allocation, you're paying through the nose for it. Cloud providers have some leverage here, but they're passing a substantial portion of those costs downstream to customers. The hope that hardware improvements would naturally drive down AI costs fast enough to save us looks increasingly optimistic.

Why This Matters More Than You Think

The affordability crisis isn't just about bigger AWS bills. It's reshaping the competitive landscape in ways that will determine which companies and which types of products can exist in an AI-powered future.

The Incumbent Advantage

Companies like Google, Microsoft, and Amazon can lose billions on AI and call it "strategic investment." They own the infrastructure, they have existing revenue streams to subsidize experimentation, and they can wait out the economics.

For startups and mid-market companies, there's no such luxury. You need to hit positive unit economics within a reasonable timeframe, or you're dead. This creates an asymmetric playing field where incumbents can underprice the market indefinitely, making it nearly impossible for challengers to compete on cost.

The result? A consolidation of AI capabilities among a small number of players, reduced innovation at the edges, and a narrowing of the types of AI products that can achieve venture-scale returns. That's bad for competition, bad for innovation, and ultimately bad for users who benefit from diverse approaches to solving problems.

The Commoditization Paradox

Here's my take, and I think this is where a lot of the current discourse gets it wrong: we're simultaneously seeing AI capabilities commoditize while costs remain stubbornly high. That's not how commoditization is supposed to work.

In a normal market, as capabilities become commoditized, prices fall and access expands. But AI's cost structure—dominated by compute that isn't getting dramatically cheaper—means we're getting commodity capabilities at premium prices. It's like if electricity became a commodity but still cost $5 per kilowatt-hour.

This paradox forces product builders into an uncomfortable position: you can't differentiate on AI capabilities (everyone has access to similar models), but you also can't compete on price (the underlying costs are too high). The only winning move is to find use cases where the value created so dramatically exceeds the cost that price becomes irrelevant—and those use cases are rarer than most pitch decks suggest.

Survival Strategies for Product Builders

If you're building AI products in this environment, fatalism isn't a strategy. There are concrete approaches that can make the difference between sustainable growth and a death spiral of rising costs.

Architect for Cost from Day One

The biggest mistake I see teams make is treating AI costs as an optimization problem to solve later. By the time "later" arrives, you've made architectural decisions that lock in expensive patterns and built user expectations around capabilities you can't afford to deliver.

Instead, treat cost as a first-class constraint from the beginning:

• Model selection matters more than you think: Using GPT-4 for every task is like driving a Ferrari to get groceries. Map your use cases to the smallest model that can handle them. Claude Haiku, GPT-3.5, or even fine-tuned smaller models can handle a surprising number of tasks at a fraction of the cost.

• Implement aggressive caching: If you're generating the same or similar outputs repeatedly, you're burning money. Semantic caching, response reuse, and intelligent prefetching can cut inference costs by 40-60% in many applications.

• Optimize prompts ruthlessly: Shorter prompts cost less. Every unnecessary token in your system prompt is money out of your pocket multiplied by every API call. Invest in prompt engineering not just for quality but for efficiency.

• Consider hybrid architectures: Not every interaction needs to hit a frontier model. Use smaller models or traditional ML for routing, classification, and filtering, reserving expensive models for high-value tasks.

Choose Your Battles

Not every problem needs AI, and not every AI problem needs to be solved right now.

The most successful teams I've worked with are ruthlessly focused on high-value use cases where the ROI clearly justifies the cost. They're not trying to sprinkle AI on everything—they're finding the 20% of use cases that deliver 80% of the value and nailing those.

This means saying no. A lot. It means disappointing stakeholders who want AI-powered features that sound cool but don't move the needle on metrics that matter. It means being honest about what you can afford to build and maintain.

Build Cost Transparency

You can't manage what you don't measure. Implement detailed cost tracking at the user level, feature level, and interaction level. Know which users are profitable and which are subsidized. Understand which features are cost centers and which justify their expense.

This visibility enables smart decisions: you might discover that 5% of your users generate 50% of your costs, allowing you to implement usage-based pricing or usage caps for heavy users. Or you might find that a feature everyone loves is actually bankrupting you, forcing a hard conversation about sustainability.

Explore Alternative Architectures

The affordability crisis is driving innovation in cost-effective AI architectures:

• Open source models: Llama 3, Mistral, and others offer capabilities you can self-host, trading infrastructure management complexity for lower per-inference costs at scale.

• Specialized models: Instead of general-purpose frontier models, consider fine-tuning smaller models for your specific use case. The upfront cost can pay off if your volume is high enough.

• Edge deployment: For some use cases, running models on-device eliminates inference costs entirely, though at the expense of capability and freshness.

• Mixture of experts: Route different types of queries to different models based on complexity, using cheap models for simple tasks and expensive ones only when necessary.

The Bifurcated Future

Looking ahead, I think we're heading toward a two-tier AI ecosystem:

Tier One will be premium AI capabilities—frontier models, real-time processing, multimodal understanding—available primarily through products built by companies that can absorb or subsidize the costs. These will be "AI-native" experiences that fully leverage cutting-edge capabilities but require either venture funding, enterprise pricing, or integration into larger product ecosystems to be economically viable.

Tier Two will be cost-effective AI—smaller models, longer response times, more constrained capabilities—serving price-sensitive segments and use cases where "good enough" AI delivers sufficient value. This tier will be where most innovation happens, because the constraints force creativity.

Neither tier is inherently better or worse. They serve different needs and different markets. But product builders need to consciously choose which tier they're playing in, because the strategies for success are completely different.

The danger is getting stuck in the middle: trying to deliver Tier One capabilities at Tier Two prices. That's where companies go to die.

What Needs to Change

The current trajectory isn't sustainable for a healthy AI ecosystem. We need:

More transparent pricing: Cloud providers and API vendors need to offer predictable, understandable pricing that allows builders to model costs accurately. The current system of complex per-token pricing with variable latency and quality makes financial planning nearly impossible.

Better tooling for cost management: We need developer tools that make cost optimization as easy as performance optimization. Observability platforms should track dollars as prominently as they track latency.

Alternative business models: Usage-based pricing makes sense for providers but creates risk for builders. We need more options: committed use discounts, cost caps, hybrid models that balance flexibility with predictability.

Hardware innovation: Yes, we need better chips. But more importantly, we need chips optimized for inference at scale, not just training. The economics of AI won't fundamentally improve until we can run models much more efficiently.

Building in the Crisis

The affordability crisis is real, and it's not going away soon. But crisis creates opportunity.

The teams that will win are those that treat cost as a design constraint rather than an afterthought. They'll build products that deliver genuine value—value so clear that pricing becomes a secondary concern. They'll architect for efficiency from day one. They'll make hard choices about scope and focus.

Most importantly, they'll be honest about the economics. No more pretending that AI costs will magically decrease fast enough to save unsustainable business models. No more assuming that scale will solve the unit economics problem. No more building products that require AI to be 10x cheaper to work.

The gold rush continues, but the easy money is gone. What comes next requires discipline, creativity, and a clear-eyed understanding of the economics. The affordability crisis will separate the serious builders from the tourists.

Which one are you?