After two years of maxing out on AI tools, many organizations are confronting an uncomfortable reality: AI costs don’t scale linearly with adoption.
As workflows move beyond simple chatbots to autonomous agents, token consumption has exploded. Although the cost per token has declined, the total number of tokens being processed has increased even faster, driving expenses sharply upward. What many executives assumed would resemble a predictable software subscription has now turned into an infrastructure cost headache that grows with every prompt, workflow, and AI agent deployed.
Companies that once encouraged employees to maximize AI usage are now introducing spending caps and closer budgetary oversight. Uber reportedly exhausted its annual AI coding budget months ahead of schedule after widespread adoption of AI development tools. Company COO Andrew Macdonald noted that rising token consumption was not always translating into productivity gains. This is now shaping up to be a challenge increasingly shared across industries as organizations seek to balance AI adoption with measurable business outcomes.
The deeper cost problem is rarely the price of the model itself. It is the volume and quality of the information organizations ask of those models to process. As it turns out, companies looking to reduce AI spending may be searching in the wrong place. It’s everything that happens before a model ever generates an answer.
The Hidden Cost Behind Every Prompt
Before it generates a response, a system has to gather relevant documentation, retrieve customer records, search internal knowledge bases, enforce permissions, rank information, and assemble everything into a prompt. A model being only one component of an enterprise AI system, that orchestration layer often consumes more engineering effort–along with costs–than inference itself.
This becomes even more expensive with AI agents. Unlike chatbots that answer a single question, agents repeatedly retrieve documents, invoke tools, call APIs, maintain memory, and perform multiple reasoning steps before completing a task. Every one of those actions consumes additional tokens. And though the average cost of tokens is decreasing, the average cost per task is staying the same. Furthermore, as organizations deploy more autonomous systems, token usage naturally rises. The question is whether those tokens are being spent efficiently.
That doesn’t mean, however, that model optimization has its role to play.
Juan Diego Raimondi, Head of Data Science at Making Sense, a mid-market technology consultancy, said that enterprises have numerous ways to reduce AI costs before they ever touch context engineering.
Choosing the appropriately sized model instead of defaulting to the largest available one, applying quantization techniques, using prompt and semantic caching, routing simple requests to smaller models, and batching workloads can all lower costs without sacrificing quality.
Raimondi told the International Business Times that “model routing is also very popular: requests can be made for cheap models and escalate to bigger (more expensive) ones only when needed.”
Those techniques matter because they compound with context optimization rather than replace it.
Even after selecting the right model, organizations frequently waste tokens by retrieving excessive information, repeatedly searching the same knowledge sources, or carrying unnecessary conversation history into every prompt. Techniques such as context pruning, context compaction, and intelligent tool placement reduce token consumption while improving response quality.
The point, Raimondi explains, isn’t simply to buy cheaper intelligence but to stop wasting expensive intelligence that has already been accounted for.
Some executives assume that rapidly expanding context windows make context engineering less important. The opposite is increasingly true. As models become capable of processing millions of tokens, organizations are tempted to solve problems by simply providing more data. But more information often leads to both higher inference costs and lower-quality decisions. As Raimondi observes, “The bigger the context, the worse the performance on generation tasks (and the more expensive it is).”
The strategic question for enterprise leaders is no longer whether AI models can process more information. It is whether they ought to. Every additional document, email, policy, or knowledge source increases cost while also raising the likelihood that irrelevant information will dilute the model’s reasoning. Business leaders that gain the greatest advantage from AI won’t be those that feed models the most data, but those that consistently provide the right data at the right time. In that sense, context engineering is becoming a core executive capability, one that shapes the quality, speed, and economics of AI-driven decision-making across the enterprise.
That shift in thinking extends beyond prompt design to the underlying quality of enterprise data itself. If context engineering is about selecting the right information, then data engineering is about ensuring that the information is worth selecting in the first place.
As Lauren Cascio, founding partner of Gulp Data, a data valuation company, explains: “Everyone is obsessed with the cost of AI models, but that’s a rounding error compared to what companies are wasting on bad data. The real advantage isn’t finding a cheaper model; it’s treating data like the asset it is. Clean, validated, productized data means lower inference costs and better outputs.”
Architecture Matters More Than Model Choice
The need for better data underscores a broader strategic reality: competitive advantage in enterprise AI comes less from choosing the “best” model than from building assets that retain their value regardless of which model is in use. Clean, well-governed data and carefully engineered context are durable capabilities. Foundation models, by contrast, are evolving at a relentless pace.
The rapid pace of model development creates another challenge. Today’s leading model may be surpassed in a matter of weeks, which can make long-term platform commitments increasingly risky.
Skylar Roebuck, CTO at Solvd, a Silicon Valley-based AI engineering company, argues that to counter this, enterprises ought to build composable AI architectures rather than tightly coupling applications to a single model provider.
As Roebuck puts it: “[A]daptability must be architected,” adding that one can “build an architecture that anticipates this pace of change and establish an intelligence backbone that keeps learning while you keep replacing the parts around it. “By separating orchestration, context management, and application logic from the underlying models, organizations can gain the flexibility to evaluate and replace models as pricing and performance evolve.
That philosophy extends beyond model selection. Organizations should also be thinking of context as portable infrastructure rather than something tied to any individual vendor. Shared infrastructure, reusable intelligence, interoperable systems, and continuous learning across the organization are also best practices according to Roebuck.
To realize this approach in practice, Manu Swami, Chief Technology Officer at modernization engineering firm Sonata Software, detailed the value of a hierarchical system to guide model assignment strategically.
“AI-native organizations will redesign roles and ways of working for continuous AI-driven decisioning,” Swami said, adding that the old ways of working are slowly giving way. “The traditional resource pyramid will give way to a ‘models pyramid’: frontier models for complex reasoning, smaller models for cost and latency, and domain-tuned models for regulated or workflow-specific tasks.”
The Next AI Race
The first generation of enterprise AI was defined by access to powerful foundation models. It’s easy to anticipate that in less than a decade, companies won’t compete based on who has access to the best model. The models will be commodities just like the company fleet or its real estate. Instead, companies will compete based on who can deliver the right context to those models faster, cheaper, and more accurately than everyone else.
In other words, it won’t be a matter of simply negotiating lower token prices but building systems that minimize unnecessary retrieval, optimize context, intelligently route requests, cache repeated work, and deliver precisely the information a model needs.
The companies that recognize that shift early won’t just lower their AI bills; they’ll build AI systems that are more accurate, more adaptable, and ultimately more sustainable.
