Token Prices Are Falling, So Why Do AI Costs Keep Rising? Gartner's 'Inference Paradox'

According to a report released Monday by Gartner, enterprise AI inference costs are projected to increase more than fivefold by 2028. As enterprise AI use evolves from simple assistive functions into multi-step processes, the cost burden that technology leaders must bear is growing accordingly.

Will Sommer, VP Analyst at Gartner, said in the report, "Each generation of AI capability requires more—and typically more expensive—tokens," adding, "There's no reliable, low-cost, do-it-all model on the horizon."

Tokens get cheaper, but the bill gets bigger

Gartner explains today's token economy through three factors. First, the cost of foundational models themselves is declining. Second, this gain in efficiency actually encourages organizations to pursue higher-value AI applications by leveraging more powerful and expensive models. Third, these sophisticated AI workflows consume far more tokens than the simple chatbot interactions of earlier models, driving up overall inference costs.

Sommer explained the difference in economics between simple chatbots and AI agents, saying, "A simple chatbot just needs to read and interpret a query and quickly produce a probabilistically plausible answer, but an AI agent must continuously reason, negotiate, and question itself." Gartner has named this structure—where the total cost of AI keeps rising without a clear path to predictable value—the "Inference Paradox."

The more agentic the AI, the higher the ROI bar

The Gartner report states that achieving ROI with more advanced systems, such as agentic AI models or reasoning agents, requires far higher returns than with basic models. Otherwise, Sommer said, companies must heavily optimize their models to complete complex tasks in order to achieve cost efficiency. In a July interview, he noted that both options are viable, but each requires a major overhaul of work workflows.

Sommer said, "Every CIO has different priorities and different issues that matter to their organization," adding, "CIOs need to work with the rest of the organization to determine what the priorities are and whether AI is a cost-effective way to solve those problems."

Failure to predict costs escalates all the way to the boardroom

According to a July report by cost-management software company Mavvrik, roughly half of organizations said they have had to report unexpected AI spending issues to their board. In a separate announcement earlier this month, Gartner projected that spending on AI-optimized infrastructure services (computing that supports training and operating large language models) will nearly double to $42 billion by the end of 2026. Sommer also said in the July interview that Gartner estimates global token consumption at roughly 300 trillion tokens per day, with an annual growth rate of 500–600%.

Implications for AX in Korean Enterprises

Gartner's warning is a caution to organizations that have optimistically projected AI adoption costs based on falling token prices. Even as unit prices decline, if the scope of use and the complexity of workflows expand in tandem, total costs can actually grow larger. Organizations transitioning from simple chatbot-style functions to agentic automation, in particular, should design their budgets on the premise that cost increases may not be linear but, as Sommer noted, can steepen with each successive generation.

As Sommer emphasized, AI adoption priorities differ by organization, and ultimately what matters is answering for yourself the question, "Is AI a cost-effective way to solve this problem?" Organizations considering an AI transition should first distinguish whether the functionality they intend to adopt is simple response-based or agentic, requiring multi-step reasoning. If it's the latter, the sensible approach is to reset ROI baselines on the assumption that token usage will increase.

Source: Advances in AI capabilities to outpace cost savings