For a long time, building a website rested on a single assumption: that a human would open the screen, read it, and click. As of 2026, that assumption is breaking down. It's no longer only humans who open a site, and no longer only humans who execute actions on it.
The shift is happening on two fronts at once. One is how AI crawlers harvest web content; the other is how AI agents connect directly to enterprise systems and complete transactions. The first is a problem that's already been measured. The second is a problem for which the specifications already exist.
The composition of website visitors has changed
Cloudflare measures and publishes a 'crawl-to-refer ratio' based on requests passing through its network. It's the number of times a given platform's user agent requested HTML content, divided by the number of visits that same platform sent back with its name in the Referer header — a measure of how many visitors a site gets back for every page a bot scrapes.
In an observation period from June 19 to June 26, 2025, Anthropic's ratio was 70,900 to 1. That means for every 70,000-plus HTML page fetches, exactly one visit came back to the site. In the same post, Cloudflare wrote of this figure: "These ratios have a direct impact on the sustainability of publishing content on the internet."the post stated.
The exchange that used to exist between search engines and websites — allow indexing, receive traffic in return — has broken down in this segment. A design built around ranking well and getting exposure to earn visitors simply doesn't function here.
What llms.txt revealed: putting a file in place accomplishes nothing on its own
The industry's first response to this problem was a protocol file. The proposal was to place an llms.txtfile at a site's root to tell AI what document structure to read.
Originality.ai tracked over 3 million websites from June 2025 to May 2026 and found llms.txt files grew 8.8x, from 4,088 to 36,120. llms-full.txt grew from 23 to 2,463, and ai.txt grew from 4 to 397.
But Ahrefs' analysis of server logs across 137,000 domains in May 2026 tells a different story. 97% of llms.txt files received not a single request that month. Of the requests that did come in, AI search and crawling bots accounted for just 1.1%, while the largest reader of these files — at 21.7% — turned out to be SEO audit tools.
The platforms themselves have been unambiguous. Google has stated that llms.txt has "no negative or positive effect"on search visibility or ranking, and that it is "not required"for Google Search. OpenAI and Anthropic direct site operators to robots.txt for crawler management, mentioning llms.txt only in developer documentation.
The response of uploading a single file produced no measurable result. Most of the files that grew 8.8x in a single year went unread by anyone.
The layer that actually works isn't the page — it's the endpoint
The channel agents actually use is being built separately — not as documentation, but as an API specification.
The Agentic Commerce Protocol (ACP)jointly maintained by OpenAI and Stripe, has its April 17, 2026 spec as the current stable version. It's a publicly available specification under the Apache 2.0 license and is currently in beta. Merchants seeking to comply don't edit a document — they implement two APIs: the Agentic Checkout API and the Delegate Payment API. The April 17 changelog lists carts, product feeds, orders, authentication, and MCP together.
When Stripe published this specification on September 29, 2025, it explained that the goal was to let merchants "avoid building separate inventory and payment integrations for every individual AI agent" by conforming to a single standard just once. Instant checkout completed within ChatGPT began with Etsy sellers and is expanding to Shopify merchants.
On the internal systems side, there's MCP (Model Context Protocol)— a specification published by Anthropic in November 2024 that is now run as a project under the Linux Foundation. In other words, it belongs to no single vendor.
What's worth noting is what MCP's 2026 roadmap lists as its priorities. One of its four top priorities is 'enterprise readiness,' and the items listed beneath it are "audit trails, SSO-integrated authentication, gateway behavior, and configuration portability."This isn't about model performance or prompting. It's the same integration, authentication, and operations problem that always comes up when connecting something to enterprise systems.
The gap agencies must fill isn't the screen — it's the control mechanism
Two surveys from around the same period show the size of that gap.
In a report published July 29, 2026 by compliance firm Schellman, based on a survey of 525 U.S. governance professionals, 86% of respondents were piloting AI agents, and nearly half had already put them into production. 90% said they had allocated budget for AI governance, and roughly three-quarters said they could pass an AI compliance audit. But only 27% said their own program was fully mature,and only 64% had an AI usage policy that was actually communicated to employees.
Danny Manimbo, Schellman's ISO and AI practice lead, summed it up in the report: "The challenge is turning individual activities into a mature, operable program that can withstand regulatory scrutiny and keep pace with fast-changing AI systems."
The cost side tells a similar story. According to data released July 29, 2026 by Harness, based on a survey of 700 FinOps and engineering leaders, one in every four dollars spent on AI is wasted.More than half of organizations had no one designated to own AI costs, and only one in five could pinpoint the cause of an unexpected cost spike within a few hours. Most organizations were simultaneously using three or more AI providers with different pricing structures.
Half of organizations are already running agents, but only a quarter have the mechanisms in place to control them. This gap defines what web and digital agencies can actually sell — not making screens look pretty, but defining the interfaces agents will access and making that access loggable, authenticatable, and blockable. Sitemaps and design mockups give way to access permission lists and audit-log design.
What's different for Korean companies
Both surveys cited — Schellman's 525 and Harness's 700 — are U.S. respondent samples. Their figures can't be directly applied to domestic companies with different organizational structures and regulatory environments.
Timing also differs. ACP's instant checkout launched first for U.S. users and U.S. merchants, and Korea's commerce ecosystem — built around simple payment services and PG-based settlement — doesn't plug into it the same way. There's not yet a strong reason to rush the payments side.
MCP, on the other hand, has no regional constraints. Connecting internal data and business systems to agents is something that can be done in Korea right now, and as noted above, the hard part of that layer isn't the model — it's authentication, auditing, and configuration management. This also aligns with the trend of domestic companies leaning toward in-house builds over off-the-shelf solutions. In a July 2026 survey by EYof 500 U.S. executives at SVP level and above, three-quarters of leaders with ongoing AI investments said off-the-shelf software didn't fit their company's IT requirements.
What to check right now
For organizations reviewing their agent strategy, it makes sense to confirm these three things first.
- The share and source of automated traffic among requests coming into your domain. Without knowing which AI crawlers are taking what and how much, there's no basis for deciding what to allow or block.
- A list of the interfaces agents can access. This should be written in terms of actually callable endpoints, not documentation. If that list is empty, uploading an llms.txt file won't change anything.
- Whether someone is designated as the owner of AI costs. If no one is, tracking down the cause after usage spikes will take weeks, not hours.
Whether to adopt agents is already a question of the past — 86% of surveyed organizations are already piloting them. The remaining question is how your systems appear to agents, and who controls that access.
Sources: Cloudflare — The crawl before the fall… of referrals · llms.txt adoption and actual request data (Originality.ai, Ahrefs) · Agentic Commerce Protocol specification · Stripe — Developing an open standard for agentic commerce · Model Context Protocol 2026 roadmap · CIO Dive — Most US companies lack mature AI governance frameworks · CIO Dive — 1 in 4 dollars spent on AI goes to waste