Why We're Still Chasing Core Web Vitals
If you're a frontend dev, you've probably lost an entire afternoon fighting the Lighthouse over 0.3 seconds of LCP. I have. I've swapped images for WebP, lazy-loaded everything that could be lazy-loaded, and debugged an annoying CLS that only showed up on mobile because a banner loaded half a second after everything else. It's tedious work, but necessary, and we all know it.
The reason is simple: Core Web Vitals is still one of the metrics Google cares most about when ranking your site, and it's also, without sugarcoating it, the difference between a user staying or closing the tab in 2 seconds. In 2026 that hasn't changed. If anything, INP (which replaced the old FID) is now firmly established as the main responsiveness metric, and Google keeps tightening the acceptable thresholds with every update.
So let's be honest: optimizing performance is still part of the job. Nobody's saying to stop doing that. But what if I told you that while you're sweating over shaving 200ms off your LCP, there's an entire "user" of your site that couldn't care less about that, one you've probably never stopped to think about?
The Blind Spot: Your Site Also Has a Non-Human User
For years we got used to thinking about two types of visitors: the human, who clicks, scrolls, and decides whether to buy or not, and the Google crawler, working behind the scenes indexing everything so it can later show your site in search results. Two audiences, two concerns: UX for one, SEO for the other.
Except today there's a third visitor that most frontend teams still haven't really stopped to consider. AI agents. The ChatGPT someone asked to research a product, the Claude summarizing an article, or even an autonomous agent carrying out a task on its own, like comparing prices or filling out a form. These agents don't browse the way we imagine. They don't click a button waiting for an animation, they don't scroll through admiring the design, and most importantly, many of them don't even execute your JavaScript.
And that's where the problem lives. If your site depends on heavy hydration, on content that only appears after a client-side API call, or on messy HTML structure that only makes sense visually, you could have flawless Core Web Vitals and still be practically invisible to this new kind of "user."
How an AI Agent "Reads" a Page (Unlike a Traditional SEO Crawler)
A traditional SEO crawler, like Googlebot, is relatively tolerant. It can wait for rendering, process JavaScript (with limitations), and still put together a reasonable understanding of the page, even if it's messy.
AI agents can work differently. Depending on the agent and its tooling, they may access a page through rendered content, HTML, Markdown, clean text, or structured interfaces. They benefit from clear structural signals such as semantic HTML, headings, and accessible server-rendered content. Beyond these established web practices, newer AI-oriented conventions and protocols, such as llms.txt and MCP, are also emerging, although adoption and support still vary significantly between agents.
In other words: it's not that the agent is "dumb" and can't read your site. It’s that your site may not be exposing its content in a way that’s easy for different agents to understand and use.
Meet isitagentready.com: A "Lighthouse" for Agent-Readiness
If you work in frontend, you've probably run Lighthouse so many times you could recite its score color palette from memory. Green is good, orange is concerning, red means stop everything and fix it. Well, there's a similar tool, except it's focused specifically on whether your site is ready to be read and used by AI agents. It's called isitagentready.com, built by Cloudflare, and it works pretty simply: you paste your site's URL, run the scan, and get an X-ray of how ready (or how far off) your site is for this new kind of consumption.
It's not some flashy marketing tool just meant to scare you. It checks real patterns that are already being adopted, some already established, others still emerging, but together they give a good sense of where the web is heading.
What the Tool Actually Checks
The scan is split into five categories, and it's worth understanding quickly what each one means in practice:
Discoverability: checks things like robots.txt, sitemap, link headers, and a newer standard called DNS-AID, which helps agents discover information about the site directly via DNS.
Content Accessibility: checks whether the site offers Markdown content negotiation, meaning whether there's a way for an agent to request a clean text version of the page instead of the fully rendered HTML/JS version.
Bot Access Control: analyzes whether you have specific AI bot rules in robots.txt, plus newer signals like Content Signals and Web Bot Auth, which help control who can access what.
Protocol Discovery: checks support for protocols like MCP, Agent Skills, WebMCP, OAuth discovery, and others. This is the more "cutting edge" part, aimed at sites that want to be actively used by agents, not just read.
Commerce: audits patterns built for agentic commerce, like x402, ACP, and UCP, meant for scenarios where an agent can literally make a purchase on a user's behalf.
Not every site needs, or should aim, to score 100% across all of this. An institutional blog has very different priorities than an e-commerce store, for example. But understanding these categories already helps you see where your project stands, and where it might be worth investing first.
Running the Scan in Practice: What Usually Shows Up
If you've never heard of these standards before, it's worth running the scan on one of your own sites just to get a feel for it. Most of the time, especially on projects that never stopped to think about this, the results tend to follow a pretty similar pattern.
robots.txt usually exists, but it's that generic file nobody has touched since the project started. It handles traditional search crawlers just fine, but has no specific mention of AI bots, so the site ends up with zero control over who can or can't use that content to train a model or answer an agent's query.
llms.txt usually doesn't exist at all. Makes sense, it's a fairly recent standard and hasn't become anyone's "must-have" checklist item yet. But it works basically like a direct map for an agent to understand what the site is about, without having to guess by rendering page after page.
On the content accessibility side, it's common for the site to offer no form of Markdown negotiation at all. Meaning, if an agent requests a lighter, cleaner version of the page, it simply doesn't get one, and has to make do with the fully rendered HTML, JS included.
And the newer protocols, like MCP or Agent Skills, almost always come back at zero. Which is expected, since those standards are aimed at sites that want to be more than just "read" by agents, they want to be actively used by them to carry out a task. That's more relevant for products and platforms than for a blog or institutional site, for instance.
In the end, the most common scenario is: a site with decent, well cared for, optimized Core Web Vitals, that in practice is speaking a language only a human's browser understands. None of these items are usually hard to fix on their own. The problem is that nobody looks to fix what they never knew needed fixing.
How to Improve AI-Readiness in Your Frontend Workflow
The good news is that none of this requires rewriting the project from scratch. In most cases, these improvements fit right into the normal workflow, the same way we already fit in performance optimization without having to halt the entire project for it.
Start With the Basics: A Properly Configured robots.txt and Sitemap
The first step is the simplest and also the most overlooked. Review your robots.txt and make sure it has clear rules for AI bots, not just for traditional search crawlers. That means consciously deciding which agents can access your content and which can't, instead of leaving it on autopilot.
An example of a specific AI bot rule:
User-agent: GPTBot
Allow: /User-agent: ClaudeBotAllow: /User-agent: CCBotDisallow: /Alongside that, keeping your sitemap up to date and referenced inside robots.txt itself helps both traditional crawlers and agents quickly understand your site's structure, without having to navigate page by page trying to guess it.
Publishing an llms.txt
llms.txt works like a straight-to-the-point summary of what the site is, what it covers, and which pages matter most, written specifically with language models in mind as the reader. It sits at the root of the domain, yoursite.com/llms.txt, similar to robots.txt itself.
There's no mystery to the structure. It's usually just plain Markdown, with a short description of the site and a list of important links, each with a one-line explanation.
SSR/SSG as a Bridge Between Human Performance and Agent Readability
This is where things get interesting if you work with Next.js or headless WordPress. The same architectural decision that helps with Core Web Vitals, rendering content on the server instead of relying on heavy client-side JS, also helps enormously with agent readability.
A site using SSR or SSG delivers ready, structured HTML on the first response, without requiring anyone, human or agent, to wait for hydration to see the actual content. That means if you've already been taking good care of performance with these techniques, you're probably closer to agent-readiness than you think. The work, in that case, is less about rewriting the architecture and more about making sure it's also properly exposed and signaled to agents, with the right files and headers in place.
The Final Checklist: What to Run Before Every Deploy
To wrap up with something practical, here's a quick list to check whenever you're reviewing a project, yours or a client's. You don't need to tackle everything at once, but it's worth keeping on your radar as part of the normal frontend and QA process.
Performance (what you already do):
LCP within the recommended threshold
Consistent INP, no lag on interactions
Controlled CLS, no elements jumping around on screen
Optimized images loading smartly
AI-Readiness (what now becomes part of the process):
robots.txt with specific rules for AI bots, not just search crawlers
Sitemap kept up to date and referenced
llms.txt published at the domain root
Main content accessible without depending solely on client-side rendering
Clear semantic structure in the HTML, headings, hierarchy, not relying only on CSS to give visual meaning to content
None of the items on that second list are complicated on their own. The point is that, until recently, nobody thought of this as part of the frontend checklist. Today it's no longer something you can treat as optional or "a distant future thing." Agents are already browsing, reading, and in some cases, even acting on behalf of real users.
Performance has always been about delivering the best possible experience to whoever visits your site. It's just that now "whoever visits" has changed shape, and it's worth making sure your work still makes sense for both sides.
.jpg)
.jpg)
