Is Your Site Agent-Ready?

Agent-ready means the content and structured data a page depends on are already present in the raw HTML your server sends, before any JavaScript runs. This matters because content that only appears after scripts run is not in the raw HTML a plain HTTP fetch downloads, which is exactly what ClarAI's own no-JavaScript audit fetch sees, and none of OpenAI, Anthropic or Perplexity publish a statement that their crawlers execute JavaScript. A page can rank perfectly well on Google, which does render JavaScript, and still read as a blank shell to a crawler that does not. The fastest way to know where you stand is to look at your pages the way a no-JavaScript fetch sees them. This post covers what agent-readiness actually means, the one fact that decides it, the two failure modes ClarAI's site audit is built to catch, the lighter-weight signals worth checking, and how to audit your own site in a few minutes.

The one fact that decides it

A plain HTTP crawler fetches a page's HTML and parses it without running the client-side JavaScript a browser would, and that is how ClarAI's own audit fetch reads your pages. Googlebot is different: it renders JavaScript, which is why a client-rendered single-page app can index and rank in ordinary Search. The AI-search crawler vendors publish no statement that theirs render JavaScript, so the honest assumption is that the raw HTML is all they get. On that assumption the same page can be a first-page Google result and, at the same time, an empty container to a crawler that never runs the script. That gap is the whole of agent-readiness. If the words and the structured data only appear after the browser executes your JavaScript, a crawler that does not render never sees them.

“To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet.”

Google's own eligibility bar is deliberately boring: indexed, crawlable, snippet-eligible (and, since 31 August 2026, not excluded through the Search generative AI setting in Search Console). Googlebot renders your JavaScript to get there, so Google can forgive a client-rendered page. The AI-search crawlers are stricter, because they stop at the raw HTML. Agent-readiness is simply meeting the boring bar in a form that survives without a browser: the content in the HTML, not assembled after the fact.

The two failures ClarAI's site audit looks for

ClarAI's Site Audit crawls your pages the way an AI crawler would, with a no-JavaScript fetch, and raises two conservative findings against exactly this gap. Both are deliberately cautious, and both read the raw fetch view. Updated 1 October 2026: when our crawler needs a full browser to load a page (for example because of an anti-bot rule), the two checks still run on what the plain fetch received, because that is what a JavaScript-blind crawler gets; they used to skip such pages.

• Content not in raw HTML (high severity). Fires only when the raw fetched HTML is an empty shell on all four signals at once: near-zero visible text, no server-rendered headings, no structured data, and no images. That stacked combination on a healthy page is an unrendered single-page-app shell, which is what an AI crawler receives. The fix is to server-render or pre-render the main content so it is in the HTML the crawler downloads.
• Structured data not in raw HTML (low severity). Fires on a real, server-rendered content page, one with genuine body text and a heading, that ships no Schema.org JSON-LD in its raw HTML. The usual cause is structured data injected client-side by a tag manager or an app script. Google renders JavaScript and reads injected JSON-LD, and whether ChatGPT, Perplexity or Claude use JSON-LD at all is not documented, so this is a low-severity note, not a citation lever. Structured data helps rich results and product data, Google says its AI features need no special schema, and a controlled study found no citation lift from adding it. If you rely on structured data for rich results or product listings, emitting it server-side is the safe default.

The two are complementary. The first catches the page that has nothing an AI crawler can read. The second is a smaller note: the page a person would call complete, where only the machine-readable layer arrives after the browser loads, which matters for rich results and product data more than for AI answers.

The lighter agent-readiness signals

Beyond the raw-HTML question, ClarAI's Tech SEO Audit surfaces four first-version agent-readiness heuristics. These are early signals, reported as pass, warn or unknown rather than hard pass-or-fail, because the practices around agent access are still forming. They are worth knowing, and worth keeping honest about what they do and do not buy you.

• Semantic landmarks. Whether your pages expose the HTML5 landmark tags (main, nav, header, footer, article). The check passes when at least 70 percent of sampled pages carry three or more of the five. Programmatic agents parse the DOM and the accessibility tree, so explicit landmarks make a page easier to navigate by machine.
• An explicit robots policy for AI and agent user-agents. Silence is treated as a warning, never a failure: addressing the known AI crawlers by name, allow or disallow, is emerging best practice rather than a requirement.
• llms.txt. Shown for information only and never scored: Google says Google Search does not use it, an Ahrefs study (May 2026) found 97 percent of about 38,000 valid llms.txt files received no requests, and no other major engine documents using it for inclusion. Its absence costs nothing.
• Markdown content negotiation. Whether your server answers a request for Accept: text/markdown with actual markdown. It is an emerging agent-readiness signal that very few sites currently support, so serving HTML here is a warning, not a fault.

How to check your own site

• View source, or fetch the page with curl, on your money pages and read the body. If the visible text, your headings and your JSON-LD are all there in the delivered HTML, an AI crawler can read them. If the HTML is a near-empty div and everything arrives later, it cannot.
• Disable JavaScript in your browser and reload a key page. What remains is roughly what a JavaScript-blind crawler sees. A page that goes blank is the content-not-in-raw-HTML case.
• Search the raw HTML for a script type of application/ld+json. If your structured data is missing from the source but present in the rendered page, it is being injected client-side, which is the structured-data-not-in-raw-HTML case.
• Run the free ClarAI grader at /grade. It checks the raw-HTML-facing signals alongside the heavier ones: semantic landmarks and markdown content negotiation carry small weights under the much heavier snippet-eligibility, Googlebot and AI-crawler checks, and llms.txt is shown for information only with no weight in the grade. No signup needed.

One honest note about us, in the spirit of this blog. When we first published this post, ClarAI's own public marketing pages were a client-rendered app and pre-rendering them was still on our list. Updated 1 October 2026: our public pages are now pre-rendered at build time, so each public route ships its content in the raw HTML, and the build fails if a public route is missing from the pre-render list. We also ship the robots.txt that allows the same AI crawlers this audit checks, a sitemap, and a title and description per page. Agent-readiness is a bar most sites are still climbing toward, and keeping ours there is ongoing work.

Quick answers to the common questions

• If a page ranks on Google, is it automatically readable by ChatGPT? No. Googlebot renders JavaScript to index a client-rendered page; the AI-search crawlers do not run JavaScript, so the same page can be a strong Google result and empty to them.
• Do I need an llms.txt file? No. Google says Google Search does not use it, so it neither helps nor harms your Google visibility, and no other major engine documents using it for inclusion. Most published llms.txt files get no requests at all.
• Will server-side rendering actually fix this? For the raw-HTML gap, yes. Server-rendering or pre-rendering puts the content and the JSON-LD into the HTML the crawlers download, which is the whole point.
• Does passing these checks guarantee I get cited? No. Being readable is the floor, not the finish line. It makes you eligible to be pulled into an answer; whether you actually get cited is a separate question you have to measure.