--- title: "Is Your Site Agent-Ready?" description: "AI search crawlers download your HTML but do not run your JavaScript, so client-rendered pages read as empty to them even when they rank on Google. What agent-readiness means, the two raw-HTML failures ClarAI's site audit looks for, and how to check your own site." published: 2026-08-14 reading_minutes: 7 tags: ["Agents", "How AI search works"] canonical: https://myclarai.com/blog/is-your-site-agent-ready source: ClarAI blog --- # Is Your Site Agent-Ready? _AI search crawlers download your HTML but do not run your JavaScript, so client-rendered pages read as empty to them even when they rank on Google. What agent-readiness means, the two raw-HTML failures ClarAI's site audit looks for, and how to check your own site._ Agent-ready means the content and structured data a page depends on are already present in the raw HTML your server sends, before any JavaScript runs. This matters because the crawlers behind AI answers, GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot, download your HTML but do not execute your JavaScript. A page can rank perfectly well on Google, which does render JavaScript, and still read as a blank shell to ChatGPT, Perplexity and Claude. The fastest way to know where you stand is to look at your pages the way a no-JavaScript fetch sees them. This post covers what agent-readiness actually means, the one fact that decides it, the two failure modes ClarAI's site audit is built to catch, the lighter-weight signals worth checking, and how to audit your own site in a few minutes. ## The one fact that decides it The major AI crawlers fetch a page's HTML and parse it, but they do not run the client-side JavaScript that a browser would. Googlebot is different: it renders JavaScript, which is why a client-rendered single-page app can index and rank in ordinary Search while being invisible to the AI-search crawlers that never run the script. So the same page can be a first-page Google result and, at the same time, an empty container to the engines assembling AI answers. That gap is the whole of agent-readiness. If the words and the structured data only appear after the browser executes your JavaScript, an AI crawler never sees them. > To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet. > > Google Search Central, "Optimizing your website for generative AI features on Google Search" (2026) Google's own eligibility bar is deliberately boring: indexed, crawlable, snippet-eligible. Googlebot renders your JavaScript to get there, so Google can forgive a client-rendered page. The AI-search crawlers are stricter, because they stop at the raw HTML. Agent-readiness is simply meeting the boring bar in a form that survives without a browser: the content in the HTML, not assembled after the fact. ## The two failures ClarAI's site audit looks for ClarAI's Site Audit crawls your pages the way an AI crawler would, with a no-JavaScript fetch, and raises two conservative findings against exactly this gap. Both are deliberately cautious: they only fire on the raw fetch view, and a page recovered through a full browser fallback is skipped entirely, because a browser-rendered page cannot tell us what a JavaScript-blind crawler would have received. - Content not in raw HTML (high severity). Fires only when the raw fetched HTML is an empty shell on all four signals at once: near-zero visible text, no server-rendered headings, no structured data, and no images. That stacked combination on a healthy page is an unrendered single-page-app shell, which is what an AI crawler receives. The fix is to server-render or pre-render the main content so it is in the HTML the crawler downloads. - Structured data not in raw HTML (medium severity). Fires on a real, server-rendered content page, one with genuine body text and a heading, that ships no Schema.org JSON-LD in its raw HTML. The usual cause is structured data injected client-side by a tag manager or an app script, so the page reads fine to a person and to Googlebot but the schema is invisible to AI crawlers. The fix is to emit the JSON-LD server-side so it is present in the raw HTML. The two are complementary. The first catches the page that has nothing an AI crawler can read. The second catches the page a person would call complete, where only the machine-readable layer went missing because it was bolted on after the browser loaded. ## The lighter agent-readiness signals Beyond the raw-HTML question, ClarAI's Tech SEO Audit surfaces four first-version agent-readiness heuristics. These are early signals, reported as pass, warn or unknown rather than hard pass-or-fail, because the practices around agent access are still forming. They are worth knowing, and worth keeping honest about what they do and do not buy you. - Semantic landmarks. Whether your pages expose the HTML5 landmark tags (main, nav, header, footer, article). The check passes when at least 70 percent of sampled pages carry three or more of the five. Programmatic agents parse the DOM and the accessibility tree, so explicit landmarks make a page easier to navigate by machine. - An explicit robots policy for AI and agent user-agents. Silence is treated as a warning, never a failure: addressing the known AI crawlers by name, allow or disallow, is emerging best practice rather than a requirement. - llms.txt. Reported honestly in both directions: Google explicitly ignores llms.txt, so its absence costs nothing with Google's AI features, while some other AI tools do read it. Publishing one is an optional, low-cost signal, not a lever that unlocks anything. - Markdown content negotiation. Whether your server answers a request for Accept: text/markdown with actual markdown. It is an emerging agent-readiness signal that very few sites currently support, so serving HTML here is a warning, not a fault. ## How to check your own site - View source, or fetch the page with curl, on your money pages and read the body. If the visible text, your headings and your JSON-LD are all there in the delivered HTML, an AI crawler can read them. If the HTML is a near-empty div and everything arrives later, it cannot. - Disable JavaScript in your browser and reload a key page. What remains is roughly what a JavaScript-blind crawler sees. A page that goes blank is the content-not-in-raw-HTML case. - Search the raw HTML for a script type of application/ld+json. If your structured data is missing from the source but present in the rendered page, it is being injected client-side, which is the structured-data-not-in-raw-HTML case. - Run the free ClarAI grader at /grade. It checks the raw-HTML-facing signals alongside the heavier ones: semantic landmarks carry 6 of the 100 points, markdown content negotiation 5, and llms.txt 4, sitting under the much heavier snippet-eligibility, Googlebot and AI-crawler checks. No signup needed. One honest note about us, in the spirit of this blog. ClarAI's own public marketing pages are a client-rendered app, and full pre-rendering of them is still on our list. We have shipped the robots.txt that allows the same AI crawlers this audit checks, a sitemap, and a rich server-rendered HTML head, and pre-rendering the rest of the marketing routes is the next fix we owe our own site. Agent-readiness is a bar most sites, ours included, are still climbing toward. ## Quick answers to the common questions - If a page ranks on Google, is it automatically readable by ChatGPT? No. Googlebot renders JavaScript to index a client-rendered page; the AI-search crawlers do not run JavaScript, so the same page can be a strong Google result and empty to them. - Do I need an llms.txt file? No. Google ignores it, so it neither helps nor harms your Google AI visibility. Some other AI tools read it, which makes it an optional low-cost extra, not a requirement. - Will server-side rendering actually fix this? For the raw-HTML gap, yes. Server-rendering or pre-rendering puts the content and the JSON-LD into the HTML the crawlers download, which is the whole point. - Does passing these checks guarantee I get cited? No. Being readable is the floor, not the finish line. It makes you eligible to be pulled into an answer; whether you actually get cited is a separate question you have to measure. **See your site the way an AI crawler sees it** Run the free ClarAI grader on your domain. It reads your pages without running JavaScript and shows you the agent-readiness gaps, with fix hints and no signup required. [Grade your site](/grade) [Start free](/register)