Comparison
Do AI Crawlers Render JavaScript? What GPTBot Actually Sees
· 5 min read
On this page
Short answer: no. As of mid-2026, none of the major AI crawlers execute JavaScript — not GPTBot, not ClaudeBot, not PerplexityBot, not Meta’s or ByteDance’s crawlers. They fetch your HTML, read what’s in it, and move on. Whatever your JavaScript would have rendered into the page never existed as far as they’re concerned.
Google spent a decade teaching the industry that client-side rendering is fine because Googlebot renders pages in a headless Chromium. That reassurance quietly stopped generalizing: an entire second audience of crawlers now feeds AI search and AI training, and that audience reads view-source, not the DOM. This post is about seeing the difference concretely — what an AI crawler gets from a static site versus a client-rendered one — and deciding how much you should care.
Who’s crawling, and what they skip
The user agents worth knowing fall into three groups: training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider), search-index crawlers for AI answer engines (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and live fetchers that grab a page because a user asked a question right now (ChatGPT-User, Claude-User, Perplexity-User). Different purposes, same relevant behavior: none of the declared crawlers, at least, run your scripts.
They do download JavaScript files, which trips people up when reading server logs. A crawl
analysis published by Vercel and MERJ measured it: GPTBot spent about 11.5% of its fetches on
JavaScript files that it never executes. Crawlers collect the files — presumably as training
data or for link extraction — but no rendering happens. Seeing main.js in your access log
under GPTBot’s user agent is not evidence that your React app got rendered.
Googlebot remains the exception — it still renders, and Google’s AI features inherit that
index. Everyone else sees your site the way curl does.
The two-minute test
You don’t need to speculate about any of this; the whole question is checkable from a terminal. Fetch a page the way a crawler does and look for your content in what comes back:
curl -s -A "GPTBot" https://astro.p4ni.com/blog/cloudflare-pages-vs-workers/ \
| grep -ci "gradual deployments"On this site that returns matches, because every article is statically pre-rendered: the full text is in the HTML response, headings, code blocks and all. A crawler that reads raw HTML gets the entire article in one request.
Now run the same test against a client-rendered app and the response is a shell:
<!doctype html>
<html>
<head>
<script type="module" src="/assets/index-Bka92mQd.js"></script>
</head>
<body>
<div id="root"></div>
</body>
</html>That <div id="root"></div> is the entire “content” a non-rendering crawler sees — no
product descriptions, no documentation, no blog posts. For Googlebot the app renders and gets
indexed; for the AI crawlers the site effectively consists of a title tag. Try it on your own
site with JavaScript disabled in the browser, or curl | grep for a phrase from your most
important page. If the phrase isn’t in the response, AI search can’t quote you.
Static HTML vs client-side rendering, from a crawler’s seat
| Static / server-rendered | Client-side rendered | |
|---|---|---|
| Googlebot | Full content | Full content (after render queue) |
| GPTBot / ClaudeBot / PerplexityBot | Full content | Empty shell |
| ChatGPT-User (live fetch) | Full content | Empty shell |
| Cost to the crawler | One request | One request that yields nothing |
The asymmetry is the point: static HTML serves both audiences with zero extra effort, while CSR serves one audience and silently drops the other. When I compared what Astro and Next.js ship by default, the numbers were 645 bytes of JavaScript against 642 kB — but the AI-crawler angle turns that from a performance argument into a visibility one. It’s not just that the static page is lighter; it’s that the content exists without any of that JavaScript running.
To be precise about frameworks: this isn’t “JavaScript frameworks are invisible”. A Next.js site with SSR or static export serves full HTML and does fine. Astro’s islands hydrate interactive components on top of complete HTML, so the content is crawler-visible either way. The line isn’t which framework you chose — it’s whether your content is in the initial response or assembled afterward in a browser the crawler doesn’t have.
Does it actually matter yet?
An honest cost-benefit, because “AI search is the future” is doing a lot of unearned work in 2026 marketing copy.
The case for caring: AI assistants increasingly answer questions with citations, those citations send real (if modest) referral traffic, and being uncitable means being absent from however large that channel becomes. The interest runs both ways — tooling vendors are now detecting AI agents at the dev-server level, and crawler traffic keeps growing in every published measurement. Content sites — blogs, docs, product pages — are exactly what gets quoted in AI answers, and they’re also the sites where static rendering costs nothing to adopt.
The case for calm: AI referrals are still a fraction of search referrals for most sites, and a working product with a CSR frontend doesn’t need an emergency rewrite because of a crawler that can’t see the settings page. Interactive app surfaces behind a login were never going to be crawled meaningfully by anyone.
Where that lands: if your public content is client-rendered, that’s now a real gap; if your app is, it isn’t. For sites already static, the marginal work is zero — you’re visible to this audience by construction.
If you’re set up right, go one step further
For a static site the interesting question stops being “can they read it” and becomes “how easy am I making it”. Two low-effort additions:
- llms.txt — a Markdown index of your site for AI consumers, generated from the same content collections as the pages. Adoption by the crawlers is still an open question (I’ve written about exactly who fetches it), but it costs a build-time endpoint.
- Structured data — JSON-LD sits in the initial HTML, so the same crawlers that skip your scripts do get your metadata: authorship, dates, what kind of page this is. The machine-readable layer works precisely because it doesn’t depend on rendering.
Both follow from the same principle that decides the rendering question itself: assume the reader of your HTML is a program that will not run your code. Googlebot spent years being the forgiving exception. The new crawlers make the strict interpretation the norm again — and a static site, whatever else you think of the architecture, is the one setup that never had to care about the difference.