Rendering, speed and access
Serve your main content as HTML from the server. Studies found that major AI crawlers such as GPTBot and ClaudeBot fetch but do not execute JavaScript, while Google renders it. Keep pages fast (Core Web Vitals), make sure your CDN or firewall isn't blocking wanted bots, and use semantic, accessible markup that people, crawlers and AI agents can all parse.
Rendering
| Crawler | Executes JavaScript? | Source |
|---|---|---|
| Googlebot (and Gemini) | Yes, renders pages | [1][2] |
| Applebot | Yes (browser-based) | [2] |
| GPTBot, ClaudeBot | No: fetch JS files but don’t run them | [2] |
| CCBot | No | [2] |
Vercel and MERJ reported in December 2024 that “none of the major AI crawlers currently render JavaScript,” apart from Google’s and Apple’s.[2] Independent study Content that appears only after client-side rendering may be invisible to them. Use server-side rendering, static generation or pre-rendering for text, links, titles and structured data.
Speed
Google’s Core Web Vitals targets: LCP within 2.5 seconds, INP of 200 ms or less, CLS of 0.1 or less.[3] Page experience is part of Google’s ranking systems,[4] and fast, lightweight HTML is cheaper for every crawler to fetch. Vendor-documented
Access
- Check robots.txt and CDN/WAF rules (Cloudflare blocks AI crawlers by default on new domains since July 2025).[5] See AI crawlers.
- Return correct status codes; avoid soft 404s and login walls on public content.
- Submit sitemaps and use IndexNow for Bing.[6][7]
- Snippet controls limit AI use:
nosnippet,data-nosnippetandmax-snippetin Google;[8]NOCACHEandNOARCHIVEin Bing.[6] Don’t set them by accident.
Agent-friendly markup
AI agents that browse on a user’s behalf read the page structure. OpenAI says its Atlas agent uses ARIA labels and roles,[9] and Google’s web.dev guidance on agent-friendly sites recommends semantic HTML and accessible forms.[10] Real <button> and <label> elements help both screen readers and agents.
Frequently asked questions
Does ChatGPT see JavaScript content?
Vercel's 2024 analysis found OpenAI's and Anthropic's crawlers fetch JavaScript files but don't execute them. Server-render important content.
Is page speed an AEO factor?
No engine documents speed as an AI citation factor, but Google uses page experience in ranking, and ranking drives retrieval.
See also
References
Pages accessed September 28, 2026 unless a date is given. See all sources and our editorial policy.
- ^ "Understand JavaScript SEO basics". Google Search Central.
- ^ "The rise of the AI crawler". Vercel (with MERJ). Published December 17, 2024..
- ^ "Web Vitals". web.dev (Google Chrome team).
- ^ "Understanding page experience in Google Search results". Google Search Central.
- ^ "Content Independence Day: no AI crawl without compensation!". Cloudflare Blog. Published July 1, 2025.
- ^ "Bing Webmaster Guidelines". Microsoft Bing Webmaster Tools.
- ^ "IndexNow protocol". IndexNow.org.
- ^ "Robots meta tag, data-nosnippet, and X-Robots-Tag specifications". Google Search Central.
- ^ "Publishers and Developers – FAQ". OpenAI Help Center.
- ^ "Build agent-friendly websites". web.dev (Google Chrome team).