Experiment: Do AI Crawlers Execute JavaScript?

Hypothesis

Most AI search crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) do not execute JavaScript when indexing pages — they parse raw HTML only, similar to Googlebot's first crawl wave. If this is true, only the HTML probe will fire for AI bots. If a bot executes JS, both the JS probes will fire, which is a significant security observation — JS execution means the crawler is a full browser engine, dramatically widening the attack surface.

Why JS Execution Matters for Security

A crawler that executes JavaScript can:

• Be redirected via window.location to internal/private URLs (SSRF-like)
• Execute timing attacks against third-party resources
• Trigger fetch() calls to arbitrary endpoints the page author controls
• Leak information via postMessage, navigator APIs, or storage access
• Be fingerprinted — we can determine browser engine, screen size, installed fonts

Any confirmed JS execution by an AI crawler will be documented and considered for VRP submission to the relevant programme (Google VRP, OpenAI Bug Bounty, Anthropic VRP, Perplexity Security).

Probe Design

This page fires four distinct beacon requests to /api/ping depending on what the visiting agent can do. Each request is logged server-side with the full User-Agent header.

Probe	Mechanism	What it detects	Requires JS?
html-img	<img src="/api/ping?probe=html-img">	HTML parsed + subresources fetched	No
css-bg	CSS background-image: url(...) on a visible element	CSS parsed and applied	No
js-fetch	fetch() inside <script> on DOMContentLoaded	JavaScript executed, Fetch API available	Yes
js-xhr	XMLHttpRequest inside <script> on DOMContentLoaded	JavaScript executed, XHR available	Yes

Interpretation Matrix

html-img	css-bg	js-fetch / js-xhr	Conclusion
✅	❌	❌	HTML-only parser — no CSS engine, no JS
✅	✅	❌	Full CSS rendering, no JS execution
✅	✅	✅	Full browser engine — JS executes. VRP candidate.
❌	❌	❌	Bot fetched the page but no subresources (no-subresource policy)

How to Read the Results

Results are persisted to Cloudflare KV and displayed on the live results dashboard. Each probe hit is logged with the full User-Agent, Cloudflare bot score, ASN, and country.

Current Results

Data collected since 2026-04-14. Four distinct capability tiers confirmed across 7 crawlers. View full live dashboard →

Bot	html-img	css-bg	js-fetch	js-xhr	Infrastructure	Date
Googlebot (natural crawl)	❌	❌	✅	✅	Google LLC (ASN 15169), US — executes JS, skips invisible subresources	2026-04-16
Google-InspectionTool	✅	✅	✅	✅	Google LLC (ASN 15169), US — full Chromium, mobile+desktop	2026-04-14
GPTBot	✅	✅	❌	❌	Microsoft Azure (ASN 8075), US — headless browser, JS disabled	2026-04-15
MetaBot	✅	❌	❌	❌	Meta (ASN 32934), US — HTML parser only	2026-04-15
AhrefsBot	✅	❌	❌	❌	Ahrefs (ASN 16276), CA — HTML parser only	2026-04-15
ClaudeBot	✅	❌	❌	❌	AWS (ASN 16509), US — HTML parser only	2026-04-14
PerplexityBot	—	—	—	—	—	Not yet seen
CCBot (Common Crawl)	—	—	—	—	—	Not yet seen

Key Findings

• Googlebot's production renderer executes JS but skips invisible subresources — natural Googlebot crawl fired fetch() and XMLHttpRequest but did NOT load the 1×1px <img> or CSS background-image probes. This means Google's Web Rendering Service is smart enough to suppress loading invisible/tracking-pixel resources while still executing page scripts. This is a meaningful difference from the testing tools (Inspection Tool, Rich Results Test) which load everything.
• Google's testing tools run full Chromium — both the Rich Results Test and URL Inspection Tool fire all 4 probes. They send a plain Chrome UA with no "Googlebot" identifier — only detectable by ASN (15169). Render twice per page (mobile + desktop).
• GPTBot has a CSS engine but no JS — processes background-image: url() but does not execute scripts. OpenAI deliberately disabled JavaScript in their crawler — a security-conscious design choice. Runs on Microsoft Azure infrastructure (ASN 8075).
• ClaudeBot, MetaBot, AhrefsBot are HTML-only — no CSS rendering, no JS execution. Smallest possible attack surface.
• Cloudflare Bot Management scores ALL crawlers as 99 (human) — verifiedBot: false for every bot tested. Cloudflare's paid bot detection product does not flag any of these crawlers.