Experiment: Do AI Crawlers Execute JavaScript?
Can we detect whether AI search crawlers render CSS and execute JavaScript — or do they only parse raw HTML?
Hypothesis
Most AI search crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) do not execute JavaScript when indexing pages — they parse raw HTML only, similar to Googlebot's first crawl wave. If this is true, only the HTML probe will fire for AI bots. If a bot executes JS, both the JS probes will fire, which is a significant security observation — JS execution means the crawler is a full browser engine, dramatically widening the attack surface.
Why JS Execution Matters for Security
A crawler that executes JavaScript can:
- Be redirected via
window.locationto internal/private URLs (SSRF-like) - Execute timing attacks against third-party resources
- Trigger
fetch()calls to arbitrary endpoints the page author controls - Leak information via
postMessage,navigatorAPIs, or storage access - Be fingerprinted — we can determine browser engine, screen size, installed fonts
Any confirmed JS execution by an AI crawler will be documented and considered for VRP submission to the relevant programme (Google VRP, OpenAI Bug Bounty, Anthropic VRP, Perplexity Security).
Probe Design
This page fires four distinct beacon requests to /api/ping depending on what the
visiting agent can do. Each request is logged server-side with the full User-Agent header.
| Probe | Mechanism | What it detects | Requires JS? |
|---|---|---|---|
html-img | <img src="/api/ping?probe=html-img"> | HTML parsed + subresources fetched | No |
css-bg | CSS background-image: url(...) on a visible element | CSS parsed and applied | No |
js-fetch | fetch() inside <script> on DOMContentLoaded | JavaScript executed, Fetch API available | Yes |
js-xhr | XMLHttpRequest inside <script> on DOMContentLoaded | JavaScript executed, XHR available | Yes |
Interpretation Matrix
| html-img | css-bg | js-fetch / js-xhr | Conclusion |
|---|---|---|---|
| ✅ | ❌ | ❌ | HTML-only parser — no CSS engine, no JS |
| ✅ | ✅ | ❌ | Full CSS rendering, no JS execution |
| ✅ | ✅ | ✅ | Full browser engine — JS executes. VRP candidate. |
| ❌ | ❌ | ❌ | Bot fetched the page but no subresources (no-subresource policy) |
How to Read the Results
Results are logged to Cloudflare Pages → Functions → Real-time logs.
Each log line is a JSON object: {"probe":"html-img","ua":"GPTBot/1.0 ...","t":"..."}.
We filter by page=exp09 and group by ua to build the matrix above.
Current Results
Waiting for AI crawlers to index this page. Check back after first GSC indexing signal.
| Bot | html-img | css-bg | js-fetch | js-xhr | Date |
|---|---|---|---|---|---|
| Googlebot | — | — | — | — | Not yet seen |
| Google-Extended | — | — | — | — | Not yet seen |
| GPTBot | — | — | — | — | Not yet seen |
| ClaudeBot | — | — | — | — | Not yet seen |
| PerplexityBot | — | — | — | — | Not yet seen |
| anthropic-ai | — | — | — | — | Not yet seen |
| CCBot (Common Crawl) | — | — | — | — | Not yet seen |