Why AI crawlerscannot readmost websites.
The most common reason a site earns zero AI citations has nothing to do with its content. It is that the crawler never sees the content at all.
Most websites are built as single-page applications that render their content with JavaScript in the browser. The crawlers behind AI answer engines — including GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot — largely do not execute JavaScript, so they fetch an almost-empty HTML shell and index nothing. The fix is to serve real, server-rendered or prerendered HTML, which is the single change that most often moves a site from zero AI citations.
Why does my content not show up for AI crawlers?
Many modern sites ship a nearly empty HTML document — often little more than a single <div id="root"></div> — and then build the visible page with JavaScript once it loads in a browser. A human sees a full page; a crawler that does not run JavaScript sees the empty document. If the content is not in the HTML, an answer engine has nothing to read and nothing to cite.
Do AI crawlers run JavaScript?
Mostly, no. Google’s primary crawler renders JavaScript, which is why a JavaScript-only site can still rank in Google. But the crawlers that feed the AI answer layer generally do not execute JavaScript. That gap is why a page can rank first in Google and still never be quoted by ChatGPT or Perplexity — one crawler sees the content and the other sees a blank shell.
How do I check what a crawler sees?
Use “View Source” in the browser rather than the element inspector — the inspector shows the page after JavaScript has run, which hides the problem. If the raw source has real headings and body copy, a crawler can read it; if it is mostly empty with a bundle of scripts, it cannot. Fetching the page with a plain request, without a browser, shows the same truth.
How do you fix it: SSR, SSG and prerendering
The fix is to make sure real HTML is served. Three approaches do this:
- Server-side rendering (SSR)
- The server builds the full HTML for each request, so every visitor and crawler receives a complete page.
- Static site generation (SSG)
- Pages are built to HTML ahead of time and served as static files — fast and fully readable.
- Prerendering
- After a normal build, each route is rendered to static HTML once, so crawlers get real content while the app still boots for users.
What else blocks AI crawlers?
Rendering is the biggest cause, but not the only one:
- robots.txt
- A disallow rule — or simply not allowing the AI bots — keeps them out entirely.
- Bot blocking
- A CDN or firewall rule that blocks unknown user-agents can silently exclude answer-engine crawlers.
- Content behind interaction
- Anything that only appears after a click, scroll or login is invisible to a crawler.
- Slow or fragile pages
- Timeouts and heavy client-side work reduce how much a crawler can retrieve.
How Blackwave Digital fixes rendering
Blackwave Digital audits a site exactly as an AI crawler fetches it, then fixes the floor: prerendering or server-side rendering so every page ships real HTML, an open crawl policy for the answer-engine bots, and structured data on top. This site itself is prerendered for the same reason — until an engine can read the page, no amount of on-page copy will earn a citation.
- Most sites render content with JavaScript, but most AI crawlers do not run it.
- A site can rank in Google and still be invisible to ChatGPT and Perplexity.
- Check with “View Source,” not the element inspector, to see what a crawler really gets.
- SSR, SSG or prerendering ships real HTML — the fix that most often unlocks AI citations.
Frequently asked.
Does Googlebot run JavaScript?
Yes, Google’s crawler renders JavaScript, which is why a client-rendered site can still rank in Google. The problem is that most AI answer-engine crawlers do not render JavaScript, so a site can rank in Google yet be unreadable to ChatGPT or Perplexity.
Is prerendering the same as server-side rendering?
They are related but not identical. Server-side rendering builds HTML for each request at run time, while prerendering renders each route to static HTML once at build time. Both ensure a crawler receives real content instead of an empty shell.
How do I know if my site has this problem?
Open the page and choose “View Source.” If you can see your real headings and paragraphs in the raw HTML, crawlers can read them. If the source is mostly empty with a set of script tags, your content depends on JavaScript and AI crawlers likely cannot see it.
Want this built for you?
Blackwave Digital engineers the rendering, schema and answer content these guides describe. Send a brief and we'll show you where the leverage is.