How to check if AI can access your website

How to check if ChatGPT and other AI can access your website

1. Enter the page people would ask about — the homepage, or a product or article page. robots.txt rules are site-wide, but noindex, snippet rules and JavaScript rendering are per page, so check the page that matters.
2. Read the summary line first. “AI systems can access this page” means nothing found. Otherwise it counts what may keep AI out, and each item below explains itself.
3. Look at the crawler verdicts by role. A blocked search crawler is the one that costs you AI answers. A blocked training crawler is a policy choice and is not flagged as a problem. The difference is the most common misreading of these reports.
4. Then the page-level checks. Indexing, snippets, content in the HTML. A page can pass every crawler rule and still be invisible because of one of these.
5. Fix and re-check. The checker reads the live site, so a second run confirms the change. Greadme Site Audit runs the same page-level checks on every page of the site.

How to check manually, without a tool

The four checks are simple enough to do by hand. This is exactly what the checker automates.

1. robots.txt — is each AI crawler allowed?

Open https://your-site.com/robots.txt. Look for a User-agent: group naming the crawler (Google's specification: “Both the user-agent field name and its value are case-insensitive”). A crawler follows “the group with the most specific user agent that matches”; with no group of its own it falls to the * group. So a site with User-agent: * / Disallow: / and no exception blocks every AI crawler, and User-agent: OAI-SearchBot / Disallow: / blocks exactly one. Check the tokens one by one; allowing Googlebot says nothing about OAI-SearchBot.

2. noindex — is the page allowed in results at all?

View the page source and search for name="robots"; a noindex in that tag keeps the page out of search results, and unindexed pages do not appear in AI answers either. The same rule can arrive as an HTTP header the source never shows: run curl -I https://your-site.com/page and look for X-Robots-Tag: noindex.

3. Snippets — may the content be quoted?

In the same robots meta tag (or header), look for nosnippet or max-snippet:0. Google documents that nosnippet “will also prevent the content from being used as a direct input for AI Overviews and AI Mode”, and that max-snippet “will also limit how much of the content may be used as a direct input for AI Overviews and AI Mode”. The page stays indexed and looks fine.

4. Content — is it in the HTML the server sends?

View the source (not the DevTools Elements panel, which shows the page after JavaScript ran). If the body is an empty <div id="root"> and the text you see in the browser is not in the source, the page is blank to a crawler that does not execute JavaScript. Chrome DevTools can also disable JavaScript to show what such a crawler gets.

5. llms.txt — do you publish one?

Open https://your-site.com/llms.txt. A plain-text file starting with a # heading is one; an HTML page or a 404 is not. See the FAQ on whether you need it.

What each verdict means

The crawlers checked, and why the difference matters

These crawlers do three different jobs, and only two of them affect whether AI systems can answer questions about you. OpenAI documents the split itself: OAI-SearchBot “is used to surface websites in search results in ChatGPT's search features”, while GPTBot “is used to crawl content that may be used in training our generative AI foundation models”. The third kind, ChatGPT-User, fetches a page when a person asks about it, and “because these actions are initiated by a user, robots.txt rules may not apply”.

Is my site visible to ChatGPT? Access is step one

This checker answers “can AI read the page?”. It cannot answer “does ChatGPT mention me?” — that depends on what the assistants have indexed, how you compare with other sources, and the question asked. Greadme measures the second part separately: the AI Visibility Checker asks ChatGPT, Claude and Gemini real questions about your topic and reports whether, and where, your domain appears. Check access here first; a blocked crawler makes the visibility result meaningless.

The four ways a healthy-looking site becomes invisible to AI

1. A robots.txt rule you inherited

Most blocked AI crawlers were never a decision. They arrive with a template, a security plugin's “block AI scrapers” toggle, or a CDN preset — and because the site keeps ranking in Google, nothing looks wrong. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot does not hurt your Google ranking at all. It just removes you from the assistants people are increasingly asking instead.

2. A noindex left behind

noindex is the standard setting on a staging site, and it survives the launch more often than anyone expects. The checker reads both the meta tag and the X-Robots-Tag response header, because a header-level rule is invisible in the page source.

3. A snippet restriction nobody remembers adding

A page can be indexed, ranking, and still structurally excluded from the AI answer above the results by a nosnippet or max-snippet:0 rule. This is the one worth knowing about, because nothing else looks wrong.

4. Content that only exists after JavaScript runs

AI crawlers read the HTML your server returns. They do not run your JavaScript. If the response is an empty shell and everything real arrives after hydration, the page is blank to them while looking perfect in your browser. Server-side rendering or pre-rendering fixes it.

Questions people ask

Written by the Greadme team, who build the checker. Last updated 2026-09-03; the crawler list is the one the tool runs, and the quoted rules come from OpenAI's and Google's own documentation.