You can write the best page on the internet and it will not be cited by an AI assistant if the assistant's crawler cannot fetch it. That sounds obvious, yet it is one of the most common and least noticed problems on the web. The page loads fine for you. Your analytics look healthy. And somewhere between the crawler and your server, a rule you forgot about is turning visitors away.
By the iSuggest.ai Team ยท Updated for 2026
This guide gives you a repeatable way to check whether GPTBot and other AI crawlers can reach your site, in the order that finds problems fastest. You do not need to be a developer to do the first half. For the second half, you may need to ask your host or whoever manages your firewall. If you want the bigger picture on why crawler access matters, start with what GEO is, then come back.
Why access is the first thing to check
Access sits at the bottom of the stack. Content quality, structured data, entity clarity and mentions all matter, but none of them can help a page that is never fetched. A study reported by LLM Pulse found that over 35% of the top 1,000 websites block at least one major AI crawler. Some of those sites chose to. Many probably did not, and inherited a rule from a plugin, a template or a security setting.
It is also worth separating two ideas. Blocking an AI crawler can be a perfectly legitimate business decision. What we are interested in here is blocking by accident, where the owner would have said yes if they had been asked. The aim of this guide is to make the decision deliberate.
Know who might be visiting
AI companies run several crawlers, and they do not all do the same job. Broadly, a company may operate one crawler that gathers material for model training, another that supports search or answer features, and a third that fetches a page when a user asks an assistant to read it. The names and roles change, so read each provider's current documentation before you edit any rules. Our overview of the rise of AI crawlers lists the main players, including GPTBot, ClaudeBot and Google-Extended, and explains how they differ.
The distinction matters because a block aimed at training can sometimes also stop the crawlers that support live answers. Decide what you want for each purpose, then check your rules do exactly that.
Step 1: Read your robots.txt
Open https://yourdomain.com/robots.txt in a browser. If the file does not exist, crawlers will generally assume everything is allowed, so that is not a problem in itself. If it exists, read it from top to bottom and look for these things.
- Named blocks. Lines such as
User-agent: GPTBotfollowed byDisallow: /mean that crawler is shut out completely. - Wildcard blocks.
User-agent: *followed byDisallow: /blocks every crawler that obeys the file, including search engines. This is a common leftover from a staging site. - Broad path rules. A rule like
Disallow: /blogblocks every path that starts with those characters, which may include pages you want read. - Order and grouping. Crawlers choose the most specific matching group of rules, so a named group can override a wildcard group in ways that surprise people.
The syntax is unforgiving. A missing colon, a misspelled agent name or a stray space can change the meaning. Our post on robots.txt syntax mistakes walks through the patterns that cause accidental blocks.
A safe example
If you want AI crawlers to read your public pages but keep them out of a private area, a rule set might look like this. Treat it as an illustration and adapt the paths to your own site.
User-agent: *
Disallow: /admin/
Disallow: /cart/
User-agent: GPTBot
Allow: /
Disallow: /admin/If, instead, you want to opt out of a particular crawler, add a group naming it with Disallow: /. Either way, write down the intent in a comment so the next person knows it was deliberate.
Step 2: Test with a real request
robots.txt states a policy, but your server decides what actually happens. The next test is to request a page while identifying as the crawler and see what comes back. You can do this from a terminal with a command that sets a user agent. Use the exact user agent string from the provider's documentation, since the examples you find online may be out of date.
curl -I -A "GPTBot" https://yourdomain.com/Look at the status line. A 200 means the page was served. A 403 means forbidden, which usually points to a firewall or bot rule. A 429 means rate limited. A 503 may mean a bot-protection layer is intervening. A redirect to a challenge page is a warning sign too. Then run the same command without the custom user agent and compare. If a normal request works and the crawler request fails, you have found a user-agent-based rule.
This test has limits. A real crawler comes from published IP ranges, and some security tools check the IP as well as the name, so a request from your laptop that pretends to be a crawler may be treated differently from the real thing. Think of the test as a good first indicator, not a final verdict.
Step 3: Check your CDN and firewall
Many sites sit behind a content delivery network or a web application firewall. These services often include bot management that challenges or blocks unfamiliar automated traffic by default. The block happens before robots.txt is consulted, so a perfectly open robots.txt can sit behind a closed door.
Ask three questions, or ask your provider to answer them for you.
- Is bot protection or "AI bot blocking" switched on, and what is its default for unknown or AI user agents?
- Are there custom rules that match user agents or IP ranges used by AI crawlers?
- Do the logs show requests from the crawlers you care about, and what responses did they receive?
Some providers now offer one-click settings that block AI crawlers across a whole account. They are convenient, and easy to forget about. If you have never reviewed them, do it now.
Step 4: Read your server logs
Logs are the closest thing to ground truth. Search your access logs for the crawler names and look at the status codes. If you see GPTBot requests returning 200, it is getting in. If you see 403 or 429, something is rejecting it. If you see nothing at all, either the crawler has not visited, or a layer in front of your server is stopping the requests before they are logged.
Keep your expectations realistic. Absence of a visit does not prove a block. Crawlers choose what to fetch and when, and a small site may simply not have been visited yet. What you are looking for is evidence of rejection, which is actionable, rather than silence, which is ambiguous.
Step 5: Confirm the content is actually in the HTML
A crawler that is allowed in can still come away empty-handed. Many AI crawlers do not execute JavaScript. If your main text is injected by scripts after the page loads, the response may contain little more than a skeleton. Open the page source (not the inspected DOM) and search for a sentence from your main content. If you cannot find it, the crawler cannot either. We show the full check in what client-side rendering hides from AI crawlers.
Step 6: Check the small things that quietly block
A few other items are easy to overlook.
- Meta robots and headers. A
noindex,nosnippetor restrictiveX-Robots-Tagon a page can limit how it is used even when fetching is allowed. - Canonical tags. A canonical that points to another URL tells systems your page is a duplicate of it. See canonical tags and URL variants.
- Login walls and cookie gates. Content hidden behind a consent overlay or a sign-in is not readable by a crawler.
- Slow responses. Crawlers have time budgets, and a page that takes many seconds to respond may be skipped. Why page speed still matters covers the reasoning.
- Sitemap errors. A sitemap that lists broken or redirected URLs wastes crawler attention.
A quick decision table
Use these patterns to translate a finding into an action.
- robots.txt blocks the crawler and you want it in: remove or narrow the rule, then re-fetch the file to confirm.
- robots.txt is open but requests return 403 or 429: look at the firewall or CDN bot settings and adjust the rule that applies.
- Requests return 200 but the page text is missing: render the key content on the server.
- Everything works and you still do not want AI use: keep the block, and write the policy down so it stays deliberate.
A worked example
Imagine a small online shop. This is an illustration, not a case study. The owner has never touched robots.txt and assumes everything is open. A test request with a crawler user agent returns a 403. The shop's host turns out to have an account-wide bot-protection setting that challenges unfamiliar automated traffic. The robots.txt file was never the issue.
The owner reads the provider's documentation, decides that product and category pages should be readable and the cart and account areas should not, and adjusts the rule accordingly. A second test request now returns a 200 for the product page and a 403 for the cart. The owner adds a comment to robots.txt recording the policy and puts a quarterly reminder in the calendar to review it. Nothing has been promised about citations, but the pages can now at least be read.
Keep it from breaking again
Access problems come back because defaults change, plugins update and people edit rules for good reasons without thinking about AI crawlers. Add the access check to your regular routine: robots.txt, firewall settings and a test request, every quarter and after any major change to hosting or security. Our advice on cadence is in how often to re-audit your website.
Let a tool check the rest
Crawler access is one part of a bigger picture that also covers rendering, structured data, identity and performance. The free iSuggest.ai audit looks at those from the point of view of a machine reader, and you can read what an audit measures to see how each item is judged. Audits are free and not metered, so you can check as many pages as you like.
When your pages are reachable and in good shape, you can publish a factual snapshot to the iSuggest.ai AI directory so there is a crawlable record of what you offer; there are pages for ChatGPT, Claude and Gemini. Publishing spends credits, improves findability and does not guarantee citations. See how it works and run an audit on your homepage today, then sign in at your account page when you want to publish.