For years, a website had one kind of visitor: a person with a browser. Then came search crawlers, a second kind of visitor that read pages on behalf of a ranking system. Now there is a third. AI assistants fetch pages while a user waits for an answer, read several sources, compare them and summarise the result. Some go further and try to carry out a task, such as finding a supplier, checking opening hours or comparing prices.
By the iSuggest.ai Team ยท Updated for 2026
This post is a practical readiness guide for that third visitor. We use the word "agent" loosely, for any AI system that retrieves and uses your pages as part of doing something for a user. The details of how each assistant works change quickly and are not fully public, so we focus on the things that are true of almost all of them: they need to reach your pages, read your text, understand what you are, and trust what they find. If you want the broader background first, start with what GEO is.
Why agents change what "ready" means
A human visitor is forgiving. They scroll past a cookie banner, wait for a script to finish, squint at a confusing menu and work out what a page is about from the design. A machine visitor is not forgiving, and it often has a short time budget. If the text is not in the first response, if a wall of consent code gets in the way, or if the page does not say plainly what it offers, the agent moves on to a source that does.
The scale of the shift is worth keeping in proportion. Industry compilations report that AI referral traffic grew about 16x between 2024 and 2026, and that AI Overviews appear on about 48% of tracked queries (SEO Sherpa). Those are growth figures from a smaller base, not proof that assistants have replaced search. The point is that machine visitors are a growing audience, and the cost of making your site friendly to them is mostly the cost of making it clearer for everyone.
The five questions an agent silently asks
It helps to think of an agent arriving at your site with five questions. Each one maps to a set of fixes.
- Can I get in? Access and crawler permissions.
- Can I read it? Rendering and page structure.
- Do I know what this is? Identity and structured data.
- Can I find the fact I need? Clear, quotable, self-contained content.
- Should I trust it? Consistency, freshness and corroboration.
The sections below take them in turn.
1. Can I get in? Access
Access is the most common silent failure, and the easiest to fix. There are several layers.
robots.txt
Your robots.txt file tells crawlers what they may fetch. Many sites block AI crawlers on purpose, which is a legitimate choice, but many others block them by accident through an overly broad rule or a copied template. One report from LLM Pulse found that over 35% of the top 1,000 sites block at least one major AI crawler. If you have not looked at your file recently, look now. Our guides on robots.txt syntax mistakes and how to check if GPTBot can crawl your website walk through it, and the rise of AI crawlers explains who the main bots are.
Firewalls and bot protection
A permissive robots.txt does nothing if a firewall, content delivery network or bot-protection service challenges or blocks the request before it arrives. Check your security settings for rules that treat all non-browser traffic as hostile. If you decide to allow specific AI crawlers, allow them deliberately and consider verifying them by their published addresses rather than their name alone, since names can be faked.
Logins, paywalls and interstitials
Anything behind a login is invisible to an agent. So is content hidden by a full-screen prompt that needs a click to dismiss. If a page is meant to be found, make sure its core text is available without signing in and without interacting.
Errors and redirects
Agents hit broken links, redirect chains and server errors like any crawler, and they tend to give up faster. Keep important URLs returning a clean 200 response, keep redirects to a single hop, and make sure your canonical tags point where you intend. See canonical tags and the URL variants AI reads.
2. Can I read it? Rendering and structure
Many AI crawlers fetch the raw HTML and do not run JavaScript. If your page is a near-empty shell that fills in after scripts load, such an agent may see almost nothing. This is one of the most consequential and least visible problems, because the page looks perfect to you.
A quick test is to fetch the page without a browser and read what comes back. If your main text, headings, prices and contact details are in that response, you are in good shape. If they are not, consider server-side rendering or pre-rendering for the pages that matter. Our post on what client-side rendering hides from AI crawlers shows how to check.
Structure matters too. Use one clear H1, logical headings, real lists and real tables. Avoid putting important information only in images, canvas elements or text rendered inside a graphic, because a text-reading agent cannot use it. If a price table is a screenshot, an agent cannot quote it.
Speed is part of readability. Agents work against timeouts, and slow pages may be abandoned. We cover this in why page speed still matters for AI search.
3. Do I know what this is? Identity and structured data
An agent that cannot tell what your site is will hesitate to recommend it. Make it obvious.
- Say it in words. Your homepage and about page should state, in plain sentences, who you are, what you offer, who it is for and where you operate.
- Add structured data. Organization or LocalBusiness markup, plus Product, Article or FAQ markup where it matches the page. Structured data gives a machine-readable version of facts already on the page. It must match visible content, and it does not by itself guarantee any result. See structured data for AI search engines.
- Use one name. Use the same business name, address, phone number and description everywhere. Variants split the signal. This is the subject of entity clarity.
- Show real authorship. Named authors, an about page and a contact route help a system judge who stands behind the content.
4. Can I find the fact I need? Content an agent can use
Agents are often looking for a specific piece of information: a price, an opening time, a policy, a specification, a yes or no. Make those facts easy to find and easy to lift.
- Answer first. Put the direct answer at the top of the relevant section, then explain.
- Make passages self-contained. A paragraph should make sense without the paragraphs around it, because a system may lift only that paragraph.
- State facts plainly. "Delivery takes two to four working days" is usable. "Fast delivery" is not.
- Use question headings. Headings phrased as the questions people ask match how assistants are prompted. A short FAQ section can help; see how to write an FAQ section for AI search.
- Put key facts in text. Hours, prices, terms and specifications belong in the page body, not only in a downloadable PDF or an image.
For more on writing in a way that is easy to cite, see how to write content AI models want to quote.
5. Should I trust it? Consistency, freshness and corroboration
Agents, and the systems behind them, are built to be wary of unreliable sources. You can help them trust you.
- Stay consistent. If your site, your profiles and your directory listings disagree about your hours or your prices, an agent has reason to doubt all of them.
- Keep content current. Show real dates, update pages when facts change and remove what is out of date. See how often to re-audit your site.
- Be discussed elsewhere. One analysis reports that brand mentions correlate with AI search visibility about three times more strongly than backlinks, 0.664 versus 0.218 (Media Officers). That is a correlation from one study, but it supports the idea that independent descriptions help. See brand mentions versus backlinks.
- Do not overclaim. Statements an agent can check against other sources should be accurate. Exaggeration, if caught, costs trust.
When agents try to act, not just read
Some assistants can go beyond reading and attempt actions such as filling a form, requesting a quote or comparing options across sites. This area is changing quickly and capabilities vary, so avoid building a strategy on assumptions about any single product. A few durable habits still apply.
- Make forms simple and standard. Use real form elements with clear labels, sensible field names and visible error messages. A form that depends on custom scripts and unlabeled fields is hard for any automated user.
- Keep key actions reachable by link. A booking page, a contact page and a pricing page should each have a stable, descriptive URL that does not require a session to reach.
- Offer a plain-text path. A visible email address or phone number is a fallback that works for people and machines alike.
- Be careful with sensitive flows. Payment and account actions should keep their normal protections. Do not weaken security for the sake of machine access.
- Consider a machine-readable summary. Some sites publish an llms.txt file that points to their most useful pages. It is a proposal, not a standard, and support is uneven, so treat it as optional. Our post on llms.txt explains the trade-offs.
A worked example
Imagine a small company that rents out meeting rooms. This is an illustration, not a case study. A user asks an assistant to find a room for twelve people next Thursday in a particular city and to compare prices.
The company's site looks polished, but it has four problems. The price list is an image, so an agent cannot read it. The room page is built with a script that loads capacity and amenities after the page appears, so the raw HTML is nearly empty. The firewall challenges unfamiliar automated requests. And the business is listed with two slightly different names on two directories.
The fixes are modest. Replace the price image with a simple table in the page text. Render the room details on the server so they are in the first response. Review the firewall rules and decide deliberately which AI crawlers to allow. Standardise the name across the site and the directories. Add LocalBusiness markup that matches the visible address and hours. None of this guarantees that an assistant will recommend the company, but each step removes a reason for an agent to skip it.
A readiness checklist
Use this as a quick pass over your most important pages.
- robots.txt does not block the AI crawlers you want, and you have checked it recently.
- Firewall and bot protection rules do not silently block legitimate fetches.
- Core text, headings, prices and contact details are in the raw HTML response.
- No essential fact lives only in an image, a PDF or a script-generated widget.
- Pages return clean 200 responses with sensible canonical tags and short redirect chains.
- Your homepage states plainly who you are, what you offer and for whom.
- Structured data matches visible content.
- Name, address, phone and description are identical across your site and profiles.
- Key answers are stated first, in self-contained passages.
- Pages show real dates and are updated when facts change.
- Forms use standard elements and clear labels.
Our AI SEO checklist expands this into 25 checks you can work through in order.
Common mistakes
Assuming a good-looking site is a readable one
Visual polish and machine readability are unrelated. Test what a machine receives, not what you see.
Blocking everything or allowing everything
Both extremes are lazy. Decide which crawlers serve your goals and configure access on purpose.
Chasing a single assistant
The market is spreading. A PPC Land analysis reports ChatGPT's absolute traffic grew 27% year on year while Gemini grew 231% and Claude 320% (PPC Land). Fundamentals that serve all of them beat tricks for one.
Treating it as a one-off project
Sites change, rules change and assistants change. A site that was ready last year may have slipped. Re-check on a schedule.
Expecting guarantees
Nobody can promise that an agent will pick you. You can only remove the obstacles and give it good material.
Check your site now
You do not have to test each of these by hand. The free iSuggest.ai audit looks at access, rendering, structured data, headings and identity signals in one pass, so you can see which of the five questions your site fails. You can read what an audit measures for the full list. Audits are free and not metered, so run one on your homepage and again on the pages you most want machines to understand.
When the foundations are in place, you can publish a factual snapshot of your business to the iSuggest.ai AI directory, which has pages for ChatGPT, Claude and Gemini. It adds one more consistent, crawlable description of you. Publishing spends credits, which are non-refundable, and improves findability without guaranteeing a citation; see how credits work. See how it works, run an audit, then sign in at your account page when you are ready.