What makes a website agent-ready? (2026)

A website is agent-ready when an AI agent can find it, read it and act on it without a person steering. It takes seven things. Put your content in the server-rendered HTML, and offer a clean markdown copy of each page alongside it, so agents get either format. Publish an llms.txt index, and let AI search and user agents in through robots.txt. Keep structured data matching the visible text, keep your facts identical on every page and profile, and keep pages fast and stable. MCP and WebMCP matter only when agents must complete tasks on your site.

I'm Brenden House, and I run Searchmaxxed, an SEO and AI search agency. On 27 October 2026 (AEST) I checked our own site against every item in this guide and measured what our markdown copies save. You'll find them below, beside what the AI companies publish about their own systems.

In summary

• Most of agent readiness is good technical SEO: server-rendered pages, clear headings and plain facts, and Google says AI Overviews need nothing extra.
• Markdown copies cut the download. On our site, the markdown copy of a page ran 88% to 93% smaller than its HTML across four pages we tested.
• llms.txt is cheap and optional. It's a reading list for agents, and Google says you don't need one.
• Robots.txt is where sites quietly fail. Block OAI-SearchBot and, by OpenAI's own docs, ChatGPT search won't show your pages in its answers.
• Readable doesn't mean chosen. The facts on your pages, and on other sites, still decide who an assistant names.

What does "agent-ready" mean?

An agent-ready website is one AI agents can discover, understand and use. An AI agent is software that reads web pages and acts for a person: ChatGPT searching the web, or a browser agent filling a form.

Four kinds of automated visitor turn up, and each one wants something different from you.

Two tools now score agent readiness. Cloudflare's Agent Readiness score groups the checks into discoverability, content, bot access control and capabilities (Cloudflare, 17 April 2026). Chrome's Lighthouse "Agentic browsing" category checks accessibility, layout stability and WebMCP (Chrome for Developers, 22 June 2026).

For most B2B and enterprise sites, the first two rows carry the weight. A buyer asks ChatGPT, Gemini, Perplexity or Copilot to compare suppliers, and the assistant reads a handful of pages to answer. Browser agents and checkout protocols come later, mostly for SaaS products where an agent might start a trial or call an API.

Why does agent readiness matter for a B2B or enterprise site?

An assistant that can't read your page can't name you. Half a page gets you half a description. Agent readiness gets your pages into the set an assistant considers, while content and reputation decide whether you're the one it picks.

Our own panel shows the gap. We put 300 Australian agency-buyer questions to ChatGPT three times each, with search on. It pulled 17,870 search results into those 900 answers and kept 4,389 as sources. Being found is the easy part. About three in four results it reads never make the answer.

So treat readiness as the entry ticket. Here's how it goes wrong. Picture a buyer asking which data platforms have SOC 2 Type II and an Australian hosting region. One B2B SaaS vendor's SOC 2 badge is an image, and its region list loads from a script after the page opens. An agent that reads the raw HTML sees neither fact, so it names a competitor whose security page states both in plain text.

That's an example, not a client story, but every part of it is a common build pattern. Our guide to how to rank in ChatGPT covers the content side: answers, sources and mentions.

What does an AI agent see when it opens your page?

Search and fetch agents read the HTML your server sends first, and many never run the JavaScript that fills a page in. If your words only appear after scripts load, the agent can get an empty shell. WebYes puts it plainly: "Many agents will not wait for or fully run heavy client-side scripts" (WebYes).

Browser agents work differently. They lean on the accessibility tree, the structured outline of a page that screen readers use. Chrome's team says "A well-formed accessibility tree is the primary way AI agents understand your page" (Chrome for Developers).

So the same habits serve both:

• Server-render or pre-render every page that matters, so the first response holds the text.
• Use one H1, then headings in order. Agents navigate by them.
• Write key facts as text, never only in an image or a PDF: prices, certifications, hosting regions, integrations, customer numbers.
• Use real buttons and links, and give every form field a label, with ARIA labels where a visible one won't fit.
• Build comparisons as HTML tables.

You can test this in a minute. Run `curl https://yoursite.com/page` in a terminal, or open view-source in a browser, and look for your H1 and first paragraph. If they aren't in that response, agents that skip JavaScript won't see them either. Fixing rendering, crawl and indexing faults like this sits at the core of our technical SEO service, alongside speed, structure and schema.

What's on an agent-ready website checklist?

Here's the list, in fix order. The first four decide whether agents can use your site at all.

Adoption is still low. Cloudflare scanned the 200,000 most-visited domains, minus categories like ad servers, and found 78% had a robots.txt, mostly written for classic search crawlers. Only 4% declared AI usage preferences with Content Signals, and markdown content negotiation passed on 3.9% (Cloudflare).

Should you serve markdown versions of your pages?

Yes, if agents read your site, because markdown carries the same words at a fraction of the weight. Markdown is plain text that marks headings, lists and links with simple symbols, with none of the layout code a browser needs.

There are two ways to offer it.

1. Content negotiation. An agent asks for Accept: text/markdown, and the server replies at the same URL in markdown. Cloudflare found that can cut tokens by up to 80%.
2. A .md URL. The same page sits at its address plus .md, which catches agents that don't send the header. Cloudflare reports that as of February 2026, only three of seven agents tested sent it by default: Claude Code, OpenCode and Cursor (Cloudflare).

Here's what it saves on our site. I fetched four pages both ways on 27 October 2026 (AEST), and the token figure is the estimate our server returns in its x-markdown-tokens header.

Bytes aren't tokens. Still, an agent gets the whole page in roughly a tenth of the download.

Does markdown count as cloaking? Not when it's the same words in another format, served because the client asked for it. Cloaking means showing bots different content from people, and Evil Martians warn that sniffing user agents to switch formats drifts toward it.

Four settings protect the HTML:

• Every .md response carries a Link header with rel="canonical" pointing back to the HTML page.
• Every .md response carries x-robots-tag: noindex, follow, so Google keeps indexing the HTML page and leaves the markdown copy out of results.
• Each HTML page names its copy with <link rel="alternate" type="text/markdown">.
• The negotiated response sends Vary: Accept, so caches don't mix the two formats up.

Do you need an llms.txt file?

No, but it's cheap, and it helps the agents that look for it. An llms.txt file is a markdown file at a site's root that says what the site is and links the pages worth reading. Jeremy Howard proposed it on 3 September 2024, and version 2 of the spec was last updated on 10 August 2026 (llmstxt.org).

Be clear on what it won't do. Google's AI features guide says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features". Cloudflare's scanner skips llms.txt unless you switch that check on.

Ours, at searchmaxxed.com/llms.txt, does two jobs:

• Its first lines describe Searchmaxxed in one sentence, written as we'd want an assistant to repeat it.
• It lists more than 260 pages, each linked to its markdown copy with a one-line description, and says the HTML page at the same path is canonical.

We also publish three extras: llms-full.txt, an /ai.txt file of usage permissions, and an AI instructions page. In fairness, there's no evidence any crawler treats ai.txt or an AI instructions page differently, and Evil Martians list ai.txt as having no meaningful adoption. We keep both because they cost little and repeat the same facts as every other page.

Which AI crawlers should your robots.txt allow?

Allow the AI search crawlers and the live-fetch agents if you want to appear in AI answers. Decide on training crawlers separately. Each company publishes a separate bot name for each of these jobs.

From each company's own docs:

• OpenAI: sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers." Changes take about 24 hours to apply. ChatGPT-User acts on user requests, so robots.txt rules "may not apply" (OpenAI).
• Anthropic: blocking Claude-SearchBot or Claude-User "may reduce your site's visibility" in Claude's search results (Anthropic).
• Perplexity: PerplexityBot surfaces sites in Perplexity's results and "is not used to crawl content for AI foundation models" (Perplexity).
• Google: Google-Extended controls Gemini training and grounding. It "does not impact a site's inclusion in Google Search" (Google).

Watch for one trap. Robots.txt can say yes while your firewall says no. OpenAI and Perplexity publish IP ranges for exactly this, so check your CDN logs as well as the text file.

Our robots.txt runs three groups: AI search and user agents, AI training crawlers, then everyone else. We allow all three. The file's first lines point agents to llms.txt and ai.txt, and its last line to the sitemap. If you want to allow search but refuse training, Cloudflare's Content Signals line does it in one rule: Content-Signal: ai-train=no, search=yes, ai-input=yes.

Does structured data make a site agent-ready?

Structured data helps machines identify who you are, but it's no shortcut into AI answers. Structured data is code, usually JSON-LD using the schema.org vocabulary, that labels facts such as your company name, products and FAQs.

Google's rule is short. Make sure your structured data matches the visible text on the page; no special schema is required for AI features (Google Search Central). Tests on AI assistants agree. In a searchVIU experiment cited by Evil Martians, ChatGPT, Claude, Perplexity, Gemini and Copilot all missed product data placed only in JSON-LD.

So write every fact in visible text first, then mark it up with the types that suit a B2B site:

• Organization, with your legal name, logo and sameAs links to LinkedIn, Crunchbase and review profiles.
• SoftwareApplication or Product for what you sell, and Service for services.
• Article, BreadcrumbList and FAQPage where the page has them.

Our /agentic-websites page, for example, carries a JSON-LD graph that defines Searchmaxxed as an Organization.

Why do consistent facts matter to AI agents?

An assistant builds its answer from several pages, and when they disagree it has to pick one or hedge. Say your site claims 400 customers, a review profile says 250 and an old press release still uses your former name. That's three versions of you.

Write one fact sheet and make every surface agree with it, which for a B2B SaaS, manufacturing or public-sector supplier usually covers:

• Legal, trading and former names
• What you sell and who it's for, in one sentence
• Markets, regions and data hosting locations
• Your pricing model, even when the price itself isn't public, plus the contract terms you're happy to state on your site
• Certifications: SOC 2, ISO 27001, IRAP, FedRAMP
• Integrations, partners and customer numbers
• Founders and leaders, with the same titles on your site, LinkedIn and every speaker bio

Then fix the gaps in order: your own site first, then the profiles your team controls, such as LinkedIn and partner listings, then press boilerplate. Assistants read what other sites say about you, which is why our plans include links and digital PR that repeat the facts your pages state. Our AI search source readiness checklist walks through the outside sources in more detail.

Do page speed and layout matter to AI agents?

Yes, in two ways. Live fetches happen while a person waits for an answer, so a heavy page costs the agent time and tokens. Browser agents also need the page to hold still.

Chrome's Lighthouse Agentic browsing audit measures Cumulative Layout Shift "to ensure elements don't move unexpectedly, preventing agent misclicks" (Chrome for Developers). It also checks every interactive element has a programmatic name, and the category is "informational and unbenchmarked", so treat it as a fix list. The usual Core Web Vitals work helps both: lighter scripts, fewer pop-ups, and no cookie wall over the content.

Do you need MCP, WebMCP or agent payment protocols?

Only if you want agents to do something on your site, such as book a demo, start a trial or query your product. A marketing site can wait.

• MCP (Model Context Protocol) is an open standard that lets AI models connect to outside tools and data, such as a SaaS product's own features.
• An MCP Server Card is a small JSON file at a well-known address that tells agents what your MCP server offers.
• WebMCP is a proposed standard that exposes a site's forms and actions to agents in the browser.
• An API catalog (RFC 9727) lists your public APIs in one place.
• Agent Skills are instruction files a site publishes so agents know how to do specific tasks on it.
• x402, the Universal Commerce Protocol and the Agentic Commerce Protocol let agents pay or check out.

In Cloudflare's scan of 200,000 top domains, MCP Server Cards and API catalogs together turned up on fewer than 15 sites. That's room to stand out. Our own site has no MCP server card, because an agent has nothing to operate on it beyond the contact form.

How do you check if your website is agent-ready?

Run six checks, in this order. The first three need only a terminal.

1. Raw HTML. `curl https://yoursite.com/page`, then look for your H1.
2. Markdown. curl -H "Accept: text/markdown" https://yoursite.com/page` and check the content-type` header.
3. Robots. Search /robots.txt for the AI bot names above.
4. Scanners. Run isitagentready.com and Lighthouse's Agentic browsing audit.
5. Ask the assistants. Paste a page URL into ChatGPT, Claude and Perplexity, ask for your pricing model, certifications and regions, then check the answers against the truth.
6. Logs. Check your server or CDN logs for those bot names, and which pages they hit.

Treat any single score with care. Suganthan Mohanadasan ran his site through three tools: 100 of 100 on Cloudflare, 4 of 10 on Agentchecker, 74 on Glippy. One homepage, three verdicts.

Analytics needs the same caution, and here's what we found on an enterprise B2B client's account. A client's GA4 showed AI-assistant visits jump from 354 to 6,050 in a month. Almost none of it was real. About 5,700 came in three one-hour bursts: same 1920x1080 desktop Chrome, all new users, no referrer, about 20 seconds each, landing on two pages. Real AI traffic was 385 sessions against 354 the month before, roughly flat.

Check a spike for bots before anyone reports it as a win.

What does an agent-ready site look like in practice?

Our site is the example I can show you line by line, so here's what was live when I checked on 27 October 2026 (AEST).

None of it is exotic. It's configuration, done once and kept in line with every page change, and we do the same kind of work on client sites.

Where does Searchmaxxed fit?

Searchmaxxed is an SEO and AI search agency for enterprise, B2B SaaS, venture-backed, manufacturing and public-sector companies. We do technical SEO, content, link building, digital PR, GEO, AEO and LLM SEO. The goal: rank in Google and get named in ChatGPT, Gemini, Perplexity and Google's AI answers for the searches that bring pipeline.

There are two ways in. The Agentic Website Full Build covers market research, sitemap, design system, page families, technical SEO, conversion paths and measurement, live in 30 business days. The Agentic Migration keeps a design worth keeping and moves the site to an owned repository, and our Agentic Websites page covers both.

The managed packages differ in volume: page releases each month, and how much work we do off your site.

Here's how we run it. We build on a website you own, our Agentic Website. AI agents pull your Search Console and GA4 data, read the Google results and check what AI answers say about you and your competitors. That feeds the Managed Search Loop, which shows which page, fix or link should bring in the most enquiries next. We build it, check the live page and measure the change. Off your site, the Off-Page Source Layer builds the links, digital PR and mentions that AI answers draw on.

Searchmaxxed's work has generated more than A$10 million in revenue through search for its clients.

Is Searchmaxxed right for you?

We suit marketing and growth teams that want the agent-ready work built, then kept current as pages change. We handle Google and AI answers in one team, and you own the site, repository, content and data from day one.

Right-hand column readers will get most of the way with a free scanner and a good developer.

Frequently asked questions

What should you do next?

Run the three terminal checks on your homepage and your best product page today. Fix any failure there first. If you'd like us to build the rest and keep it current, let's talk pricing.