AI Crawl Radar

Sample report

A real report generated by our audit engine for a demo domain.

This is a clearly-marked sample. Run your own audit from the audit tool.

acme.example

https://acme.example/

Audited in 0.0s · 21 crawlers evaluated · 4 findings

AI bots allowed
19
AI bots blocked
2
robots.txt
Found
llms.txt
Missing

AI bot access

Allowed vs blocked across all evaluated crawlers

Readiness by category

How each scoring area performs

Score breakdown

Points earned per weighted category

What we check

Each area of your AI-visibility score — open the ⓘ for what it means and how to improve it.

  • AI Bot Access
  • robots.txt Strategy
  • llms.txt Readiness
  • Content Accessibility
  • Structured Data
  • Technical SEO
  • HTTP Health & Performance
  • Sitemap & Discovery
  • GEO/AEO Readiness

AI & search crawler access

Per-bot verdict from robots.txt analysis

BotProviderAccessRiskReason
GPTBotOpenAIBlockedcriticalBlocked by "disallow: /" in matched group
ChatGPT-UserOpenAIAllowednoneAllowed by robots.txt (matched group permits crawling)
OAI-SearchBotOpenAIAllowednoneAllowed by robots.txt (matched group permits crawling)
ClaudeBotAnthropicAllowednoneAllowed by robots.txt (matched group permits crawling)
Claude-WebAnthropicAllowednoneAllowed by robots.txt (matched group permits crawling)
PerplexityBotPerplexityAllowednoneAllowed by robots.txt (matched group permits crawling)
Perplexity-UserPerplexityAllowednoneAllowed by robots.txt (matched group permits crawling)
Google-ExtendedGoogleAllowednoneAllowed by robots.txt (matched group permits crawling)
GooglebotGoogleAllowednoneAllowed by robots.txt (matched group permits crawling)
BingbotMicrosoftAllowednoneAllowed by robots.txt (matched group permits crawling)
CCBotCommon CrawlBlockedhighBlocked by "disallow: /" in matched group
ApplebotAppleAllowednoneAllowed by robots.txt (matched group permits crawling)
Applebot-ExtendedAppleAllowednoneAllowed by robots.txt (matched group permits crawling)
Meta-ExternalAgentMetaAllowednoneAllowed by robots.txt (matched group permits crawling)
FacebookBotMetaAllowednoneAllowed by robots.txt (matched group permits crawling)
BytespiderByteDanceAllowednoneAllowed by robots.txt (matched group permits crawling)
AmazonbotAmazonAllowednoneAllowed by robots.txt (matched group permits crawling)
YouBotYou.comAllowednoneAllowed by robots.txt (matched group permits crawling)
DiffbotDiffbotAllowednoneAllowed by robots.txt (matched group permits crawling)
DuckAssistBotDuckDuckGoAllowednoneAllowed by robots.txt (matched group permits crawling)
OmgilibotWebz.ioAllowednoneAllowed by robots.txt (matched group permits crawling)

Findings

What we observed on your site

  • Critical
    1 high-value AI crawler(s) are blocked
    Blocked: GPTBot. These power AI search and citations.
  • Warning
    No llms.txt found
    An llms.txt gives AI assistants a curated map of your site and citation guidance.
  • Warning
    Missing schema: WebSite, WebPage, BreadcrumbList
    Schema.org markup helps AI engines understand entities and trust your content.
  • Warning
    robots.txt and meta robots disagree
    Conflicting crawl/index signals confuse crawlers — align them.

Recommended fixes

Prioritized, with paste-ready snippets below

  1. criticalUnblock high-value AI crawlerseasy

    Your robots.txt blocks GPTBot. Unless this is intentional, allow them.

  2. mediumPublish an llms.txteasy

    Add this file at /llms.txt to guide AI assistants to your best content.

  3. mediumAdd WebSite JSON-LDeasy

    Place this in a <script type="application/ld+json"> tag in <head>.

  4. lowAdd BreadcrumbList JSON-LDeasy

    Place this in a <script type="application/ld+json"> tag in <head>.

Generated for you

Copy these directly into your site to close the gaps above

Recommended llms.txt
llms.txt
# Acme

> Acme Analytics — Real-time dashboards

This file helps AI assistants and answer engines understand, navigate, and accurately cite Acme.

## Important pages
- [Home](https://acme.example/): Overview of Acme.
- [About](https://acme.example/about): Who we are and our expertise.
- [Contact](https://acme.example/contact): How to reach us.

## Resources
- [Blog](https://acme.example/blog): Articles and guides.
- [Sitemap](https://acme.example/sitemap.xml): Full list of indexable URLs.

## Content usage policy
You may cite and summarize this content with attribution to Acme and a link to the source page.

## Preferred citation
When referencing this site, cite "Acme" and link to the specific source URL.

Last updated: 2026-06-16
Unblock high-value AI crawlers
AI Bot Access
User-agent: GPTBot
Allow: /
Publish an llms.txt
llms.txt
# Acme

> Acme Analytics — Real-time dashboards

This file helps AI assistants and answer engines understand, navigate, and accurately cite Acme.

## Important pages
- [Home](https://acme.example/): Overview of Acme.
- [About](https://acme.example/about): Who we are and our expertise.
- [Contact](https://acme.example/contact): How to reach us.

## Resources
- [Blog](https://acme.example/blog): Articles and guides.
- [Sitemap](https://acme.example/sitemap.xml): Full list of indexable URLs.

## Content usage policy
You may cite and summarize this content with attribution to Acme and a link to the source page.

## Preferred citation
When referencing this site, cite "Acme" and link to the specific source URL.

Last updated: 2026-06-16
Add WebSite JSON-LD
Structured Data
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "Acme",
  "url": "https://acme.example",
  "potentialAction": {
    "@type": "SearchAction",
    "target": "https://acme.example/search?q={query}",
    "query-input": "required name=query"
  }
}
Add BreadcrumbList JSON-LD
Structured Data
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {
      "@type": "ListItem",
      "position": 1,
      "name": "Home",
      "item": "https://acme.example"
    }
  ]
}