Can AI see
your website?
People ask ChatGPT, Claude and Perplexity for recommendations every day. Type your web address and see - crawler by crawler - whether AI assistants are allowed in. We fetch your page securely - nothing else leaves your browser.
Type your address. Get the verdict.
Or paste robots.txt and HTML manually
AI crawler verdicts
| Crawler | What it feeds | Verdict | Deciding rule / fix |
|---|
Homepage checks
Yetty sites welcome AI crawlers by default, serve llms.txt and structured data automatically, and give you a one-switch block if you ever change your mind.
The crawlers behind
every AI answer
OpenAI / ChatGPT
GPTBot gathers training data, OAI-SearchBot builds ChatGPT search, ChatGPT-User visits live when a user asks about you.
Anthropic / Claude
ClaudeBot crawls, Claude-SearchBot indexes for search, Claude-User fetches pages during a live conversation.
Perplexity
PerplexityBot builds the answer index; Perplexity-User fetches your page live when it cites you.
Google-Extended
Not a crawler of its own - a robots.txt switch that decides whether Google may use your content in Gemini.
The dataset bots
CCBot (Common Crawl) feeds datasets many AI models train on; Bytespider is ByteDance; meta-externalagent is Meta.
Applebot-Extended
Apple’s switch for Apple Intelligence training - same idea as Google-Extended.
People also ask
How do I know if ChatGPT can see my website?
Check your robots.txt for the crawlers OpenAI uses: GPTBot, OAI-SearchBot and ChatGPT-User. If a Disallow rule blocks them (or blocks everyone with a Disallow slash under User-agent star), ChatGPT cannot read your pages. Enter your web address into this checker (or paste your robots.txt) and you get a per-crawler verdict with the exact rule that decides it.
Should I block AI crawlers like GPTBot?
It depends on your goal. If you sell content itself, blocking training crawlers can make sense. But for most businesses, AI assistants are a growing source of customers: when ChatGPT or Perplexity recommends you, that traffic is free. Blocking AI crawlers means you simply do not exist in those answers.
Does robots.txt affect ChatGPT and Perplexity answers?
Yes. OpenAI, Anthropic and Perplexity all state their crawlers respect robots.txt. If your file blocks their bots, your pages cannot be fetched for browsing answers or search indexes, so competitors get quoted instead of you. Google-Extended similarly controls whether Google may use your content for Gemini.
More free tools:
robots.txt tester ·
llms.txt generator ·
All free tools
Want the whole layer automatic? See SEO on autopilot