robots.txt
tester
One wrong line in robots.txt can hide your whole site from Google - or from ChatGPT. Paste your file and your URLs, pick a crawler, and see exactly which rule decides each path. It runs in your browser - nothing is uploaded.
Which rule wins? Find out.
Path verdicts
| Path | Verdict | Deciding rule |
|---|
File health
Yetty writes and serves a correct robots.txt for every site automatically - sitemap line included - with a one-switch AI-crawler control.
How crawlers really
read your file
One group applies
A crawler picks the most specific User-agent group that names it. If a specific group exists, the * group is ignored entirely - not merged.
Longest path wins
Rules are not read top to bottom. Whichever matching rule has the longest path decides - wherever it sits in the file.
Allow beats Disallow on a tie
Equal length, opposite verdicts? The Allow rule wins. That is how you open one folder inside a blocked area.
Wildcards: * and $
* matches any characters, $ pins the rule to the end of the URL. Disallow: /*.pdf$ blocks every PDF and nothing else.
The one-character trap
Disallow: / blocks everything. Disallow: (empty) blocks nothing. Countless sites have vanished over that single slash.
Sitemap line
Sitemap: https://…/sitemap.xml belongs in robots.txt too - it is how crawlers find your page list without guessing.
People also ask
How do I test my robots.txt file?
Paste your robots.txt and a list of paths into this tester, pick the crawler you care about, and you get an ALLOWED or BLOCKED verdict per path with the exact line that decided it. It applies the real rules crawlers use: the most specific user-agent group wins, then the longest matching path, and Allow beats Disallow on a tie.
Does the order of rules in robots.txt matter?
No - and that surprises almost everyone. Crawlers do not read rules top to bottom. Within the group that applies to them, the rule with the longest matching path wins, wherever it sits in the file. If an Allow and a Disallow match with equal length, Allow wins. Order only matters for readability.
What does Disallow with just a slash mean in robots.txt?
Disallow: / blocks the whole site for whichever user-agents that group covers, because every path starts with a slash. An empty Disallow: (with nothing after the colon) means the opposite - block nothing. One character separates "welcome" from "go away", which is why testing beats guessing.
More free tools:
Can AI see your website? ·
Redirect chain visualizer ·
All free tools
Rather never touch this file again? See SEO on autopilot