Free Online Robots.txt Tester

1. Enter any URL on the site; the tester resolves the /robots.txt path automatically.
2. Press Check to fetch the live file from our server.
3. Confirm the status is found and read the raw rules for any unintended Disallow lines.
4. Check the user-agent group count and the list of declared sitemaps.
5. Remember Disallow blocks crawling, not indexing — use noindex to remove a page from results.

robots.txt is a plain-text file at a domain's root, governed by the Robots Exclusion convention, that tells compliant crawlers which paths they may or may not fetch. It is organised into user-agent groups, each with Allow and Disallow rules, and can also declare Sitemap URLs. A common misconception is that Disallow removes a page from search results — it does not. Disallow only stops crawling; a disallowed URL can still be indexed if other pages link to it. To keep a page out of results you need a noindex meta tag or x-robots-tag header instead.

We resolve the /robots.txt path from whatever URL you enter and fetch it on our server, which sidesteps browser CORS limits and returns the raw file along with a count of user-agent groups and a list of declared sitemaps. robots.txt is the canonical place to declare sitemap locations, so confirming yours appear here ensures search engines can discover them.

Review the raw rules to make sure you are not accidentally blocking important sections — a stray Disallow: / blocks the entire site.