Free Robots.txt Checker & Tester
Fetch any site’s robots.txt, see its rules grouped by crawler, and test whether a specific URL is allowed or blocked.
The page is fetched once and nothing is stored. Sites that block automated requests may not respond.
How to use this tool
- Enter the domain — the /robots.txt path is added automatically.
- Read the grouped rules to see what each crawler is allowed.
- Test a specific path to confirm it is not accidentally blocked.
- Fix anything listed under Problems, then re-check.
What is Robots.txt Checker?
robots.txt controls crawling, not indexing — a distinction that causes more damage than almost anything else in technical SEO. A blocked page can still appear in results if other pages link to it; it just appears without a description, because Google was never allowed to read it.
The other common failure is scale: a single misplaced Disallow: / on a production deploy can remove a site from search within days. Checking the live file after a deploy takes seconds.
Best practices
- Never block CSS or JavaScript that the page needs to render.
- Use noindex meta tags, not robots.txt, to keep pages out of the index.
- List your sitemap with a Sitemap: line.
- Check the live file after every deploy — staging rules escaping to production is common.
- Remember it is public: never use it to hide sensitive paths.
Frequently asked questions
Does robots.txt stop a page being indexed?
No. It stops it being crawled. If other sites link to a blocked URL, Google can still list it, showing the URL with no description. To keep a page out of the index, allow crawling and add a noindex meta tag so the instruction can actually be read.
Is there a problem with not having one?
No. With no robots.txt, crawlers treat everything as allowed, which is what most sites want. A 404 here is fine. A 5xx is not — repeated server errors on robots.txt can make Google pause crawling the site entirely.
Does Google respect Crawl-delay?
No. Google ignores it. Bing and some others honour it. Crawl rate for Google is managed in Search Console.
Related tools
Want this done properly on your site?
These tools handle the checks. If you would rather someone did the work — the audit, the fixes and the content — that is what I do.
Results are provided for informational purposes and may vary depending on the website, browser, network and available data.