Fetch robots.txt from a domain (optional)
Note: many sites block direct cross-origin reads. If fetching fails, paste the file content below, that always works.
Use * for all crawlers, or a bot name like Googlebot.
How matching works
The tester groups rules by user-agent, then for each path applies the rule with the longest match. On a tie, Allow wins. * matches any sequence of characters and a trailing $ anchors the match to the end of the path.
Frequently asked questions
How does the rule matching work?
Rules are grouped by user-agent. For each path, the rule with the longest match wins, and Allow beats Disallow on a tie. The * wildcard matches any characters and a trailing $ anchors the match to the end of the path, just like Google interprets robots.txt.
Why did fetching robots.txt from a domain fail?
Many websites block direct cross-origin reads from browsers. When that happens, open the site's /robots.txt in your own browser tab and paste the content into the tester. Paste mode always works.
Does this tool change my robots.txt?
No. Everything runs in your browser and nothing is uploaded or modified. Use the results to fix your file on your own server.
Preview
Related free tools
Free HTML Anchor Link Generator | Build <a> Tags
Free HTML anchor link generator. Build correct link tags with target, rel, and title attri
Free Canonical Tag Generator | SEO Canonical URLs
Free canonical tag generator. Enter your preferred page URL and generate a correct canonic
Free Hreflang Tag Generator for Multilingual Sites
Free hreflang tag generator. Enter each language version URL and generate correct hreflang