Robots.txt Tester

Fetch and analyze any website's robots.txt file. Test if specific URLs are allowed or blocked for different search engine bots.

About This Tool

Analyze how search engines see your crawling rules. View all user-agent rules, test if specific paths are blocked, and find declared sitemaps.