SEO
robots.txt generator
Build your rules, copy the file, and test whether a URL is crawlable — for any bot.
Start from:
Sitemaps
robots.txt
Test a URL
User-agent matching is simplified (case-insensitive exact token, else *) — fine for files you author yourself.