SEO

robots.txt generator and tester

Build your rules, copy the file, and test whether a URL is crawlable — for any bot.

Start from:
robots.txt
 

Test a URL

User-agent matching is simplified (case-insensitive exact token, else *) — fine for files you author yourself.

Sources & references