Robots.txt Generator
Create a robots.txt file to control how search engines crawl your site. Use presets or customize rules for each bot — download or copy instantly.
Quick presets
robots.txt
User-agent: * Allow: /
Robots.txt best practices
- —Always allow CSS and JS files — blocking them hurts SEO
- —Place robots.txt in your root directory (yoursite.com/robots.txt)
- —Include your sitemap URL for faster discovery
- —Don't use robots.txt for sensitive pages — use noindex or authentication instead
Enjoying this tool?
Let us know what you think