SEO & Marketing
Robots.txt Generator
The Robots.txt Generator builds a valid robots.txt file, letting you specify which paths search engine crawlers should avoid, and point them to your sitemap.
Fill in your rules and copy the result into a file named robots.txt at the root of your site.
How to Use the Robots.txt Generator
- Open the Robots.txt Generator.
- Enter the user-agent (usually * for all crawlers).
- List any paths to disallow or allow, one per line.
- Add your sitemap URL, then copy the generated robots.txt.
Frequently Asked Questions
Where do I put the robots.txt file?
It must be placed at the root of your domain, e.g. https://example.com/robots.txt, for search engines to find it.
Does robots.txt guarantee pages won't be indexed?
No, it only asks well-behaved crawlers not to crawl certain paths -- for guaranteed exclusion from search results, use a noindex meta tag instead.
Is anything uploaded when I use this tool?
No. The file is generated entirely in your browser -- nothing is sent to a server.
Advertisement
Sponsored