The Robots.txt Generator builds a correctly formatted robots.txt file from a visual form — no need to memorise the syntax. Add one or more user-agent groups, set Disallow and Allow rules for each, optionally set a crawl delay, and add your sitemap URL. The output updates live as you build your rules, and you can download the finished file or copy it directly.
1. The default group targets all bots (*). Type a specific bot name or click Common bots to pick from a list.
2. Click Add Disallow or Add Allow to add path rules. Enter the path to block or permit (e.g. /admin/).
3. Add more user-agent groups with Add another user-agent group — useful for blocking AI crawlers while allowing search engines.
4. Paste your sitemap URL in the Sitemap section (e.g. https://example.com/sitemap.xml).
5. Click Sample to see a realistic example with multiple groups.
6. Click Download robots.txt and upload the file to the root of your web server.
What is a robots.txt file?
A robots.txt file sits at the root of your domain (e.g. https://example.com/robots.txt) and tells search engine crawlers which pages or sections they are allowed or not allowed to visit. It is a convention, not a technical enforcement — well-behaved bots like Googlebot follow it, but malicious scrapers may ignore it.
What does the wildcard (*) user-agent mean?
An asterisk (*) as the user-agent applies the rules to all bots that do not have their own specific section. It is the standard way to set a default policy for all crawlers.
What is the difference between Disallow and Allow?
Disallow tells a crawler not to visit a path. Allow explicitly permits a path that would otherwise be blocked by a broader Disallow rule. For example, you could disallow /private/ but allow /private/public-page.html.
Does robots.txt prevent pages from appearing in search results?
Not directly. Disallowing a page stops the crawler from visiting it, but if another site links to that page Google may still index it (showing the URL but no content). To fully remove a page from search results, use a noindex meta tag or the X-Robots-Tag header instead.
Where do I put the robots.txt file?
It must be at the root of your domain — https://yourdomain.com/robots.txt. It will not work in a subdirectory. Upload it to the web root of your server (the same folder that contains your index.html or index.php).
Should I include my sitemap in robots.txt?
Yes, it is good practice. Adding a Sitemap: line helps search engines discover your sitemap even if you have not submitted it through Search Console. You can include multiple Sitemap: lines if you have more than one.