Robots.txt Generator
Pick a crawler, set allow and disallow paths, add a sitemap URL and extra rules, and the tool writes your robots.txt file live, ready to copy or download.
About
About Robots.txt Generator
A robots.txt file tells search engine crawlers which parts of your site they may visit. This generator writes that file for you. Choose which crawler the rules apply to, set an allow path and a disallow path, and the text updates as you type. When it looks right, copy it or download it as a ready-to-upload robots.txt file.
The crawler list includes all crawlers (*), Googlebot, Bingbot, Googlebot-Image, GPTBot and ChatGPT-User. Presets cover a standard setup, a WordPress site and a full block. Beyond the main fields, you can add a sitemap URL, a crawl delay and a host line, plus extra Allow, Disallow, Sitemap, Crawl-delay or Host rules with the Add rule button.
Each file this tool builds holds one rule group: a single User-agent line with its rules underneath. If your site needs different rules for different crawlers, generate each group here, then combine them in one file by hand, with a blank line between groups.
Robots.txt controls crawling, not indexing. A page you disallow can still appear in search results as a bare URL if other sites link to it. To keep a page out of results, leave it crawlable and add a noindex tag, or put it behind a login. Google ignores the Crawl-delay and Host lines, while Bing respects Crawl-delay. Compliance is voluntary, so well-behaved crawlers follow your file and bad bots don't.
Upload the finished file to the root of your domain so it loads at yourdomain.com/robots.txt. Subdomains need their own file. The tool runs in your browser, so your paths and URLs never leave your device, and it doesn't check your rules for mistakes. Read the preview before you publish, then test the live file with the robots.txt report in Google Search Console.
FAQ
Frequently asked questions
Where do I put the robots.txt file?
In the root directory of your site, so it opens at https://yourdomain.com/robots.txt. Crawlers don't look for it in subfolders. Each subdomain, like blog.yourdomain.com, needs its own file.
What does the Block Everything preset do?
It writes Disallow: / for the chosen crawler and clears the allow line, which asks that crawler to stay away from your whole site. It suits staging and private sites. On a live site it can stop your pages from being crawled, so don't publish it by mistake.
Will blocking a page remove it from Google?
Not reliably. Robots.txt only asks crawlers not to fetch a page. If other sites link to it, Google can still list the URL. To prevent a listing, use a noindex tag on a crawlable page, a password, or Google's removal tools.
Should I use a crawl delay?
Most sites don't need one. Google ignores the Crawl-delay line. Bing and some other crawlers follow it, reading the number as roughly the seconds to wait between requests. A high value slows crawling of your whole site, so add it only if a crawler strains your server.
How do I block AI crawlers like GPTBot?
Choose GPTBot or ChatGPT-User in the user-agent list and set Disallow to /. Generate one group per bot and join the groups in one file. This only works for crawlers that obey robots.txt, and it doesn't affect content they already collected.
Can the tool check my existing robots.txt?
No. It builds new files and doesn't read or test existing ones. Paste the result into your site, then use the robots.txt report in Google Search Console to see how Google reads it.
Related tools
View all developer tools →XML Sitemap Generator
Generate a valid XML sitemap from a list of page URLs.
Open tool.htaccess Generator
Generate common .htaccess rules for redirects, caching and rewrites.
Open toolMeta Description Length Checker
Check whether your title or meta description will get cut off in search results.
Open toolUTM Link Builder
Build UTM links so your campaign traffic shows up correctly in analytics.
Open tool