Robots.txt Generator
Create a custom robots.txt file for your website in seconds to control how search engines crawl your pages.
Configuration
The default rule applying to all search engines (User-agent: *).
Google Search (Googlebot)
Google Image (Googlebot-Image)
Google Mobile (Googlebot-Mobile)
Bing (Bingbot)
Yahoo (Slurp)
DuckDuckGo (DuckDuckBot)
Baidu (Baiduspider)
Yandex (YandexBot)
Wayback Machine (ia_archiver)
OpenAI GPT (GPTBot)
CommonCrawl (CCBot)
Pinterest (Pinterestbot)
The path is relative to root and should contain a trailing slash (e.g. /admin/).
User-agent: * Disallow: /cgi-bin/ Disallow: /admin/
Frequently Asked Questions
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests, or to keep certain pages from appearing in search results.
By setting the default rule to "Refused", you are telling every search engine and bot not to crawl any part of your site. Use this carefully, usually only for staging or completely private environments.
Any directory you add will be disallowed for crawling. The path is relative to the root of your domain and typically contains a trailing slash (e.g., /admin/). Allowed bots will still be blocked from these specific restricted directories.
The crawl-delay directive is an unofficial way to ask bots to wait a certain number of seconds between requests, to prevent them from overloading your server. Not all search engines (like Google) respect it anymore.
Stop configuring technical SEO manually.
Let AI handle your entire SEO infrastructure.
Supablog automatically enforces optimal crawling rules, handles structured data, and publishes fully SEO-ready blog content so you can focus on growing your traffic.
Get started for free 14-day free trial No credit card Cancel anytime