Web & SEO

Robots.txt Generator — Free 2026

Generate a valid robots.txt file with user-agent rules, disallow and allow paths, sitemap URL, and crawl delay settings.

User-Agent

Disallow Paths

Allow Paths

Sitemap URL

Crawl-Delay (seconds)

robots.txt Output

How It Works

Set user-agent and rules
Add sitemap and crawl delay
Copy and upload

Understanding robots.txt

The robots.txt file is a fundamental part of web site management and search engine optimization. It sits at the root of your domain and serves as the first point of contact for search engine crawlers, telling them which parts of your site they are allowed or forbidden to access. Every website should have a properly configured robots.txt file to ensure efficient crawling and protect sensitive directories.

How robots.txt Works

When a search engine bot visits your site, it first requests /robots.txt. The file contains rules grouped by User-agent (the crawler's name). The wildcard * applies to all crawlers. Each group specifies Disallow paths (blocked) and optionally Allow paths (permitted exceptions within blocked directories). Rules are matched from the longest path first, so Allow: /admin/public/ takes precedence over Disallow: /admin/.

Common Use Cases

Block admin panels (/admin/), staging environments, duplicate content, search result pages (/search), and shopping cart pages. Allow CSS and JS files that Google needs for rendering. Always include a Sitemap directive pointing to your XML sitemap to help crawlers discover your content. For generating meta tags and optimizing individual pages, try our meta tag generator. Our slug generator ensures clean URLs for better crawlability.

Important Limitations

robots.txt is advisory, not enforceable. Well-behaved bots like Googlebot and Bingbot respect it, but malicious scrapers will ignore it entirely. Do not rely on robots.txt for security — use proper authentication and access controls instead. Also note that blocking a URL via robots.txt does not prevent it from appearing in search results if other pages link to it. Use the noindex meta tag for true deindexing.

Frequently Asked Questions

What is a robots.txt file?

robots.txt is a plain text file placed at the root of your website that tells search engine crawlers which URLs they can access. It follows the Robots Exclusion Protocol and is the first file crawlers check before indexing your site.

Does robots.txt block pages from appearing in Google?

Not exactly. robots.txt prevents crawling, but blocked pages can still appear in search results if other pages link to them. To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header instead. robots.txt controls crawl access, not indexing.

What is the Crawl-delay directive?

Crawl-delay tells crawlers to wait a specified number of seconds between requests. It helps reduce server load from aggressive bots. Note that Google ignores Crawl-delay (use Google Search Console instead), but Bing and other crawlers respect it.

Where should I upload my robots.txt file?

Upload robots.txt to the root directory of your domain so it is accessible at https://yourdomain.com/robots.txt. It must be at the root level — subdirectory placement will not work. The file must be plain text with UTF-8 encoding.