Free SEO Tool

Free Robots.txt Generator

Build a valid, optimised robots txt file in under two minutes. Use this robots txt generator to control which crawlers can access your site, protect sensitive directories, block AI training bots, and declare your sitemap — all without touching a line of code.

✓ No login required ✓ AI crawler rules ✓ Multi-bot support ✓ Copy or download ✓ 100% free

What This Tool Does

The Fastest Robots txt Generator That Actually Works

Most webmasters treat the robots.txt file as an afterthought. They copy a snippet from a forum post, paste it into their server root, and forget about it — never knowing whether it's helping or silently wrecking their SEO. That's a significant problem. A single wrong Disallow directive can block Googlebot from your entire site and wipe your rankings in a matter of weeks.

This free robots txt generator removes the guesswork. You configure your crawler rules through a clean visual interface, and the tool outputs a standards-compliant robots txt file that follows the Robots Exclusion Protocol as documented by Google. It handles wildcard user-agents, specific bot targeting, crawl delay settings, sitemap declarations, and — critically — the new generation of AI training crawlers that most robots txt generator tools don't even recognise yet.

Whether you're running a WordPress blog, an e-commerce store, a SaaS landing page, or a media site, getting your robots txt file right is one of the fastest, highest-leverage SEO fixes available to you. You don't need a developer. You need this robots txt generator, three minutes, and an FTP client.

Use Cases

🔒

Protect Admin Areas

Block crawlers from accessing /wp-admin/, /dashboard/, /login/, and other internal paths that should never appear in search results. Prevents accidental indexation of staging environments and backend interfaces.

🤖

Block AI Training Bots

Explicitly disallow GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and others from scraping your content for AI model training. These bots respect robots.txt — use it to protect your intellectual property.

⚡

Manage Crawl Budget

Direct Googlebot away from thin pages, URL parameters, duplicate content, and low-value archive pages. Focus crawl budget on your money pages — product listings, service pages, and high-value blog posts that actually generate traffic.

🗺️

Declare Your Sitemap

Including your XML sitemap URL in robots.txt is a direct signal to every major search engine. It speeds up discovery of new pages and ensures Googlebot knows exactly where to find your full content inventory.

🏗️

Isolate Staging Sites

If your staging environment accidentally gets crawled, duplicate content penalties can follow. Use a blanket Disallow: / in your staging robots.txt to keep test environments completely out of search engine indexes.

🎯

Crawler-Specific Rules

Apply different rules to different bots. Let Googlebot see everything while blocking aggressive scrapers. Allow Bingbot but restrict content aggregators. Fine-grained per-crawler control without writing a single line manually.

How To Use This Tool

How to Use This Robots txt Generator in 6 Steps

Choose your crawlers

Select which search engine bots and AI crawlers you want to control. You can apply rules to all bots at once using the wildcard option, or configure each crawler individually. The wildcard (User-agent: *) applies to every bot not explicitly given its own block — it's the right choice for most websites. Add specific bot blocks if you want to give different permissions to, say, Googlebot versus an AI scraper.

Set your Disallow paths

Enter the URL paths you want to block from being crawled. Common examples include /wp-admin/ for WordPress admin areas, /private/ for restricted content, /staging/ for test environments, and any internal search result pages like ?s= that create thousands of near-duplicate URLs. Enter one path per line. Paths are case-sensitive and must start with a forward slash.

Add Allow rules (optional)

If you've blocked a broad directory but need to permit specific pages within it, add Allow rules. These take precedence over Disallow for paths that match. A classic example: you might Disallow: /wp-content/ to prevent crawling of plugin files, while adding Allow: /wp-content/uploads/ so Googlebot can still index your images. Order matters — place specific Allow rules before broad Disallow rules when targeting the same directory tree.

Enable AI crawler blocking (optional)

Toggle the AI bot section to add explicit Disallow: / blocks for GPTBot, ClaudeBot, CCBot, and other AI training crawlers. These bots are designed to harvest content for large language model training data. If you'd prefer your content not be used for this purpose, blocking them via robots.txt is the simplest and most widely respected mechanism available. This has zero effect on your Google or Bing rankings.

Add your sitemap URL and crawl delay

Paste your XML sitemap URL (e.g., https://yoursite.com/sitemap.xml or https://yoursite.com/sitemap_index.xml for WordPress with Rank Math or Yoast) into the Sitemap field. This tells every major search engine exactly where your sitemap lives without them having to guess. If your server gets overwhelmed during heavy crawl sessions, add a Crawl-delay value in seconds — note that Googlebot ignores this directive, but most other bots will honour it.

Generate, copy, and upload to your root

Click Generate to produce your robots txt file. This robots txt generator outputs colour-coded results so you can quickly spot user-agent blocks, directives, and values. Copy it to your clipboard and paste it directly into your WordPress robots.txt editor (Rank Math SEO → General Settings → Edit robots.txt), or download the .txt file and upload it to your site's root directory via FTP or your host's File Manager. Verify it's live by visiting yoursite.com/robots.txt in your browser.

73% of websites have at least one misconfigured robots.txt rule Source: Ahrefs Crawl Study

6+ major AI training crawlers active in 2026 that respect robots.txt Source: OpenAI, Anthropic, Common Crawl docs

~5KB maximum recommended file size per Google's robots.txt specification Source: Google Search Central

<24h typical time for Googlebot to pick up robots.txt changes on active sites Source: Google Search Console docs

Generator

Build Your robots.txt File

Configure your rules below — your file generates in real time.

Configuration

Search Engine Crawlers

Apply rules to

All Bots (*) Recommended

Googlebot Google

Bingbot Bing / Edge

DuckDuckBot DuckDuckGo

Slurp Yahoo

FB External Hit Meta/Facebook

Disallow paths (one per line)

Paths must start with /. Leave empty to allow all.

Allow paths (optional, one per line)

Override Disallow for specific sub-paths.

AI Training Crawlers

Block AI bots from training on your content

GPTBot OpenAI

ClaudeBot Anthropic

CCBot Common Crawl

Google-Extended Gemini training

PerplexityBot Perplexity AI

Bytespider TikTok / ByteDance

Site Settings

Sitemap URL

Include full URL with https://

Crawl delay (seconds)

Ignored by Googlebot. Honoured by most others.

Output — robots.txt

# Your robots.txt file will appear here. # Configure your settings on the left # and click Generate.

Quick reference

User-agent: Which bot this block applies to

Disallow: Path bots cannot crawl

Allow: Override a Disallow for a sub-path

Crawl-delay: Seconds between requests

Sitemap: Full URL of your XML sitemap

After generating

1. WordPress users: Go to Rank Math → General Settings → Edit robots.txt and paste directly.

2. All other sites: Upload the file to your root directory (same level as index.html/php).

3. Verify it's live: Visit yoursite.com/robots.txt in a browser.

4. Test in Google Search Console: Legacy Tools → robots.txt Tester.

FAQ

Robots txt Generator — Common Questions Answered

A robots.txt file is a plain-text file placed at the root of your website (e.g., yoursite.com/robots.txt) that tells web crawlers which pages or sections they should or should not access. Use a robots txt generator like this one to build yours without writing anything manually. The file follows the Robots Exclusion Protocol — a standard supported by Google, Bing, and most major bots. It is not a security measure; it is a directive that well-behaved bots respect. Malicious crawlers will often ignore it entirely, so it should never be used as your sole method of protecting sensitive content.

Your robots.txt file must live at the root domain — for example, https://yoursite.com/robots.txt. For WordPress sites, this usually means uploading it via FTP or cPanel to the public_html folder. If you are using a host like SiteGround or Cloudways, you can access this through the File Manager. Some WordPress SEO plugins, like Rank Math and Yoast, also let you edit the virtual robots.txt directly from your dashboard without touching the server at all, which is the safest and easiest route for most WordPress users.

Yes, significantly. A misconfigured robots.txt can accidentally block entire sections of your site from being indexed, which causes traffic to disappear with no obvious warning. Common mistakes include disallowing /wp-content/ (which blocks CSS and JS that Google uses for rendering), or disallowing the homepage by mistake. Conversely, a well-crafted robots.txt keeps crawl budget focused on your most important pages and prevents admin areas and thin content from appearing in search results.

Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. For large sites with thousands of pages, wasting crawl budget on duplicate, low-value, or internal pages means important content gets crawled less frequently — sometimes not at all. By disallowing thin pages, staging URLs, and parameter-based duplicates in your robots.txt, you guide Googlebot to prioritise the pages that actually drive traffic and revenue. For small sites with under a few hundred pages, crawl budget is rarely a concern.

That depends on your goals. If you want your content to be used to train AI models or appear in AI-generated answers, allow these bots. If you would rather protect your content from being scraped for training data, blocking them is reasonable and straightforward. OpenAI's GPTBot, Anthropic's ClaudeBot, and Common Crawl's CCBot all officially respect robots.txt rules. Blocking them has no effect whatsoever on your Google or Bing rankings — they are entirely separate crawler systems with no overlap in ranking signals.

Disallow in robots.txt stops a crawler from visiting a page entirely. The noindex meta tag (or X-Robots-Tag header) tells a crawler it can visit the page but should not include it in search results. The important distinction: if you Disallow a page, Google cannot read its noindex tag — so the page may still appear in search results (without a description snippet) if other sites link to it. For reliable deindexing, use noindex on pages you still want crawlers to technically access. Reserve Disallow for pages that must never be crawled under any circumstances.

Yes. You can stack as many User-agent blocks as needed. Each block applies rules to a specific crawler. You can also use User-agent: * as a wildcard that applies to all bots not explicitly covered by another block in the same file. Google processes each block independently, so rules in a wildcard block do not automatically apply to crawlers that have their own dedicated block. This means you can allow Googlebot full access while simultaneously blocking AI bots with a separate User-agent block.

No — Googlebot officially ignores the Crawl-delay directive. It manages its own crawl rate based on your server's response times and the settings in Google Search Console under Settings → Crawling. However, Crawl-delay is respected by Bingbot, DuckDuckBot, and many other crawlers, so it is still worth including if you want to reduce server load from non-Google bots. To control Googlebot's crawl rate specifically, use the crawl rate settings inside Google Search Console directly.

Google Search Console includes a Robots.txt Tester under the Legacy Tools section. You can paste your file and test specific URLs to see which rules apply and whether Googlebot is allowed or blocked. You can also simply visit yoursite.com/robots.txt in a browser to verify the file is live and correctly formatted. For Bing, use Bing Webmaster Tools' built-in crawler diagnostics. Always run a test after any change before assuming your configuration is live and correct — a single typo can produce unexpected results.

Related Free Tools on CyberTrickz

🗺️

XML Sitemap Generator

Build a valid sitemap to pair with your robots.txt for faster indexation.

🏷️

Meta Title Generator

Once your pages are indexed, make sure they rank with optimised titles.

Most websites are giving AI crawlers free access to all their content by default.

Your robots txt file is the only lever you have to stop it — and most site owners don't even know their file is blank. Use a free robots txt generator to fix it in two minutes. 𝕏 Share This

Block Every AI Crawler: The 2026 Bot List

Generic robots.txt tools stop at Google and Bing. The bigger question in 2026 is which AI crawlers are scraping your content to train models or power answer engines. Here is every major AI user-agent, what it does, and the exact rule to block it. Paste any line into the generator above.

Bot token	Company	What it does	Block rule
`GPTBot`	OpenAI	Scrapes pages to train ChatGPT / GPT models	`User-agent: GPTBot` → `Disallow: /`
`OAI-SearchBot`	OpenAI	Indexes your site for ChatGPT Search results	`User-agent: OAI-SearchBot`
`ChatGPT-User`	OpenAI	Live fetch when a ChatGPT user opens your link	`User-agent: ChatGPT-User`
`ClaudeBot`	Anthropic	Scrapes pages to train Claude models	`User-agent: ClaudeBot`
`Claude-User`	Anthropic	Live fetch for a Claude user request	`User-agent: Claude-User`
`Google-Extended`	Google	Gemini / Vertex AI training. Does not affect Google Search ranking	`User-agent: Google-Extended`
`CCBot`	Common Crawl	Open dataset that seeds many third-party LLMs	`User-agent: CCBot`
`PerplexityBot`	Perplexity	Indexes content for Perplexity answer engine	`User-agent: PerplexityBot`
`Bytespider`	ByteDance	Trains ByteDance / TikTok AI models	`User-agent: Bytespider`
`Amazonbot`	Amazon	Feeds Alexa and Amazon AI answers	`User-agent: Amazonbot`
`Applebot-Extended`	Apple	Apple Intelligence training (Applebot itself is search)	`User-agent: Applebot-Extended`
`Meta-ExternalAgent`	Meta	Trains Meta AI and Llama models	`User-agent: Meta-ExternalAgent`

Copy-paste: block all major AI training bots

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://cybertrickz.info/sitemap_index.xml

Tip: blocking Google-Extended keeps your content out of Gemini training while staying fully indexed in Google Search — the two are separate.

robots.txt Directive Reference

Every rule this generator can output, and exactly what it does:

Directive	What it does	Example
`User-agent:`	Names the crawler the rules below apply to. `*` means every bot.	`User-agent: *`
`Disallow:`	Path the crawler must not access. Empty value allows everything.	`Disallow: /wp-admin/`
`Allow:`	Carves out an exception inside a disallowed folder.	`Allow: /wp-admin/admin-ajax.php`
`Sitemap:`	Absolute URL of your XML sitemap so crawlers find every page.	`Sitemap: https://site.com/sitemap_index.xml`
`Crawl-delay:`	Seconds to wait between requests. Bing respects it; Google ignores it.	`Crawl-delay: 10`
`*` wildcard	Matches any sequence of characters inside a path.	`Disallow: /*?replytocom`
`$` anchor	Matches the end of the URL — useful for file types.	`Disallow: /*.pdf$`