SEO tool
Robots.txt Generator
Build crawler rules with allow and disallow paths, sitemap references, host hints, and optional crawl delay.
Output
Generated robots.txt
{stats.directives} directives
Allow
1
Disallow
3
Lines
7
Summary
3 blocked paths listed. A sitemap reference is included.
Overview
What Is robots.txt and How Does It Work?
A robots.txt file is a foundational web standard governed by IETF RFC 9309(Robots Exclusion Protocol). It serves as the primary gatekeeper for your website, providing guidelines to automated search engine spiders, social media indexers, and AI web crawlers regarding which sections of your domain they are permitted to request.
When a crawler arrives at your site, the very first request it makes is for /robots.txt. Properly configuring this file conserves server bandwidth and crawl budget, prevents administrative panels from being scraped, and points crawlers directly to your XML sitemap. This tool runs 100% locally in your browser to build and validate compliant robots.txt directives in seconds.
Specification
RFC 9309 Directives Breakdown
| Directive | Syntax Example | Function & Scope |
|---|---|---|
| User-agent | User-agent: Googlebot | Designates which specific crawler or bot the subsequent rules apply to. |
| Disallow | Disallow: /checkout/ | Prevents matching crawlers from accessing URLs that begin with this path prefix. |
| Allow | Allow: /checkout/success | Explicitly permits crawling of a specific subpath inside an otherwise disallowed directory. |
| Sitemap | Sitemap: https://site.com/sitemap.xml | Directs search engines to the absolute URL of the XML sitemap. |
| Crawl-delay | Crawl-delay: 10 | Requests a delay in seconds between consecutive crawler hits (supported by Bing, Yandex). |
| Wildcard (*) | Disallow: /*.pdf$ | Matches any sequence of characters in the URL string. |
| End of Line ($) | Disallow: /*.json$ | Designates the end of the URL pattern for strict extension matching. |
Capabilities
Key Features & Generator Controls
RFC 9309 Protocol Standard
Constructs valid, compliant directives according to the formal IETF standard recognized by major search engines.
Quick Pre-Configured Presets
One-click presets for common scenarios: Allow All, Disallow All, or Standard SEO Best Practices.
Custom Path Inclusions & Exclusions
Easily specify exact directory paths and filenames to allow or disallow with real-time syntax checking.
XML Sitemap Integration
Link your XML sitemap URL automatically to provide an immediate discovery map for visiting crawlers.
Crawl-Delay Configuration
Throttle aggressive crawlers by configuring customized wait intervals to protect server infrastructure.
Direct File Download
Save your formatted robots.txt file immediately with one click, ready for your web root deployment.
One-Click Copy & Clear
Quickly copy the clean directives directly to your clipboard or reset your rules with a single click.
100% Client-Side Privacy
Your private server directories, admin route names, and domain URLs remain strictly inside your browser.
Precedence
How Crawlers Interpret Rules & Priority
When a crawler evaluates a URL against conflicting Allow and Disallow directives, RFC 9309 mandates that the most specific rule wins based on path length:
Allow: /admin/login
→ /admin/login is allowed because its path (12 chars) is longer and more specific than /admin/ (7 chars).
If both Allow and Disallow rules match with equal path lengths, the Allow directive takes precedence under modern search engine conventions.
Workflow
How to Create and Install robots.txt
- Choose base rule: Select Allow All (recommended for public sites) orDisallow All (for staging/private environments).
- Specify disallowed paths: Enter internal paths (e.g.
/admin/,/cart/,/checkout/) that should not be crawled. - Add sitemap link: Enter your full XML sitemap address (e.g.
https://example.com/sitemap.xml). - Download file: Click Download robots.txt to save the file locally.
- Deploy to site root: Upload the file via FTP, SSH, or CI/CD to your public web root so it resolves at
https://yourdomain.com/robots.txt. - Verify in Search Console: Use Google Search Console's Robots Testing Tool to verify your rules match expected crawler behaviors.
Applications
Common Deployment Scenarios
Public E-Commerce & Content Websites
Permit full crawling of product and blog pages while disallowing search results, user accounts, and checkout funnels.
Staging & Development Servers
Disallow all crawling (User-agent: * Disallow: /) on test environments to prevent duplicate content penalties.
Blocking AI Scraper Bots
Explicitly forbid aggressive AI crawlers (GPTBot, ClaudeBot, CCBot) from mining your proprietary content.
Conserving Server Crawl Budget
Prevent search bots from indexing infinite faceted navigation filters, sorting parameters, and internal search URLs.
Advantages
Benefits of Proper robots.txt
- Directs crawler resources to high-value, revenue-generating pages.
- Protects database-heavy search and filtering endpoints from crawler overload.
- Prevents administrative dashboards from appearing in search results.
- Instantly communicates sitemap location to global search engines.
- Gives site owners explicit control over automated crawler permissions.
Boundaries
Limitations & Considerations
- Robots.txt is public: anyone can read your disallowed paths to find private directories.
- Malicious bots and scrapers intentionally ignore robots.txt directives.
- Does not guarantee de-indexing if external sites link to the disallowed URL.
- Googlebot ignores the Crawl-delay directive.
Guidance
Pro Tips for Robots.txt
- Always verify that
/robots.txtreturns HTTP 200 OK, not a 404 or 500 error. - Ensure CSS and JavaScript bundles are never blocked so Google can render pages.
- Keep paths case-accurate:
/Admin/does not match/admin/. - Use
noindexmeta tags inside pages when you need absolute exclusion from search results.
Avoid
Common Pitfalls to Avoid
- Using robots.txt to hide sensitive user data or passwords (it is publicly viewable).
- Accidentally deploying staging robots.txt (
Disallow: /) to production. - Forgetting that
Disallow: /appblocks/applicationas well as/app/. - Using relative paths for the Sitemap directive (Sitemap must be a full URL).
Questions
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a plain text file uploaded to your website's root directory that implements the Robots Exclusion Protocol (IETF RFC 9309). It instructs web robots and search engine crawlers (such as Googlebot and Bingbot) which paths they are allowed or forbidden to crawl.
Where must the robots.txt file be located?
A robots.txt file must always be placed at the absolute root of the website host (e.g. https://example.com/robots.txt). Placing it in a subfolder (like https://example.com/blog/robots.txt) makes it completely ignored by web crawlers.
Does robots.txt prevent pages from appearing in Google search results?
No. Robots.txt governs crawling, not indexing. If an external website links to a disallowed URL, Google may still index the URL in search results without crawling its content. To guarantee a page is never indexed, use a <meta name='robots' content='noindex'> tag or password protection instead.
What does 'User-agent: *' mean?
The asterisk (*) wildcard denotes a universal fallback rule. It applies the declared allow and disallow directives to all web crawlers that do not have their own specific user-agent rule block declared in the file.
How should I declare my XML sitemap in robots.txt?
Add a 'Sitemap: [absolute-url]' directive at the top or bottom of your robots.txt file (e.g. Sitemap: https://example.com/sitemap.xml). This helps search engines discover your sitemap automatically upon crawling.
What does the 'Crawl-delay' directive do?
Crawl-delay specifies the number of seconds a crawler should wait between successive requests to prevent server resource exhaustion. While supported by Bing, Yandex, and Baidu, Googlebot does not support Crawl-delay (crawl rate is managed in Google Search Console instead).
Are robots.txt path directives case-sensitive?
Yes. RFC 9309 dictates that path matching in robots.txt is strictly case-sensitive. 'Disallow: /admin/' will not block crawlers from requesting '/Admin/' or '/ADMIN/'.
Can I block AI scrapers and LLM training bots with robots.txt?
Yes. You can target specific AI bot user-agents (such as GPTBot, ClaudeBot, CCBot, or Bytespider) with a dedicated 'Disallow: /' block to request that they do not scrape your content for model training.
Why should I avoid disallowing CSS, JavaScript, or image assets?
Modern search crawlers render web pages like a browser to assess mobile responsiveness and layout stability. Disallowing CSS or JS files prevents Googlebot from rendering the page accurately, which can severely damage search rankings.
How do pattern matching wildcards (* and $) work?
An asterisk (*) matches any sequence of characters (e.g. 'Disallow: /*.pdf' blocks all PDF files). A dollar sign ($) designates the end of a URL (e.g. 'Disallow: /*.php$' blocks URLs ending precisely in .php but permits .php?query=1).
Can I download the generated file as robots.txt?
Yes. Click the 'Download robots.txt' button to save the plain text file directly to your device, ready to be transferred to your web server.
Are my site configurations uploaded to an external server?
No. All parsing, directive formatting, and file generation run 100% locally in your web browser. No paths or site architectures are ever sent across the network.
Does this tool work completely offline?
Yes. Once loaded, this Progressive Web App utility operates with zero external network connectivity.
Is the Robots.txt Generator free to use?
Yes. It is completely free with no restrictions, no account creation, and no advertising.
More Tools
Related SEO & Webmaster Tools
Terms
Robots & Crawler Terminology Glossary
REP
Robots Exclusion Protocol, formalized as IETF RFC 9309 in 2022.
User-agent
Header identifying the software agent, browser, or crawler requesting a resource.
Googlebot
The primary web crawling software used by Google to index the Internet.
Bingbot
Microsoft Bing's web crawling bot.
Crawl Budget
The volume of queries and rate limit assigned by search engines to crawl a domain.
Crawl-delay
Non-standard directive requesting an interval between consecutive bot requests.
Wildcard (*)
Pattern matching symbol representing any sequence of zero or more characters.
End Anchor ($)
Pattern matching symbol indicating the end of a URL path.
Disallow
Directive explicitly prohibiting crawler access to a path prefix.
Allow
Directive explicitly permitting crawler access to an otherwise blocked path.
AI Bot
Autonomous crawler collecting data specifically for LLM and artificial intelligence training.
Scraper
Automated script extracting content or data from web pages.
Sources
Official Standards & Documentation
Technical Verification Notice
Technical & Educational Disclaimer
A robots.txt file provides cooperative guidance for automated crawlers adhering to the Robots Exclusion Protocol (RFC 9309). It is not an access-control or security barrier. Never rely on robots.txt to protect sensitive personal records, financial data, or administrative passwords.
Calculations and transformations execute locally on your machine with full offline support. Always verify critical results.
View full legal disclaimer