KOENJP

robots.txt Generator

ROBOTS · TXT

Build a robots.txt with allow and disallow rules per crawler, Crawl-delay and sitemap lines. Includes presets (allow all, block all, admin paths, AI crawlers), validation and a plain explanation of every directive.

Quick start

PRESET

A preset replaces the rule groups below. The AI crawler block is added as a separate group at the end.

Rule groups

SITEMAP

One per line, as a full URL starting with https://.

robots.txt


        

        

          
Checks

          
    
        
    
      
    
    
    

    
    
      
    What each line means
    DIRECTIVES
    
      
    
        
    User-agent
    The crawler the group applies to. * means every crawler that isn't named in its own group. A crawler follows only the one group that matches it best.
    
        
    Disallow
    Asks crawlers not to fetch URLs that start with this path. Disallow: / covers the whole site; an empty value blocks nothing.
    
        
    Allow
    Re-opens part of a blocked folder. When rules conflict, the longer (more specific) match wins.
    
        
    * · $
    * matches any characters and $ marks the end of the URL. Example: Disallow: /*.pdf$
    
        
    Crawl-delay
    Seconds to wait between requests. Only some crawlers honor it; Google ignores it.
    
        
    Sitemap
    Full URL of a sitemap file. It doesn't belong to any group, and you can list several.
    
      
    
    
    

    
    Processed in your browser only. Nothing is uploaded.
    
    
    Good to know. robots.txt is a request, not a lock, and some crawlers ignore it. Protect private pages with a login or noindex, and upload the file to your site root (/robots.txt).
    
  
    
  
    
  
    

    



    
  
    
    
    
      HOW TO
      
    How to use
    
      
      
      
    1. Pick a preset (allow all, block all, block admin paths) or add groups yourself.
      2. 
      
    2. Choose a User-agent for each group and write Allow or Disallow paths starting with /.
      3. 
      
    3. Turn on the AI crawler block if you want it, and list sitemap URLs one per line.
      4. 
      
    4. Review the checks, then copy or download robots.txt and upload it to your site root.
      5. 
      
    
    
    
    
    
      PAIRS WELL WITH
      
    Pairs well with
    
      
    
    
  
    
  
  
    
    FAQ
    
    FAQ
    
    
    Does blocking a page in robots.txt remove it from search?
    Not necessarily. robots.txt only asks crawlers not to fetch a URL, so a page linked from elsewhere can still appear without a description. To keep a page out of results, allow crawling and add a noindex tag.
    
    
    Which bots does the AI crawler block cover?
    GPTBot, ChatGPT-User, CCBot, Google-Extended, Applebot-Extended, ClaudeBot, anthropic-ai, Bytespider, meta-externalagent, PerplexityBot and cohere-ai, in one group. Names change over time, and not every crawler obeys robots.txt.
    
    
    Where do I upload the file?
    At the root of your site, e.g. https://example.com/robots.txt. Crawlers ignore it in subfolders, and each subdomain needs its own file.
    
    
    What if Allow and Disallow overlap?
    Major search engines follow the longest (most specific) matching rule, and prefer Allow when lengths tie. So you can block /admin/ and still allow /admin/help/.