← All Tools

X-Robots-Tag Header Builder

Compose an X-Robots-Tag response header the way Google actually parses it — a global directive line for every crawler, plus optional per-bot lines (Googlebot: noindex, nofollow) that override the global one when the bot's user agent matches. Handles unavailable_after in RFC 822 & ISO 8601, the numeric limits max-snippet / max-image-preview / max-video-preview, and the newer indexifembedded. Also parses an incoming header dump and shows what each bot would see. Nothing leaves your browser.

Global directives (apply to every crawler)

presets: Fully block — noindex, nofollow, noarchive Staging — noindex, follow PDF — noindex, noimageindex News — max-snippet:-1, max-image-preview:large Widget-only — noindex, indexifembedded

Per-bot overrides

Prefix the header value with a case-insensitive user-agent token & a colon. Bots that match take the specific line and ignore the global one; other bots stay on the global line.

Generated header


  

    
    
    
    
  

  





  
Server snippets

  








  
Parse an X-Robots-Tag header

  
  
  

    
    
  

  






  
Reference — every directive Google honours

  

    
all
Equivalent to index, follow — the default; setting it does nothing new.

    
none
Shortcut for noindex, nofollow. If both none and any positive directive are present, the more restrictive wins.

    
noindex
Do not show this URL in search results, even if other pages link here.

    
nofollow
Don’t treat outbound links as endorsements — no PageRank flow, no crawl of those links from here.

    
noarchive
Don’t show a “Cached” link in results.

    
nosnippet
Don’t show any text snippet or video preview. Overrides max-snippet and max-video-preview.

    
indexifembedded
Even with noindex, allow indexing of content embedded via <iframe> in a page that is indexed.

    
max-snippet:n
At most n characters of text in the snippet. 0 = none, -1 = unlimited.

    
max-image-preview:[none|standard|large]
Cap the size of image previews.

    
max-video-preview:n
At most n seconds of video preview. 0 = a still, -1 = unlimited.

    
notranslate
Suppress the “Translate this page” offer.

    
noimageindex
Don’t index images referenced from this page.

    
unavailable_after:date
Drop the URL from the index after date. Accepts RFC 822 or ISO 8601.

  

  

    Directives are case-insensitive and comma-separated. When a bot sees a global line AND a matching per-bot line, only the per-bot line applies for that bot. When multiple directives conflict, the more restrictive wins — e.g. noindex, index is noindex.