X-Robots-Tag

HTTP response header

The robots meta tag lets you utilize a granular page-specific approach to controlling how an individual page should be indexed and served to users in Google Search results. The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in a robots meta tag can also be specified as an X-Robots-Tag

Header usage statistics

X-Robots-Tag response header information and usage statistics.

Websites using header X-Robots-Tag416,747
Percentage of websites that use X-Robots-Tag header0.42%
Total discovered header values967
Header uses directivesNo
Header values are unique or randomNo
Most popular in the country US

X-Robots-Tag Directives

X-Robots-Tag directives value information and usage statistics

DirectiveShareWebsites countUnique Values

Distribution by websites popularity

X-Robots-Tag detection in the top websites by popularity

Top 10k sites155 websites
Top 100k sites1,159 websites
Top 1m sites9,229 websites

Websites utilizing X-Robots-Tag

List of websites that use X-Robots-Tag header

DomainCountryRankContacts
plus.google.com US25
profiles.google.com US111
currents.google.com US137
support.mozilla.org US155
www.shopify.com US185
stackoverflow.com US289
See full domain list
Flat price per the report, subscription is not required.

Geographical Distribution

Header usage distribution by websites across the globe.






Common header values

List of top common X-Robots-Tag header values

Header valueValue prevalence
noindex, nofollow23.19%
noindex15.86%
noindex, nofollow, noarchive11.08%
none10.27%
noarchive8.31%
nofollow6.86%
all6.14%
all,noodp3.26%
index,follow2.18%
noindex,nofollow1.63%
noindex, nofollow, nosnippet, noarchive1.45%
noindex, follow1.28%
noarchive, noindex, nofollow1.19%
index, follow1.18%
noindex, nofollow, noimageindex0.56%
none, noarchive0.51%
noindex, nofollow, nosnippet0.37%
noimageindex0.33%
noindex, noarchive, nosnippet0.26%
notranslate0.22%