X-Robots-Tag

HTTP response header

The robots meta tag lets you utilize a granular page-specific approach to controlling how an individual page should be indexed and served to users in Google Search results. The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in a robots meta tag can also be specified as an X-Robots-Tag

Header usage statistics

X-Robots-Tag response header information and usage statistics.

Websites using header X-Robots-Tag 166,175
Percentage of websites that use X-Robots-Tag header 0.31%
Total discovered header values 553
Header uses directives No
Header values are unique or random No
Most popular in the country United States of America

Distribution by websites popularity

X-Robots-Tag detection in the top websites by popularity

Top 10k sites 114 websites
Top 100k sites 756 websites
Top 1m sites 5,289 websites

Websites utilizing X-Robots-Tag

List of websites that use X-Robots-Tag header

Domain Country Rank Contacts
plus.google.com United States of America 12
stackoverflow.com United States of America 63
support.mozilla.org United States of America 162
discordapp.com United States of America 228
discord.gg United States of America 229
moz.com United States of America 332
See full domain list
Flat price per the report, subscription is not required.

Geographical Distribution

Header usage distribution by websites across the globe.






Common header values

List of top common X-Robots-Tag header values

Header value Value prevalence
noindex, nofollow, noarchive 18.43%
noindex, nofollow 17.38%
noindex 10.96%
noarchive 10.62%
nofollow 10.19%
all 7.98%
none 7.19%
all,noodp 3.20%
index,follow 3.13%
noindex,nofollow 2.62%
index, follow 1.44%
noarchive, noindex, nofollow 1.27%
noindex, follow 0.84%
noindex, nofollow, nosnippet, noarchive 0.75%
noindex, nofollow, nosnippet 0.34%
follow 0.23%
noimageindex 0.22%
noindex, noarchive, nosnippet 0.15%
notranslate 0.15%
index 0.14%