X-Robots-Tag

HTTP response header

The robots meta tag lets you utilize a granular page-specific approach to controlling how an individual page should be indexed and served to users in Google Search results. The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in a robots meta tag can also be specified as an X-Robots-Tag

Header usage statistics

X-Robots-Tag response header information and usage statistics.

Websites using header X-Robots-Tag 39,104
Percentage of websites that use X-Robots-Tag header 0.34%
Total discovered header values 279
Header uses directives No
Header values are unique or random No
Most popular in the country United States of America

Distribution by websites popularity

X-Robots-Tag detection in the top websites by popularity

Top 10k sites 102 websites
Top 100k sites 628 websites
Top 1m sites 5,014 websites

Websites utilizing X-Robots-Tag

List of websites that use X-Robots-Tag header

Domain Country Rank Contacts
plus.google.com United States of America 12
support.mozilla.org United States of America 162
discordapp.com United States of America 228
discord.gg United States of America 229
moz.com United States of America 332
wunderground.com United States of America 369
See full domain list

Geographical Distribution

Header usage distribution by websites across the globe.






Common header values

List of top common X-Robots-Tag header values

Header value Value prevalence
noarchive 16.94%
noindex, nofollow 14.72%
nofollow 12.55%
all 10.52%
noindex, nofollow, noarchive 8.86%
noindex,nofollow 8.40%
noindex 7.35%
none 7.17%
all,noodp 5.91%
index,follow 1.09%
index, follow 1.00%
noindex, nofollow, nosnippet, noarchive 0.66%
noimageindex 0.51%
noindex, nofollow, nosnippet 0.42%
noindex, noarchive, nosnippet 0.27%
index 0.21%
noarchive, noindex, nofollow 0.21%
noindex, follow 0.18%
noindex, nofollow, noimageindex 0.14%
noarchive, notranslate 0.13%