X-Robots-Tag

HTTP response header

The robots meta tag lets you utilize a granular page-specific approach to controlling how an individual page should be indexed and served to users in Google Search results. The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in a robots meta tag can also be specified as an X-Robots-Tag

Header usage statistics

X-Robots-Tag response header information and usage statistics.

Websites using header X-Robots-Tag 39,798
Percentage of websites that use X-Robots-Tag header 0.37%
Total discovered header values 303
Header uses directives No
Header values are unique or random No
Most popular in the country United States of America

Distribution by websites popularity

X-Robots-Tag detection in the top websites by popularity

Top 10k sites 109 websites
Top 100k sites 673 websites
Top 1m sites 5,242 websites

Websites utilizing X-Robots-Tag

List of websites that use X-Robots-Tag header

Domain Country Rank Contacts
plus.google.com United States of America 12
support.mozilla.org United States of America 162
discordapp.com United States of America 228
discord.gg United States of America 229
moz.com United States of America 332
wunderground.com United States of America 369
See full domain list
Flat price per report, subscription is not required.

Geographical Distribution

Header usage distribution by websites across the globe.






Common header values

List of top common X-Robots-Tag header values

Header value Value prevalence
noarchive 16.04%
nofollow 14.41%
noindex, nofollow 13.52%
all 11.02%
noindex 8.76%
noindex,nofollow 8.24%
noindex, nofollow, noarchive 8.15%
none 7.24%
all,noodp 4.62%
index, follow 1.14%
index,follow 1.14%
noindex, nofollow, nosnippet, noarchive 0.62%
noimageindex 0.62%
noindex, nofollow, nosnippet 0.43%
noarchive, notranslate 0.27%
noarchive, noindex, nofollow 0.20%
index 0.19%
noindex, noarchive, nosnippet 0.18%
noindex, follow 0.17%
noindex, nofollow, noimageindex 0.15%