HTTP response header

The robots meta tag lets you utilize a granular page-specific approach to controlling how an individual page should be indexed and served to users in Google Search results. The X-Robots-Tag can be used as an element of the HTTP header response for a given URL. Any directive that can be used in a robots meta tag can also be specified as an X-Robots-Tag

Header usage statistics

X-Robots-Tag response header information and usage statistics.

Websites using header X-Robots-Tag 415,056
Percentage of websites that use X-Robots-Tag header 0.42%
Total discovered header values 887
Header uses directives No
Header values are unique or random No
Most popular in the country United States of America

Distribution by websites popularity

X-Robots-Tag detection in the top websites by popularity

Top 10k sites 154 websites
Top 100k sites 1,043 websites
Top 1m sites 8,422 websites

Websites utilizing X-Robots-Tag

List of websites that use X-Robots-Tag header

Domain Country Rank Contacts
plus.google.com United States of America 25
profiles.google.com United States of America 111
currents.google.com United States of America 137
support.mozilla.org United States of America 155
www.shopify.com United States of America 185
stackoverflow.com United States of America 289
See full domain list
Flat price per the report, subscription is not required.

Geographical Distribution

Header usage distribution by websites across the globe.

Common header values

List of top common X-Robots-Tag header values

Header value Value prevalence
noindex, nofollow 23.74%
noindex 15.79%
noindex, nofollow, noarchive 11.72%
none 11.00%
nofollow 9.75%
noarchive 8.34%
all 5.83%
all,noodp 3.62%
index,follow 2.18%
noindex,nofollow 1.51%
noindex, nofollow, nosnippet, noarchive 1.33%
noindex, follow 1.22%
index, follow 1.17%
noarchive, noindex, nofollow 0.91%
noindex, nofollow, noimageindex 0.59%
none, noarchive 0.52%
noindex, nofollow, nosnippet 0.43%
noimageindex 0.32%
noindex, noarchive, nosnippet 0.25%
notranslate 0.23%