Robots

HTTP response header

Header usage statistics

Robots response header information and usage statistics.
Websites using header Robots1,827
Percentage of websites that use Robots header<0.1%
Total discovered header values35
Header uses directivesNo
Header values are unique or randomNo
Most popular in the country United States

Robots Directives

Robots directives value information and usage statistics
DirectiveShareWebsites countUnique Values

Robots header usage distribution by website rank



Geographical Distribution

Header usage distribution by websites across the globe.



Websites utilizing Robots

List of websites that use Robots header
DomainCountryRankContacts
global.gotomeeting.com United States9,253
9292.nl Netherlands10,658
gotomeet.me United States36,751
rebelion.org United States77,755
iqconnect.lmhostediq.com United States102,778
ngp.zdf.de Germany147,397
See full domain list

Common header values

List of top common Robots header values
Header valueValue prevalence
none52.38%
noindex, nofollow17.57%
noindex13.25%
noarchive3.07%
noindex,nofollow2.52%
all2.35%
no-cache1.37%
index, follow1.26%
index,follow1.09%
noindex, follow0.82%
Follow0.66%
noindex, nofollow, noarchive, nosnippet0.66%
noindex, nofollow, noimageindex0.49%
NOINDEX,FOLLOW0.44%
NoIndex,NoFollow,NoArchive,NoCache,NoSnippet,NoImageIndex0.27%
noindex,nofollow,noarchive0.22%
index,follow,noarchive0.16%
no-index0.16%
noarchive, noindex, nofollow0.16%
noindex, nofollow, noarchive, nosnippet, noodp, notranslate, noimageindex0.11%