HTTP response header

you say to your clients you will crawl their sites as googlebot crawls. And then you should be sure of requesting the same HTTP headers as googlebot from their servers. The response information and later data you collect from a server depend on what you request in your crawlers HTTP headers.

Header usage statistics

X-GoogleNews-Bot response header information and usage statistics.

Websites using header X-GoogleNews-Bot 2,327
Percentage of websites that use X-GoogleNews-Bot header <0.1%
Total discovered header values 1
Header uses directives No
Header values are unique or random No
Most popular in the country United States of America

Distribution by websites popularity

X-GoogleNews-Bot detection in the top websites by popularity

Top 10k sites 6 websites
Top 100k sites 20 websites
Top 1m sites 114 websites

Websites utilizing X-GoogleNews-Bot

List of websites that use X-GoogleNews-Bot header

Domain Country Rank Contacts
lifehacker.com United States of America 167
gizmodo.com United States of America 380
kotaku.com United States of America 1,616
theonion.com United States of America 1,919
www.theonion.com United States of America 1,919
io9.gizmodo.com United States of America 2,438
See full domain list

Geographical Distribution

Header usage distribution by websites across the globe.

Common header values

List of top common X-GoogleNews-Bot header values

Header value Value prevalence
false 100.00%