you say to your clients you will crawl their sites as googlebot crawls. And then you should be sure of requesting the same HTTP headers as googlebot from their servers. The response information and later data you collect from a server depend on what you request in your crawlers HTTP headers.
Websites using header X-GoogleNews-Bot | 2,349 |
Percentage of websites that use X-GoogleNews-Bot header | <0.1% |
Total discovered header values | 1 |
Header uses directives | No |
Header values are unique or random | No |
Most popular in the country | ![]() |
Top 10k sites | 6 websites |
Top 100k sites | 19 websites |
Top 1m sites | 113 websites |
Domain | Country | Rank | Contacts |
---|---|---|---|
|
![]() |
167 | |
|
![]() |
380 | |
|
![]() |
1,616 | |
|
![]() |
1,919 | |
|
![]() |
1,919 | |
|
![]() |
2,438 |
Header value | Value prevalence |
---|---|
false | 100.00% |