With all the buzz about http://www.tellinya.com/read/2007/09/09/defend-your-website-against-google-proxy-hijacking/ I put my previously coded PHP class to good use and now filters my website traffic. And works very well.

PS:I am curious if once an IP verifies as Googlebot would it be safe to consider the entire C class as Google’s???

Thanks.

From now on, you can Check/verify googlebot ip’s very easy.
Here you can verify ip addresses:
http://www.2dwebdesign.nl/googlebotchecker.php

Notice that this tool makes use of the reverse DNS technique which google has recommended.
This tool also distinguishes all google ip’s from googlebot ip’s.

Regards,
Houman

We do use WHOIS as the first pass, it’s more reliable than DNS lookups for one. For anyone who thinks this is all meaningless, with one site I’m working with, more than 95% of the “Googlebot” visits on a daily basis fail the WHOIS test.

So we really on need to fool with DNS lookups on the 5% that pass a WHOIS.

The process (generified since Googlebot isn’t the only spider) is:
1) We have a user-agent identifying itself as a spider
2) Check the IP vs our database of good and bad IPs for that spider
3) Use WHOIS for anything that we can’t look up
4) Use DNS to verify those that pass the WHOIS test, unless it’s MSN
4a) (if MSNBot), cross your fingers and hope you didn’t block the real bot

Robin, don’t assume that those proxies would appear in any published database, respond on a predictable port, etc. If it is a public proxy, don’t assume that the IP making the request is the same as the public proxy address. Some of the biggest public proxies run multiple servers, and the IP fetching pages is not the public WWW server.

And even if there was an open proxy running the useragent string would contain “Googlebot”, and the IP address would be listed in the open proxy databases.

Robin, not all proxies are just open ports like you’re attempting to find. Some are CGI or PHP proxy sites, and they don’t all pass the user agent thru, or you can configure the user agent. Some CGI proxy sites such as Proxify by default will tell the world you’re using Safari on a Mac, so on and so forth.

Probably someone just running a stealth crawler, not Google (I’m still wondering how you concluded it was Google) but armed just with the IP address you reference, ARIN.NET claims it’s in a block of IP’s assigned to CMP Media LLC and it’s also a banned IP on the Twiki blacklist:
http://twiki.pula.org/cgi-bin/twiki/view/TWiki/BlackListPlugin

Pretty sure it’s not Google

I instituted the double lookup and am now banning all bots that don’t pass the test. I run an ecommerce site with thousands of products and have been the victim of scrapers who copy and repost my products on their pages. The question I have is that within 2 days of implementing bot traps, all but a handful of our pages have gone supplemental or disappeared and I’m wondering if the bot traps may be the cause, or purely coincidence. The majority of SERPs remaining are old links to pages that 301 to new “friendly” URLs. Googlebot is still visiting each day and scans thousands of pages each visit, but recently started hitting old URLs that were 301’d months ago. Any advice would be greatly appreciated.

Lot’s of ways it could happen, even a proxy server feeding Google a cloaked link to your site (read my posts above) but they aren’t crawling from those IPs or we would all know about it.

Besides, if Google wanted to checked to see if a web site was cloaking data, they would only have to spot check a few pages, not a full crawl, it wouldn’t be efficient to do a complete crawl to establish if a site was cloaking.

Best Regards,
J-Man