$ip =~ /^((64\.68|66\.249)\.(6[4-9]|[78]\d|9[0-5])|64\.233\.(1[6-8]\d|19[01])|216\.239\.(3[2-9]|[45]\d|6[0-3]))\./

bye

Here are my “thoughts and ideas”.

Immediately before these multiple lines with that “accelerator” I found that a person in the UK was trying to visit my site, of course “one bad apple ruined the whole bunch” so that I had the entire 81.x.x.x for the UK (and Europe) blocked. Likewise I don’t allow almost all of the APNIC group of IP’s…. too many email harvesters in Asia, thus many spam originate from Asia … first received line in emails cannot be forged… and seeing too much bad activity from Asia and Europe got most of those areas blocked. Unfortunate for innocent parties in those areas. (I block many US and most Canadian IP’s due to repeated bad activities whether its email spam or bad web activity.)

Nevertheless, found a UK visitor trying and trying to view my site, of course 403′d. Then after many of those I found the “accelerator” UA entry for this Google tool. The IP for a business in Mass, USA. My guess is that this Google tool looks for a cached copy of your site sitting on any number of NON-SE CACHES and if it finds a copy of your site or the wanted page on a cache near to the viewer then such copy is served to the viewer.

Good and bad. Here’s the good, first. That user agent is a real person, so a site owner can allow it or disallow. Site owner choice.

The bad items now. It “in effect” hides the actual viewers IP, instead showing the IP of the cache site (which may vary). A site owner can decide to allow this anonymous viewing or decide to block it via user-agent of Google Web Accelerator Cache Warmer. A good part is that with this new user agent roaming sites, there is one advantage. If a site owner monitors their log file (monthly log file would be best but could be huge), simply search the log for that UA and jot down the IP numbers. That should give a listing of some “unknown” and possibly undesired caches around the country or the world. The ending part of the full UA after the semicolon, I believe “may vary”. Not sure if it is a version number or an indicator of the visitor’s origination point, but it might be interesting to note that portion with the IP list that would be in the results.

Now, a good item. If that UA is allowed (at least at first) then a site owner can find many NON-SE CACHES around the country or world. Not knowing hold old they may allow their caches to become can be an issue. They might be serving VERY OLD copies of your site to visitors. The option is then to block those IP’s, or cidr groups for those IP’s especially if a cidr group is for a business. That might stop or block some of the “unknown” bots/crawlers.

The latter has always been an issue with me. Unknown bots! Especially if it is a bot owned by EDU facilities or by businesses. You never know how they might use your content. And whether they have a name or not if they don’t have full info about their bots at their site and if they’re not a normal SE then most usually I don’t want them. They use my bandwidth, they could be infringing on my writings or copyright and they could be using code of my own design (I only did one “visitor counter” script) thus the latter is not a big issue.

I believe this “info” may be correct about the Google accelerator, but it might take some verification by one or more people. Most definitely I “could not” see why someone at a business would view just one page of my site, but I can very well associate it with it being a “cache” source and that it was associated with the one site visitor that was blocked.

Most probably I will allow it for a short term, just to watch some of its activity (does it obey robots.txt or fetch too fast) or maybe I will block it completely.

Actually on an update…. it might be that the “accelerator” tool may have found that such Mass USA site was a super-high-speed source with less routing for getting to my server in southern california. Nevertheless, it does “hide” the visitor’s IP, and if that IP gets my pages they may in fact actually cache some of my pages.

It could be an interesting study. Who owns those IP’s and where are they located. And what is the relationship or info at the end of the UA string, a program version number or origination indicator of the visitor?

Best regards,

(ps please don’t put my email onto ANY lists of any kind and please ensure that it is not “anywhere” in the blog code so that it can’t be harvested…. it “is” a disposable email but I really don’t want to dispose of it. thanks.)

We have several client’s sites in which our stats (visitor analysis)
pages showed up in Google’s index. I submitted a
sitemap (our script is set to not crawl the filesystem, but instead the
webpages) and made sure that those pages/folders were not included. I also password protected the pages. They still show up in Google Webmaster Tools (sitemaps) section as an http error (401).

I had to explain this first, but my real question is…

Does Google toolbar use personal information or something because those
pages are definitely not linked from anywhere. The only way they could
get into Google’s index is if the Google toolbar or some other Google
application was sending info from the address bar back to Google. The only time I ever go to those pages is by manually typing in the url.

Love Google’s services, by the way. Not trying to expose anything…
just very curious.

Thanks,

Tim

May I ask a quick question?
I operate a Canada business. Can I register a .com domain, use a DNS and web host in USA?
If I do that will harm my ranking on google.ca?

Leeko

I do use If-Modified-Since, but this is a potential way of your being able to avoid downloading the 1st *redundant* copy of (say) an image ever…

Get your folks to look at the Content-MD5 header definition.

I don’t use ETag (knowingly), heck I don’t even understand it (yet) after ~10 years serving Web pages!

Rgds

Damon