Comments on: MSFT adds bot verification

By: Tom

Tom — Fri, 22 Jun 2007 07:42:34 +0000

I wrote a script (maybe you know about it ?) to ban bad bots.
There are offen questions about, how googlebot handels the robots.txt.
So i also implemented the RDNS Check for Google.

By: Bryan Ong

Bryan Ong — Fri, 01 Jun 2007 01:30:02 +0000

Hi Matt

I’ve got a question. Both visitsnw.com.au and visitnsw.com got a Pagerank of 6 and same equal amount of links. Why is that so?

All of Tourism NSW’s ranking is on their .com.au domain. Can you please explain? How does Google knows that it is the .com.au that it has to give rankings and not the .com domain even though it has same equalt amount of links and pagerank?

Thanks

By: Mike

Mike — Sat, 12 May 2007 12:48:12 +0000

The solution discussed in this post is more complicated than necessary. I’ve explained a better solution here:

http://botsosphere.blogspot.com/2007/05/automatic-verification-of-machine.html

By: Peter Linsley

Peter Linsley — Sat, 02 Dec 2006 01:05:30 +0000

Hi Matt,

The issue of whether Ask could support bot authentication was raised in Las Vegas at PubCon and we’ve updated the webmaster FAQ. No surprises here, it’s the old DNS roundtrip lookup trick; all our crawlers are under the ask.com domain:

http://about.ask.com/en/docs/about/webmasters.shtml

Cheers.

Peter

By: Multi-Worded Adam

Multi-Worded Adam — Fri, 01 Dec 2006 06:26:27 +0000

So what you’re saying is that my Gigablast prediction is an off-the-board pick, correct?

By: IncrediBILL

IncrediBILL — Thu, 30 Nov 2006 16:55:28 +0000

Matt, I did a little checking and it looks like the ExaleadGuy might’ve beaten you to the punch on this one.

Now if they could just make their user agent string a little more useful or post a page with information about their crawler I can find…

By: Mack

Mack — Thu, 30 Nov 2006 16:52:27 +0000

Your timing with this post is uncanny. This morning I checked my stats with recent visitors and noticed two totally different IP addresses that were identified as Googlebot. With a little reasearch I found that IP address 207.176.38.75, one of the two identified as Googlebot, is actually owned by Beyond The Network America, Inc. in Herndon, VA. My question would be, is this something for me to be concerned with?

By: ExaleadGuy

ExaleadGuy — Thu, 30 Nov 2006 12:57:53 +0000

That’s funny because actually, Exalead has got this reverse DNS stuff from the beginning of its existence (that is to say 2000)
Ah we should have patented it and ask Google and MSN for royalties now

In fact we did it because we were so sure that the other bots already did it for a long time.

If it is that fashion, we gonna add it to our FAQ too

> host 193.47.80.51
51.80.47.193.in-addr.arpa domain name pointer crawl15.exabot.com.
> host crawl15.exabot.com.
crawl15.exabot.com has address 193.47.80.51

By: Matt Cutts

Matt Cutts — Thu, 30 Nov 2006 05:50:43 +0000

Jojo, points for the first post I saw to get it, with joergvader coming up in second place.

M.W.A., if Gigablast does it next, I will do a post just to commemorate that.

By: Dema

Dema — Thu, 30 Nov 2006 05:10:35 +0000

Well they do have to increase the bot protection, but those bots aren’t very good, most of the software right not can by pass them easily, but it takes a brain to put that software together, which is why its not so cheap.