Archives for November 2006

MSFT adds bot verification

A couple months ago Google gave a way to do Googlebot verification. Basically, it involves two steps: a reverse DNS lookup followed by a forward DNS lookup. Now Microsoft has implemented the same DNS policies, so you can use the same method to verify MSNBot.

Good for them. Who will be next to implement this open method to authenticate search engine bots: Yahoo!, Ask, or Exalead? Place your bets. 🙂

Update: w00t! I’m digging out from comments and just saw this comment by Peter Linsley of Ask: “The issue of whether Ask could support bot authentication was raised in Las Vegas at PubCon and we’ve updated the webmaster FAQ. No surprises here, it’s the old DNS roundtrip lookup trick; all our crawlers are under the domain:”

So it looks like Ask wins the prize! Here’s the specific entry in Ask’s documentation:

Q. How do I authenticate the Ask Crawler?
A: A User-Agent is no guarantee of authenticity as it is trivial for a malicious user to mimic the properties of the Ask Crawler. In order to properly authenticate the Ask Crawler, a round trip DNS lookup is required. This involves first taking the IP address of the Ask Crawler and performing a reverse DNS lookup ensuring that the IP address belongs to the domain. Then perform a forward DNS lookup with the host name ensuring that the resulting IP address matches the original.

I’m in Omaha

I’m in Omaha, so light posting this week. I’m checking with PR if I can post pictures of my slides from the “Search Engine Smackdown” talk at PubCon. And since my slides were all nice, polite, and smackdown-less, I expect that I’ll do a “poor man’s version” of the talk on the blog sometime soon.