Comments on: Bot Obedience: Herding Googlebot

By: brandi belle

brandi belle — Fri, 16 Apr 2010 02:14:41 +0000

This comment is directed directed toward to the comment person above “dave” . I agree, I would also be interested in reading a blog post from you on the “site:” command issue. I haven’t read to much into this but am always looking for more. Dave, from what I heard, the quality from backlinks are good.

By: Kristi Wachter

Kristi Wachter — Wed, 26 Nov 2008 19:20:42 +0000

Did something change with this during the past week or so? I have a test site that has this in the robots file:

User-agent: *
Disallow: /*?

and also has this on every page:

Up until last week, pages were steadily disappearing from Google; last week, a search of

commonword site:test.example.com

returned only 5 results (down from more than 1000).

Suddenly, today, it’s returning 16,000 pages!

I’ve double-checked that the robot file and the noindex metatag on every page are still in place.

Any idea why this would have changed suddenly? Lots of folks have test sites, and of course we don’t want Google to index all that duplicate content. I can put an htaccess password on the site, but it’d be nice not to have to.

Thanks for the great info!

By: Mark

Mark — Mon, 24 Sep 2007 07:55:34 +0000

Matt, you robots.txt file is pretty open – does that result in the bots picking up duplicate content on your site from the various locations your posts are stored on your site? Great articles btw.

By: Danielle Fox

Danielle Fox — Fri, 27 Jul 2007 16:53:55 +0000

IncrediBILL, maybe it’s better to use Agent tag. I’m not sure you should stick to IP’s…

By: ZZPrices

ZZPrices — Tue, 19 Dec 2006 18:27:27 +0000

I have a similar case where I’ve tried to control Google by denying access to /cgi-bin/ and it still indexes all of those pages.

By: TheWheeler

TheWheeler — Fri, 03 Nov 2006 11:58:22 +0000

Looks like Google is spidering all of the pages, no matter what’s written in robots.txt. At least that happened in my case.

By: Data Recovery

Data Recovery — Sat, 23 Sep 2006 19:27:12 +0000

How safe is google URL removal ???

I have mistakenly uploaded two pages index.htm and index.php

Google cached index.php and yahoo and MSN has cached index.htm

Which URL should I remove ?

By: Joe

Joe — Fri, 11 Aug 2006 12:42:56 +0000

Hi Matt,

This is a late reply, but a valid “feature request”

My site has a high incidence of traffic from site rippers, spambots, image farming, and competitors stealing images and content. I find sites using my content and URLs in Google searches on a regular basis and report them as SPAM. I use traps, but a spider authentication process would be an easily implemented method of alleviating the problem of bad bots and rippers. A credit card processing company uses a simple method. I receive a payment notification via the company’s server posting a query to a designated URL on my site. I post (reflect if you will) that query, appended with a validation request back to their server, and they display a string of either “verified” or “invalid”. A method similar to this would simplify the process of setting trusted “users” and granting access to spider my site.

Best of luck and keep those videos coming!!!

Joe

By: In-House SEO

In-House SEO — Mon, 07 Aug 2006 22:30:34 +0000

Matt said: “At a page level, use meta tags at the top of your html page. The noindex meta tag will keep a page from showing up in Google’s index at all. This tag is great on any page that’s confidential.”

This doesn’t seem to be working as designed. I’m seeing pages tagged “noindex” all over Google’s search results — and ranking well. Not isolated instances either, but dozens of pages that clearly are not intended to supposed to be indexed showing up at the top of the SERPs.

By: DIO

DIO — Sun, 06 Aug 2006 16:44:37 +0000

SEO is more of an art than a science since most people are not aware of what the exact search engine algorithm is like. Nonetheless, the tips listed on this page are useful.