Comments on: Learn more about robots.txt

By: Derek Jansen

Derek Jansen — Wed, 02 Mar 2011 09:56:12 +0000

Thanks Matt – Was a little confused between robots.txt and the .htaccess but this clarifies the two.

By: Andy H

Andy H — Wed, 10 Mar 2010 17:19:03 +0000

The only way to figure out Google is to look backwards. People should forget all they think they've learnt and read The Anatomy of a Search Engine before they do anything else.In there, you will find the answer to the universe and everything. Matts video explanation above was first explained back in '97. It seems to me that very little has really changed, which is a good thing.

By: Yudh

Yudh — Tue, 02 Mar 2010 00:27:07 +0000

Thank you very much for this video, its helpful for newbie like me

By: Bobby

Bobby — Thu, 25 Feb 2010 07:09:44 +0000

Thanks for posting this. i’ve been wondering about those robots violating this for a while.

By: Evans

Evans — Thu, 11 Feb 2010 21:43:31 +0000

Well I always write ‘index, follow’ in the robots.txt I’ve never really cared about changing it into anything else.

By: Amy

Amy — Wed, 13 Jan 2010 22:06:58 +0000

I usually ignore the robots.txt file. Thanks for the clip, I will keep it higher on my priority list now.

By: Tom

Tom — Tue, 12 Jan 2010 20:18:24 +0000

Thanks Matt! Great information as always. I rely heavily on my robots.txt. It is time I double check mine…I did the old “set and forget.” Thanks!

By: zoe

zoe — Wed, 30 Dec 2009 05:09:05 +0000

Hi, Matt. I’m so disappointed that I cannot watch the video in my country. However, thank you for sharing.

By: Roni

Roni — Tue, 24 Nov 2009 14:49:47 +0000

Great simple & to the point explanation Matt. I always thought blocking in robots.txt = no crawl = noindex. Now I know noindex and no crawl are two completely different things. Besides, you can always remove specific urls you do not want showing on Google in Webmaster Tools…

By: Jeremy Chatfield

Jeremy Chatfield — Sat, 31 Oct 2009 22:50:13 +0000

Thanks for the video Matt; consistent with everything you, robotstxt.org and the Google webmaster blog have said. Useful re-confirmation and explanation.

@Anthony Von Ducci – The listing that I can see for you is rationally consistent with what Matt said. The page has not been crawled, in accordance with your robots.txt, so it consists of a link and “similar”. If you put “Disallow: /”, that’s exactly what should happen. The robots can’t see the NOINDEX you put in the home page, because they can’t crawl it. If you *had* an ODP (Open Directory Project aka DMOZ) listing, then that would be shown by Google, even if you used the NOODP robots directive on the home page, because the directive would be on a page that the robots aren’t allowed to see. Shows that the robots.txt file is working exactly as it should!

@Aery – read robotstxt.org and use the “User-Agent” lines to identify the bots by name.

@Dudibob – you need “Canonical link refs”, I think, to solve your duplicate pages problem; just search for it, it works for all the major search engines, eventually – you need to have the dups crawled, of course, to see the on-page meta tag!