Comments on: Learn more about robots.txt http://www.mattcutts.com/blog/robots-txt-remove-url/ neat fun stuff Fri, 19 Mar 2010 01:55:22 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Evans http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-467078 Evans Thu, 11 Feb 2010 21:43:31 +0000 http://www.mattcutts.com/blog/?p=3204#comment-467078 Well I always write 'index, follow' in the robots.txt I've never really cared about changing it into anything else. Well I always write ‘index, follow’ in the robots.txt I’ve never really cared about changing it into anything else.

]]> By: Amy http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-452750 Amy Wed, 13 Jan 2010 22:06:58 +0000 http://www.mattcutts.com/blog/?p=3204#comment-452750 I usually ignore the robots.txt file. Thanks for the clip, I will keep it higher on my priority list now. I usually ignore the robots.txt file. Thanks for the clip, I will keep it higher on my priority list now.

]]> By: Tom http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-452210 Tom Tue, 12 Jan 2010 20:18:24 +0000 http://www.mattcutts.com/blog/?p=3204#comment-452210 Thanks Matt! Great information as always. I rely heavily on my robots.txt. It is time I double check mine...I did the old "set and forget." Thanks! Thanks Matt! Great information as always. I rely heavily on my robots.txt. It is time I double check mine…I did the old “set and forget.” Thanks!

]]> By: zoe http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-445990 zoe Wed, 30 Dec 2009 05:09:05 +0000 http://www.mattcutts.com/blog/?p=3204#comment-445990 Hi, Matt. I'm so disappointed that I cannot watch the video in my country. However, thank you for sharing. Hi, Matt. I’m so disappointed that I cannot watch the video in my country. However, thank you for sharing.

]]> By: Roni http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-425627 Roni Tue, 24 Nov 2009 14:49:47 +0000 http://www.mattcutts.com/blog/?p=3204#comment-425627 Great simple & to the point explanation Matt. I always thought blocking in robots.txt = no crawl = noindex. Now I know noindex and no crawl are two completely different things. Besides, you can always remove specific urls you do not want showing on Google in Webmaster Tools... Great simple & to the point explanation Matt. I always thought blocking in robots.txt = no crawl = noindex. Now I know noindex and no crawl are two completely different things. Besides, you can always remove specific urls you do not want showing on Google in Webmaster Tools…

]]> By: Jeremy Chatfield http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-411868 Jeremy Chatfield Sat, 31 Oct 2009 22:50:13 +0000 http://www.mattcutts.com/blog/?p=3204#comment-411868 Thanks for the video Matt; consistent with everything you, robotstxt.org and the Google webmaster blog have said. Useful re-confirmation and explanation. @Anthony Von Ducci - The listing that I can see for you is rationally consistent with what Matt said. The page has not been crawled, in accordance with your robots.txt, so it consists of a link and "similar". If you put "Disallow: /", that's exactly what should happen. The robots can't see the NOINDEX you put in the home page, because they can't crawl it. If you *had* an ODP (Open Directory Project aka DMOZ) listing, then that would be shown by Google, even if you used the NOODP robots directive on the home page, because the directive would be on a page that the robots aren't allowed to see. Shows that the robots.txt file is working exactly as it should! @Aery - read robotstxt.org and use the "User-Agent" lines to identify the bots by name. @Dudibob - you need "Canonical link refs", I think, to solve your duplicate pages problem; just search for it, it works for all the major search engines, eventually - you need to have the dups crawled, of course, to see the on-page meta tag! Thanks for the video Matt; consistent with everything you, robotstxt.org and the Google webmaster blog have said. Useful re-confirmation and explanation.

@Anthony Von Ducci – The listing that I can see for you is rationally consistent with what Matt said. The page has not been crawled, in accordance with your robots.txt, so it consists of a link and “similar”. If you put “Disallow: /”, that’s exactly what should happen. The robots can’t see the NOINDEX you put in the home page, because they can’t crawl it. If you *had* an ODP (Open Directory Project aka DMOZ) listing, then that would be shown by Google, even if you used the NOODP robots directive on the home page, because the directive would be on a page that the robots aren’t allowed to see. Shows that the robots.txt file is working exactly as it should!

@Aery – read robotstxt.org and use the “User-Agent” lines to identify the bots by name.

@Dudibob – you need “Canonical link refs”, I think, to solve your duplicate pages problem; just search for it, it works for all the major search engines, eventually – you need to have the dups crawled, of course, to see the on-page meta tag!

]]> By: I. Ali http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-411720 I. Ali Sat, 31 Oct 2009 17:53:46 +0000 http://www.mattcutts.com/blog/?p=3204#comment-411720 Hi Matt, My one client owns art gallery website having thousands of web pages in which artists gests registered and also some times delete their profile, when they delete their profile then broken link is generated, in such way right now site is having hundreds of broken links Website is dynamic and having more and more broken links so removal of broken links from website is some tough task and major problem is that client is not providing me ftp details he says you provide me suggestion I will rectify the same. "I told him to remove all the broken links from web pages" then he told me that it is not possible becuase site is having hundreds of broken links. Now pls let me know could we fix those broken links with robots.txt also? Awaiting for a reply Thanks Hi Matt,

My one client owns art gallery website having thousands of web pages in which artists gests registered and also some times delete their profile, when they delete their profile then broken link is generated, in such way right now site is having hundreds of broken links
Website is dynamic and having more and more broken links so removal of broken links from website is some tough task and major problem is that client is not providing me ftp details he says you provide me suggestion I will rectify the same.
“I told him to remove all the broken links from web pages” then he told me that it is not possible becuase site is having hundreds of broken links.
Now pls let me know could we fix those broken links with robots.txt also?

Awaiting for a reply

Thanks

]]> By: Gareth http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-411298 Gareth Fri, 30 Oct 2009 23:51:45 +0000 http://www.mattcutts.com/blog/?p=3204#comment-411298 Should host make it simpler to offer robot and non-crawl functions for newbies? Should host make it simpler to offer robot and non-crawl functions for newbies?

]]> By: Selvam http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-408826 Selvam Sun, 25 Oct 2009 17:08:52 +0000 http://www.mattcutts.com/blog/?p=3204#comment-408826 Hi matt I have seen your most of the videos in you tube and congratulations to one million video views. And finally come up with small clarification from you. Here http://www.google.com/support/forum/p/Webmasters/thread?tid=30390cbd6a1754c6&hl=en i have created thread recently related to Google indexing my site. The problem seems peculiar.. Would you please answer my question in google webmasters thread? Hope you can able to resolve my issue. :) Hi matt
I have seen your most of the videos in you tube and congratulations to one million video views. And finally come up with small clarification from you. Here http://www.google.com/support/forum/p/Webmasters/thread?tid=30390cbd6a1754c6&hl=en i have created thread recently related to Google indexing my site. The problem seems peculiar.. Would you please answer my question in google webmasters thread? Hope you can able to resolve my issue. :)

]]> By: Sean I. Mitchell http://www.mattcutts.com/blog/robots-txt-remove-url/#comment-407226 Sean I. Mitchell Fri, 23 Oct 2009 05:17:02 +0000 http://www.mattcutts.com/blog/?p=3204#comment-407226 Great video and explanation as to be expected from 'the horses mouth'... I love this part of the text you placed at the top of the video: "You can watch it if you want:" That's like saying "Your banker has some information for you on how to increase your investments...watch it if you want." I just loved the phrase-ology that you used there. :) S Great video and explanation as to be expected from ‘the horses mouth’…

I love this part of the text you placed at the top of the video:

“You can watch it if you want:”

That’s like saying “Your banker has some information for you on how to increase your investments…watch it if you want.” I just loved the phrase-ology that you used there. :)

S

]]>