Comments on: New robots.txt tool http://www.mattcutts.com/blog/new-robotstxt-tool/ neat fun stuff Fri, 06 Nov 2009 18:35:07 -0800 http://wordpress.org/?v=2.8.5 hourly 1 By: Laxmi http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-402911 Laxmi Tue, 13 Oct 2009 14:04:09 +0000 http://www.mattcutts.com/blog/?p=203#comment-402911 Hello, Thank for providing such great information about robots.txt. I also block the pages through robots.txt & i added the code in ------------------ User-agent: * Dissallow : /*LinkClick.aspx?link= ------------------------- URL are come in www.alert-ims.com/LinkClick.aspx?link=103&tabid=127 www.alert-ims.com/LinkClick.aspx?link=105&tabid=127 in this pattern I want to block all URL which comes with this pattern Hello,

Thank for providing such great information about robots.txt.

I also block the pages through robots.txt & i added the code in
——————
User-agent: *
Dissallow : /*LinkClick.aspx?link=
————————-
URL are come in
http://www.alert-ims.com/LinkClick.aspx?link=103&tabid=127
http://www.alert-ims.com/LinkClick.aspx?link=105&tabid=127

in this pattern

I want to block all URL which comes with this pattern

]]> By: Atlanta foreclosures http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-341445 Atlanta foreclosures Sat, 06 Jun 2009 14:33:10 +0000 http://www.mattcutts.com/blog/?p=203#comment-341445 thanks for this great site. i will look forward to read from u later thanks for this great site. i will look forward to read from u later

]]> By: Atlanta foreclosures http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-341444 Atlanta foreclosures Sat, 06 Jun 2009 14:27:21 +0000 http://www.mattcutts.com/blog/?p=203#comment-341444 It seems this is a common problem in the internet forum world and nobody seems to get any success, is there a known solution to this using the text file? It seems this is a common problem in the internet forum world and nobody seems to get any success, is there a known solution to this using the text file?

]]> By: Stevy http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-269906 Stevy Mon, 09 Mar 2009 15:59:42 +0000 http://www.mattcutts.com/blog/?p=203#comment-269906 Hi Matt i would like to block evrything aroung that url "archive.womansday.com" if i do that will it also block the main url womansday.com. Hi Matt i would like to block evrything aroung that url “archive.womansday.com”
if i do that will it also block the main url womansday.com.

]]> By: Fox http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-245133 Fox Fri, 13 Feb 2009 22:39:21 +0000 http://www.mattcutts.com/blog/?p=203#comment-245133 Hi Matt.. Am i looking in the right place? the link at the top of the page is broken.. http://www.searchengineworld.com/cgi-bin/robotcheck.cgi Goes to a dead page.. Hi Matt..

Am i looking in the right place?
the link at the top of the page is broken..

http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

Goes to a dead page..

]]> By: Atlanta foreclosures http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-235745 Atlanta foreclosures Tue, 03 Feb 2009 15:14:42 +0000 http://www.mattcutts.com/blog/?p=203#comment-235745 I know that , you don’t want to restrict any user agents from crawling any files or directories - is there a point to having a robots.txt? I know that , you don’t want to restrict any user agents from crawling any files or directories – is there a point to having a robots.txt?

]]> By: Steven http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-165477 Steven Mon, 24 Nov 2008 14:13:21 +0000 http://www.mattcutts.com/blog/?p=203#comment-165477 As Alan Perkins and others mention, the url patterns ARE case sensitive. http://www.google.com/support/webmasters/bin/answer.py?answer=40360&query=robots.txt&topic=&type= Perhaps you could edit your post ever so slightly, as I trusted you more than the commenters and spent some time tracking this down: your blog post is ranked more highly than the official google answer (above) for a "robots.txt case-sensitive" google search! Thanks! As Alan Perkins and others mention, the url patterns ARE case sensitive.

http://www.google.com/support/webmasters/bin/answer.py?answer=40360&query=robots.txt&topic=&type=

Perhaps you could edit your post ever so slightly, as I trusted you more than the commenters and spent some time tracking this down: your blog post is ranked more highly than the official google answer (above) for a “robots.txt case-sensitive” google search!

Thanks!

]]> By: me http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-123742 me Sun, 09 Mar 2008 14:46:31 +0000 http://www.mattcutts.com/blog/?p=203#comment-123742 Seems searchenginewolrd have pulled their tools :( Seems searchenginewolrd have pulled their tools
:(

]]> By: Andrew http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-122716 Andrew Mon, 18 Feb 2008 05:10:50 +0000 http://www.mattcutts.com/blog/?p=203#comment-122716 Stevie, Perhaps try changing your robots.txt file to: User-agent: * Disallow: /forum/calendar_week.asp Disallow: /forum/calendar.asp Disallow: /forum/members.asp Stevie,

Perhaps try changing your robots.txt file to:
User-agent: *
Disallow: /forum/calendar_week.asp
Disallow: /forum/calendar.asp
Disallow: /forum/members.asp

]]> By: retchel http://www.mattcutts.com/blog/new-robotstxt-tool/#comment-120622 retchel Mon, 14 Jan 2008 19:02:12 +0000 http://www.mattcutts.com/blog/?p=203#comment-120622 Hi, When googlebot crawled my urls, they found 112 ulr's that were restricted by robot.txt, Do you know how to fix it..Cause I want no restriction. I want google can crawl all my files. Thank you. Hi,

When googlebot crawled my urls, they found 112 ulr’s that were restricted by robot.txt,

Do you know how to fix it..Cause I want no restriction. I want google can crawl all my files.

Thank you.

]]>