Comments on: Googlebot: Keep out! http://www.mattcutts.com/blog/googlebot-keep-out/ neat fun stuff Fri, 19 Mar 2010 01:55:22 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: John S. Britsios http://www.mattcutts.com/blog/googlebot-keep-out/#comment-208068 John S. Britsios Mon, 29 Dec 2008 14:52:32 +0000 http://www.mattcutts.com/blog/?p=247#comment-208068 Matt, can "?googlecrawl=no" be implemented as an alternative to the "nofollow" attribute? If not, why is that? Thanks, John Matt,

can “?googlecrawl=no” be implemented as an alternative to the “nofollow” attribute? If not, why is that?

Thanks,

John

]]> By: Sunil http://www.mattcutts.com/blog/googlebot-keep-out/#comment-131482 Sunil Fri, 08 Aug 2008 15:22:50 +0000 http://www.mattcutts.com/blog/?p=247#comment-131482 Hi Matt, what is your stand on this issue http://www.youtube.com/watch?v=nLB1-Kc4CWE all Blogger blogs using custom domains are not being indexd and cached after 30 July. No Google employee is ressponding on this issue in webmaster help group. Where elsse we can hope for the answer? Please take a look of this issue. Thanks, Sunil Hi Matt, what is your stand on this issue

http://www.youtube.com/watch?v=nLB1-Kc4CWE

all Blogger blogs using custom domains are not being indexd and cached after 30 July. No Google employee is ressponding on this issue in webmaster help group. Where elsse we can hope for the answer?

Please take a look of this issue.

Thanks,
Sunil

]]> By: Jenny W http://www.mattcutts.com/blog/googlebot-keep-out/#comment-98560 Jenny W Sun, 04 Mar 2007 20:16:42 +0000 http://www.mattcutts.com/blog/?p=247#comment-98560 Awesome! I love the ?googlecrawl=no idea! <a href="http://www.askapache.com/2007/seo/seo-with-robotstxt.html" rel="nofollow">robots.txt examples for phpBB and WordPress</a> Awesome! I love the ?googlecrawl=no idea!

robots.txt examples for phpBB and WordPress

]]> By: SEOboy http://www.mattcutts.com/blog/googlebot-keep-out/#comment-95011 SEOboy Sun, 28 Jan 2007 03:17:41 +0000 http://www.mattcutts.com/blog/?p=247#comment-95011 Hi Googlebot search logic. I still don't know the logic of Google to find brandnew website. But I know some tactic for them to find you. You can post your URL to some famous forums in which googlebot frequently visits. Then googlebot will see the link to your site there and start to crawl you. Hai :) Hi Googlebot search logic.
I still don’t know the logic of Google to find brandnew website. But I know some tactic for them to find you.
You can post your URL to some famous forums in which googlebot frequently visits. Then googlebot will see the link to your site there and start to crawl you.
Hai :)

]]> By: Googlebot search logic http://www.mattcutts.com/blog/googlebot-keep-out/#comment-90387 Googlebot search logic Thu, 30 Nov 2006 19:30:55 +0000 http://www.mattcutts.com/blog/?p=247#comment-90387 Can anyone provide me the basic logic with which google starts crawling . This is not the logic of crawling a particular website , but the logic with which Google finds out a newly introduced Website ( with new IP and Domain name) Urgently needed Regards - Can anyone provide me the basic logic with which google starts crawling . This is not the logic of crawling a particular website , but the logic with which Google finds out a newly introduced Website ( with new IP and Domain name)

Urgently needed
Regards -

]]> By: trevor http://www.mattcutts.com/blog/googlebot-keep-out/#comment-62830 trevor Sat, 05 Aug 2006 17:38:35 +0000 http://www.mattcutts.com/blog/?p=247#comment-62830 Could collecting info by crawling url's/content that are forbidden by the webmaster be construed as theft or invasion of privacy? If not, why not? Could collecting info by crawling url’s/content that are forbidden by the webmaster be construed as theft or invasion of privacy? If not, why not?

]]> By: Keith Ort http://www.mattcutts.com/blog/googlebot-keep-out/#comment-19437 Keith Ort Mon, 27 Mar 2006 21:58:17 +0000 http://www.mattcutts.com/blog/?p=247#comment-19437 Matt, The removal finally went through. Thanks for looking into it. Matt,
The removal finally went through. Thanks for looking into it.

]]> By: Jon Payne http://www.mattcutts.com/blog/googlebot-keep-out/#comment-19321 Jon Payne Mon, 27 Mar 2006 15:56:39 +0000 http://www.mattcutts.com/blog/?p=247#comment-19321 Matt - regarding the first couple sentences of your post - it looks as though Google is in some cases indexing pages with "id" as a parameter - see the first result (teleflora) on this search: http://www.google.com/search?q=dozen+premium+red+roses The resulting page is http://www.teleflora.com/product.asp?id=34529 Does Google recognize that with Teleflora "id" is not a session ID but rather their product sku? Does Google SiteMaps help is this is a concern one might have? Matt – regarding the first couple sentences of your post – it looks as though Google is in some cases indexing pages with “id” as a parameter – see the first result (teleflora) on this search:

http://www.google.com/search?q=dozen+premium+red+roses

The resulting page is http://www.teleflora.com/product.asp?id=34529 Does Google recognize that with Teleflora “id” is not a session ID but rather their product sku? Does Google SiteMaps help is this is a concern one might have?

]]> By: Sebastian http://www.mattcutts.com/blog/googlebot-keep-out/#comment-18532 Sebastian Tue, 21 Mar 2006 22:18:22 +0000 http://www.mattcutts.com/blog/?p=247#comment-18532 Servus Matt, read your article twice but I guess my english is just not good enought to understand. Anyway maybe someone has a solution for my problem: I'm the admin of a German karting website, just for fun and private but I think it is quite good (around 120 unique visitors per day). So now after the site is up for 1,5 years it is time to look what could be improofed. Well there is for sure a bunch of stuff to do and that is how I found your blog. Q1: in my robot.txt i have: User-agent: * Disallow: /administrator/ So why do i see google bot checking a page that you can only reach after loging into the admin backend O_o http://www.domain.de/administrator/index2.php?option=com_content&....... Q2: The Site has the feature to print articles (usually about races and stuff like that) as a pdf well seems like the google bot really likes that feature - it indexes the pdf files rather than the actuall page with doen't help the user cause the pdf engine doesn't support pictures. Here is a link as it could be: http://www.domain.de/index2.php?option=com_content&do_pdf=1&id=80 So how do I tell google bot not crawl these links? As the site is based on the joomla/mambo cms it is not easy to make changes on the system itself so I would prefer a work around. Btw. what would you guys guess, will google then index any site at all? Maybe I should leave it with that but now that I have started here is another question - witch url is the better one ;) Q3: http://www.domain.de/index.php?option=com_content&task=view&id=76&Itemid=1 or http://www.domain.de/content/view/76/1/ and what about this one? http://www.domain.de/component/option,com_docman/task,cat_view/gid,80/ So thanks to anyone, Sebastian just in case someone wants to mail directly: #azreael#ät#web#pünkt#de# Servus Matt,

read your article twice but I guess my english is just not good enought to understand. Anyway maybe someone has a solution for my problem:

I’m the admin of a German karting website, just for fun and private but I think it is quite good (around 120 unique visitors per day). So now after the site is up for 1,5 years it is time to look what could be improofed. Well there is for sure a bunch of stuff to do and that is how I found your blog.

Q1:
in my robot.txt i have:
User-agent: *
Disallow: /administrator/
So why do i see google bot checking a page that you can only reach after loging into the admin backend O_o
http://www.domain.de/administrator/index2.php?option=com_content&…….

Q2:
The Site has the feature to print articles (usually about races and stuff like that) as a pdf well seems like the google bot really likes that feature – it indexes the pdf files rather than the actuall page with doen’t help the user cause the pdf engine doesn’t support pictures.
Here is a link as it could be:
http://www.domain.de/index2.php?option=com_content&do_pdf=1&id=80

So how do I tell google bot not crawl these links? As the site is based on the joomla/mambo cms it is not easy to make changes on the system itself so I would prefer a work around. Btw. what would you guys guess, will google then index any site at all?

Maybe I should leave it with that but now that I have started here is another question – witch url is the better one ;)

Q3:
http://www.domain.de/index.php?option=com_content&task=view&id=76&Itemid=1
or
http://www.domain.de/content/view/76/1/
and what about this one?
http://www.domain.de/component/option,com_docman/task,cat_view/gid,80/

So thanks to anyone,
Sebastian

just in case someone wants to mail directly: #azreael#ät#web#pünkt#de#

]]> By: CrankyDave http://www.mattcutts.com/blog/googlebot-keep-out/#comment-18525 CrankyDave Tue, 21 Mar 2006 21:37:49 +0000 http://www.mattcutts.com/blog/?p=247#comment-18525 **Hmm. A more correct way to put it would be that there is a regular Googlebot and a supplemental Googlebot (though their user agents will be the same), and uncrawled urls from the regular Googlebot will go in the regular index while uncrawled urls from the supplemental Googlebot will go in the supplemental index. Hope that makes sense; I believe that’s correct.** Matt, Can you, or anyone else, show me an example of a uncrawled URL from the supplemental index in that shows up in the search results? Thanx :) **Hmm. A more correct way to put it would be that there is a regular Googlebot and a supplemental Googlebot (though their user agents will be the same), and uncrawled urls from the regular Googlebot will go in the regular index while uncrawled urls from the supplemental Googlebot will go in the supplemental index. Hope that makes sense; I believe that’s correct.**

Matt,

Can you, or anyone else, show me an example of a uncrawled URL from the supplemental index in that shows up in the search results?

Thanx :)

]]>