Comments on: noindex test http://www.mattcutts.com/blog/noindex-test/ neat fun stuff Fri, 06 Nov 2009 18:35:07 -0800 http://wordpress.org/?v=2.8.5 hourly 1 By: AjiNIMC http://www.mattcutts.com/blog/noindex-test/#comment-240711 AjiNIMC Mon, 09 Feb 2009 08:26:48 +0000 http://www.mattcutts.com/blog/?p=1279#comment-240711 Hi Matt, As Bill mentioned >> Matt, please see if your associates would enhance the URL removal tool to remove https pages or permit http or https to be used as the preferred domain. Can we have some preferred option for http and https under webmaster console or even as a new parameter to robots.txt. It is sometimes so difficult to have two robots.txt seomoz.org/ugc/solving-duplicate-content-issues-with-http-and-https? It will help a lot of people, looking for a reply, please do consider answering it, I will also appreciate if I get to know that I am answered :) (Though I will keep following it up under this blog post) Big thanks for all your hard work Matt. Regards, Aji Hi Matt,

As Bill mentioned
>> Matt, please see if your associates would enhance the URL removal tool to remove https pages or permit http or https to be used as the preferred domain.

Can we have some preferred option for http and https under webmaster console or even as a new parameter to robots.txt. It is sometimes so difficult to have two robots.txt seomoz.org/ugc/solving-duplicate-content-issues-with-http-and-https?

It will help a lot of people, looking for a reply, please do consider answering it, I will also appreciate if I get to know that I am answered :) (Though I will keep following it up under this blog post)

Big thanks for all your hard work Matt.

Regards,
Aji

]]> By: YellowSEO http://www.mattcutts.com/blog/noindex-test/#comment-159602 YellowSEO Tue, 18 Nov 2008 20:03:17 +0000 http://www.mattcutts.com/blog/?p=1279#comment-159602 could'nt help but pay attention to the Interesting results, seeing it indexed by both Google, Ask, Live and Yahoo. Means as a best practice as a webmaster you must take note to double check robots.txt is not blocked if going to use no follow meta. But it still seems that just using robot.txt still seems to be the best method. could’nt help but pay attention to the Interesting results, seeing it indexed by both Google, Ask, Live and Yahoo. Means as a best practice as a webmaster you must take note to double check robots.txt is not blocked if going to use no follow meta. But it still seems that just using robot.txt still seems to be the best method.

]]> By: Bill http://www.mattcutts.com/blog/noindex-test/#comment-152557 Bill Thu, 13 Nov 2008 05:49:35 +0000 http://www.mattcutts.com/blog/?p=1279#comment-152557 Matt, please see if your associates would enhance the URL removal tool to remove https pages or permit http or https to be used as the preferred domain. A client has a site, running on the worst CMS I have come across. No root access is provided and it is hosted on a Windows server. :( So I have no ability to add meta tags, can't serve separate robots.txt files, and can't even use 301's. What a nightmare and the company providing this horrible service is not compelled to help. If we had the ability to remove just the https pages, or set the preferred domain to http or https, this could quickly solve some problems and remove duplicate content from the index. As it stands now, the homepage is indexed with https, http, http://www, and previously had http://www.domain/default.asp. The client wonders why their homepage no longer has its first page listings in Google Search... Have a good day Matt. :) Matt, please see if your associates would enhance the URL removal tool to remove https pages or permit http or https to be used as the preferred domain.

A client has a site, running on the worst CMS I have come across. No root access is provided and it is hosted on a Windows server. :( So I have no ability to add meta tags, can’t serve separate robots.txt files, and can’t even use 301’s. What a nightmare and the company providing this horrible service is not compelled to help.

If we had the ability to remove just the https pages, or set the preferred domain to http or https, this could quickly solve some problems and remove duplicate content from the index. As it stands now, the homepage is indexed with https, http, http://www, and previously had http://www.domain/default.asp. The client wonders why their homepage no longer has its first page listings in Google Search…

Have a good day Matt. :)

]]> By: Dave (Original) http://www.mattcutts.com/blog/noindex-test/#comment-139078 Dave (Original) Fri, 31 Oct 2008 00:28:31 +0000 http://www.mattcutts.com/blog/?p=1279#comment-139078 <blockquote>I just want to check on how Yahoo/Live/Ask treat pages with noindex meta tags.</blockquote>Yahoo at least does the same as Google, they ignore the tag, flex their muscle and Index it. I guess might is right?

I just want to check on how Yahoo/Live/Ask treat pages with noindex meta tags.

Yahoo at least does the same as Google, they ignore the tag, flex their muscle and Index it. I guess might is right?

]]> By: youfoundjake http://www.mattcutts.com/blog/noindex-test/#comment-138933 youfoundjake Thu, 30 Oct 2008 21:03:00 +0000 http://www.mattcutts.com/blog/?p=1279#comment-138933 So Matt, I guess the question is "How can Google display similar pages to a page that has been blocked by robots.txt?" So Matt, I guess the question is “How can Google display similar pages to a page that has been blocked by robots.txt?”

]]> By: Tom http://www.mattcutts.com/blog/noindex-test/#comment-138880 Tom Thu, 30 Oct 2008 20:14:14 +0000 http://www.mattcutts.com/blog/?p=1279#comment-138880 Page isn't spidered because it's blocked, but it is added to index (with a best guess at page title) due to links. Take delicious.com as an example. Same thing is happening there. Meta noindex on the home page, but all files blocked by robots. So you do a search on Google for "delicious" and the new home page shows up, but with no snippet or meta description because the page wasn't crawled, but there is a page title associated with it, not because it actually is the page title, but because it's the domain name and it is used in anchor text (google will attribute a page title to any page that doesn't have one, and it's often the name of the root). If you do search for "delicious" on MSN, the page doesn't come up, so it appears that MSN is accessing the file (despite what robots.txt says) and finds the noindex meta. Yahoo! gives page title and a description, which actually isn't in the meta description or found anywhere on the page. It's pulled from the Yahoo! directory listing of the site. Yahoo! tends to assign Y directory data to pages that don't have it (and even often times when they do). So it would it appear that Y also follows the robots.txt directive. Of course delicious.com is a Yahoo! property, so you could draw a different conclusion, but this is my thought on the subject. Page isn’t spidered because it’s blocked, but it is added to index (with a best guess at page title) due to links. Take delicious.com as an example. Same thing is happening there. Meta noindex on the home page, but all files blocked by robots. So you do a search on Google for “delicious” and the new home page shows up, but with no snippet or meta description because the page wasn’t crawled, but there is a page title associated with it, not because it actually is the page title, but because it’s the domain name and it is used in anchor text (google will attribute a page title to any page that doesn’t have one, and it’s often the name of the root).

If you do search for “delicious” on MSN, the page doesn’t come up, so it appears that MSN is accessing the file (despite what robots.txt says) and finds the noindex meta.

Yahoo! gives page title and a description, which actually isn’t in the meta description or found anywhere on the page. It’s pulled from the Yahoo! directory listing of the site. Yahoo! tends to assign Y directory data to pages that don’t have it (and even often times when they do). So it would it appear that Y also follows the robots.txt directive. Of course delicious.com is a Yahoo! property, so you could draw a different conclusion, but this is my thought on the subject.

]]> By: guy http://www.mattcutts.com/blog/noindex-test/#comment-138062 guy Thu, 30 Oct 2008 04:07:28 +0000 http://www.mattcutts.com/blog/?p=1279#comment-138062 lol, page is indexed, so noindex wont work :D lol, page is indexed, so noindex wont work :D

]]> By: Dave (Original) http://www.mattcutts.com/blog/noindex-test/#comment-138015 Dave (Original) Thu, 30 Oct 2008 03:00:41 +0000 http://www.mattcutts.com/blog/?p=1279#comment-138015 <blockquote>Dave, but the noindex shouldn’t be spidered since it’s blocked by robots.txt.</blockquote>I completely agree and have mentioned this to Matt about robots.txt and noindex is his post ASKING for opinions on the subject. Unfortunately Google is a GIANT that often gets its own way, even when it's morally wrong.

Dave, but the noindex shouldn’t be spidered since it’s blocked by robots.txt.

I completely agree and have mentioned this to Matt about robots.txt and noindex is his post ASKING for opinions on the subject.

Unfortunately Google is a GIANT that often gets its own way, even when it’s morally wrong.

]]> By: Peter (IMC) http://www.mattcutts.com/blog/noindex-test/#comment-137741 Peter (IMC) Wed, 29 Oct 2008 20:25:02 +0000 http://www.mattcutts.com/blog/?p=1279#comment-137741 huh? No spam protection anymore? huh? No spam protection anymore?

]]> By: Peter (IMC) http://www.mattcutts.com/blog/noindex-test/#comment-137738 Peter (IMC) Wed, 29 Oct 2008 20:24:33 +0000 http://www.mattcutts.com/blog/?p=1279#comment-137738 So Matt, how are the others treating the noindex? G, Y and M haven't indexed it. what else can you tell about it? So Matt, how are the others treating the noindex? G, Y and M haven’t indexed it.

what else can you tell about it?

]]>