Comments on: What should NOINDEX do? http://www.mattcutts.com/blog/google-noindex-behavior/ neat fun stuff Fri, 06 Nov 2009 18:35:07 -0800 http://wordpress.org/?v=2.8.5 hourly 1 By: Umang http://www.mattcutts.com/blog/google-noindex-behavior/#comment-352398 Umang Tue, 23 Jun 2009 16:11:39 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-352398 My understanding is that the no follow tag is is used to google indexing and crawling your pages. Someone please help elaborate on my understanding. Thanks My understanding is that the no follow tag is is used to google indexing and crawling your pages. Someone please help elaborate on my understanding.

Thanks

]]> By: Scanto http://www.mattcutts.com/blog/google-noindex-behavior/#comment-239567 Scanto Sun, 08 Feb 2009 00:53:25 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-239567 Ok guys, so I just mistakenly moved a site live that had that noindex tag from the test site. It was there for about a week. So how long before my rankings return? Is there anything I can do to encourage the spiders to come back and crawl in the meantime? Ok guys, so I just mistakenly moved a site live that had that noindex tag from the test site. It was there for about a week. So how long before my rankings return? Is there anything I can do to encourage the spiders to come back and crawl in the meantime?

]]> By: Robert http://www.mattcutts.com/blog/google-noindex-behavior/#comment-134285 Robert Fri, 03 Oct 2008 11:34:57 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-134285 @ Alan Perkins "If somebody links to your site, and Google indexes that link, then your gripe is with the person who linked to your site … not with Google." This is the precise reason that the robots.txt file is flawed and the Meta attribute of "NoFollow" is required. @ Alan Perkins

“If somebody links to your site, and Google indexes that link, then your gripe is with the person who linked to your site … not with Google.”

This is the precise reason that the robots.txt file is flawed and the Meta attribute of “NoFollow” is required.

]]> By: Robert http://www.mattcutts.com/blog/google-noindex-behavior/#comment-134283 Robert Fri, 03 Oct 2008 10:54:29 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-134283 Well I see it as a pretty simple open and shut case myself. If I've specified NoIndex for any reason then don't index. Yes, I might have shot myself in the foot but isn't that every business (read website) owners choice? We can't tell you how to run your search engine, but we can ask you not to tell us how to run our websites. Let me put it this way Matt, if I come into your home and take photos of you sleeping, can I post those anywhere? No? But you might be an exhibitionist wanting me to do so, right? I think you know where I'm going with this. If the door is closed, if the sign says do not enter, then it should be respected by everyone, everybot, everything. No matter how good or bad the reason. Well I see it as a pretty simple open and shut case myself.

If I’ve specified NoIndex for any reason then don’t index. Yes, I might have shot myself in the foot but isn’t that every business (read website) owners choice? We can’t tell you how to run your search engine, but we can ask you not to tell us how to run our websites.

Let me put it this way Matt, if I come into your home and take photos of you sleeping, can I post those anywhere? No? But you might be an exhibitionist wanting me to do so, right? I think you know where I’m going with this.

If the door is closed, if the sign says do not enter, then it should be respected by everyone, everybot, everything. No matter how good or bad the reason.

]]> By: SEO http://www.mattcutts.com/blog/google-noindex-behavior/#comment-130005 SEO Thu, 10 Jul 2008 13:04:53 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-130005 I’m surprised this debate is going on but hats off to you for giving us a say. noindex should do what it says. Don’t index. I feel MSN and Yahoo should follow suit. Google already provides the best user experience and I don’t think this issue affects that. It would be great to see a across the board approach to this on all search engines. Most people reading your blog will have an interest in what happening with Google and other search engines but there are many web developers who don’t keep up with what’s happening. How could you inform all those people of the change? What about wee Johnny who has a hidden page dedicated to his childhood sweetheart? What if Google changed the rules on the noindex tag? Imagine the embarrassment at school when he’s back from his summer break! So for wee Johnny please keep noindex. I feel it’s common sense – sure some people will make mistakes with it but if a listing on Google is important to them they’ll work it out. I’m surprised this debate is going on but hats off to you for giving us a say.
noindex should do what it says. Don’t index. I feel MSN and Yahoo should follow suit. Google already provides the best user experience and I don’t think this issue affects that. It would be great to see a across the board approach to this on all search engines.
Most people reading your blog will have an interest in what happening with Google and other search engines but there are many web developers who don’t keep up with what’s happening. How could you inform all those people of the change? What about wee Johnny who has a hidden page dedicated to his childhood sweetheart? What if Google changed the rules on the noindex tag? Imagine the embarrassment at school when he’s back from his summer break! So for wee Johnny please keep noindex.
I feel it’s common sense – sure some people will make mistakes with it but if a listing on Google is important to them they’ll work it out.

]]> By: Ray Paseur http://www.mattcutts.com/blog/google-noindex-behavior/#comment-129069 Ray Paseur Sat, 21 Jun 2008 13:02:31 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-129069 The Atomz web site search engine uses [noindex] and [/noindex] and the corresponding [nofollow] tags embedded in HTML to control its spider, so that only parts of a web page will be spidered. It also respects the robots.txt and meta tags. It would be great if Google could also support an inline [noindex] tag of some sort. My $0.02. ~Ray The Atomz web site search engine uses [noindex] and [/noindex] and the corresponding [nofollow] tags embedded in HTML to control its spider, so that only parts of a web page will be spidered. It also respects the robots.txt and meta tags. It would be great if Google could also support an inline [noindex] tag of some sort. My $0.02. ~Ray

]]> By: Xel http://www.mattcutts.com/blog/google-noindex-behavior/#comment-126777 Xel Sun, 04 May 2008 15:16:19 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-126777 SAM: As allready mentioned Google will not stop indexing your files when you use robot.txt. As this poll tells us (Google never *really* cared for webmasters whishes, they will not start to do so now!) this will be the same with the meta. You can simply leave your pages as they are. But it might be an genious idea to hide categories which are unfilled (at least for *not logged in* users) - since your visitors will dislike empty pages anyway. SAM:
As allready mentioned Google will not stop indexing your files when you use robot.txt. As this poll tells us (Google never *really* cared for webmasters whishes, they will not start to do so now!) this will be the same with the meta. You can simply leave your pages as they are. But it might be an genious idea to hide categories which are unfilled (at least for *not logged in* users) – since your visitors will dislike empty pages anyway.

]]> By: SamK http://www.mattcutts.com/blog/google-noindex-behavior/#comment-126618 SamK Tue, 29 Apr 2008 20:30:38 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-126618 With all of the discussions about NOINDEX, I've been unable to find an answer to a question I have. I'm currently programming a community-based web site that has several thousand pages (basically categories), but at the beginning, will not have any meaningful content for a good chunk of those pages. If I put a NOINDEX robots tag on the pages without content, will the search engines come back and try to index the page again at a later date? If they will not come back, what is the best way stop bots from spidering the pages until there is some meaningful content? Thanks, Sam With all of the discussions about NOINDEX, I’ve been unable to find an answer to a question I have. I’m currently programming a community-based web site that has several thousand pages (basically categories), but at the beginning, will not have any meaningful content for a good chunk of those pages.

If I put a NOINDEX robots tag on the pages without content, will the search engines come back and try to index the page again at a later date? If they will not come back, what is the best way stop bots from spidering the pages until there is some meaningful content?

Thanks,
Sam

]]> By: herbi http://www.mattcutts.com/blog/google-noindex-behavior/#comment-125483 herbi Sun, 06 Apr 2008 10:44:53 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-125483 >>> I’d also be interested in (constructive) >>> suggestions in the comments about how >>> Google should treat the NOINDEX meta tag. matt, this is a funny question ... read my lips: NOINDEX ^^ i don't really see why this should be discussed in the first place. we had a similar discussion at the german google-webmaster-help-group about robots.txt, the google-webmaster trying to tell us something about how hard google's task was in "interpreting" what publishers meant by excluding files from crawling and i didn't understand the point there either ... so i wonder what you're at? take over total control letting publishers no choices at all anymore? >>> I’d also be interested in (constructive)
>>> suggestions in the comments about how
>>> Google should treat the NOINDEX meta tag.

matt, this is a funny question …
read my lips: NOINDEX ^^

i don’t really see why this should be discussed in the first place. we had a similar discussion at the german google-webmaster-help-group about robots.txt, the google-webmaster trying to tell us something about how hard google’s task was in “interpreting” what publishers meant by excluding files from crawling and i didn’t understand the point there either … so i wonder what you’re at? take over total control letting publishers no choices at all anymore?

]]> By: R. Richard Hobbs http://www.mattcutts.com/blog/google-noindex-behavior/#comment-124848 R. Richard Hobbs Tue, 25 Mar 2008 18:35:44 +0000 http://www.mattcutts.com/blog/google-noindex-behavior/#comment-124848 I havent finished reading thru all the comments but for myself I will say what I am seeking is a way to keep all the flotsam and jetsam (archives of several species and other secondary and likely duplicated content directories and files neccesary for my site to run itself) out of sight, but I want (and expect Google would want same) to take advantage of any opportunities to discover new links as might be available on the site as well as spider and index any unique content. So here's my 2 cents / suggestion: - you dont want your site / content seen /examined at all = disallow - you dont want the content appearing in the results = noindex BUT either by default or by including i.e follow (?) the spider will follow any links - noindex / nofollow could be tantamount to disallow ...anyway this the way my mind works about this - So my next question is how to put this work? the syntaxes are necessarily strict and somewhat arcane to most hobby coders (like moi) and certainly any newbies - and mistakes can be costly... what needs to go in the robots.txt do what I want to do - without a computer science degree ? noindex - will this *just* keep the specified content from appearing in the index? I tried adding "noindex,follow" and "noindex, follow" to robots.txt they both threw syntax errors Hmm this is interesting - noindex: follow: DIDNT throw an error but whats the Googlebot going to do with that? All I really want to completely disallow is administrative / code / backend / internal stuff that nobody with good intentions is likely to be interested in - read: potentially security sensitive areas.... Hope any of this makes sense is there anybody else out there with similar objectives? thx I havent finished reading thru all the comments but for myself I will say what I am seeking is a way to keep all the flotsam and jetsam (archives of several species and other secondary and likely duplicated content directories and files neccesary for my site to run itself) out of sight, but I want (and expect Google would want same) to take advantage of any opportunities to discover new links as might be available on the site as well as spider and index any unique content.

So here’s my 2 cents / suggestion:

– you dont want your site / content seen /examined at all = disallow
- you dont want the content appearing in the results = noindex BUT either by default or by including i.e follow (?) the spider will follow any links
- noindex / nofollow could be tantamount to disallow

…anyway this the way my mind works about this –

So my next question is how to put this work? the syntaxes are necessarily strict and somewhat arcane to most hobby coders (like moi) and certainly any newbies – and mistakes can be costly… what needs to go in the robots.txt do what I want to do – without a computer science degree ?

noindex – will this *just* keep the specified content from appearing in the index?

I tried adding “noindex,follow” and “noindex, follow” to robots.txt they both threw syntax errors

Hmm this is interesting – noindex: follow: DIDNT throw an error but whats the Googlebot going to do with that?

All I really want to completely disallow is administrative / code / backend / internal stuff that nobody with good intentions is likely to be interested in – read: potentially security sensitive areas….

Hope any of this makes sense is there anybody else out there with similar objectives?

thx

]]>