I run a Web site that was recently blocked from Google’s index without explanation. It’s a noncommercial site run as a hobby (i.e. there are no ads, commerce, ‘SEO’ type activities or motivation for them), so I was/am quite at a loss to find out why.

After some non-fruitful tests and attempts to find/guess at the source of the problem (and changing my search pages to Bing.com in protest , a message appeared in Webmaster Tools claiming to have found hidden spammy text as justification for the (or new?) blocking, suggesting the site has been compromised. “The following is some example hidden text we found at http://cexx.org/: order viagra discount… (list of similar naughty words here)”

A thorough grep of the server found the page it is most likely triggering on: as part of the anti-malware research we publish on the site, one file lists all the URLs and search keywords a particular ad-popping malware product triggers on. The page is intentional and legitimate (not compromised), however, you can guess the types of keywords this pest triggers on (viagra, mortgage…), and on review, I can fully understand how a bot would classify it as spam. In any case, it’s legitimate data, but I’ll agree that it has no place being indexed in any search engine. If I read the above correctly, adding the NOINDEX meta tag should at least keep this from appearing in Google and a few other engines.

My question is – although NOINDEX’d pages are not shown in the index at all, is the page content still read/parsed by Google for other purposes internally (e.g. the autobanhammer), or does Google essentially abort the HTTP transfer upon encountering the noindex tag? More to the point, is noindex an appropriate way to exclude content from the index that the Algorithm would (either rightly or questionably) use for site classification or assess penalties for?

PS. If you are still maintaining this page, the answer to this question would be helpful incorporated into the discussion above on how Google handles pages with noindex tags. I’m sure I’m not the first/only person to wonder about this!

Thanks
Tim

I would also vote for WC3 standards should be followed…

cheers!

Swegill

Now what can I do to get my pages re-indexed as quickly as possible?

pageB has noindex meta tag. Will page rank flow from pageA to pageB?

Or, if we put noindex on pages that we don’t want indexed, should we also put nofollow to all internal links that are leading to those pages?