I was reading an interesting question on Google’s webmaster help group that was posted a few weeks ago. The question was
Is there any official Google statement regarding that search result on
one’s own site ought to be disallowed from indexing (e.g. via
and the questioner went on to mention that YouTube’s search results were showing up in Google. Vanessa Fox showed up to tackle the answer:
Typically, web search results don’t add value to users, and since our
core goal is to provide the best search results possible, we generally
exclude search results from our web search index. (Not all URLs that
contains things like “/results” or “/search” are search results, of
I’ll take a look at the YouTube example. Thanks.
As a result of that question, YouTube added a “Disallow: /results” line in its robots.txt file. That’s good because as Google recrawls web pages, we’ll see that and begin to drop those search results.
Google already does similar things with our web search results, Froogle, etc. to try to prevent our web search results from causing problems for any other engines’ index. In general, we’ve seen that users usually don’t want to see search results (or copies of websites via proxies) in their search results. Proxied copies of websites and search results that don’t add much value already fall under our quality guidelines (e.g. “Don’t create multiple pages, subdomains, or domains with substantially duplicate content.” and “Avoid “doorway” pages created just for search engines, or other “cookie cutter” approaches…”), so Google does take action to reduce the impact of those pages in our index.
But just to close the loop on the original question on that thread and clarify that Google reserves the right to reduce the impact of search results and proxied copies of web sites on users, Vanessa also had someone add a line to the quality guidelines page. The new webmaster guideline that you’ll see on that page says “Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.”
This hasn’t been a burning issue for many people, and for people that pay attention to search I’m sure it’s a well-known fact (e.g. see here where someone asked me about a particular site copied via a proxy, and my reply later that day), but it’s still good to clarify that Google does reserve the right to take action to reduce search results (and proxied copies of websites) in our own search results.
Philipp, thanks for asking the question originally. It was good that you pointed out we had some of our own web search results showing (so that we could correct that), and it’s also good to make sure that site owners get clear guidance.