Five things you didn’t know about Google’s search

(This is all my personal opinion.)

To be completely honest, I was a little worried about Saul Hansell, a journalist for the New York Times, sitting in on some of our confidential quality meetings at Google. Even though everything was off-the-record, you can’t help but be slightly nervous talking about evaluation methodologies and confidential projects with a reporter in the room. You can read the article now, and in my opinion it does a good job of describing search quality at Google.

I think it was worth the risk of letting a journalist attend our quality meetings. To see why, I’ll highlight five things from the article that you might not have known:

Google continues to have a strong focus on search

All the time I hear things like “If Google doesn’t pay attention to search…” or “If Google loses its focus on search.” That’s not likely to happen, but let me explain why people might worry that Google will lose our focus on search.

– Something like Street View is splashy, cool, and easy to understand, so launches like that tend to get more coverage. It’s much easier for someone to write about a new product or feature than about how Google has improved its semantic understanding of the web, or when we get better at scoring documents. I love Street view, Google Gears, and mobile Calendar, by the way. I’m just using them as examples because they’re easy to understand and recent.
– We don’t always talk a ton about core search quality. Part of the reason is that some reporters are less interested in changes that can’t even be seen (“Google’s search just got a little better in Thai. You can’t see it, but it did!”). Sometimes core search is hard to get other people excited about — kinda like it’s hard to make a picture of someone working on a computer exciting. And sometimes as a business you don’t want to give hints to competitors about how you do things. I’ve got a funny story about “url.host” that I’ll tell someday. Maybe someone will ask me about it in the Q&A tomorrow at the conference.

What happens when you put these two trends together? People see media coverage on neat/wild/fun things that Google does, and they don’t read many stories about core search quality. From those two facts, they extrapolate to what seems like a reasonable conclusion: Google is focusing less on search. But that’s just not true. Hundreds of engineers pay attention to our search quality in ways big and small. Google is practically designed from the ground up so that we can’t lose that search focus. It’s natural to combine these two trends and come to the wrong conclusion. By giving a glimpse at what our search quality engineers do on a daily basis, this article dispels that misconception.

Google makes lots of improvements that most people never notice.

Some people think that Google changes a few things every few months. At least in search quality, it’s more like a few things every week. From the article: “the search-quality team makes about a half-dozen major and minor changes a week to the vast nest of mathematical formulas that power the search engine.” I don’t think we’ve discussed our pace of search quality changes before.

Getting search right is really hard

The article quotes John Battelle:

“People still think that Google is the gold standard of search,” Mr. Battelle says. “Their secret sauce is how these guys are doing it all in aggregate. There are 1,000 little tunings they do.”

In my experience that’s correct. Running a search engine at Google-scale means that you have to get lots of big things and hundreds of little things right. Missing even a few of those things will annoy users (sometimes subconsciously) and they won’t use your search engine as much. I would never claim that we get all of those hundreds of things right ourselves, but we try to. I read a quote from someone from a different search engine last year. They essentially said that “there was no more secret sauce left” in search. After reading that claim, I walked around happy for days. 🙂

Google has some good internal tools

This article was the first time that I know of that we’ve mentioned our internal debug tools. When you get hundreds of millions of queries a day, it’s inevitable that some queries won’t return the ideal set of results. At Google, we love hearing about those queries because we can dissect them and plan how to improve our algorithms.

There are a lot of people “behind the curtain” at Google that improve search

I think it’s important to get more Googlers out into the spotlight. Sometimes search engine optimizers attribute (say) some crawl change to me when the most I might have done was relay a problem report to the experts in the crawl/index team, who then do the real work of deciding how to tackle an issue and implementing that idea.

So I’m glad that the article sheds light on some new people in search. The article discusses Amit Singhal, who is a wonderful guy and a strong influence at Google. The newspaper article also includes a picture of Jianfei Zhu. Jianfei is a colleague that works with me and others on Chinese webspam; Jianfei also spoke at SES China recently and has done interviews about SEO and Chinese search.

Most importantly, the article mentions that there are hundreds of engineers that pay attention to search and quality at Google. These are phenomenal people who work on everything from international issues to evaluating our quality to crawling/indexing to personalization to fixing bugs to new quality initiatives. (Not to mention all the other people who make a difference at Google in hundreds of ways outside of search.) I know that Saul Hansell talked to several other engineers when he visited Google, so over time I believe we’ll get even more Googlers out into the spotlight.

So, five things you might not have known about Google’s search until you read this article:
– Just because Google doesn’t always talk about search and journalists don’t always write about core search doesn’t mean stuff isn’t happening. Google devotes a ton of effort to improving our search in many different ways.
– Google makes a go/no-go decision on several different quality changes each week.
– If you want to build search loyalty, you have to get a lot of different things right.
– Google has many ways to prioritize feedback and tools to look at how to improve search.
– I’m glad we’re shedding light on some additional people at Google. Many people work behind the scenes to improve the user experience at Google, and we should look to highlight even more of those people.

121 Responses to Five things you didn’t know about Google’s search (Leave a comment)

  1. If Google cares so much about search then how come nobody’s webmaster console has been updated for many months (inlinks) and the just updated index is chock full of cheating sites?

    It would help if someone would answer my spam reports occasionally. Google makes $99sqillion and yet cannot have the decency to give any kind of reply to my many reports into very obvious spam techniques being employed to the detriment of my business. E.g. a new site and it’s 9 sister sites interlinking themselves up to (as of today) #5 in the serps for my keyterm. THIS SITE IS STRAIGHT UP RIPPING OFF IT’S CUSTOMERS IN AN OBVIOUS WAY!!

    Google consistently allows cheating to be the winning strategy…

    If you gave a damn you’d sort situations like this one out!

  2. I have to say I don’t and never have doubted that Google are sitting back on their search. Just the other day we found Google spiders in log files that could only have been accessed by crawling flash….

    However the more diverse Googe gets the less people think of them as a search engine, but then… is that a problem?

  3. I think one of the force of Google is the company *doesn’t* focus on search, it focus on user. THERE is THE difference. =)

  4. Plus Google visibly takes pride and joy in search – if it is ‘just a way to monetise the web’ – as some claim – you guys hide it well. 😉

    Keep doing what you’re doing!

  5. Google has changed the way search takes place now. I foresee a strong personalized search in the coming years. Though I agree with Andrew that Google Visibility takes pride and joy in search.

  6. Google is a massive advertising agency. It makes no money from search. Matt says above that there are hundreds of engineers working on search. I wonder how many thousands (tens of thousands) work on the advertising side of Google..

  7. Matt, I remember thinking early last year that maybe Google had started to stretch itself too thin, and maybe hadn’t the resources to focus as much on search because there was a period of time there where a lot of spam was getting through and the usually reliable results were slipping a little. Of course it was a temporary glitch, but at that time Google also had a BUNCH of new products kinda taking the spotlight like you say – so it’s easy to see why sometimes search seems secondary.

    However, as long as Google remains the go-to search engine of the masses you can know that you’re doing something right.

  8. It was nice to read a well written and deeply researched article about Google for a change as most journalists seem lazy and don’t appear to know what they are writing about when it comes to search engines. The extensive background research and subsequent information that was included in the article was really interestring too for us on the outside so well done to opening up those closed doors for the interview guys!

  9. “Google is focusing less on search” its really hard to take. Infact sometimes i think search is the only product google has and everything built round around it. Search for web then search for email and then search for friends…. All comes from the same cooker.

  10. I also think that most people probably dont understand or know about how the different teams inside Google work. Just because you have a new product (Gears, Street View) rolling out doesnt mean that there arent the numbers of “ranking algorithm” engineers locked away keeping the basics of what makes Google Google (if that makes any sense).

    Long time reader, first time commenter, great blog Matt.

  11. Great post and great article in the times. So you guys do work hard on keeping up search quality 😉 Haha, just kiddin’

  12. Very interesting article, thank you to Google and NYT.
    I think a weak point of Google is the ability to detect if a website has an original content for user or a content to sell an affiliation link for a product.

    I’m working in a company which gives the opportunity for webmasters to promote our product and earn money if they recruit clients.
    When I search the name of my company in Google Web (french) I’m surprised to see that one of the poorest website are very high in the google ranking. Even this website has 2 results in the top 3.
    What is funny is that this website has a better ranking than the company website itself ! 😀

    In fact I know why he’s trapping Google.
    He makes a lot of link exchanges with the anchor of the company where I’m working. In fact this anchor is diriged to his homepage. Of course he writes the name of the company in his title and he has optimized is homepage with it. I can add that the brother of this webmaster has a huge website and gives him some very powerful backlinks.
    At the contrary, some very good websites are lower.

    Of course I don’t pretend learn something to Matt Cutts.
    But I just want to explain what problem I see with google, maybe it could be usefull to detect some spams.
    I don’t give the name of the company because it’s not the goal but if Matt cutts wants more information he can contact me or just look my ip 😉

    Otherwise I think that such a news in your blog is a good thing. Google works not only in search engine but in mails, softwares, divertissment, etc. It’s a good thing to notice that you are focused for search engine, it’s thanks to this that google is a so big company.

  13. is that because of the new methodology used by google make my websites can’t get reach by googlebot? all my websites are currently have problem because googlebot can’t reach the robots.txt. this things get weird because other bots still can reach my robots.txt file. only googlebot can’t reach it. or is my site has been penalised by google?

  14. What I would do to be in the shoes of “Saul Hansell” in that meeting :-)….

  15. I found the article informative and thank you for your follow up to it. The QDF itself is subject to a lot of buzz that should be hitting blogs by the 1000’s today. Will be fun to see who ranks on the first page tomorrow.

  16. Saul writes some very, very good stuff IMHO. Seems like his equivalent at the Wall Street Journal is Kevni Delaney who also writes from great pieces.

    Would be interesting Matt to hear your two cents about who out there in Mass Media Land does a good job convering your business/industry.

  17. Google crawlers are revenue driven.

  18. “When you get hundreds of millions of queries a day, it’s inevitable that some queries won’t return the ideal set of results. At Google, we love hearing about those queries because we can dissect them and plan how to improve our algorithms.”
    Great, where do I send them? I get them all the time with queries relating to my field.

  19. Finally, something from the mainstream media that’s actually worth reading as opposed to propaganda designed to appeal to the masses.

    I’d second alek’s question about what is actually worth reading as well, although I suspect it would be a short list at best.

  20. Agree with what’s been said — the NYT’s is a good quality, reasonably sophisticated article that focuses on the people as well as the technology, that’s insightful for the knowledgeable and maybe groundbreaking for the uninitiated.

    I think people feeling Google is “focusing less on search” is actually good, in a backwards sort of way. That means that core search is of such a quality that the work that goes into it is transparent, and technical shortcomings or fireworks aren’t getting in the way of people’s experience of the web.

    Very much appreciate the opportunity to get a little “behind the scenes” perspective!

  21. I think that its important to note this is actually Matt’s personal view, in that respect, search is pretty much the main focus.

    However, I feel that with every blooming business, the desire to diversify is every present and so far Google has done a great job in keeping in touch with their core “search” technology in the expansion of their products and services.

    For those constantly monitoring the detailed changes in search can appreciate the work that goes on to improve the key focus of Google – Delivering information to the users.

  22. When I started reading the article, I had hoped for the keys to the kingdom of top ranking listings – but no silver bullet for us.

    BTW, how do us mere bloggers get a chance at playing with “Debug” : )?

  23. TOMHTML, well said. One way to think about it is that we try to optimize our search quality and everything else to lead to happier users.

    alex, I agree that Kevin Delaney is very good too. I’ll have to think about other good pieces I’ve read. Mark Buckingham’s piece in .NET magazine was a really good intro to SEO, I thought.

    Chris Jara, there would have to be a slip-up somewhere for an external person to see it. 🙂

  24. This sounds like a whine session by Google. Waaahhh…not getting press for search (so much so you have to an “in-depth” interview in a paper?!). Waaaah…why do the maps guys get all of the love? Waaah…how the heck did those YouTube guys make money on stolen videos, escape the lawsuits (G’s problem now), and still retire bizillionaires? Waaahhh…

  25. Nice to see an article in mainstream press with some real depth of reporting.

    The description of Google’s approach using signals and classifiers was interesting. Seems to me this would lend itself to a “guided search” feature that lets the user manually specify some factors for a single search (unlike using all of a person’s history). For example, I might want a “person – not famous” and this would influence my results. This sort of exists already in that I go to Books for a book, and Products for shopping, but there are many more subcategories/refinements that would be useful and could be exposed to the user more elegantly.

    In some ways this is a complement to Universal Search, where I manually limit or influence the set of results and factor weights used to get my results. It probably comes in after trying a universal search and not spotting any great results.

  26. Matt,

    I thought you might be interested in this piece from UK newspaper, The Guardian:

    http://education.guardian.co.uk/higher/news/story/0,,2095238,00.html?gusrc=rss&feed=8

  27. Has any thought been given to allow webmasters to provide you with query + query results which provide non-optimal customer experience? While you accept web spam reports from users and webmasters I haven’t seen anything for reporting poor query results.

    As with anything quality, the more eyes you have, the more things you can spot. And while the article touches on the cases where the results for certain queries don’t provide the best mix of web sites to feature, there’s a parallel problem of picking the best page within a site, particularly very large sites such as e-commerce. Folks familiar with those sites also may have a good sense for which of the many possibly matching pages on their site may be the best customer experience, either by intuition or actual data from conversion analysis, etc.

    I would imagine that you would have to put some filter on this, or your set of queries to look at will explode from the quoted 100/day to thousands/day. But maybe you could add this only to the webmaster tools, and require a certain amount of evidence/detail which adds both a trust component, and sets a bar so that only well intended feedback gets through.

  28. Anytime that you have a product or service that is offered to the public (ie. Search) the focus must be on the user. Google understood this from the beginning and continues to focus on the user. You can be as innovative as you want, but only if the user accepts that innovation will that innovation be a true success. It doesn’t take a Jedi-Master to figure out that the user is ultimately the one in control who determines how successful Google (with search) or any business (with their product/service) will be…

  29. Nice insight Matt. I keep on forgetting about how Google is improving their search engine algorithm. The thing is that Google has too many pet projects and many more in their queue which we don’t get to hear about most of the time. Those are the once that are interest to most people.

  30. Really interesting, but seems risky from a corporate perspective. Sure, we learned some interesting new stuff about Google. But where’s the benefit to G? Isn’t it risky to give competitors these little tidbits about how it all works? Or is it assumed that they already know?

    The QDF thing, for example, seems it would be kept quiet. I guess everybody is working on a similar program, just seems like the type of thing that would be kept under wraps 🙂

    Re: French Revolution fix: So in some cases, results are tweaked to be more of an “exact match”? And they’re made on a one-off basis? That seems like a shift from the traditional “we make algo changes to fix problems, and usually don’t do manual ones”.

    Really interesting article though, glad they did decide it was worth the risk.

  31. Oops, I skimmed over the conslusion to why you think it was worth the risk on my first read. I definitely see your point, I do see a lot of “they need to stick to search” comments (mostly on forums). Still seems risky, but I see the value from a PR perspective now.

    And I guess there were safeguards in place to protect vital stuff from getting printed. Ok, I think I’m convinced. Ignore previous post…

  32. So Google never stops dancing, only sometimes it dances it it’s sleep…

  33. Matt,

    Google clearly remains the gold standard in search while Yahoo and MSN apparently keep hoping the domain name or path and filename will lead them to the best content. But there’s a reason some people are growing uncomfortable with Google’s status as THE search engine. Webmasters who work at it full time may know where to go, what shows to attend and what forums to frequent to get questions answered, but others, in the midst of nervous breakdowns when their decade of Internet publishing slips on Google’s radar screen, don’t find Google much easier to deal with than the old phone company.

    On one hand, when I made a blog post questioning whether Google Books could be creating duplicate content problems for publishers with online content, I got a phone call from somebody at Google anxious to explain it wasn’t so and to generally “spin” me like I’ve never been spun. On the other hand, reporting a copyright infringement to Google with a DMCA complaint after multiple reports through the spamreport.html page and webmaster console failed to get the infringements removed is like pulling your own teeth. Submit a DMCA report, get “outed” in Chilling Effects as meanie, and the infringements still remain in the Google index.

    While it’s clear Google has limited human resources to devote to the exceptions that creep up, it’s clear as mud to outsiders how those resources are prioritized. I’m glad to hear that the main priority remains search, and I’ll have to admit that it’s comforting to hear it from the NYT as well. But I know I’m not the only person worried that the very success of the search engine has led too much web influence concentrating in Google’s hands, and the fear that power is a stronger force for corruption than any other.

    I also worry that as smart as Googleans appear to be, there’s always room for “Long-Term Capital Management” errors. Everybody called them the smartest guys in the room, but their algorithms got so complicated that they couldn’t predict (correctly) the outcome of their moves. I sometimes suspect that’s happens at Google, that you’ve tweaked and tweaked, and supplementaled, and I don’t know what else, but the presence of duplicate content still hurts the originators, even when not beating them out on searches.

    I guess I’m saying, keep up the good work but don’t let it go to your heads, if that’s possible. I know it would go to mine.

    Morris

  34. > Really interesting, but seems risky from a corporate
    > perspective. Sure, we learned some interesting new
    > stuff about Google. But where’s the benefit to G?

    I think before you can even *think* about utilizing the information in the article in any Google-competitive you need:
    – the skilled engineers Google has
    – the ~100,000s of servers they have
    – all of their established work processes & traditions
    – all of their internal software
    – the basic seach framework
    etc. etc.

    So that might leave only Yahoo, really (and to some extent less, Microsoft and Amazon). And for Yahoo, the insight of QDF might have been too trivial, or if not, it’s not the thing that would give them a real advantage over Google in the big scheme of things. To the rest of the developers who read along the insights might serve as inspiration for tools of their, own but none of those tools would be competitive (or it might serve as inspiration to apply for a job with Google). And for the masses, the insights are a reminder that Google cares a lot about search, which might more strongly bind them to Google, and might serve as an antidote to the next “Google killer released, solves search problem” press release by a startup.

    However, I bet that if you work at Google, you’re much more likely to think “this or that inside information can be useful for a competitor!”, simply because yes, Google has all the rest of the infrastructure & brain power in their HQ, so they *would* be the one party that might actually make use of such a piece of knowledge would they not already have it.

    In the end, my guess is that if you sent let’s say 20% of all Google search-related source code to the Ask.com team they wouldn’t be able to improve their search engine. What you would need for a search competitor to get started is a detailed blueprint of how to build a search engine, how to structure the organization, how to hire the right people, etc. etc. (think a giant encyclopedia to fill your bookshelf). Basically, you’d need the “Google Company DNA”. And then there’s still the issue of how to pay for all those 100,000s of servers, or where to get all those workers (as most are sitting at Google)… and even if you finally have ALL THAT, well, then you basically need half a decade to grow all that into today’s Google… the final problem being that in 5 years Google isn’t where it is today, but advanced into something else entirely!

  35. I think it’s great that Google is being transparent about most of the algorithms of their natural search, now only if they can share some idea on how their “quality score” and “minimum bid prices” are calculated for paid……

  36. Matt –

    I think you raise an interesting point in general about development that I wonder if some other programmers could comment on.

    My bosses get more excited about a word changing color than all code being made modular, 100% dynamic, implementing new unseen features, etc. For the lay person, they get more excited about changes that they can grasp visually and conceptually (Captain Obvious here).

    So how does a company like Google, and on a smaller scale the lone developer, get people excited about the backend, small change stuff? ROI numbers? Other metrics? Fancy demos? It seems that the publci can lose its mind over pictures of some girl in a truck with her thong showing while items like the Google Mashup Creator sign up goes almost completely unnoticed.

  37. I have to admit I’m rather disappointed with Google’s search of late (and with its image search for much longer). Way too many spammy pages and way too many parked pages have made their way into the results. To top that off the vast majority of search results are commercial and only marginally related to my queries. I wish I could provide feedback on every search and every result. Let me flag a site as spammy, as unrelated to my query, as a parked page, or as one I never want to see in a result again. If Google could do that to improve search results for ME it would tremendous. If Google could find a way to use those ratings to improve search results for others, without allowing jerks to game the system, it would be even better.

  38. Matt

    “Mr. Singhal has developed a far more elaborate system for ranking pages, which involves more than 200 types of information, or what Google calls “signals.” ”

    IMO the most facinating thing is the speed by which Google deals with all those 200 signals and return a relevant serps within seconds!
    Unfortunately that article of NYTimes doesn’t mention much about it.

    Btw, why 200 signals, Matt? why not only 10 or 20 signals and give that poor SEO a chance to do optimization in more comfortable way?

    To take into account during optimization those 200 signals would make any SEO the most unhappy person on this planet 🙂

  39. I just hope my sites get a better listing with the new search 😀

  40. Matt,
    (please comment on anything you feel you can, and ignore the rest)

    I don’t believe it, tweaking 200 “signals” a day? Surely that is begging for some sort of semi-supervised machine learning, lol. If you can search that many pages, then surely you can solve a 200 dimensional minimisation problem for each search!

    Seriously, though. It is interesting in general how Google write code to deal with specific problems (think “oneboxes”), and then try to tie them all in together with what (sounds) like fairly hard-coded logic (It almost feels like there is a database of regex strings sometimes). It sounds like this flows over into the overall design of search – just a thought, although it may be the terminology used in the article that is giving me that impression.

    On another note, is the work on QDF documented/published/patented anywhere – a quick search finds nothing (Which either means the information isn’t out there, or that Google personalised search hasn’t worked out what I normally search for yet.)

    One final question, out of interest, does the “Search Quality” group include Crawling, Data extraction (html->words), Indexing (words->[secret]->page stored for search), and all of search? I always assumed they just worked on handling the query, and you had a separate “indexing” group somewhere that were kept far away from prying eyes.

  41. Hello Matt.

    >

    I have a query for you…

    I type in “boot camp in san francisco” and vyayr.com comes up in the top results in Yahoo & MSN. I don’t see it at all under Google’s results? Why is this happening?

  42. Hey Matt,

    Isn’t it a bit ironic for the NYTs to be on Google’s VIP meeting list after Search Engine Journal and SE Land accused them of spamming less than one month ago?

    Maybe the word I am looking is ‘coincidence’?

    -Ryan

  43. Yes Matt, I agree with you, even i have posted some recent and valuable updates of Google in my blog. 🙂

    Pratheep

  44. There are a lot of people “behind the curtain” at Google that improve search

    I think it’s important to get more Googlers out into the spotlight. Sometimes search engine optimizers attribute (say) some crawl change to me when the most I might have done was relay a problem report to the experts in the crawl/index team, who then do the real work of deciding how to tackle an issue and implementing that idea

    They must also be studying the posts in this blog, and following and debating the more controversial posters like SearchEnginesWeb (based on recent changes). 😉

    It would have been extremely interesting to get some feedback on their discussions and debates on the posts from the past two years that prompted policy re evaluations and re analyses.

  45. Matt

    Btw, how could you keep focus regrdless of that fancy red chess board shoes of Danny?

    Shouldn’t we give Adam Lasnik such red shoes next time he celebrates his birthday 🙂

  46. Matt –

    Here’s one that’s come up on the google webmaster help forums just right now.. http://groups.google.com/group/Google_Webmaster_Help-Indexing/browse_thread/thread/6e2a6126cca76934

    I quote:-

    ‘I have never seen this before. If you do a search for seamonds.com,
    the result comes back with [email address] as the URL. Is this
    a search error or a DNS setting?

    When you click it you get a “you are trying to log in to this site”
    error.’

    It’s one of the WEIRDEST things I’ve seen from a search before – is this a genuine goglebug?

    Cheers,

    Matt

  47. I wish more people came out from behind the curtain too – it would be better for your relevancy if Googlers answered and corrected the questions and poor info out there more frequently… I think the innocents wouldn’t make as many of the mistakes that are clogging the arteries. They wouldn’t feel as cut off at the knees. Not to mention what it does for PR; trust and loyalty. The organic nature of your design would still stand heartily on its own even with more Google inside insight.

  48. Matt,
    I’m sorry to bother you in your comments, but my site has been banned and I can’t figure out why. I started to ask for reinclusion, but I have to admit wrongdoing to submit the form, and I’m not willing to do this. Can you take a look and let me know why the ban has happened?

  49. QUOTE:Google continues to have a strong focus on search

    aa aaan!!! you than said therez more focus on the searcher or the user!!!

  50. Robyn, if it’s the site linked to your comment, I’d start looking at the paid links at the bottom first, NOFOLLOWing them would be a start.

    http://www.google.com/support/webmasters/bin/answer.py?answer=66736&topic=8524

    Or start a new thread at The Google Webmaster Help Forum for more thorough reviews.

    http://groups.google.com/group/Google_Webmaster_Help-Indexing/topics

  51. Hey Robyn – Why don’t you pop over to the google webmaster help forum and post your url and problem there – there are heaps of willing volunteers there who should be able to put you on the right track.

    I’ve had a quick look at your site and seen a couple of problems but this is not the place to discuss them – It’s probably not really on topic.

    Check out http://groups.google.com/group/Google_Webmaster_Help-Indexing

  52. Dave (original)

    I thought Google read all the SEO forums and measured the amount of self interest “Google’s Broken” posts to determine how they are doing for relevancy 😉 That is, the more “Google’s Broken” type posts the better Google are doing for relevancy 🙂

    Seriously though Matt, Google have always had a “different” approach to business than most big business and it’s a breath of fresh air IMO.

  53. Excellent post Matt, now when will the (some would say evil) google redirect be removed? I noticed it was gone yesterday, and it’s back today.

  54. Matt,

    About quality of search, does Google allow someone to develop 50 websites, each of them are talking about one of their services, interlink together, and get 43 out of the top 50 positions?

    I found a company is doing this, and is very successful in Google.

    Please advise whether it is spam or not.

  55. The NY times report is indeed a good information sharing by G.

  56. Very interesting article thanks 🙂

    Never ending changes happening to the algorithms, good to know.

  57. “If you want to build search loyalty, you have to get a lot of different things right.”

    That’s exceptionally vague, bordering on cryptic.

    And those different things would be?

  58. Dimple Mankani

    The best thing about google is that it provides the information acording to user’s convience and its always updated which really attracts evryone..
    keep going.!!

  59. Okay, okay. I’ll bite.

    What is the “url.host” story?

  60. Well done publicity.
    I see here a lot of people that remind me of bible “thumpers”. Just believe what God says and don’t question. Doing that interview was the benefit for Google and knowing corporate world I know that everything was prepared. It is a good business practice. As for Google focusing on improving internet search… I am still a skeptic. I search the net many many times a day and am amazed at how many times the results that come up have no relevance to the search what so ever. And especially the sponsored search results. If Google does stand for quality and not simply about monetizing than there should be some sort of control of what they advertise. Anyway. I am obviously the nay sayer. Although I do like reading Your articles even though many times it is only for a laugh.

  61. Anyone who thinks that Google isn’t working to improve their search functions needs a reality check!

  62. It would be a good idea if you published a “how too” somewhere, showing all the ways that search can be used. I bought a dummies book on Google, there is a lot more than first appears.

  63. Google has always focused on the user… I love the way the Google philosophy runs. Its all about the quality, not the revenue.

    BTW, Matt, why do you use WordPress? 😛 What happened to Blogger?

  64. Matt

    I’m really disappointed that neither Adam nor you have posted anything, neither on your blogs nor on webmasters related forums, regarding the recent improvements within Google Webmaster Guidelines (:-(

    Maybe you and the friends at the plex see such improvements as minor events. However, for publishers any change happening within Google Webmaster Guidelines is an important event which might affect their sites directly or indirectly.

    Oh well….

  65. im a little annoyed at googles decision to display cross media results as part of their listings. i for one dont want to see video or image listings as part of the general search results instead i opt to view them independently as and when i want to. For me the new results just push the content i want to view below the fold!

  66. Your write up on why it would seem Google has lost its focus makes perfect sense.

    I would also add that if you look at the search results provided by the other search companies, they are not as accurate, by and large, as Google results. With the constant changes to how websites / webpages are designed, Google will have to keep tweaking the search algorithm to maintain such accurate results.

    How much does the amount of spidering have to do with Google’s success? I just launched a new site about a month ago and Google has crawled thousands of its pages while the other engines have only done a few hundred.

    Thanks again for making my job easier,

  67. Matt, you plagiated the other initiative 🙂

    5 things you don’t know about Cristian Mezei (and others) 🙂

    I read the whole article … Maybe a little bit more technical aproach, for us enthusiasts, was highly appreciated…

  68. Matt,

    Someone else mentioned that QDF should be a semi-hot topic by now, and seeing as it is a few days later, and I have seen some buzz around it, and doing a little bit of Google Blog Search I do see there is some… but nothing relevant to the article is showing up in searches, including the article itself (which I’m sure got plenty of link loving, and appears to be on a semi-authority site).

    Is the QDF factor then reserved mostly for near internet-phenomena sized topics? Or is it not fully implemented yet?

    I mean, the query [qdf] itself only returns 693k results, so it’s not like it was a highly competitive term in the first place, right?

  69. Ah, I missed it somehow, the Times story is actually at #6, and another post discussing it at #7. My bad.

    I would have thought it would have appeared higher, seeing as how it is a non-competitive term.

    PS – you seem to be having DB problems, in case you didn’t know.

  70. Matt, you know that many of us in the SE News / Media industry will remain critical of your work… but you are correct and applauded for taking the high and diplomatic road to point out these things.

    Why though, is there so much smoke and mirror these days? Back in the days when we spoke (back at the Boston PubCon) I was amazed at how willing you were to explain things in great detail… Now, we get commentary from such a high level, that we’re unable to speculate, take the ball and run, and form these highly opinionated rants that we love so much! 🙂

  71. Speaking of Quality control, your blatant censorship of the Something Awful forums IS irritating and consciously and subconsciously making me have resentment towards Google. Could you please fill me in WHY this happens: http://www.somethingawful.com/d/hogosphere/google-hates-sa.php

  72. Keep up the great work! Google has already set the standard so high for internet search,and I know you guys will continue to set it higher in the future! All the Best, -Terra at http://www.betteforbusiness.com

  73. You know I have been a webmaster for some time that has to deal with Google’s changes which have lead to many headaches over the past 7 years for me. We run very ligit sites and they are constantly up and down in rankings in fact some of them that where listed as first page results have dumped down to page 83 with out any understanding why. While this might not matter much to your average joe when you rely on search results to bring you customers outside of actually paying Google (which some times i wonder if thats the reason sites dump to lower pages).. anyhow just a bit of frustration im not sure if Google is listening in or not.

    Steve

  74. Heather Paquinas

    Perhaps google should have a search changelog newsfeed for us search geeks,
    http://www.google.com/search-changelog.rss

  75. Grat article from NYTimes. I’ve looked also for the ‘street walk’ in SanFrancisco, and worked excelent:) Cruising from home, was fun.
    “But at its core, Google remains a search engine” I dissagree. There’s lot more that google offers to people. Features from code, image/video hosting, and many many more. And also, latest aquisition(PeakStream) and partnership with SalesForce should be mentioned. Search engine requires lot of work, and having also a colateral activities, should be only praised.
    My words:)
    Victor

  76. I kinda figured Google made changes often, which we never see. If you ever worked in the corporate world then you know businesses are ever-changing. I’m sure code drops (algorithm changes) are made frequently to improve the search experience.

    Matt,
    One thing I still don’t know about Google is “Does Google like search engine optimization or do they only deal with it because they can’t stop it?” When I use the term search engine optimization…I mean “aggressive white hat SEO”

  77. Its kind of stupid to have a reporter in your meeting since its a confidential quality one and you do have alot of competition !

    Anyway, you work inside google so you know all about these weekly updates, webmasters on the other hand, have their main focus on their own sites and would ignore any changes that does not effect them, unless its a major update. The avg joe would even hardly notice a major update so don’t expect him to notice your weekly tweakings 🙂

    Think of it as a flying plain. Passengers will not notice anything unless the plain diverts from its course or have been hit by something. The things the captin and his crew “Google Staff” would do to keep the plain “Google” flying its course safely, is hardly noticed by the passengers and they wouldn’t care to know or ask, as long as you keep the plain flying. But if something goes wrong…..you get my point 🙂

  78. Matt, I finally got around to actually reading the NY times piece – that article is, well, just plain exciting, interesting and (for want of a better word) inspirational.

    Non-googlers with a fascination for search (like the majority of people that post here) just love that kind of stuff – I liked hearing about the culture, the way you do things, the approach you take and the kind of tools you use to break down immense problems into (relatively) bite sized chunks you can deal with quickly.

    I also liked the little anecdote about the search quality engineer in NY munching on cake while on the video conference – it’s easy to see Google as the huge corporate behemoth it is and forget that, really, at the end of the day it’s just – well – people like us 🙂

    Thanks for the great article.

    Matt

  79. I think, its not just google focusing on a good search.

    Actually, Google really needs some more changes even, if we can see, after every quater of a year, webmasters are also doing some new things to rank up high on Google Search.

    Its a fact of time. What you think Matt ??

  80. Nice to Google opening up more to the public. I meet you and some of your team at SES London and it was also nice to see you openly answering peoples question there. Google certainly seems more open to me now than 2 years ago.

  81. Search results are one thing.
    Relevant search results are another.
    For broad terms I rarely get a relevant result returned and that’s because Google doesn’t learn about me.

    Sure the toolbar / agent will cache my entries but I always thought that the experience would benefit from a little more insight.
    Perhaps registration as a casual user with a questionaire that builds a profile of me would help to return more relevant results both paid and natural.

    DoB, occupation, country, boolean list of hobbies and interests, income… that kind of thing.
    It’s not fool proof but it’s a huge step toward learning about me the user.

  82. Great article! Thank you for that great information!

    Jonathan

    http://www.photosforsouls.com

  83. Matt,
    With all due respect, and I did not read the whole blog, but I get the jest of it. I just wanted to say that I guess the quality of Google search depends on what industry you are in. Personally, it’s a nightmare for me, as I am in a very competitive field, and there are a lot of spammers in my industry.

    Google forces a site to over keyword, because they allow sites with NO relevance in the rankings. BUT, because they have all the right words in the right places, and they’re “popular”, with thousands of links, they get ranked before a local company, which is what the consumer is really looking for. (in my case) They’re not looking for a company that ship’s it’s products.(again, in my industry) I am a local company, and can’t even keep a high ranking, because all the others do is add another word, change their page, buy more links, and they get all the traffic, which just keeps them at the top, while sites like mine, once they fall, continue to fall, because people don’t get past the first 4 links. This has happened to me for the last 5 years. And I do update my site regularly.

    Today, I discovered that G’ did not index my site right on the 1st, that only a few words were indexed, as evidenced by the highlighted text in the cached page. It’s even the wrong words, not the words I typed in to look at the cached page. Apparently, reading the help boards, I am not alone. It’s happened to others as well.

    This wil have a HUGE impact on businesses like mine, that can not afford to pay for internet advertising. Now I don’t know if I will be able to recover from this drop. I went from # 3 for most of my keywords, to the middle of the 2nd page, and worse, in a few days.

    Things like this, that have an impact on small business is something I would like for G’ to consider when they make changes to how things are done.

    I don’t know what the glitch is, but nothing from a website building aspect was done to my site for this to have occurred. But this year I will miss out on Father’s Day sales, which I count on to pay my bills. I was # 2 for it, now I am nowhere to be seen….really makes me worried about my business. I still show up on all of the other search engines, but this was the bread winner.

    So, while people are raving about G’ and their technology, people like me are trying to figure out what to do next about their sites. If anything can be done…..

    Thought you’d like to know the OTHER side of the Google search.

  84. “Long ago, Google figured out that users who type “Brittany Speers,” for example, are really searching for “Britney Spears.” ”

    well, if that’s so, then why can’t google correct a simple error such as recognizing the duplication of http://www.www. or recognizing a comma for the dot after the www. and automatically correct it? and this has to be one of the easiest errors to correct because the format is a fixed prefix

    but I can’t tell you how bloody annoying it is to get the duplicated www because my spastic hands hit the comma rather than the dot and google just happily sent me a 404 message instead.

    but the game on search terms for images is really horrific and many of the images that come up are not good for indexing recall. they were meant for family and friends and have little meaning or content beyond a small circle and have the same weight as images that are valuable for research and education and general information. the google keyword game is hot topic in a stockphoto forum/newslists because it examplifies bad keywording — and trying to search through google images is a nightmare anyway. Search for a Czech frog and you come up with a military barricade against tanks and not hylidae

  85. I should also say that the blogpost was listed in PC Mech for today

  86. The precis of that post is: Google does stuff.

  87. Google should put out some kind of monthly video. You could go around and interview the programmers and other Google staff and give sneak peeks at upcoming Google features. It would be a hit!

  88. Matt,

    I was browsing this page and checking out some of the posters sites. The first 2 are OK. But one of the next 4 sites I went to absolutely loaded my computer up with adware, malware, and tons of junk that almost caused me to have to format my hard drive.

    Just trying to let you know. I think it was the diamond site but can’t be sure. I am afraid to go back and check, that is for sure. The reason I don’t know exactly which one is because I did a right click on 2 or 3 of the site links at once to open up a couple and then go back and look at them so I opened 3 sites real quick but I am almost sure it was the diamond site that hit me.

    I was using AVG Anti-Spyware and it promptly just shut it right down and proceeded to bomb my entire computer. I am sure it shut down my AVG software, I watched the little icon disappear at the bottom right of my toolbar. Couldn’t believe what was happening.

    I am not trying to tell on anybody or anything like that, they probably don’t even know their site is infected. I don’t wish what happened to me on ANYBODY.
    Fortunately I was able to cleanse my computer of the mess with my spyware software. It took forever.

    OK, maybe you can find out which one it was as you probably have better spyware software to protect you.
    It was definitely not the first 2 but any one of the next 4.

    Sorry for the bad news.

  89. Hi Matt!

    You give above a very good example of misunderstanding by intend I think.

    ……. and everything else to lead to happier users.

    I don’t think that the statement of TOMHTML was meant as you interpreted it.

    I think one of the force of Google is the company *doesn’t* focus on search, it focus on user.

    He named Google a prospect searcher!

    The genius saw where normals look over, because of their prospect.

    I think it is a big part of mankind’s psychology,
    to prefer what they are prospecting
    but please don’t muddy this up with a description of reality.

    Greetings Karl

  90. Matt I am apologize but could you tell me why my comments not published in your blog?? If you don’t mind.

  91. Where can we report queries that return essentially worthless results? I currently work for a company that sells its products almost exclusively on the internet. The top 5 search results for our major terms are always “SEO” type companies who have little or no knowledge of our industry, and people who search for our products don’t want to find that type of website at the top.

  92. The internet is still growing up and I think Google still has to learn from its mistakes. Its good they make the attempt to fix the wrongs, no matter what it be.

  93. I’m sure this is the wrong forum, but since you’re the only human face I know of @ google search: I would like to use google to search in my own blog in order to link related posts to earlier posts on the same subject. Even if i use the site: parameter i get lousy results that got even lousier of late. (Anything more than a month old often doesn’t show up in the results at all) I’m forced to use technorati to search in my blog (and from this i deduce other specific blogs as well)

  94. “Google is focusing less on search”. Hmm. Nice headline. Once an opinion-leader journalist dares writing it down, every other journalist on the planet will begin talking about this until it begins to sound true.

    If I am not mistaken, Yahoo! has been around long before Google first shown its ridiculously simple “white-page with a form-field and button” design for web search. If Yahoo is now the “search-engine Avis” (“only #2”) certainly it has nothing to do with Google not “trying harder”.

    At least once a week, I run that day’s every search query on Google, Yahoo AND Windows Live, using their toolbars and a stopwatch. I have been running this test for a more than year, both for text and pictures. Unmistakably, I spend less time in Google SERP’s than in any of its major competitors. Period.

    Search is about leaving the search page as fast as we can. We don’t access the internet to watch at great-looking result pages with lots of cool stuff, but to find the fastest possible whatever we are looking for.

    Google simply does it better than its competitors. One may cry, whine or scream in despair but that will not change this simple fact anyone can check by running their own tests.

    That’s sad, because true competition always helps quality improvement. Google doesn’t need to run 100 mph faster than Yahoo to keep it beaten down anymore. You only need to stay five or six steps ahead… But, as a safety measure, stay ten steps ahead and feel happy with that.

  95. To people “behind the curtain” – thank you.

  96. I have to say that this is a great post and I would love to hear more about what you guys do. I think it is really impressive for me as a young blogger to see what is going on with the great google. What do you feel that Google did to get to where it is at and what can other small companies and bloggers can learn from it?

  97. Thank you for the useful info, Matt!

  98. Matt – thanks for the post. My stance personally is that the google™ algorithms will just get
    better and better over time and will in fact continue to reward the ‘legits’ whilst weeding out the weeds if you will…

  99. david sutherland

    two words: LATVIAN SPAM

    (okay more than two words: please hire some Latvian/Letish speakers and help stamp that junk out! It’s the only thing that slips through into my gmail acct.)

  100. Hi Matt, I do understand your engeneers work hard to get the best results, as well as SEO engineers and webmasters do a good job. I’m none of them, a simple person, I mantain my wifes website.

    I do have a question;

    Many times on the web I see link campanies advertising and express themselfs as the best place to add your website, some times even automatic as you add there links on your website.

    wenn you look to the url and the searchengines results they have excellent scores, mostly 5-6 PR even 7 PR and huge links resuls hundred of thousands in Alltheweb.com => thousands in Google

    is it safe to add your url in such linkdirectory’s because they give a trustful impression as they are listed in the SE as well?

    please reply

    best regards

    Frank

  101. Hi Matt,

    with all those great “few improvements every week” , what about all that .edu spam? Can you tell us something about it? I see serps gets manually edited for high spam area like “viagra” and so on, but in general, what is going on?

    I can show you how to get on top of google for ANY keyword in 24 hours if you have content on .edu domain. Another concern, if content is not a doorway page, why it is getting removed from serps?

    I will appreciate your comment on edu spam

  102. Google is like an anthill… there’s a lot of moving inside and you never exactly know what’s happening.

  103. Most people only care about how the changes would affect the ranking of their sites. Nonetheless, thanks to those “behind the curtain”.

  104. I think the most damning thing from this article is the lack of any acknowledgement from the author regarding articles that have obviously been written by ‘normal’ people who have evident and valid concerns regarding Google and they way the search engine works.

    Unfortunately its the $$ that govern the decisions that companies make in this day and age regardless of the insane amounts of profit they are making.

    Google is just another example of this happening.

    Not answering the concerns of the small people who are the ones most needy of a fair, indescriminate means of Web advertising are being negelected in the face of the conglomarate machine whos only goal is to amass as much fiscal income as possible.

    what a shame, that the people who have a voice that WILL BE heard only use this privilage to line their own pockets……….

  105. Hi !!

    Happy Belatd B’Day to google..
    how did you celbrated Matt was it an off aor a party .
    If got tiem then mail me abt that !!!

    HAve a nice day

  106. Google has deinately changed the way search takes place now.

  107. Google (and everyone in software business) makes a lot of improvements that most people never notice

    Absolutely TRUE.

    In my experience UI is what in most noticable. You can make a million little improvements, and when you add new icon or change logo for holidays – people talk about it.

    Sad but true.

  108. Keep up the good work!

  109. I am still so amazed at how fast Google made it to the top. i used to work for microscoft and I heard was about they were going to rip Google up with their new search. We heard this for so many years but nothing ever happened. i think that google figured out something that msn could never get and still don’t have. That is how we try to be int eh hair industry but not to Google’s level – YET!

  110. Well changes need to be made that often from a company as big as Google anyway. I hope these continous and regular change will help SME’s rather than the BP.

  111. Yes, I was also part of Microsoft team and there were so much news, development, research and dollors going around for MSN Live Search!! But Google stayed ahead untouched. Now I know why they are the best.

    They have so many updates and constant research to bring in something new and impressive!!

  112. Hi matt,
    Google is still doing some funny things.
    Yesterday i was doing some image search in google. first i typed “wooden table folding” and i got one picture which ilike and make a plan to build somethin like that. Now, after say one hour my wife came and i wanna show her the model but i forgot the exact query, so i tried a variety of things like, “wooden folding table / folding wooden table and folding wood table” and i viewed the results upto 10 pages but coult able to retrieved the one i was looking for, except google.com, i also tried in my local google but the end resullt is same. Today morning i thought maybe i have put something else and tried the first one, and there it is sitting in the first page.
    I agree, you guys are doing a great job and google is best compared to others but still way to go.

  113. Hey Anshuman,

    You got it wrong. It is not Google’s mistake. When you query a search keyword, the keyword is searched in a Google server which is closest to your location or a server which is near you and not that busy. So each server database might vary. You also have to know that Google has more than a million servers world wide. So its not surprising to get different results.

    You can blame Google if it produced the wrong results. But it did not do so… 🙂

  114. Google is the most successful website because it does its job well. They have been voted as one of the top companies in the world to work for. I am sure they would not achieve this if they were sat around gloating about how much money they made.

  115. 2 questions Matt:

    1. Is every incoming link important? If i have a shoe website, would a good, reputable laptop link for example do me any good?

    2. Do no follow links account for anything at all in terms of SEO? I have seen sites linking from wikipedia only, ONLY, ranking in the top 10. If they do count.. then whats the no follow fuss anyway?

    You wouldn’t answer these anyway, but maybe someone else will..

    Another one – Google has just banned a site of mine that ranked very high, over 2000 uniques a day.. submitted reconsideration request. no one replied for many months, did it again.. no one replied.. whats the form for? Btw, that site was mentioned in all local newspapers, heavily advertised.. and now that its banned, its a complete waste.. as everyone searches on Google for the site anyway..

  116. I realize this is an old post, but thought I’d comment on a post that’s germane to my question/comment.

    I just read on one of the top SEO websites out there:

    “Google’s algorithm is based on links. Links are the most important metric they use, but of course content promotes the natural growth of links. Google is very good at figuring out which sites’ links are acquired naturally and which are not.

    This said, spam makes its way to the top of the SERPs via links and links alone (even though it rarely stays there for very long). Even so, we can thus deduce that links are valued very highly in Google’s algorithm.”

    If this is really true, isn’t this a sad state of Google search? I can’t argue that there’s NOTHING better out there & so you guys are doing something right, but isn’t this why we see spam more at the top vs. quality content for a number of searches? Or is this simply untrue? I’m just curious if this statement is true or not.

    Thanks in advance Matt.

  117. If you read this and believe it, you may want to ask yourself one glaring question – Are you prone to fits of uncontrolled gullibility – and is there a cure? Google is in the business of doing several things, but there ultimate goal is to control the web. You control the web and ultimately, you control how the money flows on the web and then to a mid-level degree, how money flows off of the web. There are certain rules that apply to those who decide to conform to them and there are certain rules to apply to those who do not have to conform. large entities on the web, don’t even need relevant keywords because their connections to other sites makes them relevant. Smaller sites, without big budgets for marketing have to do more to get listed because they don’t have the connections to the largest sites or the content. For instance, there are two things that get you listed very high on a search engine, internal and external links and Web 2.0 content. Search terms mean virtually nothing, and keywords mean virtually nothing. When you take into account those two factors (2.0 content and the links you have to other websites), only large websites are going to be listed highly. The web is no longer an even playing field, the rules have changed based on what these large corporations have said they wanted from google and google is changing based on that feedback. It used to be that almost every site could get listed highly for weeks based on keywords used and to a degree content and then the press that drives users to their site…now if you aren’t big, or have the budget to spend on marketing or the links to some of the largest websites, it will be alot more difficult than in the past.

  118. Allow me to give an example of what I mean…I own a small golf club sales site http://www.sj-golf.com and we specialize in game improvement equipment from smaller golf companies – we do that for two reasons
    1. I found that these smaller companies come up with great innovation long before the biggers ones do…i.e. changeable shafts etc, using different materials…etc. (by the way, Nickent Golf and Dynacraft came out with the changeable shaft long before Taylormade, but not many know that because Taylormade has a huge marketing budget)
    For me to get listed on google I have to have the internal and external links to larger website, good content or a huge marketing budget (not a keyword ad budget because those are not longer that important). So, that means I have to link to PGA tour or Golf magazine or ESPN. Well, all those site advertise for the big name companies, Taylormade, Callaway, Cleveland etc…they do not advertise for smaller companies. So, when someone clicks on my rss feed to the PGA Tour site, there going to see ads for these other clubs. So, it is conceivable (and I am not saying this is written in stone) but it is conceivable that all the money is going to flow back to one place…The biggest sites, the biggest companies etc. Now, I really do not believe this for my site or myself..I believe I can dictate what happens – but if you look at from a logical standpoint, then thats what it looks like.

  119. well, the most bizarre thing related to google i have ever seen is this :

    http://makemylogic.com/2009/12/11/wtf-the-most-bizarre-searches-on-google-lol/

  120. Matt, the thing about Google and its tools is that people mainly utilize and love google because it treturns the best search results. The google tools are based around giving webmasters better tools to imporve the focus and relevance of their site which leads to better visitor experience. All this is a bonus to Google.

    Dont worry about the NY times listening into your conversations. The market loves realistic and ‘open’ companies. Thats why we all love google it seems.

  121. I can only imagine all the tiny things that Google deals with on a daily basis but I am hoping and at the same time realizing that Google keeps its eye on the ball when it comes to search. There is no such as too big to fail. I am a Google fan and think it is the top search engine but keep your on the ball or a better way of saying it dance with the one that brought you to the dance – search is why we know Google!

css.php