PubCon 2009 talk: State of the Index

If you followed @googlewmc on Twitter you would already know about this, but I recently recreated my “State of the Index” talk from PubCon in November 2009. Here’s the video of the talk below:

And here are the slides if you’d like to follow along:

The talk is almost half an hour, so I hope you enjoy it and learn something new!

53 Responses to PubCon 2009 talk: State of the Index (Leave a comment)

  1. Thank you for doing this recap for us… as always, excellent info. Cheers!

  2. Thanks for doing the recap! Excellent info, as always. Cheers!

  3. Sorry for the double post. Got a 500 error!

  4. Very good info, thanks for keeping us informed, its what keeps Google on top. Nothing like free for sure

  5. Is there a reason you’ve installed Thesis into wp-content/themes/thesis_15/ and not wp-content/themes/ ? That won’t boost SEO would it? 😉

  6. Matt,

    Question about webmasters help section on “Moving your site”. I’ve talked with webmaster forums ect and none can give a good answer. I did a correct 301 redirect for all my pages correctly. I did the change of address and submitted a sitemap. It’s been 5 1/2 months and rankings are nowhere near back. Withing you webmaster help section it states “help you retain your site’s ranking in Google’s search results.” I did the redirect based on this and can’t understand why google can’t be more transparent about loss of rankings??

    Could you please post more clearly about moving a site so other small business owners won’t lost 2/3rds of their business?? Why offer it if you will be penalized??

  7. I think Google can do more to categorize the Index so folks can find information easily. I found this cool site – see my “author’s link”- they seem to have it right. All nicely categorize, no need to type anything just click the category links and you have beautiful results from major authority websites.

    Google should be doing the same, or it will be left behind.

  8. Don’t give up your Google job for a singing career 🙂

    Matt, you seem to be aloof to “average google users” if you think open source projects are of interest to them. Heck, I doubt most “average google users” are even aware of any other Browsers outside IE. IMO, you spend to much time in the company of “Google Geeks”.

  9. Sorry, Matt, I have to disagree with you guys on something here. Disclosure: I’m doing so because I have a vested interest, but I also have a greater-good interest at heart as well.

    While everyone wants faster sites, the logic you’re using to look at multiple CSS files doesn’t make all that much sense. Yes, multiple CSS files create multiple HTTP requests, but they’re also extremely useful in some cases.

    Media types: Print and screen CSS, if presentation needs to be different and yet still maintain some sense of CSS styling, requires two files. You can’t merge these together, no matter what…at least until the W3 comes out with some sort of way to merge print and screen together. A lot of people print a lot of web information.

    Organization of page/section/type-specific CSS: A shopping cart might require its own CSS file(s). Certain pages may have CSS-specific information that is useful only to that page and therefore should not be loaded on other pages.

    I also personally tend to separate my site-wide CSS files into elements, classes and IDs (both screen and print) just so that I don’t have to waste time either typing in a search in my editor or scrolling through trying to find the one item that I need to change. I also have four standard CSS files that I use across multiple sites that contain common aspects that I don’t want and shouldn’t have to recreate. Yes, that amounts to 10 CSS files, but one CSS file would be completely unmanageable in that scenario. The time it takes to make those server requests is very minimal as well. Doing things in this manner also reduces my own development time, which in turn allows me to put out a better website with more features for the end user.

    Besides, most of the slow-loading sites are the ones that have 1000 lines of internal CSS and Javascripts in an include or master page file because the webmasters/mistresses couldn’t be bothered to externalize their files. If you want to focus on bad, slow-loading code, start with those, man.

    Don’t get me wrong: I’m all for sites that adopt good coding practices. But this at best is a grey area, and if you start telling people they have to merge multiple CSS files together when there’s a much better reason to keep them separate, you’re going to hear some backlash about the topic, and rightfully so.

    Anyway, I’m done venting. Big up yaself.

  10. @adam – I completely agree with you.

  11. dont devalue what you offer matt and also, please, don’t sing again 😉

  12. sorry. that sounds harsh. didn’t mean it to come across like that.

  13. @damian – JERK! lol

  14. Yeah, Damian…don’t hate on Matt’s karaoke skillz. He may end up in a bar one day outsinging you and you’ll live to regret it.

  15. Content is still king and always will be 🙂 IMO, so long as site pages render in the most popular Browsers, Google cannot afford not to rank it based on content (real PR as apposed to TBPR), both directly and indirectly.

    Given a choice between good content (with non strict code) and average content (with strict code) I’ll choose the former because most SE do.

  16. Matt, I really really appreciate these videos. Great to hear what is communicated at conferences that I can not attend.

  17. Matt
    I definatly see the benefits of speedier sites and social stuff. I am however concerned about those genuine webmasters who care more about the content they offer than the design or structure of their sites.
    I struggle to understand this stuff and I feel i will be penalised, because I am not technical and I know I am not alone.
    I hope it is the best sites that offer the best content that will rank, not those that are just built better to suit Google’s needs. M.O.O.

  18. Don’t quit your day job Matt. Viva Las Vegas! Good overview. Thanks!

  19. I am really excited about some of the visible changes to the search interface that’s been rolled out in January. Highly appreciate that rich content such as vids and images are being lifted from the web content and placed next to the post they’re embedded in.

    However, when are you guy’s going to focus more on the searches than the pages? Wrote some more on that note here – jesperastrom.com/search-engine-marketing/google-challenges-2010/

    Also: When are you going to remove Mashable’s “facebook is down – again” posts from your index? 😀

  20. i have very important question and wait for answer please
    question is : it is links with NoFollow will help in SERPS or no ?

  21. Whilst some have said that the videos are difficult to view in a work environment etc, I must say I think they’re great. Keep up the great work Matt, and thanks for the very much appreciated info. Cheers, Todd.

  22. Thanks for the post Matt
    @Mark Flanighan – I don’t think the implication was that you will be penalised for not gzipping every file on your site. I think Matt was saying if you can make your site load faster it will help a little – just like most seo; everything helps a little but having content which is good helps much, much more.

  23. Is the social search function supported globally? I’m in New Zealand but seems it’s not happening here yet..

  24. That’s all fine and good but don’t you think that by incorporating things like “social search” and web history into SERP delivery that you’ll lose the advantage of offering the best web pages for a search. Another way of asking the question: don’t you think that all these little personalizing ranking add-ons are going to dilute the results, even propel pages that don’t merit ranking.

  25. Thanks for doing the recap! I really appreciate these videos. 😀

  26. every thing was ok…but what about duplicate content. let me clear the things…i have site http://www.arnavgupta.com i have created very good site and getting very good rank on Google. now one of company krishaweb has buy one domain hire seo expert .com and copied all my site content on it (stolen my entire site)…now Google is giving equal value to this domain and getting very good rank as well..

    Matt…you know this is now effecting my business…i have report to Google as well (also posted on WPN comment)…but still not action has been taken and your search engine still giving them good value…

    this is really serious issue with your algorithm..that you are not able to filter original content and not penalize stolen website….

    its my request to remove spam websites as they are continue hurting me..

    i hope you will look into this matter and if you want to talk to me then plz contact harryzalaa@gmail.com

  27. Nice overview of what was covered at Pubcon – thanks Matt!

  28. Matt I think Google has made a serious error in thinking people wanted you to hold off on caffeine. The general consensus I’m hearing at just about every major webmaster forum I go to is that peoples sites that they are working their butts off on rank BETTER in caffeine than they do in the old setup. That caffeine is returning more relevant results and less crud. Holding back caffeine is actually annoying most people at these forums and basically we feel cheated that its STILL not fully kicked on. Many of us were excited for it in Nov. Then told hey just kidding wait till the holidays are over. We were all holding our breath on JAN 1. By Jan 10 we were checking our watches and wonder double-u tee eff. Now we are seeing those results pop in sporadically at best and its actually to the point that many are getting fairly annoyed by it and by Googles complete silence about what may be going on or what the delay is. So far (and I’m not positive… just a guess) these are the DC’s I BELIEVE are showing caffeine results. I love these results and want to know when I can expect to see them all the time. Many others wondering the same:

    66.102.7.*

    66.102.7.18
    66.102.7.19
    66.102.7.44
    66.102.7.81
    66.102.7.84
    66.102.7.99
    66.102.7.100
    66.102.7.101
    66.102.7.102
    66.102.7.103
    66.102.7.104
    66.102.7.105
    66.102.7.106
    66.102.7.115
    66.102.7.132
    66.102.7.147
    66.102.7.161
    66.102.7.162
    66.102.7.184
    —————

    72.14.213.*

    72.14.213.18
    72.14.213.19
    72.14.213.44
    72.14.213.81
    72.14.213.84
    72.14.213.99
    72.14.213.100
    72.14.213.101
    72.14.213.102
    72.14.213.103
    72.14.213.104
    72.14.213.105
    72.14.213.106
    72.14.213.115
    72.14.213.132
    72.14.213.147
    72.14.213.161
    72.14.213.162
    72.14.213.184
    —————

    74.125.19.*

    74.125.19.18
    74.125.19.19
    74.125.19.44
    74.125.19.81
    74.125.19.84
    74.125.19.99
    74.125.19.100
    74.125.19.101
    74.125.19.102
    74.125.19.103
    74.125.19.104
    74.125.19.105
    74.125.19.106
    74.125.19.115
    74.125.19.132
    74.125.19.147
    74.125.19.161
    74.125.19.162
    74.125.19.184
    —————

    74.125.95.*

    74.125.95.18
    74.125.95.19
    74.125.95.44
    74.125.95.81
    74.125.95.84
    74.125.95.99
    74.125.95.100
    74.125.95.101
    74.125.95.102
    74.125.95.103
    74.125.95.104
    74.125.95.105
    74.125.95.106
    74.125.95.115
    74.125.95.132
    74.125.95.147
    74.125.95.161
    74.125.95.162
    74.125.95.184
    —————

    209.85.225.*

    209.85.225.18
    209.85.225.19
    209.85.225.44
    209.85.225.81
    209.85.225.84
    209.85.225.99
    209.85.225.100
    209.85.225.101
    209.85.225.102
    209.85.225.103
    209.85.225.104
    209.85.225.105
    209.85.225.106
    209.85.225.115
    209.85.225.132
    209.85.225.147
    209.85.225.161
    209.85.225.162
    209.85.225.184

  29. Hi Matt
    Thanks for putting the presentation up here too. I really appreciate it, my connection is iffy for videos so its great to have a written version, thanks again.

    Now the only thing I need to know is how do I get my pic up and not the little piece of wallpaper!

  30. quick follow up. seems 3 of the 5 datacenters I pointed out in my last post have reverted back to the old results. the only ones continuing to display what i believe to be the new results are

    209.85.225.*

    74.125.95.*

    which is exactly the kind of frustrating thing I was describing. Seems like you guys are making progress.. up to 5 data centers.. then reverts back down to 2.

  31. Matt
    You probably won’t read this as it doesn’t really fit in to the post. But…

    We’ve been contacted by a ‘Google Partner’ in the UK who claims to be working directly for Google. His claim is that all sites in Google’s index will now disappear from the organic serps as a result of Caffeine which – in his words – ‘has been designed by Google to banish all sites which are ‘optimised’ from the serps! He further went on to claim that only 10% of all organic listings on page #1 of the serps are ever clicked with 70% of clicks going to the sponsored results at the top of the organic listings, and 20% to the sponsored listings on the right. He claims these to be “Google’s own figures”. Surely not?!

    I like to think we’re reasonably aware of how to ‘optimise’ a site. We use WP as a CMS; all page titles and descriptions are unique; lots of fresh and original content; good site structure and linking (thanks WP); not a single solicited incoming link and everything totally ‘White Hat’. Blimey, we’ve even started using Gzip to cut down on the html (why did WP stop offering it ‘out-of-the-box’ after 2.5?) and are looking at minifying javascript and css. But what about the poor harassed business owner who gets a call from ‘Google’ telling him that you can guarantee him to be top of the tree and because you (Google) are changing the organic results he has no other way of appearing on the first page of the serps unless he coughs up a few quid?

    Not a good scenario.

  32. Hi matt i just wanna say that what google had done during 2009 for webmasters is amazing.
    back in israel we are lecturing a professional SEO course and we teach people to really practice only whitehat methods and do what’s best for the surfers on the web.

    on our website we embed every youtube piece you release and translate what’s important to hebrew so people can get answers for their questions from an offical source (which is you of course 🙂 )

  33. Dude, just grow your hair back, really, the information is very useful, thanks but hey let it grow!

  34. We’ve been contacted by a ‘Google Partner’ in the UK who claims to be working directly for Google. His claim is that all sites in Google’s index will now disappear from the organic serps as a result of Caffeine

    Correction, you’ve been contacted by a ‘spammer’. IF “all sites in Google’s index will now disappear from the organic serps as a result of Caffeine” wont that leave Google with no results to display? (No need to answer)

    Just another spammer cashing-in on Google fear. Something SEO have been doing for Years 🙂

  35. Dave (Original) – Agree with you completely. The anger comes from companies who gain a ‘Qualified Google Advertising Professional’ status and then use that to claim they are ‘Google Partners’, and use scare tactics to push people in to spending money. Surely Google should remove the ‘Professional’ status from companies like this? It can’t do Google any good and the completed fabricated ‘Google Facts’ they us to justify their pitch should be stamped on.

    And yes, the claim that all ‘Optimised’ sites will disappear from the serps due to Caffeine is a barking mad opening gambit, but one that would surely put the wind up a business owner who has no real knowledge of how search results work?

    Needs a bit of input from Google to penalise the abusers.

  36. I had nothing but problems on my (wordpress) blogs to optimize them for speed, 90% of the CSS sprites/merger plugins out there are nothing but hacks. CSS merging and Javascript merging (to get a good YSLOW rating) always results in weird errors, themes and other plugins breaking or behaving oddly. (For example the sharethis plugin).
    For the sake of that my sites actually display RIGHT i had to revert some of them again. Then rather a slow site, but no display errors.

  37. John, agree. IMO, Google spends too much time banning/penalizing the victims (effect) of “SEO” companies, rather that the “SEO” company themselves (cause). If they took a ‘top-down’ approach everyone would win.

    Starting a SEO business is like starting a lawn-mowing business. I.e. any fool can jump on the bandwagon.

    IMO, the concept of SEO is a myth/fallacy. If one optimizes site pages for Humans, good ranking will follow in time. Know your content and know your site visitors. After all, why would any SE prefer sites optimized for Search Engines when their users are Humans?

  38. Hi Matt,
    I hope your twitter diet is going well. I really appreciate all the tools, advice, and insight you provide to the Google community. I am currently using many of these tools like page speed, the rel=”canonical” tag, and the js compiler… to name a few. However, I am concerned (and a little frustrated 🙁 about some the search changes I am seeing. For example I worked very hard to make my site a faster, friendlier, quality content enriched place for visitors and I find 3 cloned spam sites owned by the same person creeping up the search results some how. I am really concerned as these sites are cross linked, offer no unique content, load in several seconds, and are filled with page slowing ads, affiliate content, and keyword stuffing. Meanwhile I slave and update my site daily with fresh content and seem to be punished for the effort. I am not sure what this guy is doing but I am sure it is black hat. If you care to see what i am talking about check out these three sites: infiniteadvice/.com (Ranked second for the keyword psychic), freepsychicsight/.com, and prophet4you/.com. The text on the page is so keyword stuffed it does not even make sense. The large obscene banners(their are seven of them on the home page!) make the site an eyesore. I know everyone makes mistakes and my site is far from perfect but I do try very hard to continuously improve site quality and user experience. And, to add insult to injury some person filed a DMCA complaint for a syndicated article they submitted to 100,000 different article sites back in 2009. They did not even follow up on it and of course I took the article down but I can’t help but think it was a personal attack by a black hat expert. Well, needless to say, I check all my article submissions in copyscape now and will be faxing a counterclaim in to Google (thanks to Google webmaster tools for the info.). Thanks again Matt and God bless.

  39. Frederick Gimino: you mean you didn’t see it coming? Given the nature of your site…you probably should have.

  40. Hi Matt,
    After reading and listening to a mixture of both good and bad SEO advice over the years, it was like a breath of fresh air to discover your YouTube videos recently. I really appreciate your clear, open and informative approach on your videos which provide the Google user community with a wonderful insight. Thank you.

  41. Frederick Gimino: you mean you didn’t see it coming? Given the nature of your site…you probably should have.

    Always took for the kind of guy who would kick a man when he’s down.

  42. As a deaf user, I would like to let you know that I cannot access any auditory information of your videos or slides without transcripts and/or captions. It is not only one of basic requirements of web accessibility, but also benefits everyone, including website owners because it improves SEO rankings of their own websites. Would you please make all of your auditory contents accessible via text equivalents? Thank you

  43. Nice hair cut Matt hehehe that was really cool. Thanks for the recap… Google rocks!

  44. Matt thanks for the recap of your Pubcon talk. I missed this years unfortunately due to some projects, hopefully will make it next year. It was a blast hanging out at the beer garden in Vegas with SEO’s from all over the world. We’re planning a large SEO Camp here in Nashville for the Fall if you have any interest or are visiting the family in Tennessee. Hope all is well!

  45. Matt, nice to hear that Google appreciates and rewards the tech efforts of SEO. Excellent video, audio, ppt all together = nice presentation. Thanks. /r Scotty

  46. Hi Matt I have a question concerning the “state of the index” Why does Google favor negativity? Google is the only search engine that does this but any site with the term, scam, sucks, complaint, or ripoff in their domain gets top priority in your listings, and the webmasters of those type sites know it, and abuse it. Take for instance I own a drop shipping company I fight hard to keep my rankings, follow Googles rules and some clown comes along with a site names it dropshipping sucks .com and guess what he’s number one, He has a one page website, that’s a mile long, and to top it all off, he’s selling other drop shipping companies affiliate programs. I think Google needs to evaluate how much rank they give to these type sites, and stop promoting negativity. The drop shipping example is just one of many here’s a few more examples : Survivor- People who Google the term survivor to see who is stuck on that island also are treated to survivor sucks. com a few spots down, Pay Pal =pay pal sucks. com. Bill Oreilly I guess this ones a coin toss. oreilly-sucks . com. Even something like fencing, come on now fencing sucks. com.
    The funny thing is this one is nowhere to be seen google search sucks. com.

  47. Great. The hair cut is superb…. Veryy cute Matt..

  48. Matt – the state of the index, if you are talking about Google then it isn’t good! Google developed into the primary search engine because it delivered relevant results quickly and clearly, it is still number one but you will loose your market share from now on. The simple reason, the grab for money. I’ve been searching for a commercial service defined by geographical location for a client – the results screen that Google now returns is just appalling! The left hand column has 3 sponsored links at the top, the right hand column is just sponsored links as per usual. The rest of the page is filled with business listings. The top of the organic results is at the bottom of the page on the left hand column – how appallingly bad is that! With a PhD academic background, i used Google from the early days for academic research and it was fantastic! The results were clear and obvious, but there is now an evident push to make ‘money’ being the primary reason for Google’s existence. As with many big companies, you are fast loosing touch of where you came from and it is a real shame because this will inevitably be the beginning of the demise of google. People use google because of the organic results – period. To show just one organic listing above the fold for any search is a very sad return from what used to be a great search engine!

  49. Hola Matt! buen trabajo el realizado en 2009, espero que sigan siendo un poco más cercanos a nosotros los webmasters para mejorar en conjunto!.

    Hi Matt, goo work you made the last year, I hope you follow doing them for a better WWW working together!

  50. thanks matt, it is good to have some of the myths cleared up. i like it when you confirm things like ‘keywords tag’ not a factor.

  51. Excelent post. Thanks.. I really appreciate it, my connection is iffy for videos so its great to have a written new version..

  52. Adam… I completely agree with you.

  53. Thanks Matt alot of these tools I did not know google offered. I did try looking in Google Webmaster help but could not find a compilation of all the intuitive tools available specifically for webmasters. So this video has really helped me along.

css.php