Debugging Dean

Battelle points to this post by Steven Johnson. Mr. Johnson just had a boy, named him Dean, and asked his friends to point to that post. Battelle asks because the post was in the top 10 for the search [dean] for a while, and then it wasn’t.

By the time I checked today, it was back in the top 10. Bear in mind that rankings do change all the time, especially for bloggy items. One common reason is when multiple people blog about something, it’s often on the front page of their blogs, and front pages typically have the highest PageRank. As stories scroll off the front page of peoples’ blogs, there’s often a dip in ranking because the links tend to be from deeper pages with lower PageRank.

There can be other reasons why a page temporarily drops, too. I saw at least one day when we weren’t able to include the page in the index (could have been that the server was unreachable for a short time). When a server is down for a short while, we normally recrawl the pages again the next time, and then the page often returns to roughly where it was before. Our webmaster console in Sitemaps is a great place to see errors like that.

On the other hand, is that really the post you’d like Dean-watchers to find? Can you imagine when Dean is 13, and mortified that one of the top results is baby pictures of him? “Dad, you might as well have posted picture of me in a bathtub! Why do you always have to embarrass me?” πŸ™‚

33 Responses to Debugging Dean (Leave a comment)

  1. Of course, all of this would have been moot if the new arrival had been named Raffey Silvertongue McHernandez. Rafe for short.

  2. That middle paragraph is good thanks for confirming that Matt, who cares about the other stuff. πŸ˜‰

  3. β€œDad, you might as well have posted picture of me in a bathtub! Why do you always have to embarrass me?”

    Yup, that would certainly be embarrassing. πŸ˜€

    However, it was a nice idea to give PR to his son.

  4. hmmm….This is interesting that Google can change the SERPS so dramically for competative keywords. This is interesting news….

    There have been a number of interesting examples of Top ranked Blogs or UGC with Topics that appeared high on the SERPs for competative keywords (partially because of the TITLEs used) – then abruptly dropped a few days later

    Right after the Miss USA pageant – A topic from this Blog appeared on the First Page for MISS USA and stayed for a month. πŸ™‚

    Also, many hi profile online news sites will sometimes have a popular article that – because of the TITLE used – abruptly skyrocketed into the top 20 in a few days.

    One interesting factor, was the Google Webmaster Guidelines – that for the past two years was in the (SERPs) TOP 20 for

    SEARCH ENGINE OPTIMIZATION

    then made it to the TOP 10 for about three months – then suddenly DISAPPEARED for about two weeks – now, it has re-debuted again at position 5 for the past two weeks. πŸ˜•

  5. Dave (Original)

    RE: One interesting factor, was the Google Webmaster Guidelines – that for the past two years was in the (SERPs) TOP 20 for Search Engine Optimization”
    ==========================================

    I wish Google would fix that page at #1 for any/all terms like SEO. Would go a long way towards innocent site owners forking out big dollars to black hats only to have their site pages fall, banned or penalized!

  6. Good morning Matt

    The truth is that the lovely Dean was hit by one of those Data-Refreshes of yours and lost his ranking.

    Btw, have you told Mr. Johnson that in future he should expect Dean ranking to be “Refreshed” each week πŸ™‚

  7. Dave, I’m a little surprised (but happy) that it ranks naturally in the top 10 for [seo] and [search engine optimization].

    S.E.W., I’m letting that comment stand, but do you have to be so CAPITAL and boldy all the time? πŸ™‚ And sometimes CAPITAL BOLDY?

  8. Dave (Original)

    RE: “Dave, I’m a little surprised (but happy) that it ranks naturally in the top 10 for [seo] and [search engine optimization].
    =========================================

    Yes, but it be even better it it was stuck at #1 forever πŸ™‚ After all, it should be the foundations of any/all SEO.

  9. Impatient_fanboy

    Are those data refreshes results just temporary?

  10. I dunno, that strikes me as not-quite-right somehow. The page should have to stand on it’s own.

    Hey, I wonder what the results on Yahoo! are like for [seo]. Pause. Pause. Holy shit?! I’m #1 on Yahoo! for SEO?!? And #7 for [search engine optimization]? Woohoo! Where do I collect my check? πŸ˜‰

  11. Matt, I’ll be happy to give you a check at SES if I can advertise my SEO services on here. πŸ™‚

  12. Nah, I thought people just magically made money from being #1 for [seo]. πŸ˜‰ You hear people talking about it all the time..

  13. >I saw at least one day when we weren’t able to include the page in the
    >index (could have been that the server was unreachable for a short time).
    >When a server is down for a short while, we normally recrawl the
    >pages again the next time, and then the page often returns to
    >roughly where it was before.

    So if a server is not available for a day, Google removes the page from the index? Wow. Are you certain about that, Matt? Can Google crawlers determine if it is a connection problem and possibly on their side? (bad DNS cache, etc)

    Why is it that pages which have been 404’ing for a year are still in the index? πŸ˜‰

  14. “I’m #1 on Yahoo! for SEO?!?”

    Compared to other countries you must have a very bad reputation in Germany
    http://de.search.yahoo.com/search?p=seo
    Try a post in German, you will be amazed.

  15. >Nah, I thought people just magically made money from being #1 for [seo]. πŸ˜‰ >You hear people talking about it all the time..

    Looks like you got a monetarization problem … We can talk about it in detail next week. πŸ˜‰

  16. Dave (Original)

    RE: ” dunno, that strikes me as not-quite-right somehow. The page should have to stand on it’s own.”
    =======================================
    Oh I don’t know. Google has started showing some results *different* to the search term used after #3 or so, so why not this? Even just a little tiny one….please πŸ™‚

  17. I can’t see Google screwing with results like that. They weren’t even AIMING for that in all likelihood.

    Plus, if they did that and it didn’t even turn up on the radar in Yahoo! and MSN, then every conspiracy theorist webmaster alive would be talking about how big G shows favouritism to itself and how evil it is and how Matt, Sergey and Larry (the only three employees…they don’t have any more, you know!) are redoing the algorithm late at night in their own best interests.

    I’m with Matt on this one, Dave. Sorry. They gotta leave this alone.

    I’m just impressed that Matt finally swore. About time he let loose. I knew it was in him. Come on, Matt, go Tourette’s on your blog. Open up a can of Stone-Cold-level whoopass on something. That’d be cool! (Just not me. I didn’t do anything wrong.)

  18. > Why is it that pages which have been 404β€²ing for a year are still in the index?
    ===============

    This was a fantastic question. I’ve often seen URLs that 404 or have been 301 redirected stay in Google’s search results for ages. It would be really nice if webmasters could make bad URLs/pages on their sites disappear/redirect automatically in Google’s serps.

    While maybe a page shouldn’t be removed the first time it turns up a 404 error, maybe it should after the second or third try. This could greatly improve user experience and it would help webmasters get bad pages removed from their sites if they so wished.

    I know in my case, Google is claiming to have indexed many times more pages for my site than actually exist and I think these phantom pages are causing duplicate content/doorway page penalties for my site. I aggressively go through my 404 errors trying to remap bad inbound links with 301 redirects to the correct page to improve user experience. I sometimes think that sometimes Google is treating the redirected URL as a separate page and then dinging me with a duplicate content penalty.

  19. “duplicate content penalty” πŸ™‚ penalty or filter? πŸ˜€

    404’s for ages? The usual time I see 404’s in Googles index for is about a year. What about 403’s or 410’s? Wouldn’t it make sense if the webmaster explicitly deleted and returned “forbidden” or “gone” to respect that? It’s not like it’s an accident…. like a short-term server outage or anything like that …

    Many people seem to have trouble getting into the index, they should try getting out πŸ™‚

  20. > Many people seem to have trouble getting into the index, they should try getting out

    ======

    You know Yahoo is way worse about this than Google. Six years ago I implemented 301 permanent redirects from one domain to another as part of a site move and would you believe that Yahoo is still pointing to the old domain? What’s even worse, they are pointing to the old domain using current title and description stuff pulled from the new domain. I’m certain, Yahoo’s inability to handle 301 redirects correctly is killing me in their search results. πŸ™ You’d think that after six years their bots would have figured out the 301 redirect thingy.

  21. Matt ranking #1 for “SEO” in Yahoo shows how literal an engine Yahoo is and also how they can easily be gamed by Digg-bloggy-comment manipulation. Google’s ranking for SEO Matt is about right, seems fair.

    LOL, I have been looking around for money in “SEO” and can’t find any, also how the hell do you rank for something “SEO” if you are not a Matt Cutts these days?

    Great post, highly educational.

    Lesson: Webmasters can work with their existing PR no matter how big or small it might be, a beautiful thing really.

  22. > Nah, I thought people just magically made money from being #1 for [seo]. πŸ˜‰ You hear people talking about it all the time..

    Has anyone told you about Adsense? πŸ˜‰ I’ll send you my affiliate URL.

  23. Perhaps you could auto remove bold from posts by S.E.W. and drop full caps to initial caps for sentences? πŸ˜€

  24. Hi Matt,

    I attended SES 2006 in San Jose as an exhibitor and enjoyed Google Dance which’s really exciting. After nice food and beer, I spent all my time at Google Lab chatting with Google engineers. Amanda said she knew you well and you’re in Spam dept. I wished I could meet you there.

    Now I have a problem with my company site, after coming back from San Jose…

    The Google Indexed pages in our site has been down to 70,000 from 1,500,000 pages. This is the second time since this year. The first time it happened in early April due to Bigdaddy switch. But what’s going on now? Is it another Google Dance or what?

    I just want to find out whoes problem it is. If something is wrong in our end, please let me know what it is in order for us to correct it ASAP.

    BTW, I asked Googlers in San Jose why Google more and more discriminates new sites and only favors big guys.

    Thanks for your time.

  25. and like I said… spelling – I know it, its my typing, it sucks and I forget to check before I submit… dagnabbit…

  26. Matt,
    Does Google crawl more often if the a website content keeps changing ? Or the crawling rate is still the same and still depends on quality inbound links ?

  27. ”Does Google crawl more often if the a website content keeps changing ? Or the crawling rate is still the same and still depends on quality inbound links ?”

    In my opinion, changing content doesn’t make google crawl your site more.
    but I think pagerank do have a big influence on this.. so yea inbound links matters.

    Keep in mind nobody can be sure how it really works.. Google always keep their little secrets. grr πŸ˜›

  28. Please Dean, don’t be embarassed. There is no reason for that…hhhhh….Thanks for the nice entry and I hope that the post remains on the top ten forever.

  29. LOL, I can just see it, the next generation of birthday gifts becomes backlinks. You get your 25 friends to put a link to your site on their blogs…

    But seriously, Google SERPS fluctuate so much, and it’s often hard as a web developer to figure out why your site went up or down. Often it’s something out of your control, like someone with a decent PR who had linked to you moved their site, or something in the news caused a lot of other sites to nab the top spots for your keywords.

    I suggest we all link to THIS post, drive its PR up, and thus “donate” to the Dean story.

  30. My question is why, when a page drops, does it drop so much? I recently made no changes to one of my sites, but it fell from about 7th to somewhere on the 5th page of results. Since then, it’s bounced back, but it took a couple of weeks. I don’t understand why it fell in the first place, and why it came back, but especially why it fell so far…

  31. Touch: It’s called a “Google Dance”…. this is when Google shuffles listings around when it updates it’s data.

    So that means when your site moved up, it wasn’t really up. It just got moved until the sites that really was supposed to be up was updated.

  32. Lol, Poor Dean, affected by the power of the Google search πŸ˜›

  33. Does Google crawl more often if the a website content keeps changing ?

css.php