Debugging Dean
Battelle points to this post by Steven Johnson. Mr. Johnson just had a boy, named him Dean, and asked his friends to point to that post. Battelle asks because the post was in the top 10 for the search [dean] for a while, and then it wasn’t.
By the time I checked today, it was back in the top 10. Bear in mind that rankings do change all the time, especially for bloggy items. One common reason is when multiple people blog about something, it’s often on the front page of their blogs, and front pages typically have the highest PageRank. As stories scroll off the front page of peoples’ blogs, there’s often a dip in ranking because the links tend to be from deeper pages with lower PageRank.
There can be other reasons why a page temporarily drops, too. I saw at least one day when we weren’t able to include the page in the index (could have been that the server was unreachable for a short time). When a server is down for a short while, we normally recrawl the pages again the next time, and then the page often returns to roughly where it was before. Our webmaster console in Sitemaps is a great place to see errors like that.
On the other hand, is that really the post you’d like Dean-watchers to find? Can you imagine when Dean is 13, and mortified that one of the top results is baby pictures of him? “Dad, you might as well have posted picture of me in a bathtub! Why do you always have to embarrass me?” 
Matt Cutts Said,
July 30, 2006 @ 8:06 pm
Of course, all of this would have been moot if the new arrival had been named Raffey Silvertongue McHernandez. Rafe for short.
Aaron Pratt Said,
July 30, 2006 @ 8:34 pm
That middle paragraph is good thanks for confirming that Matt, who cares about the other stuff.
Mark Said,
July 30, 2006 @ 8:41 pm
“Dad, you might as well have posted picture of me in a bathtub! Why do you always have to embarrass me?”
Yup, that would certainly be embarrassing.
However, it was a nice idea to give PR to his son.
SearcH EngineS WeB Said,
July 30, 2006 @ 9:19 pm
hmmm….This is interesting that Google can change the SERPS so dramically for competative keywords. This is interesting news….
There have been a number of interesting examples of Top ranked Blogs or UGC with Topics that appeared high on the SERPs for competative keywords (partially because of the TITLEs used) - then abruptly dropped a few days later
Right after the Miss USA pageant - A topic from this Blog appeared on the First Page for MISS USA and stayed for a month.
Also, many hi profile online news sites will sometimes have a popular article that - because of the TITLE used - abruptly skyrocketed into the top 20 in a few days.
One interesting factor, was the Google Webmaster Guidelines - that for the past two years was in the (SERPs) TOP 20 for
then made it to the TOP 10 for about three months - then suddenly DISAPPEARED for about two weeks - now, it has re-debuted again at position 5 for the past two weeks.
Dave (Original) Said,
July 30, 2006 @ 10:42 pm
RE: One interesting factor, was the Google Webmaster Guidelines - that for the past two years was in the (SERPs) TOP 20 for Search Engine Optimization”
==========================================
I wish Google would fix that page at #1 for any/all terms like SEO. Would go a long way towards innocent site owners forking out big dollars to black hats only to have their site pages fall, banned or penalized!
Harith Said,
July 30, 2006 @ 10:44 pm
Good morning Matt
The truth is that the lovely Dean was hit by one of those Data-Refreshes of yours and lost his ranking.
Btw, have you told Mr. Johnson that in future he should expect Dean ranking to be “Refreshed” each week
Matt Cutts Said,
July 30, 2006 @ 11:44 pm
Dave, I’m a little surprised (but happy) that it ranks naturally in the top 10 for [seo] and [search engine optimization].
S.E.W., I’m letting that comment stand, but do you have to be so CAPITAL and boldy all the time?
And sometimes CAPITAL BOLDY?
Dave (Original) Said,
July 30, 2006 @ 11:52 pm
RE: “Dave, I’m a little surprised (but happy) that it ranks naturally in the top 10 for [seo] and [search engine optimization].
=========================================
Yes, but it be even better it it was stuck at #1 forever
After all, it should be the foundations of any/all SEO.
Impatient_fanboy Said,
July 31, 2006 @ 12:06 am
Are those data refreshes results just temporary?
Matt Cutts Said,
July 31, 2006 @ 12:20 am
I dunno, that strikes me as not-quite-right somehow. The page should have to stand on it’s own.
Hey, I wonder what the results on Yahoo! are like for [seo]. Pause. Pause. Holy shit?! I’m #1 on Yahoo! for SEO?!? And #7 for [search engine optimization]? Woohoo! Where do I collect my check?
ThomasB Said,
July 31, 2006 @ 12:33 am
Matt, I’ll be happy to give you a check at SES if I can advertise my SEO services on here.
Matt Cutts Said,
July 31, 2006 @ 12:47 am
Nah, I thought people just magically made money from being #1 for [seo].
You hear people talking about it all the time..
JohnMu Said,
July 31, 2006 @ 1:00 am
>I saw at least one day when we weren’t able to include the page in the
>index (could have been that the server was unreachable for a short time).
>When a server is down for a short while, we normally recrawl the
>pages again the next time, and then the page often returns to
>roughly where it was before.
So if a server is not available for a day, Google removes the page from the index? Wow. Are you certain about that, Matt? Can Google crawlers determine if it is a connection problem and possibly on their side? (bad DNS cache, etc)
Why is it that pages which have been 404′ing for a year are still in the index?
Ralf Said,
July 31, 2006 @ 1:21 am
“I’m #1 on Yahoo! for SEO?!?”
Compared to other countries you must have a very bad reputation in Germany
http://de.search.yahoo.com/search?p=seo
Try a post in German, you will be amazed.
ThomasB Said,
July 31, 2006 @ 2:39 am
>Nah, I thought people just magically made money from being #1 for [seo].
>You hear people talking about it all the time..
Looks like you got a monetarization problem … We can talk about it in detail next week.
Dave (Original) Said,
July 31, 2006 @ 3:15 am
RE: ” dunno, that strikes me as not-quite-right somehow. The page should have to stand on it’s own.”
=======================================
Oh I don’t know. Google has started showing some results *different* to the search term used after #3 or so, so why not this? Even just a little tiny one….please
The Adam That Doesn't Belong To Matt Said,
July 31, 2006 @ 7:00 am
I can’t see Google screwing with results like that. They weren’t even AIMING for that in all likelihood.
Plus, if they did that and it didn’t even turn up on the radar in Yahoo! and MSN, then every conspiracy theorist webmaster alive would be talking about how big G shows favouritism to itself and how evil it is and how Matt, Sergey and Larry (the only three employees…they don’t have any more, you know!) are redoing the algorithm late at night in their own best interests.
I’m with Matt on this one, Dave. Sorry. They gotta leave this alone.
I’m just impressed that Matt finally swore. About time he let loose. I knew it was in him. Come on, Matt, go Tourette’s on your blog. Open up a can of Stone-Cold-level whoopass on something. That’d be cool! (Just not me. I didn’t do anything wrong.)
Ken Barbalace Said,
July 31, 2006 @ 8:15 am
> Why is it that pages which have been 404′ing for a year are still in the index?
===============
This was a fantastic question. I’ve often seen URLs that 404 or have been 301 redirected stay in Google’s search results for ages. It would be really nice if webmasters could make bad URLs/pages on their sites disappear/redirect automatically in Google’s serps.
While maybe a page shouldn’t be removed the first time it turns up a 404 error, maybe it should after the second or third try. This could greatly improve user experience and it would help webmasters get bad pages removed from their sites if they so wished.
I know in my case, Google is claiming to have indexed many times more pages for my site than actually exist and I think these phantom pages are causing duplicate content/doorway page penalties for my site. I aggressively go through my 404 errors trying to remap bad inbound links with 301 redirects to the correct page to improve user experience. I sometimes think that sometimes Google is treating the redirected URL as a separate page and then dinging me with a duplicate content penalty.
JohnMu Said,
July 31, 2006 @ 11:40 am
“duplicate content penalty”
penalty or filter? 
404’s for ages? The usual time I see 404’s in Googles index for is about a year. What about 403’s or 410’s? Wouldn’t it make sense if the webmaster explicitly deleted and returned “forbidden” or “gone” to respect that? It’s not like it’s an accident…. like a short-term server outage or anything like that …
Many people seem to have trouble getting into the index, they should try getting out
Ken Barbalace Said,
July 31, 2006 @ 3:25 pm
> Many people seem to have trouble getting into the index, they should try getting out
======
You know Yahoo is way worse about this than Google. Six years ago I implemented 301 permanent redirects from one domain to another as part of a site move and would you believe that Yahoo is still pointing to the old domain? What’s even worse, they are pointing to the old domain using current title and description stuff pulled from the new domain. I’m certain, Yahoo’s inability to handle 301 redirects correctly is killing me in their search results.
You’d think that after six years their bots would have figured out the 301 redirect thingy.
Aaron Pratt Said,
July 31, 2006 @ 6:59 pm
Matt ranking #1 for “SEO” in Yahoo shows how literal an engine Yahoo is and also how they can easily be gamed by Digg-bloggy-comment manipulation. Google’s ranking for SEO Matt is about right, seems fair.
LOL, I have been looking around for money in “SEO” and can’t find any, also how the hell do you rank for something “SEO” if you are not a Matt Cutts these days?
Great post, highly educational.
Lesson: Webmasters can work with their existing PR no matter how big or small it might be, a beautiful thing really.
Jason Murphy Said,
August 1, 2006 @ 2:39 pm
> Nah, I thought people just magically made money from being #1 for [seo].
You hear people talking about it all the time..
Has anyone told you about Adsense?
I’ll send you my affiliate URL.
Ian Said,
August 8, 2006 @ 4:02 am
Perhaps you could auto remove bold from posts by S.E.W. and drop full caps to initial caps for sentences?
Barry Su Said,
August 14, 2006 @ 1:30 am
Hi Matt,
I attended SES 2006 in San Jose as an exhibitor and enjoyed Google Dance which’s really exciting. After nice food and beer, I spent all my time at Google Lab chatting with Google engineers. Amanda said she knew you well and you’re in Spam dept. I wished I could meet you there.
Now I have a problem with my company site, after coming back from San Jose…
The Google Indexed pages in our site has been down to 70,000 from 1,500,000 pages. This is the second time since this year. The first time it happened in early April due to Bigdaddy switch. But what’s going on now? Is it another Google Dance or what?
I just want to find out whoes problem it is. If something is wrong in our end, please let me know what it is in order for us to correct it ASAP.
BTW, I asked Googlers in San Jose why Google more and more discriminates new sites and only favors big guys.
Thanks for your time.
Laura MacKenzie Said,
September 20, 2006 @ 9:57 pm
and like I said… spelling - I know it, its my typing, it sucks and I forget to check before I submit… dagnabbit…
Sterling Silver Jewelry Said,
January 5, 2007 @ 1:42 pm
Matt,
Does Google crawl more often if the a website content keeps changing ? Or the crawling rate is still the same and still depends on quality inbound links ?
Remi Said,
April 27, 2007 @ 3:10 pm
”Does Google crawl more often if the a website content keeps changing ? Or the crawling rate is still the same and still depends on quality inbound links ?”
In my opinion, changing content doesn’t make google crawl your site more.
but I think pagerank do have a big influence on this.. so yea inbound links matters.
Keep in mind nobody can be sure how it really works.. Google always keep their little secrets. grr
butlimous Said,
June 24, 2007 @ 7:26 am
Please Dean, don’t be embarassed. There is no reason for that…hhhhh….Thanks for the nice entry and I hope that the post remains on the top ten forever.
Jen Said,
August 26, 2007 @ 10:35 am
LOL, I can just see it, the next generation of birthday gifts becomes backlinks. You get your 25 friends to put a link to your site on their blogs…
But seriously, Google SERPS fluctuate so much, and it’s often hard as a web developer to figure out why your site went up or down. Often it’s something out of your control, like someone with a decent PR who had linked to you moved their site, or something in the news caused a lot of other sites to nab the top spots for your keywords.
I suggest we all link to THIS post, drive its PR up, and thus “donate” to the Dean story.
Touch Said,
September 8, 2007 @ 6:55 pm
My question is why, when a page drops, does it drop so much? I recently made no changes to one of my sites, but it fell from about 7th to somewhere on the 5th page of results. Since then, it’s bounced back, but it took a couple of weeks. I don’t understand why it fell in the first place, and why it came back, but especially why it fell so far…
mark Said,
September 13, 2007 @ 9:36 am
Touch: It’s called a “Google Dance”…. this is when Google shuffles listings around when it updates it’s data.
So that means when your site moved up, it wasn’t really up. It just got moved until the sites that really was supposed to be up was updated.
JS Said,
November 5, 2007 @ 4:48 pm
Lol, Poor Dean, affected by the power of the Google search
طبيب Said,
February 5, 2008 @ 5:19 pm
Does Google crawl more often if the a website content keeps changing ?