Bigdaddy status update: almost there

We’re down to just 1-2 data centers left in the switchover to Bigdaddy. It’s possible that the Bigdaddy switchover will be complete in the next week or two. Just as a reminder, Bigdaddy is a software upgrade to Google’s infrastructure that provides the framework for a lot of improvements to core search quality in the coming months (smarter redirect handling, improved canonicalization, etc.). A team of dedicated people has worked very hard on this change; props to them for the code, sweat, and hours they’ve put into it.

90 Responses to Bigdaddy status update: almost there (Leave a comment)

  1. Hey, I finished my posts before midnight this time! ๐Ÿ™‚

  2. Good morning Matt

    Thanks for update and looking forwards for the new infrastructure.

    Just to be sure. Did you say “improved canonicalization” in the coming months?

    Thats for sure good news for Stephen and Co, ya know ๐Ÿ™‚

    Have a great day

  3. Congratulations! Go to bed.

  4. “smarter redirect handling, improved canonicalization”
    is there a way to get more technical details?

  5. Harith, I did. milius, let’s get Bigdaddy out first and then we’ll start to talk about it more in the next few months.

  6. Good morning every one.

    I find a strange result until last changes of Bigdaddy, if you try to find keywords linke “SEO” in some result appear in bold text the word “sexo”, translated is “sex”

    Try searching “consejos seo”. In these IP

    I think these example can efect to other words, I will try to make a list to be much specific.

  7. Yep, now the fun really starts (hopefully)

    Cant wait for improvements that Big Daddy supposed to introduce ๐Ÿ™‚ ๐Ÿ™‚

  8. Is Big Daddy live in UK and Ireland yet?

  9. “bei gott” you’re right, used the search and found all the stuff I wanted to know – and more ๐Ÿ™‚ in the future I’ll finish my coffee before posting here …

  10. โ€œsmarter redirect handling, improved canonicalizationโ€

    So, does this mean that the subdomain spam we see so much of in Google these days is over? ๐Ÿ™‚

    We actually just had the issue of subdomains and clustering up in this weeks Strikepoint at WebmasterRadio, with my co-host DaveN, and we couldn’t really figure out if what we see is a bug in Google or on purpose …

    Not so long ago Google would cluster results from the same domain so a max of two listings was presented but now we see entire SERPS filled up with just one domain.

    Just do a search for “jubii” (Lycos in Denmark) and you’ll see that 7 out of 10 results is from the domain. As a search user I don’t like that because I know there is much more websites out there that writes about Jubii. As a consultant for Jubii i love it ๐Ÿ™‚

    So, Matt, is this a bug or is it a new strategy? ๐Ÿ™‚

  11. Hi Matt,

    I look forward to see Big Daddy in action. I was wondering if you had contemplated in its features something to detect and remove a model of websites I’ve detected lately, that I think are doing a real harm in the SERPs. I give you one example: How does Google regard these kind of websites? Is Big Daddy going to take any measure towards them.


  12. Mikkel deMib Svendsen

    >>>>Just do a search for โ€œjubiiโ€ (Lycos in Denmark) and youโ€™ll see that 7 out of 10 results is from the domain.

    How about a search on Yahoo – looks like 90% of the first 100 results is with a sub-domain.

    Personally sub-domain clustering would be great IMO ๐Ÿ™‚

  13. Matt, once Big Daddy has been rolled out (pun intended :-)) may I suggest to create a list of search examples that help us understand the nature of the innovation in a fast and easy way? I don’t mean to put the work on your shoulders alone, everybody could contribute his or her findings. Let me also thank you for your efforts here, thus giving non-Googlers the opportunity to feel Google’s pulse a bit better.

  14. I can’t wait for BD to switch over. I jump from the 4th to the 3rd page then.

    The results are much nicer on BD too (in the results above me).

  15. OK, all whitewashed and ready to go. Flip the switch.

  16. Handle redirects better? Does that mean that the 302 PageJacking glitch is going away?? I have a client that has gotten indefinately hurt in with their G Index because of it…

  17. So does this mean we’re finally going to be seeing stable results?

  18. Why don’t we get more infos with respect to the “supplemental” glitch?

  19. Sorry… I haven’t read this post:

  20. When will the adult serps be back to normal?.
    Spam makes up the majority of adult serps right now.
    The dec 27th refresh got rid of allinanchor or something?.
    Does anyone at Google know that the adult serps are spam filled?.

  21. Matt-

    I have been waiting for an oppurtunity to post about the issue that I have been trying to deal with. Here it goes…

    Short version: I believe Google’s new way of picking the best looking url to display (i.e. sf giants) is causing Google to index my url without www but cache the www version.

    Here is why I believe this is happening.

    –Indexed by Google

    –Gets 301 redirect to

    –Google indexes

    –Google caches

    I know this because the non-www version of new-page.html is a blank page that returned a 404 status. The Google cache was not blank.

    I even looked at my server’s log file to verify, and I sent a portion to They replied with, ” we do not review individual sites”.

    I think it has to be Google’s new way of picking the best looking url to display (sf giants) because there were no links to the non-www version and all of my internal links are absolute urls with www.

    Sorry so long. Thanks for any help.

  22. Matt-

    What is going on at

    Is that one of the few without BigDaddy?

    The reason I ask is because this is one of the few dc’s that appear to not be affected by the above mentioned problem (The comment above this one).

  23. Pretty much every 216.239.59.* is not BigDaddy.

  24. Well as long as these “software updates” don’t throw us all in the trash and replace our spots with Google widgets I am good with it. I would like to see less of Google when I do a search in Google in the future, Is this going to be possible?

  25. Ben – that looks like a Non-Big Daddy DC (the only C-Class left ?)

    Matt – would it be reasonable to expect a PR update based on the new infastructure pretty soon ?

    If Big Daddy can do things Little Daddy cant then this seems logical to me.



  26. Matt,

    I have seen another search engine try something different to weed out the spam sites. The one thing that they had tried was to impliment a “sentence agrigator”, i.e it will read the sentences and check for nouns, verbs etc in the sentences. It only needs to read the first 2-3 lines. It does take alot of processing power but then again Google has money to impliment such things. After all its only hardware!

  27. Stephen, shows our site pretty well indexed. The other Big Daddy DC shows most of our old url’s marked in supplemental results and does not index our new site. Will Big Daddy help supplemental results?

  28. Matt: Great news, thanks for the update.

    Mikkel/Stephen (Re: Sub-Domain spam): I couldn’t agree more.

    Roxana: Love the index page of that domain… not much keyword spam on there!

    Ulysee: Personally I’m quite happy with the results right now, my (very non-spammy) adult sites are finally beginning to dominate in some areas… You’re right though, there is still a lot of ultra-spammy stuff in there (particularly sub-domain spam).

  29. BigDaddy has definitely helped our site. As a newly launched site (with a 2 year old 301 redirect) we went from a lowly 53 pages to over 350 pages indexed. Now if I could only put away my sand shovel….
    On another note, thanks Matt for the updates. Your info has helped alot of people.

    – Scott

  30. ClickyB, Matt:
    I’ll give you an example of a adult serp that’s 90% spam
    If you do a search for “p0rn0 clip” with the “s” or without 90% of the sites that show up for that term are blackhatted.

    Most of the adult serps are 50-90% spam/404. Allinanchor seems to be messed up with spammy redirects, 404’s and sites with no links ranking high in the adult serps.
    On March 8th for a few hours spam sites and 404’s were knocked out and good relevant sites were put in place, what happened to the good serps?.

    I hope that this big time spam problem is fixed soon.

  31. Thanks for the update Matt. It’s much appreciated

    I know you and your team have had to endure a lot of pressure in the last few weeks internally and from some of us. Best wishes to all for a successful completion of the transfer and fixes.

  32. SEW: after having watched your seemingly endless and perpetually inane rantings for the last few months, I think I’ve come up with not one, but two solutions for every one of your alleged problems.

    Solution 1: Get a job. Oh, but not just any job. Get a job in Mountain View, California, improving relevancy, providing better results to end users, and bringing Google up from the gutter you seem to think it’s in.

    I’m sure Matt would be more than happy to take your resume and forward it on to HR, which in this case would likely be the closest available paper shredder, garbage can, or for the environmentally conscious, recycle bin.

    But, since I suspect your answer to Solution 1 would be that they wouldn’t pay you what you’re worth (and I can see why!) I suggest Solution 2 as a backup.

    Solution 2: build your own damn engine. Seems to me someone with the intimate knowledge of relevancy, spam, and what users really want should be able to build an engine that will address all of the issues you seem to raise, become extremely popular, and make mad coin.

    Pick 1, or pick 2, or pick neither….whatever you decide, just shut up (or find another tack.)

  33. Thanks for the update Matt!

    Thanks to Big Daddy the results that I am getting are the best results ever for Google. Congratulations to Google, yourself and the engineers who have done an excellent job. Thanks also to you Matt for keeping us informed as no other search engine does what you do.

    The university where I work (over 75,000 students, faculty and staff) use Google many millions of times a month and are very much wanting good relevant search results and your update will definitely give then the results they need.

    Thanks again!


  34. SEW,

    One man’s junk is another man’s treasure. I happen to like the relevant search results which deliver my content to tons of happy Google users.

    Ever thought of using ASK? They have this cool talking monkey…

  35. RE: “So, does this mean that the subdomain spam we see so much of in Google these days is over?”

    Bit rich coming from a SE forum member (and Mod) who has been condoning their use for years. As with all blackhats, they are last to the party ๐Ÿ™‚

    Adam, RE: SEW and “Get a job”. That is all that is needed to be said IMO ๐Ÿ™‚

  36. Search Engines Web, I’m moderating your comments. Several people complained specifically about the quality of your comments at SES NYC.

  37. Hi Matt,

    I’ve got it and thanks for the update too. BUT… What is BigDaddy?


  38. Adam, RE: SEW and โ€œGet a jobโ€. That is all that is needed to be said IMO

    Yeah, but it’s not as venomous. Sometimes, ya gotta get a little rattlesnake bite out of your system, and this was a LONG time coming. ๐Ÿ™‚

    The fact that Matt’s moderating SEW’s comments (WOO HOO!) reinforces that.

    Big up for that, by the way. I’m just surprised you put up with it as long as you did, dude.

    Based on SEW’s comments and Matt’s reaction to them (very calm), I am now convinced that Google’s healthcare plan includes marijuana for what a client of mine calls “preventive medicinal purposes. An ounce of bud is worth at least 1000 pounds of bodies that would end up taking up hospital space caused by work-related stress caused by constant complaining.”

    So yeah, can I come work there sometime? ๐Ÿ™‚

  39. Ok, good to here.
    We will see hope it will be fix some Problems.

  40. Good morning Mikkel deMib Svendsen ๐Ÿ™‚

    I have a question about your site, Mikkel.

    When run a query on a BigDaddy DC “demib”,

    I see your own site homepage listed as #1 with this title:
    Svendsen , Mikkel deMib

    While the source of your homepage page shows
    Advanced Search Engine Optimization Services

    Svendsen , Mikkel deMib
    Maximize your share of search engine traffic from the major search engines – get help from today Mikkel deMib Svendsen. – 11k – Cached – Similar pages

    Then when run command, I get a result of your same site with homepage listed with title:
    Advanced Search Engine Optimization Services
    which is in fact matching the Advanced Search Engine Optimization Services of your homepage source.

    Advanced Search Engine Optimization Services
    Years of professional experience with search engines. He is a long-term moderator at Search Engine Forums where he established and managed the two forums: … – 11k – Cached – Similar pages

    Thoughts from you and possibly from Matt?

    Wish you both a great day.

  41. I have a site which has some rankings before bigDaddy update, but now only the home page was indexed, lost all traffic. The site only uses unique content and updated daily. Could you please tell me if it’s a known problem? Thanks!

  42. nearly over… time for a sigh of relief….

    one thing i don,t understand is why the dc,s are giving 4 different results for backlinks to my site. How long will this last for ?

  43. Dude Adam, you are like a happy puppy, LOL! ;-o

    Good call on the SEW moderation Matt, some of his linkbaits are gettin’ a little lame and most people see right through them. Offer something new to the conversation dude, don’t bore us to death!

    Blogs are self policing, I like that.

  44. It would appear to me Google have rolled back most datacenters. Most of my websites indexed pages have dropped massively, a new site that went live a few days ago appears to be no longer indexed and most of the cached pages showing are 6 months old!

  45. You have no idea, Aaron.

    The problem I have with people like SEW is that they yell just loud enough for just long enough that other people start believing them and picking up their ranting.

    It’s what I call The Big Yell…not necessarily The Big Lie, because sometimes there are kernels of truth (not in SEW’s case, but in some cases).

  46. I am curious to note the impact of this evolution of BigDaddy.

  47. I notice the security code to do this post was PRKMW so I guess I need to ask a question about PR : )
    Matt could you give us the word on the PR export. Has one happened? if so what date? is it in the works? is it being done per dc as bigdaddy is being down?
    Oh yes are you going to the TO SES?

  48. Let’s get back to Bigdaddy. Would it launch on April 1st? ๐Ÿ™‚

    [quote]I notice the security code to do this post was PRKMW so I guess I need to ask a question about PR : )[/quote]

    Matt, you can rewrite the security code as a fortune cookie? Use some good poems as the security codes. ๐Ÿ™‚ That’s would be fun.


  49. Um, now that Bigdaddy is almost done rolling out, I think you’re more likely to see a PR update soon. After the Boston pubcon, I don’t plan to hit any search engine conferences until San Jose.

  50. Matt,

    I have a question for you. I am the business owner of the domain name This site is #1 in Google for the keyword ‘mortgage’ and has a PageRank of 6. When I bring up (non-www) I get a PageRank of 4, same when I bring up All the same site but Google seems to be seeing it differently. Should I do a 301 redirect on the sub sites to the www version or is this what BigDaddy is doing….figuring this all out?

    Please let me know anything you may know of.


  51. Hi Matt

    You don’t need to reply to this one if you can’t for any reason.

    Why these two BigDaddy DCs show DMOZ titles of the sites when run command


  52. Matt,

    Based on the above I hope the homepage only effected sites are back in the index before the PR toolbar update otherwise they will suffer another 3 months with a 0 tb rank when they return.

  53. Oh no, now we have people obsessing over toolbar PR again ๐Ÿ™

    It’s funny how those that don’t bother with PR have more than those that obess over it ๐Ÿ™‚

  54. Totally off-topic post…well, partly.

    Does the BD update have anything at all to do with Google News RSS feeds or their displays? I’m working on a site right now for a client where the RSS feed shows up fine on my server, but not theirs.

    I doubt that BD has anything to do with it, but if nothing else, I figure throwing it up there may trigger someone’s thoughts as to what I can do to fix this. (Client’s not mad or anything, but it’s pissing me off to no end.)

    Thanks to anyone who might be able to shed some light.

    Although this does lead to another suggestion:

    Google should have a page full of general contact emailss for the various services, so that if users have questions, they can fire off support emails and get them resolved.


    Search Engine Support: (note: this probably wouldn’t be a good one to use due to every idiot writing in about why their site doesn’t rank, but it’s an example)

    Newsgroups Support:


    Google Pages (or whatever they’re called):

    And so on and so on.

  55. Seem to have spotted something very strange, I updated one of my clients home pages a few days ago. Yesterday the new the new home page displayed on and, today the page has reverted to the pre-update version. Is this something to do with Big Daddy?

  56. Hi Matt

    I have a question that could stop all the insanity.

    Why not tell everyone PR is just one part of a 100 part algorithim and to stop having canipition like fits over the damn number??

    Insane thoughts from the SEO deranged

  57. Matt

    Its time the truth be told:

    Search Engine Web and Matt Cutts are the same guy! Probably not, but maybe?

    I’ve often wondered why Matt allows Search Engine Web’s comments. Sometimes the comments could be taken as insulting (although obviously not the intent) and they many times promote spamming Google. But yet they are allowed? At first, I thought ‘at least his comments are entertaining’, since I’ve found myself skipping over other’s comments looking for S.E.W.. And that, maybe, just for the ‘entertainment value’ Matt was allowing his comments. But, could it be that Matt and S.E.W. are really the same person? How else better for a Google Search Engineer to beat spammers than to take on that persona. Could this be a Jekyll and Hyde experiment gone awry by the mad scientist?

    Either way, good job Matt – S.E.W. adds a little spice to your site.

  58. Slartibartfart

    Matt, just curious: how does google feel about 3 way linking schemes? Is this a punishable offense?

  59. I wonder if Google well ever be sued for creating a little green bar obsession ๐Ÿ™‚ Some SEOs would have a really strong case IMO.

  60. Yeah, a basket case.

  61. Now, now. It’s not their fault they are obsessed.

  62. Not others faults they dont understand the value of PR either and that due to Hijack/Canonical issues a PR update is very very very important for some sites.

    If (as has been said) Mozilla Googlebot is for Big Daddy and Normal Googlebot was for Little Daddy ๐Ÿ™‚ – then Big Daddy is going to have a different calculation of PR/BL (and other ranking factors).

    EG. A Hijacked page could have gone to PR0 in the old index – however, now Mozilla Googlebot (hopefully) can understand the destination and reallocate PR to the correct page.

    Fine – Dont obsess if you have moved from a PR5 to PR4 homepage – but as Mozilla Googlebot can technically do things that normal Googlebot can not then the reallocation of PR/BL (and the other ranking figures) could be a very important step in the Big Daddy process for lots of sites.



  63. Hello Matts,

    It seems, BigDaddy don’t solve the spamdexing problem.

    In fact for my website, on BigDaddy if I search , there is no result.
    But on :
    I find my website with good description.

    And some days ago, when I searched my domain, I find the link page of another website (at the place of my homepage). (This website use xoops CMS) Who use a metarefresh to redirect to my homepage.

    So it seems that if google/BigDaddy think that a link page is like my homepage, it delete both.


  64. Kevin-

    Your site where only the homepage comes up using the site: command. Check www version and non-www version (i.e. and See if you get different results.

  65. Well I hope with these updates, we can find more accurate SERPS.

    Just try this *it is mainly same SERPS on all other search engine , buti would like Google to offer always the best.

    Opposite of next…..result in Google and other search engines give always opposite of sex *the movie IMDB

    Better Big Daddy with cooler algo..

  66. Ben: It’s the same. Google liked the site prior to the BG update. But after the update, Googlebot crawls it once per week according to the cache date, but didn’t index any internal pages for over a month.

    The site is 4 months old. I have a speculation that BigDaddy update didn’t pick up the latest data (like one month) in the old data centers, also somehow ignored it in the new data center. I think it could be a common problem for all sites will similar age.

  67. Cani use 301 redirect to move site to a new domain without loosing PR and positions in SERPs?

    Many webmaster using 301 redirects lose their positions…


  68. http:// – This looks like only new data spidered in the last few weeks.

    Not an index built on old data with new stuff added in, but purely stuff that has been recently spidered.

    What say you?


  69. Does bigdaddy have anything to do with switching from lynx based (text only reader) to mozilla based (reads anything it darn well pleases)?

    I’ve heard a small mention of the lynx to mozilla switch and don’t know what it really is about.

    (curses the security code is case sensitive!)

  70. “smarter redirect handling, improved canonicalization, etc.”
    I’m interested in the “etc” ๐Ÿ™‚ any more details? In fact, why not invite one of the BigDaddy engineers to write a guest post here about it?

    “After the Boston pubcon, I donโ€™t plan to hit any search engine conferences until San Jose.”
    It would be nice if you/some other Google Engineers could attend SE conferences in the UK sometimes too ๐Ÿ™‚

  71. This one is not yet align :

  72. Roxana wrote ..

    “Hi Matt,

    I look forward to see Big Daddy in action. I was wondering if you had contemplated in its features something to detect and remove a model of websites Iโ€™ve detected lately, that I think are doing a real harm in the SERPs. I give you one example: How does Google regard these kind of websites? Is Big Daddy going to take any measure towards them.


    I did not see an answer to this and I was wondering the same thing as well. Though in the Big Daddy center the number of these that show up in the top pages of the search results has greatly diminished from before, they are still very prevelant and I believe that are using sponsored links.. also they often seem to be the same company with a different url which is spamming yes??

    Thanks for all your help Matt – I so appreciate your site for making clear what for me is often mud ๐Ÿ™‚

  73. What’s with the addiction with so many on the different Google data centers? Nothing helpful can be learnt from them just as nothing helpful can be learnt via Toolbar PR.

    All I can see happening is Webmasters chasing their tails.

  74. It gives geeks yet another set of meaningless objective data to paralyze by analysis, Dave.

  75. That along the lines of what I thought. ๐Ÿ™‚

    What’s the old sayings?

    If you can’t dazzle them with science, baffle them with bull %^$# ๐Ÿ™‚

    Never let the facts get in the way of a good story ๐Ÿ™‚

  76. Dave, no one baffles anyone with bull%&#* anymore! We all have scalable, robust infrastructures that are adjustable to effect a solution for the ever-changing paradigms of a global marketplace and allow us all to become more customer-centric.

    Now THAT’s bull%&#*.

  77. Yep, seems to me that many SEO collect tonnes of useless stats, info and numbers to justify they extortionist fees.

  78. It appears that results on and are dramaticlaly different. Yesterday results from those 2 DCs propagated to about 12 DCs but later reverted back to just these 2. What are those results?


    stressing out way to much about these DC’s what are they???????? They show results much differnt then the rest.

  80. Big Daddy is probably the worst – I thought between march – apr 8 the results were good for the search term. But now meaningless sites come up. The numbers of links were incorrectly shown. The alogorithm is getting worse by the day. If google really doesnt correct itself it will probably get buried in the hole it is digging for itself.

  81. How to find out if the website is listed as supplemental? When I type my domain name in google search and select ‘containing the term’ – the website is not listed in the results at least in the first 5 to even 7 pages. Any input will help. Thx

  82. Is this BigDaddy switchover the reason why a lot of websites are having the problem of most of their pages being dropped out of the search results? I think this problem should be addressed. Everyone is left wondering what they should do to their sites to get themselves back into the search results.

  83. Hi, got a big problem my site selling 1000’s products online with 1000’s of indexed pages has recently rebranded from a different website ltd unfortunately we have dropped out of the first rankings and second page ranking and third… I know this has something to do with BD rollout as the sudden dissapearance coincided perfectly. i also know that biglink redirects to buildsaver and where product pages dont exist the buildsaver redirects again to a default.asp page. Google search probably thinks we are up to something. Do you think its wise to deindex all biglink pages? how could i do that

  84. My question had nothing to do with “PageRank” so to speak, more to the fact that google saw my website as two different site (non-www and www version) based on the page rank results.

  85. Hi Matt,
    Have you finished the BigDaddy project? You see, since later April, most of the pages of my site drop from Google, in the past, we have more than 20 keywords can list in the top of the result. I don’t know how long it would last, as more than twenty thousand customers are finding us every day on the web. I think it’s a huge loss for my company, not so convenience for the cusotmer either. help us please, my site is

  86. We have to assume that Big Daddy is all completed? Do you know when the next update approximately ?

  87. Seems like the total number of search results are less now compared to it was before Big Daddy ? Looks like G drops a lot of spam sites…

  88. Seems like even though there is no algo update, the DC’s are not showing the same results.

  89. I wonder if Google’s employes manually check some of the Serp listings for quality of websites… I mean i have a really quality website targeting Debt Consolidation but it will not rank for the big terms… I think Google’s Sandbox comes into play

    For a site to be successful, it has to have quality content, professional design and strong inbound links built over a 2-3 year term… that’s when you will actually rank for some big keywords… What do you guys think?

  90. Since I suspect your answer to Solution 1 would be that they wouldnโ€™t pay you what youโ€™re worth (and I can see why!) I suggest Solution 2 as a backup.