Reminder: check your sites

I’m catching up on my RSS and my email. There’s the typical run-of-the-mill emails, like the emails to Googlebot:

Good day Googlebot!
We are an international escrow company.
Now we are looking for a new partners.

You can earn some money – do not lose this opportunity!

It is easy and completely free for you.

Please contact us for more details: XXXXXXXXXXX@XXX.com

I also saw a different email that a group sent to Google. The email is a little prickly. They said things like

our-domain.com is still listed on other search engines and we find it
very difficult to understand why it is no longer listed on Google. We
implore you to reconsider your censorship of our domain name, so that the
spirit of freedom and the hopes of many are allowed to flourish.

Just as a reminder, if your website isn’t showing on Google, you should check your own domain. After reading the email from this group, I checked out their domain on Yahoo! for a couple of minutes. I tried doing a site: query on Yahoo! and saw doorway pages on the first results page. With almost no refinement, I found pages like this:

Bad Site Search

Hmm. Looks like multiple doorway pages on the site (not even in just one subdirectory). So I go to check out one of these pages and it looks like this:

Madonna Star Sex!

Hee hee. It says “Copyright 2004 madonna star sex porno Company. All Rights Reserved.” Heh. All the pages have this doorway template.

Google’s webmaster quality guidelines are pretty clear that autogenerated doorway pages are not welcome in our index. So this is a case where checking out your domain would be just a productive as contacting Google. The best advice I could give would be to remove all the doorway pages and then submit a reinclusion request.

156 Responses to Reminder: check your sites (Leave a comment)

  1. Matt,

    Glad to see you back in action, I believe you were refered to as the “Spam Cowboy” in irvine. Keep up the good work!

  2. Matt, I just wanted to say Welcome back from one Matt to another.

  3. Back to work, eh, Matt. Speaking of spammy pages, what are your views on the 5 billion pages index by Google, http://forums.digitalpoint.com/showthread.php?t=97090. I think a lot of people would like to hear your opinion…

  4. Hi Matt,

    Welcome back !!

    Nice to see you posting again..Hopefully you can help us..

    Our site http://www.printe-z.com has no spamming issues and is following all Google guidelines, however our site is only partially indexed on Google with lots of supplemental results to it.

    I know we are not the only site suffering, many many other sites are in the same boat.

    You have not posted much about this. I would appreciate for you to post more about this problem, and what to expect from Google regarding this update.

    Kindly look into our site, or have someone else look into our site and please try to include our entire site in your index. Thier is really no reason for you to omit most of our pages.

    Thank you..

  5. Hi Matt,

    I bet you do get a laugh when you see sites like this! It just amazes me how people think they can get away with this spam. Dont they know that Cutts is on the job!

  6. Welcome back Matt. We missed you. I hope you had an excellent vacation in the real world and far away from the world of search.

  7. Welcome back, good timing, I was just venting this morning about how Google is slower than Yahoo or MSN to index my site:

    http://www.anotherblogger.com/2006/07/05/hello-google-anyone-home/

  8. Matt, you quite often mention the obvious ones like doorway pages and hidden text but what else can you suggest to look for if you don’t have and never had those?

  9. Is Google ever going to fix the mutiple domain issues?

    http://www.google.com/search?hl=en&lr=&q=site%3Auup99x.org%2F

  10. Right, that’s an easy one.

    I’d also like to know what happened to the index on June, 28th. Our indexed pages were cut in half in 1 day after we recovered so well from the BD deindexing disaster earlier this year.

  11. Which I could cloak work activities as a means to surf pron. :p

    Funny stuff Matt, thanks for the advice as allways 🙂

  12. Hello Matt,

    Being frank, I expected a specific mention about what happened the 27th of June, but having said this, any advice from you is always welcome.

    Talking about your post, I’m not sure what Google thinks is a doorway page, in fact I always thought that the page in the root of your domain was a doorway to your blog, but you are not being penalized in any way by Google…

    I would like to read your definition for ‘doorway page’, so I can remove any of them from my site. Maybe what I think is a ‘guide to my site’ page is considered spam.

  13. Matt,

    Clearly the right thing to do is to put that international escrow company in contact with the Madonna porn video doorway page generating company 🙂

    Welcome back!

    Michael

  14. Baaah,

    The Google Webmaster Guidelines also state:

    – Avoid hidden text or hidden links.
    – Don’t load pages with irrelevant words.

    But although I have reported, via my sitemaps interface, the top two results for the search phrase “Bahamas Diving” every week for the last 3 or 4 months, Google seems to think repeating a completely hidden phrase over 70 times on the homepage alone is perfectly fine! Or simply doesn’t check out the reports they specifically make a point of asking for in the sitemaps interface.

    I guess I shall continue my weekly reporting until someone looks at them.

    Welcome back, BTW, sorry to be a miserable old git for your return, its just that these two sites are annoyingly spammy, and I can’t seem to get G to have a look…..

    Ben

  15. Sounds like that company wasn’t using very smart logic on their webpages? (too subtle?)

    Also, you get stuck with the googlebot emails? That would suck, no wonder you never get time to blog as much. You should bounce those to Adam. I always thought you’d have had a single letter @google.com email address. Are those still a prestige symbol?

  16. Bantar.. You asked: ‘What is a ‘doorway page’?’ That’s like asking a surgeon ‘what is a disease?’. Not the best ‘Matt, what is…?’ question. ‘http://en.wikipedia.org/wiki/Doorway_page’ should help you. But I doubt your being serious.

  17. Dear Mr. Googlebot,

    I kindly saw you visiting my site today and I wanted to see how your experience was. I am very interested in making your experience on my site the best as it could possibly be, so please take a moment to fill out this survey.

    =======================

    oh man…the emails you get much be so much funnier and better than what I get 😛 I’ll be up in Santa Cruz all of next week, too bad I won’t have a car — otherwise I’d drive up to Mountain View to meet up with you and Bharat hehe. hope everything is going well back at the office now 🙂

  18. Matt, did you go to the root URL of the domain creating the spam in question?

    I think the greater crime against humanity is their “legitimate” content. Especially with that crappy classical Vivaldi music and the moving things. I wanted to play Duck Hunt with my monitor and blast the stupid “Make Money” cursor.

    Nevertheless, quite amusing when the clueless attempt to inform the ones with a clue, as always.

  19. I too would like to hear about what happened on June 27. There appear to be many legitimate sites which took a huge hit on SERPS and look to be “white hat”. A common symptom seems to be that the homepage no longer appears with the results of “site:www.example.com”.

  20. And another thing that I just noticed (sorry for two posts):

    They’re running Adsense on their “old blog” (click News Weblog, then Old Weblog, then scroll to the middle of the page).

    When, if ever, is there going to be a tie-in between sites that are penalized/banned from Google and those penalized/banned from Adsense?

  21. I agree, its going to be had to ban all the spam when the AdSense is one of the prime motivators in the creation and implementation of these pages.

    What if google give users the option to only view pages without Adsense? Hehe, not going to happen, eh?

  22. Matt,

    First off, this is the first time I’m commenting. Been reading your blog for months now, and eagerly awaiting your return from vacation.

    However, as I can see your point, and validate that indeed the site in question is definetly spammed doorway pages, does this mean that Google is working on fighting spam like this subdomain spamming? For instance,

    http://www.google.com/search?hl=en&lr=&q=site%3Auup99x.org%2F

    And how does Google respond to sites that are specifically made to serve up advertising, such as,

    http://www.google.com/search?num=100&hl=en&lr=&q=site:s168897611.onlinehome.us

    Doesn’t these fall into doorway pages? It’s hard to understand because alot of these pages are supplanting pages that webmasters have been working years on just getting to the first page, even 10th position,offering real content sweated over by and revised over hours by someone who has “done no evil”

    Just trying to learn here Matt. I’m trying to follow the guidlines as best as possible, but it really hurts to see these type of results, especially when we all work very hard at SEO.

    Welcome back. Hope your not PO’d at this comment.

  23. Thanks, Matt. As I’m looking at those screenshots just now, my boss walks up behind me. I’m going to go try and save my job now, I’ll finish reading the article later… AT HOME!

  24. Aloha, Matt ~

    Yes, a detailed explanation of what happened on June 27th, 2006 would be much appreciated. Our three long-on-line, Google webmaster guidelines compliant sites have suddenly lost 95% of their Google search referral traffic. Many other webmasters of legitimate sites have experienced same. Our three sites all went down at the same time, but are not cross-linked in any way. (Two of our other similar sites were not effected at all.) The sites that died in Google have been on-line since 1998, 1998, and 2001, respectively, and have been stable in past Google storms.

    site: command SERP behaviour:

    1.) Index/default/home page is not in the list (in nearly all DCs);
    2.) Text below Title is exact Meta Name Description tag content;
    3.) Sites do not appear to be listed in any order of importance.

    Thx in advance for any light you can shed on the subject.

    Best,
    Kirk out.

  25. lol, I would like to know how to do doorway pages, only for seo experiments 😛

  26. Hey Matt, it’s great having you back…posts like this make me happy.

    Small, tiddly problem – if I do a site: search for the /support/ directory on my site, I get only 2 results. But site: search on the domain shows that there are lots lots more support forum posts cached. Are there still problems with the site: command?

  27. The smain site is in fact a fairly quality site, and non-porn related –

    the concern is should the ENTIRE Domain be Banned – as opposed to those Doorway pages?

    The information from other areas of the site, could be potentially useful to searchers seeking related information

  28. We had to change our name from pixoh.com to snipshot.com to avoid some potential trademark trouble. A couple days after we set up the 301 redirect, pixoh.com *and* snipshot.com seem to have been dropped from the index. pixoh.com had a respectable PR, and snipshot.com has links pointing to it. The webmaster tools don’t have any warnings, just a reminder to follow the webmaster guidelines.

    So what did we do wrong?

  29. “Hee hee. It says “Copyright 2004 madonna star sex porno Company. All Rights Reserved.” Heh”

    Looks like a pretty easy fingerprint to spot and filter!

  30. You have to love the porn spam, they are usually the biggest offenders.

  31. If nothing else http://www.slk-action-group.com deserves the ban for having background music on their front page 🙂

    BTW, interesting that you need to use Yahoo to search for spam… I’d figure there’s some easy way for you to get all that banned-or-not-banned data from some Google tool.

  32. C’mon Matt, return that black marker to the spy agency you used to work for. I mean, why bother when what you didn’t cover up makes it so easy to find the actual site 🙂

  33. “Danny Sullivan Said,July 5, 2006 @ 3:40 pm , C’mon Matt, return that black marker to the spy agency you used to work for. I mean, why bother when what you didn’t cover up makes it so easy to find the actual site”

    I just held my monitor up to the window and could see right through the black mark. Learned that one on CSI Sometown.

  34. Very interesting, that site is obviously a spammer/scraper site. However sometimes google does get it wrong. One of my websites – http://www.brangelina.us – was banned in google. (actually, as I write this I see that the front page was back in – no other pages are being indexed however, so maybe just a big daddy hiccup?) I’m guessing that the reason that Google didn’t like the site is because all of the articles come from other new sources, I could be wrong.

  35. Hey Matt,

    Welcome back.

    If a website isn’t showing on Google – we need to check the site to figure out why. Good. But, what should we do if a site is only showing where it serves Google needs – and banned when it doesn’t? Isn’t this a case where Google should be doing the cheking?

    —————–
    Example:

    Search for: % of African American in Los Angeles. At the top of the page is a Google Q&A answer – 11% – based on data retrieved from this page.

    But the site itself has been removed from the main index. No pages in site command. Not allinurl command. Not in info command. Not in domain search.

    Crawling been done pretty recently. Here is the June 2006 cache.

    Thus – here is an example of a site that has been deemed good enough to serve as an Answer reference for the Google Q&A feature – yet has been banned from the main index.

    Sneaky.

  36. If any clients come to me from another SEO I always check their sites over for rubbish like this. Some so-called SEOs insert doorway pages into their clients sites without their clients knowledge or permission, with the inevitable results. I have found so many, it’s not funny any more.

  37. Dave (Original)

    I love it when spammers are caught out. It makes my day!

    RE: “Matt, you quite often mention the obvious ones like doorway pages and hidden text but what else can you suggest to look for if you don’t have and never had those?”

    Valentine, the best place to start is the Google guidelines. It’s all spelt out there in black & white.

  38. I hate spam as everyone but i do enjoy seeing the ideas these guys sometimes use. Some are very creative to do their “business”, though at the end, they suck.

  39. Danny, I felt no need to call out the site. But red-widgets.com without a picture doesn’t convey the full connotations of most situations. 🙂

    Philipp, I wanted to demonstrate how if you’re not doing well in one search engine, there are many tools you can use to investigate your own domain.

    S.E.W., notice that the doorways weren’t just in one subdirectory. At the point where someone is adding doorways in different places around a site, that’s worse (in my opinion).

  40. Dave (Original)

    RE: “…. there are many tools you can use to investigate your own domain.”

    Nah, that would assume the problem is in their own backyard. Much easier to come here and say “Googles broken” 🙂

  41. I only have one thing to say on this topic….

    BWAHAHAHAHAHAHAHA… ROTFLMAO

  42. Philipp, I wanted to demonstrate how if you’re not doing well in one search engine, there are many tools you can use to investigate your own domain.

    Now Matt, with all due respect, this is precisely the kind of comment that can get you into a lot of trouble.

    If you tell people that there are tools they can use to investigate their own domains, they’re going to go onto every webmaster forum and try to use the tools who claim to be experts to analyze their domains and come up with 1,000,000 shortcut answers that never work.

    So if you see a drastic increase in spam in the next 48-72 hours, it’s all your fault. You didn’t specify which tools. 😉

  43. Sorry Matt, but the “It’s all your fault” line just doesn’t ring true any more. We’ve been waiting for improvement in the day to day results for over a year now and still when I search for my local greengrocer, I get someone in holland or california offering everything from CD’s to Hairpieces.

    This isn’t our fault, Yahoo & MSN are doing a better job nowadays … sad, but true.

  44. Toby, yes I already know what a doorway page is, but I was not joking. The definition of doorway page in Google guidelines is so vague that I would like know what the anti-spam boss considers a doorway page.
    Does it need to have a redirect using a meta refresh? or does Google consider other things?

    BTW. If I’d like to know what a disease is, I think I’ll ask a surgeon, not a computer analyst, for example.

  45. Dearest Matty, Here’s the thing, Check it out..
    Where, oh where, should we begin telling you about how quixotic google manipulating drug related .info sites are? How about here: Google spamming Drug related .info sites are trying to force us to become clinically depressed just to prove they can. Although my approach may appear a bit pedantic, by setting some generative point of view against a structural-taxonomical point of view or vice versa, I intend to argue that if we are going to talk about higher search quality standards, then we need to live by those higher standards. The first thing we need to do is to get the supporters of drug related .info sites to admit that they have a problem. Said supporters should be counseled to recite the following:

    I proponent of, drug related .info sites, am a drugged-out, sappy galoot.
    I have been a participant in a giant scheme to drive google searchers into a state of apoplexy.
    I hereby admit my addiction to fanaticism. I ask for the strength and wisdom to fight this spam addiction.

    You know what? these .info sites want to replace intellectual integrity with grotty sloganeering. Personally, I don’t want that. Personally, I prefer freedom. If you also prefer freedom, then you should be working with all to justify condemnation, constructive criticism, and ridicule of these sites.. The only thing that has any meaning to these drug and other .info sites is anti-intellectualism. Why? I’ll tell you the answer in a moment. But first, let me just say that if we don’t soon tell these spammers to stop what they are doing, they will proceed with such short-sighted, google-hate-filled tactics, considerably emboldened by the lack of resistance. You will have tacitly given such drug related .info sites permission to do so. These related .info site moron’s have put a proud search engine on a path which, if left unchanged, may well cause it to follow the Roman Empire into historical oblivion. And that’s why I say to you: Have courage. Be honest. And reverse the devolutionary course such .info sites have set for You. That’s the patriotic thing to do, and that’s the right thing to do.

    For shame! Unfortunately, I can already see the response to this comment. Someone, possibly google or even one of the said spammers drones, will write an unforgiving piece about how brutal I am. If that’s the case, then so be it. What I just wrote sorely needed to be written.

    Until Later
    Erik

  46. Dave (Original)

    Bontar, it’s quite simple IMO. If the page is for SEs and only exists to try and improve rankings, it’s likely spam.

    Simply build good informative content pages for *humans* and you will NEVER need to worry.

    Don’t listen to the blackhats out there, they constantly try to blur the lines and add confusion about what is sapm and what’s not.

  47. Thanks for the advice Dave, I do try to build good content for humans, not for search engines, but the problem is that Google doesn’t think so, well, at least since 27th June… I’m being penalized and I don’t know why.

  48. Dave,

    Thanks for that explanation – but what exactly is ‘only to try and improve rankings’? Can you give a simple example? I have been doing web content management for years, but has never been completely sure what it means. If I have a badly designed sitemap page (which incidently also distributes PR) will it be a doorway page? Or is it something only for googlebot, and a visitor to the page is meta refreshed itno some other page – or are there many many more such possibilities?

  49. Dave (Original)

    Why do you say you are being “penalized” by Google?

  50. Dave, I think my site is being penalized by Google because my site suddenly dropped 200 positions in the serps for many searches, because when I search for “www.mysite.com/myfolder” my site appears at position number 10 after 9 scrappers.

    Yes, It could be that all of a sudden 200 sites became more relevant than mine, but, 9 scrapper sites more relevant than my site, when searching for myself???

  51. Matt, I have to ask you, if you could actually do an even small review on what webmasters (that have the recent indexed pages problem) should do. I mean I wrote some of the steps I could think about (and that helped me, and others), but it would be a much better thing for you to give some advices.

    I’m not saying you should admit that Google has indexing problems and flaws (at least recently), but instead try to help people that are facing those problems.

    Thanks man, and glad you’re back. I hope you had a real nice, quiet and peacefull vacation, because now you’ll be nuked again with all the pressure.

  52. Matt, 4th result from your screenshot cannot be found, may be it’s not doorway pages, but just some simple html pages

  53. Oh, snap! 🙂 Matt, we’re glad we could help you with your research. But surely we don’t want to get into posting example queries where one engine is heavily spammed and another isn’t…. I mean, do you _really_ want to go there? 🙂

    By the way, let me just say that I, for one, don’t believe what people have said about Google’s spam problems getting worse recently just because you’ve been on vacation. Even if your spam-fighting effort is more manual than automated, I very much doubt that losing just one person for a while could have a noticeable effect(?)…

  54. I find it more disturbing that someone actually can get traffic from any search engine for “madonna sex film star” Who are these people searching for that string of terms?

  55. It’s good to see examples where Google is attempting to combat SPAM, and this is a very good example. I read in a webmaster forum someone complaining because he had been dumped by Google, and didn’t know why – I did a bit of digging, and found 2,000 doorway pages – um and he didn’t know why they were dumped.

    Good to see your blog getting active again.

  56. heh… example from your post is so innocent… this guy have been caught, you show us an example of a site banned months ago. you know what? we dont care.

    we care about those shitty number-dot-six-random-characters-dot-info sites. because they steal our content and then mercilessly crawl up the SERPs.

    All hosted on theplanet, biggest spam isp in google’s short history. They scrape sites – i have it in my logs – and two weeks later… ooo… their spamsites rank higher than my sites – even though they just copied and manipulated my content!

  57. Dave (Original)

    RE: “Thanks for that explanation – but what exactly is ‘only to try and improve rankings’? Can you give a simple example?”

    A page that is intented for SE only and is of no use to site visitors, or never see by them.

    RE: “I have been doing web content management for years, but has never been completely sure what it means. If I have a badly designed sitemap page (which incidently also distributes PR) will it be a doorway page?”

    So long as it’s visible and accesible to your site vistors and they are no redirects taking place, no. A site map will NOT distrubute PR very well at all. This would be done by sensible linking from various internal (and Home) page(s) to other internal pages.

  58. Dave (Original)

    RE:”Dave, I think my site is being penalized by Google because my site suddenly dropped 200 positions in the serps for many searches, because when I search for “www.mysite.com/myfolder” my site appears at position number 10 after 9 scrappers.

    Yes, It could be that all of a sudden 200 sites became more relevant than mine, but, 9 scrapper sites more relevant than my site, when searching for myself???”
    =========================================

    If you have read & understood the guidelines are your site is NOT outside them in anyway, then you are very unlikely to penalized.

    You seem to be focused on 1 single page though? A site should aim to obtain vistors from as many pages as possible. Then, these molehills(page dropping in ranking) do not become Mountains.

  59. “Is Google ever going to fix the mutiple domain issues?

    http://www.google.com/search?hl=en&lr=&q=site%3Auup99x.org%2F

    This one has gone from 125k to 1.23 million in 2 days. And they keep on growing…

  60. ilovedomains,

    They are up to 1.8 million on some datacenters.

    http://oy-oy.eu/google/pages/?url=http%3a%2f%2fuup99x.org

    I think someone should graph this.

  61. Dave (original), I don’t focus on a single page, it is the whole domain that has dropped in the serps.

    My site has + than 3000 pages, some of them are articles, other are product reviews, also I have two small directories of specific niches, user created image galleries, forums… As you can see I have put a lot of effort on not depending on a small set of searches. When one of the mini-sites of the domain become interesting (number of visits, good revenue, or whatever other reason), I purchase a new domain and create a spin-off site about this topic with more features than the previous mini-site.

    Working that way, I (we) have grown up to 12 domains right now, and only one has been hit hard by 27J. Sadly, the domain that I *think* is being penalized is the 50% of my revenue, so I’ve lost that 50% since 27th June.

    Not only I’ve lost the 50% of my monthly payment, I’ve lost the domain that helped me finding good niches to work on.

    There are two possibilities about what happened the 27th June:

    – Google is broken. We don’t know, and I can’t rely on this.

    – Google implemented some kind of filter that affects my domain badly. If this is the reason, I have to find out what kind of filter is, and change my site to meet what Google wants.

    I know this is not the place to discuss this, Matt’s silence about this issue seems to be a very pollute way of telling us “go with it somewhere else”, and I understand him. At the end, it is Matt’s personal blog, not the ‘Western Wall’ of webmasters.

  62. greg from france

    I’m in france and a lot of webmasters have the 27 june dropped pages syndrom…

    When i analyse the cached pages for my own site (who is dropped too) i find a lot of olds pages cached by google (2004-2005), before the 27 june these pages were not appear ; and now they appear on result of search before my recent content (2006). my recent content is then take as duplicate content by google.

    I think there’s a mixing of current and old index !
    on this DC i can approximatively find correct search result and the cached pages are very recent : http://64.233.189.104

    What about you ?

  63. Great example Matt – I haven’t seen one that blatant for ages.

    Well – not one that wasn’t running Adsense….

    🙂

  64. Matt interesting post on what no to do, I believe in the Google Guidelines for webmastering and really enjoy the new sitemap feature Google has offered to webmasters.My question is a simple one, why is it that money making keywords like casinos , Paris Hilton sex tape, and sex toys are still tookk over by spammers and that Google has done nothing to take the spammers out of the top 10? First General rule I use is that if a site is at the top for a keyword not in the domain itself more then likely it is a result of a Google Bomb, second you harped on a post about not having more links than everyone else in your industry well the top 3 for sex toys have nearly 10 times the amount of links as everyone else in the top 20, now I am sure there are millions of people searching the Madonna nude but really why not focus on the top 100 money making words and see who is at the top?Sure you might say that hte number one guy has a great looking website nad covers the topic well but it is just a result of you making them so much money they are able to afford the better programmers and newest software and hardware.My reccommendation is to make flip pages so that when a keyword is searched for such as casinos every couple of minutes the second page results appear on the first page then on the next flip the thrid page then on the next back to the first one.

  65. Wow, I am impressed with the discussion here. I can see how google would not want to index these types of doorway pages… I do have to agree with others that the guidelines are quite vague in some respects, trying to figure out what doorway or many other terms discussed on the guidelines could leave a lot of grayspace between different people’s opinions.

    Chris_D – good point – that is funny.

    Bontar – I feel sorry for you. Ihave heard other people say that sites they created even non commercial were heaviliy affected by new algorythm changes that were meant to fight spam.

    Matt C –
    RE: “…. there are many tools you can use to investigate your own domain.”

    Would it be possible one day to get some of these tools without having to install the sitemaps thing? I am not hosting my site / blog in a way that is easy to install this, and really don’t have the patience to learn even more techy stuff about google scripts and stuff. Couldn’t this become a web app or something? That would certainly help you all get more new web people to get things right and avoid anything that causes issues with google, apparently with the hopes of making a better internet right?

    Ps your captcha thing resets pretty fast for a long read blog. had to hit back button and reload the page.

  66. Matt,

    I don’t want to be a pain in the arse, but when are you going to do a search for:

    “our top * resource” “top * resources”

    .. and delete all of the results? Those pages ( like this one http://www.finnlo.nl/finnlozoekmachine/finnlo299.html ) really piss me off.

    Cheers,
    Tomas

  67. Bontar Said :”At the end, it is Matt’s personal blog, not the ‘Western Wall’ of webmasters.”

    Very true… Yet unfortunately it seems to be the only way to communicate with Google. If you do something horrendously bad and against their guidelines they will at least take you out of the index and put a nice note into your sitemaps account notifying you that you did something really black hat and got caught… The provide you a way to clean up the pages and file for reinclusion.

    Great… these black hat people have yet another advantage over people trying to do no evil since Google provides a means for them to communicate. Yet those of us who are purely white hat yet get dropped in the SERPs overnight by more than 200 positions (from top 5 to 300 ore more) are left with no mechanism to communicate with Google and resort to venting our frustrations on Matt’s personal blog.

    Instead we have a generic “feedback” form that results in an email telling us to read the FAQ that says “As you may know, our search results change regularly as we update our index.”. According to that answer, over one night Google found more than 200 new pages that it thought were more relevant than each of the 700 pages in my site that it had throught were wonderful for many years.

    The sad part of this is that there are more and more people who make a living from the internet. This huge change on June 27 also destroyed the site of my biggest client. Their original site had been designed using almost no text and instead was 95% graphics. They wanted to at least show up in Google for their company name so they hired me. I rebuilt the site with text, put an h1 tag on each name with the product name, and changed the page title to something appropriate for the product name.

    Things had been good on that site for the last few weeks as Googlebot finally had some content (text) to crawl. However on June 27 that site also was destroyed in the SERPs. The client blames me and told me they will be hiring someone else for future work because of what they lost on June 27.

    I have no means other than Matt’s personal blog here to communicate with Google. However if I had created some hidden text on the page and been dropped by Google I could have at least filed the reinclusion request form and had a means to communicate with Google.

    I think I’m done venting now…

  68. And what about the results for searching “e-aanbiedingen” ?

    Absolutely normal? Then try “forum e-aanbiedingen”.

    Your SE flips like hell, no need to answer me. Visitors and time will tel if Google has problems or not.(i see pages/ sites disappearing and come back after time(without changes and then it’s difficult ta say that the reason is this or that. If they disappeared why do they come back?). The crap sites which didn’t rank in 2005 all rank much better now)

  69. Bontar

    I understand your concern. We have been experiencing many problems for the last 17 months. Yes a significant revenue drop in the 6 figures. The revenue is sorely missed in light of a recent accident and brain surgery my son just had. (Hell the helicopter flight alone ran about $17,700 the surgery will be much higher than that.)

    The only advice I can give is to be patient and don’t expect enough information to pass your way from anyone from Google (reinclusions/webmaster queries/Cutts/Adam) or from other webmasters that will give you enough information to know who is at fault and how to correct it. I don’t think this is the place to discuss it but Matt usually throws a discussion or two out there that may be a better place to post your experiences, complaints, praises, etc. Lots of luck to ya! If you find some send it our way too!

  70. Amit Patel, idcide.com looks interesting. I can’t speak to why that domain is highlighted in a “one box,” but the part that I noticed was that back in November 2004, this was a completely different company:
    http://web.archive.org/web/20041109094751/http://www.idcide.com/
    Looks like they were an enterprise privacy corporation. Now someone else is using that same domain, and has ~40,000 hotel pages in Yahoo.

    Tim, it was a mark of respect that I went to Yahoo! first to check it out–I didn’t at all mean to say “oh look, Yahoo! has spam.” I wanted to say “Before you send us an angry email because you aren’t ranked where you want to be, check out how you look in different search engines and see if you can think of a reason why that might be so.” When you can’t find something in Google, other search engines are a great place to go check things out and see if you can figure out why. Sorry if it came across as “look, there’s spam in Yahoo!” because I didn’t mean to sound like that.

    Bontar, I believe any changes on the 27th were refreshing data used by an existing algorithm.

  71. “Bontar, I believe any changes on the 27th were refreshing data used by an existing algorithm.”

    Hmmm…it’s very worrying that you believe that. Those of us that we’re affected (see any of the forums) know that that is not what happened. Is it possible that you guys threw a switch that you didn’t even mean to throw? Anyone care to look into this? Thousands and thousands of legitimate sites were surgically penalised on the 27th of June. This cannot have been caused by a mere “refresh”.

  72. “Bontar, I believe any changes on the 27th were refreshing data used by an existing algorithm.”

    Bontar, see what I mean. Still not enough to go on to base a decision. Was the data bad or is there something wrong with your site. The world may never know.

    “Hmmm…it’s very worrying that you believe that. Those of us that we’re affected (see any of the forums) know that that is not what happened. Is it possible that you guys threw a switch that you didn’t even mean to throw? Anyone care to look into this? Thousands and thousands of legitimate sites were surgically penalised on the 27th of June. This cannot have been caused by a mere “refresh”.”

    You beat me to it. Yeah unless the data was incomplete, corrupt, or just pure junk, from what I have seen from other webmasters this just does not equate. But what do I know I don’t work there nor can I get any clue as to what is going on with our site.

  73. Well I have redirected most of my sites inner pages to the homepage to try to regain my position in the serps somehow.
    It still seems like page ranking factors were turned off or some algorithm change has caused many spam pages to rise up from their graves since big daddy took over.
    Can anyone at Google explain why for six months straight now I can search for many adult terms and find 500+ spam/redirects pages out of 1000?.
    Has Google given up on it’s “search”?.

  74. “You beat me to it. Yeah unless the data was incomplete, corrupt, or just pure junk, from what I have seen from other webmasters this just does not equate. But what do I know I don’t work there nor can I get any clue as to what is going on with our site.”

    You speak for a lot of people. To say webmasters are frustrated would be the understatement of the year.

    But now its not just the drop in SERPS for thousands of sites but its the million of spam domains that seem to be helping it out.

    http://www.google.com/search?hl=en&lr=&q=site%3Auup99x.org%2F

    1.8 million and counting. People getting deindexed, supplemental problems, dropping SERPS, and spam domains without much of an explanation doens’t sit well with many people.

  75. Ilovedomains,

    For uup99x.org yahoo doesn’t have one page listed, MSN has 80. Google has 1,630,000.

    So unless this is another “Bad Data Push” *cough*cough it looks like this guy has found a loophole in google that just doesnt work on the other engines. Considering it was registered on 6/18/06, http://whois.domaintools.com/uup99x.org I’d say he’s got it down. I’d be happy if my site came out of the supplimental index, I’m not even going to be greedy and ask them to index 1.6 million pages in 2 weeks, 10 a week would make me happy!

  76. JLH,

    It doens’t matter if they reindexed 100 per day, this types of site will just scrape you or the search engines and outrank you. So whats the point 🙂

  77. “Googlebot” making money with a free and easy tool? The next email could read something like:
    ——
    Dear google,
    We are seo company based in xxx and we can help you get top ranks in search engines like google, yahoo and many more…

  78. Hi,

    Arubicus, I really hope your son will recover soon from the accident, and of course I’ll share any information with you and the rest of webmasters if I find a solution.

    Matt, thanks for the note, although I don’t understand it ( maybe it is because my english is limited to what I’ve learned reading the MSDN ).

    So there is a data refreshed from an algorithm that exists… hmmm, I see….It will be our secret 😉

    Can this existing algorithm be refreshed again with the data from the 26th???

  79. AKA, that’s what happened. Arubicus, the change went through evaluation just like any other change and showed a positive improvement. This is not something that’s radically new; this set of algorithms has been live for a while. I’m pretty sure that there will be another refresh of that data in the next 2-3 weeks, so we keep iterating to improve things based on our evaluation and the feedback that we get.

  80. I wrote that before I read your most recent comment, Bontar. It will continue to be refreshed (just talked with someone about it few minutes ago), but not with data from the 26th. We keep iterating based on evaluation, the changes that we see, and the feedback that we get as well.

  81. Hi Matt,

    It’s like chatting realtime 🙂 Ok, I understand that the algorithm has been online before the 27th and that it will be iterating and changing serps until you tune it. So we can expect more flux in the following weeks.

    That’s what I wanted to know, sorry if I didn’t express myself well, and thanks a lot for make it clear to all of us that there is still hope to regain our positions.

  82. Purely academic question here:

    Where would you guys get feedback that didn’t come from webmasters? Like how much feedback do you get from pure end users who are satisfied (or dissatisfied) with the results?

  83. Matt — No worries, I’m just playing. I probably came off as more serious than I meant to.

    Agreed that having spam in the index is a very different thing from spam that ranks for real (not site:) queries…. 😉 Personally, I think that (to the extent possible) engines should always allow you to find particular docs or sites if you’re looking for them explicitly…

  84. No worries, Bontar. The Adam That Doesn’t Belong To Matt, we do get emails from users about lots o’ stuff, including quality. Webmaster feedback is important though, of course. I know AdWordsAdvisor sometimes get frustrated when someone reports something like conversion data not being up-to-date, because without specifics it’s hard to diagnose. I think AWA stickied a couple people who reported it, but then you have to wait for the sticky to be answered, and sometimes it isn’t, etc. etc.

  85. >>>I believe any changes on the 27th were refreshing data used by an existing algorithm.

    Translation: More SPAM is called an update.

    Can you fix the site:domain.com command before giving us more SPAM?

    For example, for one site, a message board, I get 121 results. If I search as

    site:domain.com “keywords keywords”

    I can get 228 results, most of them being from one huge thread that has a lot of pages (Not duplicate content).

    On the datacenters, it’s correct at 37,100 results. If they can fix the datacenters, how about geting them to fix google.com!!

  86. “Arubicus, I really hope your son will recover soon from the accident, and of course I’ll share any information with you and the rest of webmasters if I find a solution.”

    Yes my sone is doing fine (1 year and 10 months old). It has been a week and he is running around being his usual self. Not much has changed except a nice zipper from ear to top of his head from the surgery. So that part is good. Actually, it is amazing how fast those little guys can recover. We will have another scan next week to see how well he is recovering. His brain shifted from the pressure of the bleed 1/4 inch so we will see how that looks. Now comes the bills! Ouch. That spanks the pocket book a bit.

    As far as sharing information, if you have any clues let us all know! I am out of ideas. All I know is that we get fully crawled frequently and that is as far as it goes. Pages in pages out slowly climbing as far as indexed pages go. Home page seems to be corrected finally. The link command doesn’t provide any information for the home page and nothing at all for internal pages. We have thousands and thousands of links to the home pages and throughout the site. Some junk links and some not – 90% natural as we enjoyed decent rankings for years so that comes with the territory. After this rollercoaster ride for the last year and a half plus the many changes Google has been through we are completely lost now.

  87. Dave (Original)

    Bontar, your late mention of 12 domainns and “mini sites” could be a reason if they are all linked up.

    I would again urge you to read the Google Guidelines. If your site(s) are doing anything for SE’s in an effort to boost your ranking (like linking non-themed sites together) then you should ‘clean up’ ASAP.

  88. Hi Matt,

    Many thanks for the feedback:

    “This is not something that’s radically new; this set of algorithms has been live for a while.”

    Interesting, but it’s effect was certainly radical. I’m not sure by what channels you get your feedback, but the 27th was certainly a Big-Bang scenario for many legitimate websites, rather than just some kind of iterative tweak. I’ve rarely seen such an instantaneous and unanimous commotion on all of the webmaster forums. I am therefore hoping that this counts as some very serious negative feedback.

    Are there any official channels by which we can register critical feedback when, as on the 27th, something goes seriously wrong? From my reading of the post-27th SERPs, it looks to me like a problem that would be virtually impossible to spot, unless you happen to be one of the affected sites. For all of the keyword phrases I have looked at, our site, usually along with 4 or 5 others, have simply been neatly spliced from their high-ranking positions and shifted downwards 40 or 50 slots. All of the other results have maintained their relative positions and simply shifted up to fill the gaps. I can’t imagine how any of your evaluation procedures could possibly detect this kind of problem, unless the affected sites happen to be cnn.com, or similar household names.

    Is there anything we can do to have the 27th of June’s “refresh” evaluated relative to our site (and the many other affected sites that are being reported), in addition to your usual test-suite. Clearly, Google’s test suite does not expose this problem.

  89. Yay Matt. Spammer is down. You should develop some sort of mechanism where people could advise of such sites. I’ll start with the worst subdomain spammer I’ve ever seen. About.com.

  90. Matt,

    Thanks for the reply. It is an honor to get one, but I must say it was a reply, not an answer.

    So, with your permission, I’d like to take another look at the case at hand.

    First let’s review the facts: One, not a single page from IDcide.com can be found by searching Google’s main index. The site is Blocked; Two, the SAME content has been deemed good enough to serve as an Answer reference for the Google Q&A feature.

    Now let’s move a step forward in search for answers.

    Why was the site banned?

    These are the Google Webmaster Guidelines. Would you like to highlight the paragraph that says Google will block sites about certain topics? Or, point out where the page count that will cause blocking is specified? Or, Maybe help people find where does it say that Google blocks domains that change ownership?

    I’ll save you the effort – it ain’t there.

    I’m now certain that IDcide.com was blocked for reasons we don’t speak of nor hear of. While I’m not naive enough to think this is the time and place to start speaking – I would like to coin a phrase:

    Black-Hat Banning – the unethical removal of sites from a search engine’s index.

    Other search engines have been doing it. Now we see Google does it to.

    So the tables turn – you’ve been caught – here is a new question – what do you do?

  91. Dave (Original)

    RE: “I’ll start with the worst subdomain spammer I’ve ever seen. About.com”

    I’ll second that! Of late they have become nothing but SE spammers.

  92. I would again urge you to read the Google Guidelines. If your site(s) are doing anything for SE’s in an effort to boost your ranking (like linking non-themed sites together) then you should ‘clean up’ ASAP.

    Just a question about this, how unrelated is too unrelated? Let’s say for example that I own three webmaster related sites, one just for graphics one just for coding and one just for SEO. Their context isn’t related but they target the same audience, which means that by linking them, each one of them can send traffic to the others. Should I use nofollow? I don’t see the reason, I trust my sites (nofollow if I am not mistaken was intended only for non trusted links). Will Google penalize them? Most likely yes.
    The Google guidelines suggest that we should build our websites for our users not for the SE, however they are driving us to do the exact oposite. Building for the users means that the content of the site and the links from the sites should be added without having to think how Google is going to treat them, yet nowdays we think carefully before adding any link. To be honest I am not prepared to do that, if I want to link I will link without second thought the same way if they want to penalize me they will penalize me without second thought.

  93. Matt, thanks. Now I know what you do.. 😉
    [P.S – you can remove this comment too. I would have sent a thank you e-mail – but have no idea where too – so I used a blog comment]

  94. Hi Matt

    First welcome back to work – fun isnt it ? 😉

    This data refresh – why have sites seemed to lose the domain name when being first on a site:domain.com check and hence lose lots of traffic.

    Changes like that dont feel like a refresh and sites seem to drop out of rankings when this happens.

  95. Dave (original), you are right, cross-linking can be an issue, although I’m quite cautious when linking across domains, I haven’t had enough care when linking across mini-sites in the same domain (different subdomains or folders, different topics).

    Jim, I agree with you, ‘related’ is a very generic word that doesn’t say much. A human can understand that a wedding site is related to a site that offers travels to the Niagara falls, but a machine may need a few ‘iterations’ to know. Having said this, it seems a bit arrogant to penalize a site for ‘unrelated links’ if you are not sure what a ‘related link’ is, but…

  96. …” an effort to boost your ranking (like linking non-themed sites together) then you should ‘clean up’ ASAP.”

    How would this affect me as a web designer – I have lots of clients with sites covering many different themes and from my portfolio I link to their sites, also many of the sites I design carry a “designed by xxx” link back to me?

    Why should people be penalised for linking to other non-related sites, especially for personal and blog sites ?

  97. Bontar, I believe any changes on the 27th were refreshing data used by an existing algorithm.

    Interesting this algo seems to refuse earlier top ranked listing pages, in my case a listing using a lot of dl dt dd coding having short descriptions in it.
    At least the site in my email. I wait and see for the “refresh of that data in the next 2-3 weeks” if this is the case.
    Btw, off topic on quality, de Adsense language issue seemed to be fixed but the Sitemaps language issue still exists.

  98. Matt, perhaps you can help. I have been penalized for some reason on April 26th, then finished off on June 27th of this year. I have a site that’s been around for years, ranked well. No tricks. It’s a fitness website offering information, video clips, talk radio clips, training and diet guides, fitness galleries, as well as fitness supplement sales. The supplements are linked to affiliate supplement company, have been since 2002. I offer unique and usefull content to the users since I have a REAL review section. I was the FIRST site to do a “review” option, however it seem now that fake or “blank” review sites are on top while I have been penalized for some reason to pages 100+ instead of page 1. 🙁 Also I have about 2000 real pages yet google site: command says it’s 10,000+! I may also be victim of the 301 issues, which I have tried to solve as well as hundreds of spam and scrapper sites linking to me. 🙁

    Any advice? I am running out of options or ideas as to what has happened.

    Jirka

    ps – I am happy to give you my site yet not posted online since there are dozens of companies ripping me off and yes ranking high as a result of it.

  99. Dave (Original)

    RE: “Just a question about this, how unrelated is too unrelated? Let’s say for example that I own three webmaster related sites, one just for graphics one just for coding and one just for SEO. Their context isn’t related but they target the same audience, which means that by linking them, each one of them can send traffic to the others. Should I use nofollow?”
    ========================================

    Why would you want different sites for related content? The only reason I can think of is to somehow boost your rankings.

    If sites (all owned by 1 webmaster) are related enough to link up, then why not make one big site? 1 big site is cheaper, easier to maintain, SEO, better for your visitors & risk free.

  100. Thanks for refer me to this discussion Matt about june 27 that i missed.
    Looking through your reply
    ” I’m pretty sure that there will be another refresh of that data in the next 2-3 weeks, so we keep iterating to improve things based on our evaluation and the feedback that we get.

    i have again hope.My site is also from one of the sites that was never affected by any Google changes in past and if changes are made by existing algorithm as you already said
    (o.p
    “Bontar, I believe any changes on the 27th were refreshing data used by an existing algorithm.” )

    then i can conclude that i’m victim from some kind of experiment and need a little patience for another refresh.
    Will be back when happen another refresh to give feedback.

  101. [quote]
    Why would you want different sites for related content? The only reason I can think of is to somehow boost your rankings.
    [/quote]

    I can see a few more reasons, Dave, but what about this one: If Google drops one of the two sites 200 positions in the serps, maybe the other site remains in the top 10 and you can pay your bills, although cancelling your vacations.

  102. Matt – Is there anything you can say about sites where the homepage is no longer shown in the results of a “site:www.example.com” query? The homepage is still cached, but is missing when using the site operator.

  103. Matt, having one of the sites that have been butchered by the 27th’s updated , Do you recommend i start making changes to the site ? or just wait till google completes their “update” ? I dont want to do a major overhaul only to find this is more of a thing google needs to tweak rather than i.

    The site use to rank great prior to June 27th and now spam sites that scrape our content rank in top 10 while we are no where to be found . So i guess google does like our content just not us 🙂

    Any insight would be appreciated..

  104. Nah, this whole subject must be a belated April fools … seriously, it’s too absurd to even contemplate.

    Anyway, Google would be one of the main culprits – how do maps advertising email photo editing – what’s the relationship there ?

    This would apply to most companies with their fingers in different pies. Microsoft ? SQL Server Games Console … the list is endless.

    [quote]
    Why would you want different sites for related content? The only reason I can think of is to somehow boost your rankings.
    [/quote]

    Don’t even know where to start with this one. How about some kind of rationing system, one website each.

  105. Dave (Original)

    RE: ” can see a few more reasons, Dave, but what about this one: If Google drops one of the two sites 200 positions in the serps, maybe the other site remains in the top 10 and you can pay your bills, although cancelling your vacations.
    ======================================

    Bontar, what is YOUR reason though?

    *Google indexes and ranks pages, not sites*.

    If you had one big site, rather than x related mini sites you are not running ANY risk. Of course, this assumes you are within the guidelines.

    SEO 101 = “Content IS King”
    Build 1 big site with many good useful unique content page. Arbitrary numbers here, but aim to get 1000 visits from 100+ pages as apposed to 1000 visits from 10, or so pages. This is your single BEST insurance against SERP fluctuations & rank dropping etc.

    You can stick to your guns and say “well I’m not changing” if you like,
    the choice is yours. Just keep in mind that Google are many steps ahead on how Webmasters try and boost site(s) by creating a mini network of smaller sites.

    Google can keep going without your site(s) in their Index just fine, but: can you afford to be dumped from the Worlds #1 SE?

  106. Hi Dave (Original) –

    I’m trying to learn here and much of what is being talked about is going way over my head. But you mention that Google indexes and ranks pages, not sites. While I believe this to be partially true, it isn’t what I’ve experienced recently.

    I have a site with about 800 articles. All written by myself, no duplicate content. The site has been working well for many years… until June 27th. All of a sudden most of my 800 pages which used to rank well for their specific topics have nearly disappeared in Google’s search results. Most pages that ranked in the top ten prior to June 27 now are at about number 200. Pages that ranked in the top twenty now show up in the 500-800 range.

    So while I sailed along for years without needing to learn “seo” and just writing good quality content…. .now all of a sudden Google things the entire site is cr@p.

    So to me this doesn’t equate to “risk free”. There is obviously something about the domain that Google no longer likes and the “site”, not just “pages” are seemingly punished.

    Bob

  107. Hi Matt,
    What do you mean by:

    ““Bontar, I believe any changes on the 27th were refreshing data used by an existing algorithm.”

    So, is this an update or just an experiment? Is Google happy with this results?

    Thank you

  108. *Google indexes and ranks pages, not sites*.
    ==========================

    Dave (Original), based on my experience, the domain itself can be penalized at a global level, not only page by page. That’s why I try to not to put all the eggs (pages) in the same basket (site).

    Having said this, I can understand your point of view, a big site is better than 10 smaller sites. It’s easier to maintain, to promote and to create a community of users.

    But, I must admit that it scares me a lot. 🙂

  109. Ouch Thathertz

    Experiment gone awry, I’d venture. Lets hope that the data loaded on 27th presents a temporary aberration that will soon be corrected.

    My site is long running and whitehat and I got walloped.

    (Yes, I run AdSense … and foolishly depend on the income to keep my mortgage paid. I can only guess that my site got caught up as an innocent victim with a flushing of Blatant AdSense Abusers.)

    My kids would scream “it’s not fair!” but silly me for expecting that Google would let my little handful of nice rankings ride for years on end.

    Or like the saying goes, “if life was fair, the horse would ride half the time.”

  110. I am very curious as to the effects of the 27th. I understand that this was an existing algorithm, most likely a few values where changed and certain effects are to be expected. 2 of my entire sites have lost almost all of its traffic.

  111. Just a reminder for google.com to check their site too for errors

    Do a google search for therapy products
    what the heck is Yahoo show up for the middle of results page?
    Looks like a google error?

  112. Dave (Original)

    RE: “But, I must admit that it scares me a lot”
    ====================================

    Google will no penalize a site UNLESS the site in questions is doing something outside their guidelines. Yes, I know we hear the screams from “innocent” Webmasters. They are NOT as “innocent” though as they would have us believe. If you have a mini network of sites and 1 is penalized (like for cross linking in an attempt to fool Google into thinking the links are votes) then they will ALL eventually suffer the same fate.

    One big site that houses lots of good orginal content and is always within the Google guidlines, will survive long term as a business should. All you then need to worry about is getting the pages ranked well and not wake up each morning wondering if Today is the day Google catches me out.

  113. “One big site that houses lots of good orginal content and is always within the Google guidlines, will survive long term as a business should.”
    ================================

    I believed this too until June 27th….

  114. Dave (Original),

    As ever. You are spot on. You are so astute, it’s amazing.

    It’s worth noting, though, that what you say is ONLY true, because Google employ a new development methodology known as BFD (Bug-Free Development). Google figured out that bugs were a significant waste of time and money. It turns out that writing code without them is much, much more efficient and offers substantial savings. For one, you no longer require a testing department.

    Were it not for BFD, your unquestioning faith in Google would be as naive as it is touching.

  115. Dave (Original)

    AKA, sorry no idea what you are trying to say.

    BTW, my faith in Google come from always sticking to their guidelines and never being penalized. Same applies to the other big 2. The White Hat SEO forums I frequent consists of member who experience the same. Going on 7 years now.

    RE: “naive as it is touching”

    It’s really funny reading many of the other forums after Google sneezes and hearing all the “why me”. Now that’s what I call naive and touching 🙂

  116. Dave (Original)

    I’m sorry if what I said went straight over your head, as so often seems to happen.

    I’d love nothing more than to knuckle down and try to explain to you what a bug is, but I’m afraid I haven’t really got the time. If you’d like, I could maybe recommend some nice basic “How computers work” type books to get you started.

    Let me know if interested.

  117. PS: I just thought…maybe you could also use a primer in statistical analysis?

    It turns out, the sample size on which you are basing all of your assumptions – namely, 1 site out of around 8 Billion – may not be as statistically significant as you think.

  118. Matt, sorry to post this again, but you said you will take a look and maybe tell me and others if the things I wrote could help somebody.

  119. For the last six months my site disappeared from Google after spending 7 or 8 years at #1 or #2. A few months ago I came across sitemaps and added my site to it. For some reason every time Google crawled my site it would get a 404 error when attempting to access my site at the root level. I checked my access logs and found that starting from December 2005 when Google and a few other bots were doing a “GET / HTTP/1.1” they would get a 404 error even though there was always an index.html file there.

    I immediately informed my hosting provider of this error and was told that it was a problem with Google not the server I was hosted on. I asked to be migrated to another server and my request was ignored. So I switched to another hosting provider and two days later Google crawled my site and got a 200 instead of a 404. Another 2 days later my site was back to #2 on Google. Has anyone else reported or seen this type of problem with a server before?

  120. Dave (Original)

    AKA, Oh, you were *trying* to be sarcastic (apparently), the lowest form of wit isn’t it? Maybe try walking before trying to run.

    But hey, I have “unquestioning faith” that you will able to make a semi-intelligent post one day, perhaps even funny. Until then, ta ta 🙂

  121. Dave (Original)

    Cristian, the page titled “Are your indexed pages WAY lower than they should be ?” is rather a funny way to look at things IMO. Doesn’t every Webmaster think their pages “should be” on page 1 of Google?

    Also, the length of the page name is waaaayyy over the top IMO. It looks, well, spammy.

  122. If cross-linking sites is such a big fat no-no and gets one penalized, how do you guys explain the success of Weblogs, Inc? Check the bottom of this page:
    http://www.thecancerblog.com/

    They link to all their other blogs, which happen to all be on separate domains.

    They aren’t penalized. Explain?

  123. So, Matt, am I right in understanding that the update on the 27th June was an ‘experiment’ and things should go back to ‘normal’ soon – or should I start aggressively SEO’ing to try and gain ranking again – something I’ve never actually done with my site? (it still has some font tags!!). I’d rather not have to do this though, as for 6+ years Google has deemed the CONTENT the most appropriate, now suddenly every page on the site has dropped down pages in the SERPs overnight!

    In addition, backlinks have dropped from hundreds down to 26! Is this due to perceived recripricol linking? All external links are industry appropriate but is it wise now to remove them and ask others to remove theirs back to me?

    Also, the severe drop in ranking came 1 week after adding Google Analytics to the site. However, by only adding the tracking code to a handful of pages could this have made it look like people were bouncing off the site when in fact they were going to pages with no GA? Does GA send it’s data to the main search to improve rankings?

    Lots of questions I know!
    thanks
    steve

  124. Matt:

    It’s time for a reality check.

    Tens of thousands of very useful non-spam sites have been dropped by Google in the last few rounds of barely-tested algorithm updates.

    Posting the details of a few spam sites that you have also caught may amuse you personally, but it does nothing to improve that fact that _overall_, the quality of Google’s search results has sufferred from the last few rounds of incompetent tinkering.

  125. Matt,

    How are things? My guess would be busy!

    I’m writing this post in hopes of just unloading my frustration to someone who seems to have a clue of the inerworkings of Google.

    If I’m sticking this post where it’s breaking the flow of the blog, or if it seems too wordy, I apologize.

    I started my site about 15 months ago and have attempted to build a quality site with quality products at great prices with good content and a solid link structure. I’ve had my site SEO’d and started to make some progress. I went from PR0 to PR2, this lasted until the beginning of 2006. After Big Daddy we dropped back to PR0 on the next update (3 months) we got a PR5 (34 backlinks) and our related blog got a PR3 and we were happy just to see things getting better. (Maybe trying to do the right things IS the best way to go.) However, we weren’t showing up in the any search results for our keyword phrases in Google. We show up quite often in MSN and Yahoo.

    I keep working my site with our customers first and search engines second, building links, adding more relative content and writing some press realeases (all written by me), but as of our last update of July 16/06 we dropped back to a PR0 on our site, dropped to PR0 on our blog, dropped to 15 backlinks. MSN=1515, Yahoo=483 (Maybe doing things the right way is just a waste of time!)

    Needless to say, today I’m really disappointed. I’ve worked my _ss of and this is the result and anything else I do won’t be seen for another 3 months, if anything is seen at all, or maybe we’ll move even further backwards.

    Obviously, I’m not as SEO savvy as most of your readres and maybe I don’t have a clue whether I have doorways pages or not that’s hampering the site, but I would really like to understand what the heck am I doing wrong for us to move backwards so radically? I certainly can’t seem to find the answer anywhere else and you know the answer of any SEO company, more money,maybe some results and possibly getting banned, because they really don’t know much more than META Info, quality backlinks and quality content. Everything alse they provide is just a guess, in my opinion. (No offense to your SEO readers, this is just what I’m seeing. I’m sure there’s quality SEO firms out there but how can you tell them apart, I can’t!)

    Any how, thanks for providing an outlet for me to vent, even if you don’t reply to this. I guess I’ll just keep plugging away in hopes that someday people with real product sites and not just affiliate sites start making some progress in Google.

    Cloud9 (someday)

  126. Quotes:
    “Bontar, I believe any changes on the 27th were refreshing data used by an existing algorithm.”

    AKA, that’s what happened. Arubicus, the change went through evaluation just like any other change and showed a positive improvement. This is not something that’s radically new; this set of algorithms has been live for a while. I’m pretty sure that there will be another refresh of that data in the next 2-3 weeks, so we keep iterating to improve things based on our evaluation and the feedback that we get.
    I wrote that before I read your most recent comment, Bontar. It will continue to be refreshed (just talked with someone about it few minutes ago), but not with data from the 26th. We keep iterating based on evaluation, the changes that we see, and the feedback that we get as well.

    You said you wanted some feedback and if you read all posts in the DP forum about the Black Tuesday (June 27th) then you can find all the feedback you need.
    It seems like when you checked the changes you missed all spam sites like the ones mentioned above in comments, but instead of removing spam sites you removed sites like mine that has been around since 2000. On that particular day, thousands, if not millions, of sites wen’t supplemental and I can’t figure out why mine did!! I have been trying to change whatever I am THINKING I had to change but nothing helps!!
    Your bot is getting wrong paths, can’t handle sessionid’s and querystrings in php pages! MSN and Yahoo don’t have any problems with it so why does Google have problems?
    What is going on with Google?

  127. Hi Matt,

    The whole results/index thing does seem to be a problem. Sometimes legitimate content that can’t be found in Google can be found in Yahoo.

    For example “Old ADSL firmware!!”

    At least one of the sites returned by Yahoo is indexed by Sitemaps and crawled regularly.

    I have always used Google, and it used to be the case that if it wasn’t in Google it didn’t exist, but now I have to check with Yahoo just to make sure.

  128. Here’s some information that Google might need!
    I found a site about MSN’s new Strider Search Defender and on that site they point out among other things a site that are spamming Googles index. This was done in May and I did a search for Fendi Handbags in Google and the site is still there. Maybe Google needs to go together with MSN so they can fix the index 🙂
    Here’s the page:
    http://research.microsoft.com/searchdefender/Google_freett_Spam_Examples.htm
    and here’s the original page with some really interesting information!
    BTW, Matt, when are you guys going to fix the broken index? It’s been weeks now since you almost deleted my site but there are still spam sites like the one I mentioned above! How come?

  129. Dave (Original)

    RE: “read all posts in the DP forum about the Black Tuesday (June 27th) then you can find all the feedback you need.
    =========================================

    Is that the one where most of the members of yet another new linking scheme were dumped? It’s always the same forums that have the most casualties (disproportionate to member numbers) when Google tackles spam. No wonder though when you read some of the high risk black hat advice coming from these places.

  130. RE:”No wonder though when you read some of the high risk black hat advice coming from these places. ”
    This is the forum that found billions of spam pages in Google and they are against Black Hat things is you actually read the posts. I even think Matt referred to it in another post.

  131. Dave (Original)

    I read many posts over there at DP. The whole place is obsessed with links for ranking purposes. I don’t however doubt that 1 in 10 threads have some value.

    Let’s not forget that DP is the home of the coop spam network linking scheme that has many site banned/penalized.

  132. Dave, I have read posts about people there talking about the coop thing and that they thought that might be one reason for getting banned and if that is so, then they only have themself to blame, which is right!
    I never used any linking scheme or anything else and if and when I link to other sites I have then it is just because I want a place for SE’s to pick up the link to my site(s) or to any other site that I create for other people/companies!
    I know it is good to have IBL and that kind of stuff but my number one priority is to “submit” to SE’s by having a link on another site that’s already indexed.

  133. Dave (Original)

    George, like most DP forum members you are in total denial of the facts. You state that: “I never used any linking scheme or anything else ….” yet your link exchange page tells a different story. If you haven’t already, you will one-day be busted for your links, which exist for no other reason other than to try and boost PR, link pop etc.

  134. Dave (Original),
    You are WRONG!
    I do NOT use links or links exchanges with anyone that I don’t know personally and I also think that people put too much into all links in and out, even if Google say they do or did!
    The links I have are either to my own sites or sites that friends have and like I said, it is for the only reason that SE’s can pick up my sites instead of trying to find a submission page and do it manually.
    I see that you don’t list your site (if you have one) but it would be interesting to evaluate your site and see if I can learn something from you since it seems that you defend Google even if there are tons of innocent victims of their latest downgrades. I was hoping for Matt to check out my site and see what is wrong with it so I can change it but he hasn’t responded to any posts yet. If you do have a web site, Dave, please post it so we all can learn from you!
    Thank God for MSN and Yahoo, if we didn’t have them to rely on, we’d have no traffic at all. Hoping that Live.com will go live soon so there will be some real competition out there to Google!

  135. Dave (Original)

    No George, I’m not wrong. I saw your link exchange page and I HAVE read Google’s guidelines. I’m not going to bother giving you details as, like all DP forum members, you are in total denial and only shoot the messenger.

  136. George,

    Don’t even bother wasting your time with Dave (Original) – he’s a self-righteous trouble making time waster with nothing constructive to say other than to quote Google’s guidelines parrot fashion and has an “I’m alright jack” mentality.

    He’s probably a traffic warden or something.

  137. I can’t believe that the “new index” will settle as long as I find domains that have been burned before in there and doesn’t even exist any more(404). Or are you re-evaluating the whole thing??

  138. I am in the unneviable position of actually having seen Dave (Original)’s site. While it was some time ago – a few months maybe – I doubt that much has changed.

    Put it this way. I am not surprised that he isn’t too keen to air it for public scrutiny. To my mind, it is the worst kind of Spam, because, unlike most Spam, which is blatant, it is Spam masquerading as “content”. Loads of “articles” written purely as SE fodder. Pity the happles user whos stumbles on Dave’s site, and mistakeny asumes that its content was actually designed for human consumption.

    It’s like Dave (O) generated a site based solely on Google’s guidelines…nothing else. As you can tell from so many of his posts, he seems to believe that these guidelines eminate some kind of magical power.

  139. Dave (Original)

    RE: “with nothing constructive to say other than to quote Google’s guidelines parrot fashion…”

    Err, you might be shocked to hear this but Google’s whole SE is based on their guidelines. Only fools and black hats ignore them. Which are you? I’m guessing the former with ambitions of one-day being both.

  140. Wow – another constructive comment Dave

    Looking through all your postings in this and other threads, I don’t think you’ve managed even one useful snipped of advice or information, just streams of “I’m doing things right, so I’m ok jack whereas you must be a spammer/black hat/idiot/not following the guidelines” – well believe me, lots of people here follow the guidelines but things just don’t work as expected – if you are so above reproach, how about imparting some of that knowledge and actually helping people.

  141. Dave (Original)

    Congratulations Francis, you exceeded my expectations and have proven yourself as a fool and black hat.

  142. I’ve said enough – I think your last post asserts my point perfectly.

  143. Amen Francis, you are not alone and I am glad you speak for the 1000’s of us lost in the choas and accusations of being black hat.

  144. Matt Cutts, you started the thread with:

    “Google’s webmaster quality guidelines are pretty clear that autogenerated doorway pages are not welcome in our index. So this is a case where checking out your domain would be just a productive as contacting Google. The best advice I could give would be to remove all the doorway pages and then submit a reinclusion request.”

    Do a search in Google for the keyword “search engine spam” (no quotation marks). #1 sponsored link at top left is this ad:

    Doorway Page Generator
    Multi Domain. Editable templates.
    1000+ page generation. Synonyms.
    http://www.xxxx.com

    Made me chuckle when I saw it. To put my search in context, I’m doing research for proof of what can hurt search engine rankings to take to upper management.

  145. Dave (Original)

    Francis, if you already proven yourself a fool (twice), no need to keep driving home the point.

    Why don’t YOU help George? I admit he doesn’t like hearing facts, but I’m sure you are the type who gives the answer he wants to hear 🙂 Or better still, show me why my advice to George is not sound.

  146. Dave (Original)

    RE: ” To my mind, it is the worst kind of Spam, because, unlike most Spam, which is blatant, it is Spam masquerading as “content”. Loads of “articles” written purely as SE fodder. Pity the happles user whos stumbles on Dave’s site, and mistakeny asumes that its content was actually designed for human consumption”

    VS

    RE: “It’s like Dave (O) generated a site based solely on Google’s guidelines…nothing else”
    ================================

    Bob (bullshit), you are not a very good at lying. You stick with 1 lie only when trying to lie 🙂

  147. Building just one large site is the worst advice you give anyone. I know because on June 27th my one big large site lost 90% of it’s traffic, I had a healthy income and 2 part time employees to work on and improve my ‘one large site’.

    In one day (after 7 years with no changes in traffic regardless of google updates and changes) all the traffic (except about 150 visitors) from Google suddenly vanished. So, I had to let my employees go and now I don’t even earn enough to pay my expenses.

    So yes, Google can whipe out a site, not just individual pages. My site is very large and searches came in from thousands of different search phrases, not just a few ‘high ranking keywords’. The whole site was wiped out, not just rankings for individual pages.

    Im not crying or complaining, Ill make it back and I’ve learned something valuable from this experience. I just want to point out that relying on one Web site (and one search engine) is a big risk, especially when you have people working for you.

    It’s clear that something went wrong on the 27th. It doesn’t make any sense that people that weren’t affect begin to preach and condemn the affected sites claiming they take part in shady SEO.

  148. I am in the unneviable position of actually having seen Dave (Original)’s site. While it was some time ago – a few months maybe – I doubt that much has changed.

    Put it this way. I am not surprised that he isn’t too keen to air it for public scrutiny. To my mind, it is the worst kind of Spam, because, unlike most Spam, which is blatant, it is Spam masquerading as “content”. Loads of “articles” written purely as SE fodder. Pity the happles user whos stumbles on Dave’s site, and mistakeny asumes that its content was actually designed for human consumption.

    It’s like Dave (O) generated a site based solely on Google’s guidelines…nothing else. As you can tell from so many of his posts, he seems to believe that these guidelines eminate some kind of magical power.

    What is it? Drop the link if it’s so bad.

  149. Dave (Original)

    I think your bluff has been called Bob (Bullshit). I KNEW you were lying and now everyone will 🙂

    BTW, why no help for George from you or Francis? Or, do you 2 only abuse those who do try and help others? I get the impression you would both abuse the Red Cross if they knocked on your door.

  150. Hold up, Dave. I’d do that too. I hate door-to-door salesmen. 😀

  151. Dave (Original)

    Oh dear, better hope you don’t need blood one day 🙂

  152. No problem. If I do, I’ll just get it the old-fashioned way….I’ll beat people up and break their noses until one of them matches my blood type. 😀

  153. Matt – I still do not understand what key change has affected the rankings.

    To demonstrate the ludicrous nature of the change:

    A rival site to mine – who a customer emailed me to point out they have stolen my content – is now ranking in my previous position while my site – with the original content – is out of the top 100!!!

    I find this totally bewildering.

    QUESTION: Does a ‘links’ page elsewhere on a site (with about 30 relevant links on it) affect SERP ranking this much? Will removing this page or placing nofollows on the links reverse the situation?!

  154. Until today I had been following this update from the sidelines, but much to my horror I woke up this morning to discover many of my pages had plummeted in Google’s SERPs. Something to keep in mind is that for some key phrases like “periodic table of elements” my site has lived on page one most of the time since before Google came out of beta in 1999, I even managed to orchestrate a domain name change in 1999 with 301 redirects and still retained my standings in Google’s SERPs. This isn’t a matter of going from page one to page two in SERPs which happens I know, this is a matter of going from page one to page five or lower. Phrases I was #1 or #2 just yesterday I don’t even appear for today.

    My site isn’t a lame MFA site (I started it in 1995) and I’d rather spend my time producing content than playing SEO games, but something totally ran amuck with this update. I have noticed that Google’s reported number of pages indexed for my site is many times more pages than exists on my site and I’ve been trying to figure out how to make sure Google only indexes the pages it should (I uploaded a site map with all valid URLs today). It is just really upsetting to wake up and discover that traffic to the site I have been working very hard on for over ten years suddenly dries up while finding millions upon millions of sub-domain spam pages being indexed for domains that aren’t even ten days old.

    I’d love to know what happened that suddenly removed my site from the favored status it has held since the days Netscape 3.0 & 4.0 ruled the Internet.

    (oh and yes I’m refering to the site my name links to)

  155. Maybe it is related to the Adsense?

    I have just done some basic research and no websites with Adsense on them came up in the first 3 pages of the SERPS (for the queries previsouly used to get to my site). Is this just coincidence or is Google trying to clean up the Internet??? Of course, that is assuming that any website with Adsense doesn’t have a product to sell or orginal content – which isn’t always true.

    I am starting to wonder if this goes deeper…

    I had been running at between 5000-6000 page impressions a day for around 5 years before I added Adsense. Within a month of adding Adsense I dropped in the SERPS and consequently page impressions went down to between 2000-3000 a day. Then in mid-June I added Google Analytics and a week after I was dropped from the top 100 results in the SERPS and page impressions are currently around 600-700 a day. Quite a drop!!

    Is this coincidence or is it related to the recent changes?

  156. Wow, im 13 years old now, but i wanna be like you matts. Thank you for this blog. Even i dont already understand all your posts.

css.php