Okay, it’s been a while since my last infrastructure status report, so I’ll briefly cover the things that I know are going on. The executive summary is that things are relatively quiet.
The quarterly-ish PageRank export is underway. As always, don’t expect traffic or rankings to dramatically change, because these PageRank values are already incorporated into our scoring. The same quarterly-ish data push that updates PageRank in the toolbar also updates the data for related:, link: and info: (remember that operator?). You can read more about PageRank from this previous post if you swing that way. Also remember that the link: operator only shows a subsample of the links to a page that we know of. I’ve mentioned before that some data centers (I believe 64.233.183.xx and 72.14.203.xx) continue to show PageRank values from a slightly older infrastructure. Not a big deal, but I wanted to mention it for the hard-core data center watchers so that they don’t get confused.
There were some situations where site: would show supplemental results ahead of regular results. I believe we’ve changed that so that regular results will usually show ahead of supplemental results for site: queries.
As a reminder, supplemental results aren’t something to be afraid of; I’ve got pages from my site in the supplemental results, for example. A complete software rewrite of the infrastructure for supplemental results launched in Summer o’ 2005, and the supplemental results continue to get fresher. Having urls in the supplemental results doesn’t mean that you have some sort of penalty at all; the main determinant of whether a url is in our main web index or in the supplemental index is PageRank. If you used to have pages in our main web index and now they’re in the supplemental results, a good hypothesis is that we might not be counting links to your pages with the same weight as we have in the past. The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit).
I think going forward, you’ll continue to see the supplemental results get even fresher, and website owners may see more traffic from their supplemental results pages. To check out the current freshness of the supplemental results, I grabbed 20 supplemental pages from my site and checked out their crawl date using the “cache:” command and looking in the cached page header. The oldest supplemental results page that I saw was from September 7th, 2006 (and I only saw 2-3 pages from September; most were from December or November). The most recent of the 20 pages was from January 7, 2007, which shows that supplemental results can be quite fresh at this point.
Let’s see, what else? I think we’re going to change the “filetype:” operator so that it doesn’t require an additional query word, so that you could do filetype:doc or filetype:pdf or whatever. That isn’t live at this point, but I believe it will be down the road.
I’ve mentioned this before, but one of our data pushes that used to happen every 3-4 weeks is now happening more like every 1-2 days. Regular searchers won’t really notice this, but if you see more variance in your rankings, I believe it’s probably due to that data push happening more frequently.
An SEO or two has been holding my feet to the fire about root pages of .com’s that are hosted outside the US. Barry has talked about the issue a little bit here. In some (pretty rare) circumstances, you’ll see the root page when you search site:domain.com on google.co.uk for the “search the web” option but you won’t see the root page when you switch to “search pages from the UK”. I thought we’d nailed this issue in December, but we found another way that this can happen. I believe a fix has been submitted and is percolating its way through the system. Of the ~7 examples that I know of, I believe all but one is working now (and the remaining site is doing a chain of like five 302 redirects to weird/long/deep urls). However, if you 1) have a .com that is hosted outside the US, 2) searching on (say) google.co.uk for [site:yourdomain.com] returns your root page and all your pages for “Search the web”, 3) if you switch to (say) “pages from the UK”, the root page does not appear but the rest of your pages do, then this paragraph applies to you. I’d wait 4-5 days to let this second change percolate completely into our index, and if you still see the behavior after 4-5 days, please leave a comment with the name of your site.
Right now I’m not expecting any major infrastructure-related upheavals to our rankings. Should that change in the future, I’ll be here to talk about it then. 🙂
Update: A few people were seeing PageRank 0 for their site. There was a small auxiliary push that needed to happen to complement the PageRank push, and that push happened a few hours ago (i.e. Jan 11, 2007). If you were getting stressed, you might want to re-check now. If you never even noticed, well, good for you. 🙂
Thanks for the comprehensive update, now if you could only stroll over to http://groups.google.com/group/Google_Webmaster_Help and log in so that you get a ‘Green G’ by your name and post this (Or maybe Adam is reading this, hint-hint, wink-wink). Then sticky it to the top. It would really help clear up a lot of noise (imagine noise crossed out and replaced with FAQ). You’ve re-re-re-reconfirmed some answers to very common questions; TBPR, link:, .com/co.uk, supplementals, etc.
Once again thanks again for a most authoritative informative post in an unofficial-your-personal-blog kind of way.
Happy to help, JLH. I’ll ask Adam if he can point it out over on the webmaster group.
Hi Matt,
Thank you for your posting. It’s very timely, because we’ve been noticing a sharp drop in our site’s PR from 7 to 0 spreading across Google Datacenters all of the last few hours.
We’ve also noticed that some other established sites are also having the same problem. Is this something you think we should be worry about, or just a glitch that will come to pass?
Thank you
Matt
Thanks for detailed weather report. A theoritical question 🙂
How can we see the difference between changes in rankings as a result of those data pushes that happening like every 1-2 and the traditional Everflux. Can we assume that from now on rankings of sites wouldn’t be as stable as before?
Thanks.
Much thanks for the supplemental update.
I can see users avoiding visits to anything that says supplemental when they do a search. It kind of screams second class.
Best to avoid it in the first place but when you have some new blog entries they likely won’t have any links so they’ll end up in supplemental. Its just the nature of blogging that people might link to a site but not the actual post so once the post moves off the front page it’ll be supplemtal no matter how good the content.
Thanks again Matt.
Philipp Lenssen reveals how page rank really works.
http://blog.outer-court.com/archive/2007-01-11-n25.html
I did that calc that reveals you are ranking 3778 pages per second, very impressive. So I’d guess the 90 seconds I cost you to read my comment and send a note off to Adam means 340,020 pages will not get updated next time. To those affected, I apologize.
Matt –
Thank you yet again for another EXTREMELY informative post. I think you answered every question I had from noticing SERP changes recently.
To follow up on SEW’s post, I heard your interview with Mike Grehan. You mentioned you understand more about what its like to be a webmaster now that you are a blogger. You also mentioned how many SEOs are getting creative in obtaining links. However, they represent a small segment of the web.
Try being just mom and pop who dont know how to play the Digg game, dont blog endlessly and dont go to SES or PubCon. They wont attract many editorially given links without a little help.
Harith, that data push is on a slightly different cycle (not pushed as part of the normal daily-ish index-data pushes), but it will be harder to distinguish between everflux and the more frequent data push.
S.E.W., too much bold and italics. Pruning.
JLH, you saw the comment to Philipp’s post that it was really Marissa doing all the PageRank? Floating point math hurts my head/CPU, anyway. 🙂
Hagrin, glad it helps. I’m taking next week off, so I wanted to throw at least one in-depth post up before I left.
Kirby, there are steps that we take to try to help those Mom/Pop sites as well.
Matt-
Referring to your comments
“The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links”
Can multiple links coming into a site from another to different pages hurt the site that is receiving the links? Example: State related linking to state related pages. If so, how can one get the penalty lifted and back in good graces with Google? We have a site that was ranking alright and just dropped of the map. Is it a problem to get such links that actually drive traffic?
Feed back would be greatly appreciatet!!
Aloha
Strange thing about Supplemental Results:
I have several domains which have been parked and redirecting via 301s to a main domain for a long time, but, even with the 301 redirects, these parked domains still appear in the Supplemental Index.
I thought 301 redirecting a domain was a way of telling Google not to Index it at all, but instead to index the domain they’re redirecting to?
Ok, this kinda explains things – I think. I was just surprised to see that my PR dropped from PR4 to PR0. Is this normal? The site is http://www.mikelopez.info.
Follow up to above post, a friend in the US (I’m in the Philippines) says that he sees PR5 on my site so I guess it will be PR5 after all.
Hi Matt
Great news about the UK, .com thing finally getting some resolution – if you think you’ve had your feet held to the fire, you should try explaining it to people who think you’ve done something to get their site banned!
Do you know if the .com sites are likely to regain the rankings that they previously had on “pages from the UK” or will the preference still be given to .co.uk TLDs even after the update?
Cheers for the info anyway!
An issue I’d love adressed is relevant link trading. My site has reviews of adult sites, and we now link with other quality sites like ours where we have reviews in common, review to review. My site got obliterated in the rankings in the last days of June 2006, and I think this was why. Too many links too fast looked spammy.
I’ve heard a lot of other people suggest google is not liking these kind of link trades, but they’re the most relevant results I can provide to my customers!! I’m giving them a second opinion for free, sending people to my competition. These links have always been designed to be a great feature for our users more than to drive traffic.
Great to see things are happening over there quickly in 2007, my page was actually not in the index on Christmas Eve, that was scary!! Then it went all supplemental, and now it’s back to about 10% supplemental, and completely fairly so, some old articles that deserve little to no attention.
Would love to get some tips, we have no idea about search engines, doing our best! socks @ if someone is interested.
What would be really useful, would be a more intelligent way of telling which country is website is based in. I’m in the UK and recently moved to a host in Utah, where I got a good deal. Now Google’s taken me out of the UK search results. Okay, the server’s in the USA, but that’s not useful information to my visitors.
How about making this information something we could provide via Sitemaps?
Oooh
Thanks for that (though I have to rewrite my general comments for January that goes in all my reports)
Time: 09:43am GMT – 11-01-07 – My pagerank score in the toolbar has just dropped from 4 to 3 – is this the toolbar update????? None of my other sites have changed, but I visit iwebtool.com alot and I’m sure they used to be PR5 and they are now 0
So, Matt, can you confirm that the only effect of being in the supplemental index is that the page gets crawled less often, or does it have an impact on ranking?
If it’s only to do with link frequency it seems curious to base it on PR – since the critical factor is surely how often a page is updated rather than how popular it is. I can think of several very highly linked, presumably high PR, articles which *never* change. Surely, brand new pages that nobody even knows about, let alone links to, are more likely to be changed frequently by their authors?
An unequivocal statement of what the effect of being in the supplemental index is would be a good idea. The webmaster help page talks vaguely about sites in supplemental having “fewer restraints”, but doesn’t make it clear what the end result is going to be.
If crawl speed is the only effect, maybe you should remove the “Supplemental Result” thing from search results altogether? I can’t see what it adds for the average searcher (I doubt if they even notice it), and it would remove half the noise from SEO forums.
@michael lopez
Many people has droped the PR by this update export.
Hi Matt,
I wonder if this new update has caused the rather bizarre event that has happened to me. Almost all my new pages have been hit by a penalty since last August. In December I requested re inclusion. On 3rd January all my rankings had returned (a truly euphoric feeling). Checked again later that week and they were still there. Checked this week and they have been bombed again (a truly un euphoric feeling). I would beg you not to do this to anyone else. Second time around it feels twice as bad. It is very tough to take having my new pages bombed again. I would love any form of explanation and I would be delighted to put right whatever I am doing wrong.
Michael Lopez – My site has just lost all PR on the toolbar. This has happened before for no apparent reason – though not usually on the whole site. It usually rights itself next time around. I know of no reason why Google would penalise my site, which is non-profit, does not buy links and has links from universities, the BBC and governmental sites, etc., which certainly should be passing PR.
If completely clean sites with no penalties, which have plenty of good (not bought, not spam) incoming links, suddenly go PR0, I’d say it’s just a little fault in the export of PR values to the toolbar display.
RE: “Not a big deal, but I wanted to mention it for the hard-core data center watchers so that they don’t get confused.”
==========================================
Those types are in a perpetual state of confusion. A dog chasing it’s tail is more productive than them 🙂
Hello Matt
The site:www.naukri.com does not show up when searched on Google.co.in with option Pages in India. Is this because the IP address of the site is in US.
also @michael lopez: check out my site at http://rss2pdf.com
it dropped from PR 7 to PR 0.
…and i don’t even have any ads on this site – just links to three other websites that i also built. to me, this is pretty evil :/
Could anyone please confirm is there is any fault exporting the PR data into the toolbar? I’ve seen a number of websites with high PR going to PR0. Big SEO company websites like seo-guy or highrankings have PR0 at the moment, from PR6. Could please someone show us the light?
If you guys took a moment to read matt’s comments, you’d realize the toolbar isn’t showing accurate information at the moment. Matt said that they were doing PR updates, so clearly this is the reason! He said to wait a few days and see what happens. This page is showing a 0 pr at the moment, as is mine.
If your site: operator is looking good, I don’t think there’s need for alarm.
I’m from The Netherlands but have a site, mscha.net, mscha.net, which is actually hosted in the US (cheaper hosting with better specs).
Therefore, it (properly) doesn’t show up when I search google.nl with “pages from The Netherlands” enabled.
Is there a way around that? Something like putting the country in a META tag or a header, perhaps?
Thanks,
– Michael
Matt, I’ve watched the video you posted regarding this topic with a few suggestions on what webmasters might do who are impacted. I’ve also read a few of your posts regarding these data refreshes.
Are there any other hints you can provide without spilling the secret sauce? I oversee a couple of sites which are nearly identical in operation and layout with the exception of the topic/industry. One of them is very frequently experiencing significant changes in ranking (up, down, up, down, up, down) and we have been banging our heads against our desks since June trying to figure out what might be unique about that site or make it look different in the eyes of Google.
Again, any additional tips or suggestions would be helpful. Thanks again for all of the great information you do provide.
Thanks for the info, Matt. I have a question relating to updates and data centers.
Is the crawl date (as shown in green next to the page size of some results in the SERP) a good indication as to whether the data center is up to date? Let me explain why I’m asking…
This morning, it seemed that the home page of a PR8 site I manage dropped out of the index completely (it’s not even listed when you use the site: operator and there’s nothing in Google Webmaster Tools to suggest there’s been a problem with our site) meaning a competitor’s site was in top position for our most popular keyword. The date next to their result was shown as “9 Jan 2007”. When I checked some other data centers, I saw they had our site back at the top for the same keyword with the dates for both sites showing as “10 Jan 2007”.
So, does a more recent date mean a specific data center is using more recent index data – or is the fact that the site’s home page is missing be something I should be concerned about?
Well, current mess with google updates it’s really weird.
Thanks for explain some things!
TBPR showing odd numbers during an update is not uncommon, as others have said, do not worry much about it until the update has finished rolling out.
Matt: The new data pushes have been showing interesting variations in rankings, but do we have any idea on what variables change and how so over the course of say 4-5 different pushes? In other words, what measures are being modified continually, and for what purposes?
These answers may give some a better understanding of the flux that they are seeing every few days.
@Socks Manly:
Check your Yahoo mail bro….
@Matt:
Excellent post man. Very informative and quality! However I would like to ask one question regarding wordpress (or any other) posts’ feeds. I mean, I got all feed pages of my blog with around 200 entires into supplemental. What’s the reason of putting feeds into supp. index and why even index those when you know the content is exactly the same as on the original page, just with different schema.
Should I be concerned about this and block all feed/ pages in robots.txt or just leave them as they are?
Another quick question….do pages go to supp. index if they are not being crawled for a long time (what’s is causing that?) ? From what I can read pages hit the supp. index even though they’re not contain duplicate content. Am I right?
Thanks a lot for your time!
Manca
Thanks for the detailed information, Matt. I run my client’s reports over the weekend so we’ll see just how drastic the changes affect them. But atleast I’ll be able to explain to them what happened and why.
Nice post!
How can one tell which index a site and/or page is in?
Thanks.
Big Daddy, I don’t see any special reason why that would hurt off-hand. Bear in mind that lots of cross-linking all over the place may look bad though. In general, if you think you may have had a penalty and you think the site is clean now, I’d follow the directions here:
http://www.google.com/support/webmasters/bin/answer.py?answer=35843
Tom Churm, sometimes it takes a while for the supplemental results to refresh and register that 301, but it sounds like you’re doing everything right. It doesn’t hurt for those pages to show up (if I user clicks on them, they’ll get the 301 to the final page), so I wouldn’t worry about it much.
Michael Lopez, we have correct PageRank stored for you internally, so my advice is not to worry about this. One thing you can do if you’re concerned is to help us pick the right preferred host (www vs. non-www) for your domain. There’s a setting for that in our webmaster console.
I will give you a heads-up (since you mentioned your site) that in a completely unrelated matter, your site is selling text links and has participated in various SEO contests (an SEO contest where you’re required to place the anchortext “online kasino”? C’mon, Marc.). That doesn’t affect the PageRank shown in the toolbar, but it does affect your site’s ability to flow PageRank and the trust in your site.
Socks Manly, “relevant link trading” can definitely be done to excess. I wouldn’t rely on that as your sole method of gaining links.
Chris Hunt, a page that’s a supplemental result now may well get more traffic in the future than it gets right now. Personally, I am a fan of dropping the label “supplemental results” but there are some valid reasons for it, including debugging for example.
Andy, if your site is over-optimized in the sense that users aren’t getting as much value-add, it may be that our algorithms are repeatedly registering your site as not as helpful as other sites. The best recommendation I can give is the solid white-hat treatment (things like great, compelling original content that users love). Without the specific site, that’s the best advice I can give.
Jean Manco, that tends to happen because we’re storing the toolbar PageRank value under a different url, e.g. http://www.example.com instead of example.com.
Bob, there’s not much more I could offer other than the tips that I gave to Andy a couple paragraphs up.
Tony, a more recent date typically means that that data center has the more recent copy of the page. So I wouldn’t worry about it given what you’ve told me so far.
Manca, I wouldn’t worry about the /feed urls showing up. It’s not hurting anything, so I wouldn’t block them. I don’t believe pages go into the supplemental results just because they haven’t been crawled for a long time. I believe that it’s a separate crawl for the supplemental results.
Keith Ort, there won’t be any traffic changes due to the toolbar PageRank changes. If the traffic is different, it’s likely to be normal everflux or other stuff.
David Dalka, all the supplemental results say “Supplemental result” at the end of the snippet for that url.
Ok, but I have following problem:
1. .com site outside the US (hosted in UK)
2, Google.co.uk – was ok for “Search the Web”, but there was nothing in “Pages from the UK”.
3. I sent “reinclusion request” and described situation.
4. After a week my site was added to the “Pages from the UK” index and was placed on 1st positon for my chosen keyword. – SUPERB!
5. Unfortunately, after about 3 weeks the main(root) page disappeared from “Pages from the UK” index. I sent “reinclusion request” and described what happened again.
6. The main page came back after few days. – SUPERB!
7. Unfortunately again – about 5 days ago main page disappeared from “Pages from the UK” again.
8. I sent “reinclusin request” again.
9. Today main page also disappeared from “Search the Web” index!! :-((
What can I do about that?:-( Why it is happen so often?
Thanks for the update Matt.
When a PageRank export happens, when is the PR snapshot taken? The day of the export, or some prior date?
This has been debated about on various forums and I wonder if you could help us get over this issue.
You know Matt, I think this may become one of your most linked entries like the BidDaddy review you posted a year ago. Maybe if you did more of these comprehensive overviews, you would feel less pressure from people to perform on the SEO side and you’d have more freedom to dwell on the other topics you like so much.
It would be nice to see a Quaterly Google Happenings update from you on this scale, maybe a little slightly more extensive.
Or, if not from you, then maybe from the Official Google Blog or the Webmaster Central blog.
Thanks!
Hi Matt,
Thanks for the post and the comments. Myself and a lot of other webmasters in the forums are expressing concern about this recent update. My problem is that I have recieved page rank on about 10% of my internal pages but nothing on the remaining 90% and more importantly 0 on the domain. Is this normal? Or does it take time for page rank to filter through the ranks.
My main concern is that fact that I have recently taken a couple of smaller property websites off line and 301’ed the domains to the relevant areas of my main site. I thought this would be fine.
Any help is much appreciated.
Thanks
Bob
Matt,
Can you make any comment about the status or future of google directory (odp)? Google’s copy of it is over a year old.
Hi Matt,
nice post,
in your reply to M Lopez, i see you’ve said:
“selling text links… doesn’t affect the PageRank shown in the toolbar, but it does affect your site’s ability to flow PageRank and the trust in your site.”
For a site that has realised the errors of it’s ways, can you clarify what the procedure is to regain it’s ability to “flow PageRank” once the offending links have been removed? Is it to file a re inclusion request?
Thanks Matt!
One more question: If a site has has 92,800 pages indexed and only 700 aren’t supplemental, could that cause rankings to disappear and the site not even rank for its own http://www.domain.com? If not, would this be some sort of penalty?
Enjoy your time off!!
Aloha
We have been found all but 3 links of about 7,5000 links we have are in te Google supplemental database, also our page rank of 6 now shows as 0.
After following the Google Webmaster Rules. We could see that we had a number of domains pointing at the one location which I understand can be classed as page duplication. So after following what we could see we redirected the following domains with a 301 re-direct which is best practice into one domain such as http://www.example.com
example.co.uk
example.com
http://www.example.co.uk
This I feel was our issue and think that http://www.example.com may have been flagged as duplicate content of http://www.example.co.uk so was classed as supplemental results by the new Google supplemental database.
How would we combat this from happening, we have now lost over 10,000 visits a day for the past 3 months, and have always tried to be search engine friendly and good, well written content.
Awesome information, your extra effort here inspires a nation of often confused webmasters trying to do their best, I thank you for your dilligence and patience!!
Very difficult in the adult industry to get a quality link that isn’t reciprocated!! We do get a lot of natural links just because we’re a quality resource, but other big adult sites would AVOID linking to us, because we’re such a great site 🙂
People are talking now that one way links are better, so to work on A->B->C trades, meaning A links to B in return for a C to A link. Since I only have one big site (I know, I’m a radical) and never really like playing these Google games, I’m not concerning myself much with it. Am I stupid to not pay attention to this?
We have over 1,350 quality articles, that are generating about a click a day from G at the moment. One of our competitors is 1st paging virtually every term he targets, on the other hand, for similar reviews. Oh, for a return to the glory days 🙂
One more thing I don’t understand, when I look at my log analysis from 2002-2005 I see a good deal of traffic from google.de, google.it, .be, .co.uk, etc etc. Now in Analytics I just see “google” and we don’t seem to be getting traffic from these worldly googles. What’s the story there?
Seconds on the Google directory too, a refresh would be lovely!
There is a lot of NEW Data Centers where Google itself has PR 0 and where those new DC are not cached on other DC’s
Any explanation for this ?
I am talking about :
64.233.171.107
DC: 64.233.179.19
DC: 64.233.179.100
DC: 64.233.185.80
72.14.209.19
209.85.129.99 etc
Thanks for the info, Matt.
Just last night I found a bunch of cached pages for one of my sites with early August 2005 dates on ’em. Hadn’t seen any that old for a while, so I was a little surprised.
I have to assume something is going on – on that same site, all but the first five entries are listed as supplemental – not surprisingly; I just redesigned the site and I don’t think it’s been completely indexed yet. The weird thing is that the first four pages are always the same, but the FIFTH page is different every couple hours. The site is database-driven, so all the pages are dynamic, but it just seemed kind of odd.
Is it normal during all this moving and shifting around for things to completely drop out and pop back up a couple days later? I have another site with maybe 7 pages; it’s been in the index a couple years, and the day before yesterday it completely disappeared out of the index – the site command returned no pages. Then yesterday one page showed up – the plaintext sitemap. Last night four pages showed up, two of which were supplemental. Today there’s two pages – but no home page. Meanwhile, webmaster tools doesn’t report any problems with it (although there’s definitely been some issues with webmaster tools)
Just wondering – seems like every time I look, I see something different.
(opps, I meant August 2006, not August 2005 on those cache dates)
Matt:
Your statement ‘…In general, if you think you may have had a penalty and you think the site is clean now, I’d follow the directions here:
http://www.google.com/support/webmasters/bin/answer.py?answer=35843…”
In particular, what if your site is clean, have filed 3-4 reinclusion requests and your site is still penalized? What do you do then?
I’ve been penalized 56+ weeks now, have removed all the programming work the outside programmers put up that evidently triggered the penalty, and have added new content-rich pages. As far as I know, the site is squeaky clean. Google sure crawls it aggressively but I can’t get my 2005 rankings back.
But a search for hotelmotelnow.com in any of the data centers comes up from #32-#45. That is weird and indicates some form of penalty doesn’t it?
Is that any way to treat a fellow UK Wildcat?
blogger.com
digg.com
myspace.com
All these established sites lost their pagerank do you have any reason for it. Its a penalty or just a glitch??? Similiarly lots of well established sites lost pagerank can you please comment MATT,
Hey Matt,
Thanks for a good answer, however I am still not clear about the real cause of pages being flagged as supplemental. I sure have many internal links and I believe all my pages are accessible to Gbot (at least there is a sitemap on the site). I am speaking about a blog in this example, and as you know blogs contain archives and categories where posts that belong to certain category are being listed in that category, showed on main page and also put in archives. The post itself has its permanent page where it’s displayed as well. I am not sure how google thread those? Are they marked as supplementals just because we’ve actually got the same content on couple different places?
Having that in mind I did block category and archive pages from being indexed (meta noindex) so they’d be only available on the main page and on the posts’ pages. Did 301 all across the site (I even setup 301 from post to post/ as well) so there shouldn’t be ANY duplicated content at all on the site….. But now, when I checked site: operator it showed bunch of supplementals (besides those feed/ urls) that are just original posts.
My question is: What is causing the page (url) to become a supplemental result when having all listed above done?
Thanks a ton!
Manca
Hi Matt,
Can you comment all those WebmasterWorld threads please (the last thread is herre http://www.webmasterworld.com/google/3216506.htm).
A lot of webmasters lost 50-100% of their Google traffic for no reason in December.
I have website with 7 subdomains. The strange thing is only 5 of 7 subdomains have lost 99.99% of Google traffic, but the traffic of other 2 subdomains is stable. My income crashed. Result of years of building and promotion is 5-10% of my usual income, thinking about finding “real job” now. 🙁 Same with some other webmasters who have just few websites.
Looks like a bug for me and for some other webmasters. Different pages have no Google cache, other ones have. When I search http://www.mydomain.com, I can’t see my main page in Google SERPs etc. Strange site: results…
Subdomains are crosslinked (each page has links to all 7 subdomains), have similar 5-10 words signature (duplicate content).
But I think crosslinked pages and signature is no problem, because all subdomains have same structure, same design, same promotion methods etc. Same situation with other webmasters’ websites.
Can you answer to all of us please. Is this temporary bug (some kind of data loss) or it’s some kind of website’s penalty? We just want to understand what’s this.
Thanks.
Hi Matt,
Thank you for taking the time to give advice. I would like to think my site is truly white hat. For example in the last year I have written detailed and original reviews for around 100 cameras. It is the reviews that appear to have the problem. I have also started to add my own video guides. User feedback is excellent. If you can spare a moment or two the site is http://www.cameras.co.uk. It would be really appreciated.
I’m in the same boat as everyone else. Why did so many sites move from PRx to PR0? I had quite a few PR6 sites that are now showing PR0…is that some sort of delay problem or why is that happening?
Heya Matt,
Feels good to first time post here,
I know the niche i work in is hard and not widely accepted
1.
Why does Google say in wmtools : not indexed, when i suspect it is banned?
Is there any change it will be in the index in the future? If not, why doesn’t Google just say you are penalized? (site:command=zero)
2
.Also 2 sites don’t show the index page when doing a regional search in the Netherlands, i thought maybe you liked to know that’snot only restricted to the US
3.
I would like to contact Google, even when i would have to pay…..
cheers
US=UK
New Data Centers where Google itself has PR 0 and where those new DC Any explanation for this ? ummmmmmmmmmmm
Thanks for the supplemental update.
The geo-targeting stuff is one of the only issues that really irks me because I think mostly you (Google) do a great job. But you know Matt, until there is a ‘US only’ option in the primary .com (sorry an mean that non existent .us search page) so that the US users get treated the same as the rest of the world, and you guys publish clear guidelines on how to differentiate real international content from local, and that includes local as in US only, geo-targeting will always be a bit of a mess.
What would be wonderful is if users defaulted to their local countries results but could choose to select ‘The Web’ and would then actually get results from the web meaning its entirety and glory – whichever site from whatever country in whichever language deserves to be highest and not a consistently biased subset of it 🙁
Maybe country targeting options within Sitemaps Vanessa?
Does this mean sites without PR since April will finally get a little green in that bar?
Of course this is assuming that they have enough links and presence to get a PR1 or higher, but it seems there hasn’t been an update for new sites lacking PR completely in quite a while.
Am I right about that? If not, how long on average does it take for a site to get their initial pagerank? Or does it vary site to site greatly based on page strength?
Thanks.
Hi Matt
I believe I had mentioned this before a little while ago in my blog and welcome the opportunity that you ahve now mentioned it here for actioning.
Our main development site is: southbourne.com and for years now we have had no problem with .co.uk and pages from the UK. In the last few months there appears to be a problem with our root page missing from search results for ‘pages from the UK’ at one point quite recently it made an apearance again, but now it has gone.
We are hosted on UK servers (WIndows) for this particular site.
Thanks
Vince
I like Teddie’s idea, but from a slightly different point of view.
Here in The Great White North, registering a .ca is an unnecessarily long process that people get confused by and quite often abandon. (Side note for those of you who have registered .ca domains in the past: I do realize it’s only an extra 6-7 mouse clicks, but to the non-techie it’s like climbing Mount Everest, ESPECIALLY with the CIRA wording of things.) So a lot of companies elect to register a .com as a result, and understandably so.
These same companies also quite often select US-based hosts simply because Canadian hosts aren’t on a par with their US counterparts; the pricing seems to be getting more in line, but the service is about as reliable as a National Enquirer story sometimes. (Anyone who has ever had to get write permissions set on a directory via a Canadian host will know exactly what I’m referring to.)
It would be nice to allow for these webmasters to select their country of origin and be included in SERPs for that country. If you’re worried about people selecting a non-US country of origin just to rank in non-US SERPs, you could use the same IP geotargeting (except in AOL’s case…not really sure what you’d do there.)
Either that, or find the contact addresses listed on each site and use those. You could probably get 99% of them algorithmically just by matching up contact, contact.*, contactus.*, contact-us.*, and any common address patterns.
Anyway, it’s just a thought. Not anything I’d consider to be a stroke of genius that should be implemented 10 minutes ago.
can anyone advise on when will a new site be evaluate and assigned a PR? … would new sites get their PR the same time with the old sites? …
will there be difference in PR ranking for a website (www) and a blog (blogger for example) which does not has the “www” initial? …
would the PR be assign or re-assign on quarterly basis? sorry if my queries sounds too 101 …
cheers …
I agree with Michael Martinez, this has to be one of the most useful posts Matt has ever done. Think of all the incorrect speculation about “supplemental results” put to rest with just a couple of words.
The only problem is that these questions will continue to be asked in areas where incorrect speculation still occurs.
New pages on Google that accurately descibe these things would allow people to reference correct data.
Adam, there’s a pretty rare set of circumstances where this happens. I would give it a few days, and then if it’s still happening, pipe up then and mention your site name on this thread.
Pittbug, it can vary. I think it’s pretty much when this one dude in the crawl/index team decides to run a certain script to export the values. So whenever he presses the enter key, I guess. 🙂
Michael Martinez, I do end up doing these every 2-3 months or so, it seems. I would do it on the official webmaster blog, but it’s always late at night, and by the time it’s done I just want to hit submit and go to bed, so it’s easier to do on my blog. 🙂
Bob Sheth/Stuart @ Vip/Elias Kai/chris/Jonathan, I added an update to my original post. I’d read that and maybe do a re-check of your PageRank now.
Jon Goldberg, that’s not really my area. I know that when the ODP was down for a bit, that kept us from downloading a fresh copy.
vimto kid, exactly right. I’d do a reinclusion request.
Socks Manly, I wouldn’t bother with triangular links or “all one-way links, all the time!” or whatever the link fad du jour is. If your an original content site in the adult industry, why not stop by Tony Comstock’s site, or get to know Chelsea Girl at http://prettydumbthings.typepad.com/ or Violet Blue. Don’t neglect the educational value of getting closer with some of the people in your industry.
Jim Clouse, I’d take your site over to the Google webmaster help group and ask for advice from other users. Expect some tough love though. They may say, “Hey, your entry for Hyatt Regency Lake Las Vegas Resort appears verbatim on 40+ other sites! In fact, you’re not even handling the quoted bullet points correctly on that page. What value are you adding? Why would someone go to your page instead of the other 40 copies of this hotel page?” But that tough love will also give you guidance on what your site could do better.
Manca, the main reason would be if you don’t have enough PageRank coming into your site, or if your link structure is set up so that not much PageRank is getting to those pages that are showing up as supplemental results.
Al, I believe that would be data push that has become more frequent. The advice that I gave to Jim Clouse isn’t bad; we’re looking for sites that provide value-add (original content, good information, other aspects of a site that look great to visitors, etc.).
elgoogdotnl, I think we know that the root page issue can affect any country right now, even if it’s rare. If you think a domain is penalized, I’d try to figure out why (site hacked? hidden text? doorway pages? etc.). Enlist help on the webmaster group if you want some more eyes. When you feel comfortable that the site is within our quality guidelines, then I’d do a reinclusion request.
Vincent Marcello, thanks for mentioning it. If it’s still happening in a few days, this will be a good datapoint to help make sure we get the issue squashed.
Aaron Pratt, glad if it helps. I’ll try to chat with Vanessa after vacation to see if she wants to fold some of this into our web documentation.
And with that, I’m turning off the computer, going to bed, and planning to get ready for a vacation with many books and few emails. 🙂
Hi Matt,
we have a site that, from early December, was in the situation you’ve described where site: operator would show supplemental results ahead of regular results. I can see that this has been fixed and that the site: operator is working consistently again but…
this behaviour of the site: operator was accompanied by the drastric ‘up down’ ‘up down’ swings similar to those decribed by Bob in his post (tho’ just from December for us).
Throughout all these swings, the site: operator showed results for the site. The swings seem to have stabilised now but we’re left in the ‘down’ position :o( These aren’t the small variances in rankings that we’d expect from the more frequent data pushes you’ve mentioned. There seem to be other sites affected similarly too – can you give us some pointers as to where we should be focussing?
cheers for the excellent post,
PS.Enjoy your week off next week.. – I’m sure you’ll have an especially good time if you can relax, happy in the knowledge you’ve replied to this post ;o)
Matt. A big thank you for the time you take to address invidual webmasters. My homepage PR is still showing 0 but I’m not bothered because my site is growing daily, the serps become better and more relevant daily and my incoming links are growing slowly but steadily.
Thanks again. I hope you enjoy your time off and maybe I’ll catch you at SES London.
Well, I’m having a nice work morning with PR jumping around between 3 and 5 and subpages between 0 and 3. Before the export the page had 4s and 5s on the different datacenters so I expected a transition towards 5. Now it seems to point towards a 3. This is too confusing, I need to get some coffee and relax. And I thought there is no more “Google Dance”. This is “Google Headbanging” 😉 Ah, yes, coffee.
Hi Matt,
Great post.
With regard to .com sites hosted outside the US and the root not showing on goggle.co.uk UK only sites, I think I can make your ~7 cases that you know of into ~8. The root also doesn’t appear in google.com – only google.co.uk all the web.
If you get the chance I’d really appreciate if you could look into http://www.evictstop.com and see if this bug applies.
Many thanks
Richard
Thanks for update Matt…
Yeah we got all of those signs – was just wondering what was happening – now we know 😉
All seems preety positive to me – just to summarise then – the higher your pagerank (toolbar or real) the less pages will be in supplemental index. Presumably vice versa also applies – the less PR the more pages will be supplemental?
Does this apply to all websites or can well connected/ optimised sites escape SI irrespective of pr? And (finally) does this mean that pages in SI can/will appear in the main SERPS?
Thanks again.
Let me get this right: you have “got pages from my site in the supplemental results” and to get pages out of ‘supplemental’ you recommend “solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit)”.
Er, your site has over 200,000 quality, relevant inbound links! So how many such links might be enough to get your pages out of ‘supplemental?
Matt, I’m an avid reader of your blog; read it every day, but I never comment. I just try to absorb everything you write, well the info regarding SEO. I do have a few questions for you though.
If you have say 10 pages of your website in the supplemental results does that mean your website will not be ranked for keywords that you are targeting?
Another thing is why do pages get put into the supplemental results? If the pages haven’t been updated in say a year does that mean they are more likely to end up in the supplemental results?
Thanks for your informative blog!!!
Was wondering why internal pages with no links are getting a higher PR than the front page which has all the external links pointing to it ?
Example :
http://www.webscutest.com = PR 3
http://www.webscutest.com/archive/index.php/ = PR 4
Thanks for answer Matt.
There are still few strange things.
1. Not all of my subdomains were affected. Affected and not affected websites have same structure, same promotion. All the same except keywords and website’s theme (one is celebrity photos, another one is car photos). Same PR, same links, everything is the same.
2. Also, top 20 websites in SERPs are websites with dublicate content too. Look at song lyrics websites – they all have duplicate content. Original content is impossible here. Still my website was removed from SERPs (I was #1-#10). Other websites (even few ones who copied my whole DB about year ago) are still there at #1-#10.
3. Some webmasters websites’ have returned their #1-#5 SERPs in few weeks after ‘disaster’. As they say they did not change anything on their websites during those few weeks.
4. Websites with completely unique content, high PR, white hat SEO were removed from index after years of beeing #1-#5.
These are very strange things.
Hi Matt
I may have that bug as described regarding .com and pages from the web etc..
website is http://www.quinn-direct.com, hosted in ireland.
on google.ie when searching site:www.quinn-direct.com pages from ireland, index page doesn’t seem to appear
Dara
Hi Matt
I must have spent hundreds of hours on google in the past 5 months trying to find out why some .com sites have been dropped from search UK listings. So it was great to read your reassuring post the other day. You said ”please leave a comment with the name of your site.” so hope this helps.
So here is that comment!
My site http://www.portraits4pets.com appeared back on UK listings for 2 days on 6-7 Jan then dissappeared and reappeared again on 9-10 Jan. It has now gone yet again.
Until August 27th http://www.portraits4pets.com had consistently ranked at number 1 or 2 on google UK for ”pet portraits”. But round about that time it was moved to a server in the USA, and dissappeared from UK listings. I had it moved straight back when Vanessa Fox kindly suggested that this was the cause of the problem. Since then the site has been intermittantly returning to UK listings and then going again and I definitely have more grey hairs now as a result.
I have been desperately searching for a solution for some time and now wonder could interlinking sites be a problem?
http://www.portraits4pets.com is an gallery of over 1000 of my past portraits, an archive of my work which I can easily update on a daily basis. http://www.sallylogue.co.uk is primarily information about the artist, forms, database, e-commerce etc managed with help from elsewhere. Could it be I’m being penalised for having 2 websites? As far as I can see theres no duplicate content, althought the root pages do now have a similar focus.
Thank you for your time
Sally
Hi Matt,
Just wondering if you could tell us where webmaster tools fits into the pagerank side of things, is the pagerank data there from the toolbar server, from one of the datacentres, or from another snapshot? The reason I ask is that the data in there seldom appears to be the same as in the toolbar, and the page on my site it says has highest pagerank seems to have PR 0 however else I look at it.
I’ve seen a couple of people ask about this around the web before, any chance you or Vanessa could enlighten us?
I am seeing some odd things with the latest data…
For one site, anything not linked to from the home page is losing PR. It uses a strong 301 from non-www to www in the .htaccess. A site: query (either www or not) returns about 5,000 pages, but allinurl:www. only shows 62 (guess how many links on the homepage, win a prize!) while allinurl: (without) shows about 5,000 pages as well. I have not used the canonical url tool on this site, but there does seem to be a real problem with the canonicalization that is beyond my control.
Another thing I picked up on is this:
http://www.google.com/search?q=allinurl:imdb.com/r/
These are 302 redirects that are being crawled and cached… the content is amazon.com’s, however. With many sites going to PR0 in the update (amazon.com seems to be getting fixed with the auxillary push) is this the cause? More on this here…
Yet another problem seems to be with higher PR on inner pages, some pages getting higher PR while there are no incoming links from outside, some with all incoming links pointing to the homepage only. This has been around before, but seemed to resolve itself.
Have there been some changes in the cannonicalization rules, the handling of 30x redirects and such?
I don’t think Webmaster Tools is terribly current – I have 62 sites in there, and NONE of my query stats have updated in over a month.
Thanks for the update Matt, you rock!
Matt:
Thanks so much for your feedback. I sincerely appreciate it. I will definitely take my site over to the help group. I respect tough love. Hey, I want the best site on the web.
With that said, I am confused by some of your remarks and would like clarification when you have the time. After reading your comments, I wonder if your comments were directed at http://hotelmotelnow.com/ or to hotel sites in general. Here is why I say that.
The hotel site you mentioned (http://hotelmotelnow.com/hotels.aspx/10204612/) is not verbatim to 40 other sites. And there are no bullet points on the page at all—never have been—as you state in your comments.
We designed this hotel page to be different than the millions of other hotel pages out there. We took the XML and formatted the page to be different. Accordingly, our 78,000 hotel’s layout is different than the other hotel sites. We want to be different and value-added.
Our original content is on the front end. Our technology is unique and was developed in 2001. We provide visual searching of hotels in large cities by depicting their exact location on city maps via clickable dots. Sorting can be done by price range, hot rate hotels, or chain name. Start with the home page and you will see. No other website does it this way. This is unique, compelling content. This type of searching adds tremendous value (re visitors are not looking for hotels outside their desired area this way. Therefore, it is a tremendous time saver.)
In summary, we have original content (our visual searching), good information (hotel information presented differently for all our 78,000 hotels) and are improving other aspects of the site that will indeed make it look great to visitors.
In just the past 52 days we have 1) changed the landing page of all 78,000 hotels, 2) deleted all duplicated content for those hotels and, 3) improved content of our remaining 4,000 webpages.
Under construction is the redesign of those remaining 4,000 pages, inclusion of user-friendly options (sorting by amenities, pet friendly, etc.), implementation of other XML-enabled features, and other features that will continue to set us apart from the competition.
We want to be as different from the other hotel websites as is humanly possible. Since we all, more or less, pull from the same database being totally different is impossible. However, our front end visual searching is what sets us apart from the competition and has for 6 years. Our back end is looking different and will continue to do so as we work on the site.
Given the changes Google has seen in http://hotelmotelnow.com/ specifically since mid-November and over the past year, Google should be impressed enough to lift the 56+ week penalty. We have shown good faith. Our best is yet to come!
Wow, great entry, but I wonder if this applies to what has happened to my site. I used to be in the top for most of my keywords, and now i don’t register for any of them. I still have a good PR and backlinks. This is discouraging!
Well – I guess the update to the update didnt complete or didnt work – even http://www.mattcutts.com/blog/ is showing a PR0 in my toolbar. My homepage at http://www.vbulletin-faq.com/ went from a PR4 to a PR6 in some datacenters, but not others. Whats disappointing is that some newer sites of mine have no PR anywhere, despite having links and being crawled and indexed.
Matt
Responding to your request for .com / co.uk site delistings of index pages, my site http://www.selfadhesivelabels.com did initially suffer from being delisted in the UK only search, but I thought I solved it with a lang=”en-GB” list in the opening html tag. This has worked for about 6 weeks now, but the new rolloutout has delisted it again…
We are a well established UK site, hosted in the UK, and with a UK focus, and suspect that the new infrastructure isn’t exactly helping us… any comments appreciated.
Thanks
Adrian
Hi Matt,
Thanks for this detailed report,
I like this country-website based system, nice job.
Matt,
I know that this is probably a tired question, but I have heard you speak and write about the fact that SEOs shouldn’t consider PageRank in their optimization efforts, however, here you said:
“the main determinant of whether a url is in our main web index or in the supplemental index is PageRank.”
I can understand this and appreciate it, however, this further reinforces the argument of why doesn’t Google update toolbar PageRank more frequently? (i.e. Pandia’s Xmas Wish)
To which you continued (from quote above), “The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit).”
The building of links goes back to the age old question: “What came first, the chicken or the egg?” I build quality content and make it accessible to my visitors, however, if my pages are in the supplemental results (due to lower PageRank) and an editorial researcher is working on a piece using Google to find information, they won’t find my pages.
I practice only white-hat techniques, but the only true white-hat approach is “build it and they will come.” I would love to hear your thoughts on this and thank you for being such an approachable person, in real-life and across your blog.
Matt,
I have a request. You have stated before that SEOs focus too much on PageRank, but the most recent version of the toolbar only gives us backlinks and such if PageRank is enabled, can this be changed?
Thanks again!
Just an update… My PR is now back to normal. In fact it increased by 1. Matt is right – of course he is. LOL
Matt
Great blog!
Thanks for the latest update – if you get a chance please would you have a quick look at our site – http://www.churchill.com – as it may be impacted by the ”pages in the uk’ problem. The rankings have been jumping all over the place, not sure why.
Thanks very much.
Thanks Matt for your time/trouble in clarifying all this for us.
Above you stated:
“Adam, there’s a pretty rare set of circumstances where this happens. I would give it a few days, and then if it’s still happening, pipe up then and mention your site name on this thread.”
This is is respone to the comment you made:
“An SEO or two has been holding my feet to the fire about root pages of .com’s that are hosted outside the US.”
As of today, I know of at least 3 sites where this is still happening.
Should we just ‘sit tight’ and be a bit more patient? Or is there still a problem?
Hope you enjoyed your hol.
Hi Matt.
You say the update is a snap-shot of the live PR, can you tell us when the snapshot was taken? Is it taken on the date the update is pushed or sometime before hand? My site has been online several months and I have a reason number of backlinks to the site, some from some highish PR pages, but I’m still not see a PR for my site on the toolbar.
Thanks.
Andrew.
We get this issue with http://www.wiisearcher.com
It has sections for UK and USA people looking to find a Wii, so is relevent to both sets of users.
http://www.wiisearcher.com – Hosted in Germany – google.co.uk (UK Only search) thinks it doesn’t exist.
We have the same for a bunch of other similar sites for other products which simply do not show on UK google event though they are half US-centric and half UK-centric. But Wiisearcher.com would be a good start!
This isn’t my site, in fact it’s just a random site I found whilst looking for more sites affected by the disappearing sites in regional indexes problem.
http://www.google.com.au/search?hl=en&q=site%3Awww.balancebodybenches.com&btnG=Google+Search&meta=
Now select “Pages From Australia” and the root of the domain disappears!
Thanks for the update on the geo-location issues. The fix is working for me so far. Any chance of telling us what went wrong?
More examples of UK Web sites appearing in “the web” results but not appearing at all in “pages from the UK” results on google.co.uk are: gsiglasers.com, autodriveoptions.com and vance-miller.com – and an example of a UK site with many pages listed in “the web” results but only two in “pages from the UK” results (which seems the wrong way about if any distinction has to be made at all) is acrlimited.com
Hi Matt – good to see the focus on Google UK results (particularly for .coms) which myself and other forum members have frequently chewed over at WebProWorld.
I did have a related question concerning this: does Google take into account local listings (e.g. Google Maps in UK) in determining a site’s origin. If not, are there plans to do so? (after all, is based on a PIN code address verification by post so should defeat spammers).
I ask this, as many of my own sites (and some of my clients) are hosted with 1and1.co.uk which is one of the UK’s most popular hosting providers but has servers in Germany (causing Google to recognise them as “Seiten aus Deutschland”).
Another thing I notice when comparing site:yourdomain.com results on google.co.uk using “the web” and “pages from the UK” is that as well as fewer pages being listed via “pages from the UK”, the default or Home Page of the site (which typically heads the list using “the web”) is usually missing on the much shorter list that you see using “pages from the UK”. The contact page is usually included on the “pages from the UK” list, as if that is a rule of some kind, but the default/Home Page is not. A typical example is matcreations.com. The site belongs to a UK business and it is hosted in the UK.
Matt 2 weeks ago we submitted our 2nd re-inclusion request after having a “minus 30 penalty” since late October 2005. This re-inclusion request and our site fixes worked and we had the minus 30,40,50 penalty lifted last week which was great obviously.
We assume that our discontinuation of text link advertising on our site helped us get out of this mess.
Now as of today we appear to have dissapeared in the SERP’s again for most searches bar a few major ones which aren’t really traffic generating. Should I submit another re-inclusion request or bide my time. I’m concerned about p!$$!ng Google engineers off with uneccessary re-inclusion requests and at the same time am also concerned with what has now happened.
Hi Matt, this certainly has been a most informative post and run of comments – perhaps by the time you return from your vacation this issue will be resolved but I too am wondering about the page rank of home pages being pr0 while the internal pages have pr3/pr4.
I’m talking about new sites, for example http://www.emailhoaxesurbanlegends.com This is surely the puzzle du jour for many webmasters.
Thanks
Lisa
Hi Matt,
I have a question for you: how come a website like Searchengineland.com (and many other sites) isn’t showing its PR on the toolbar yet?
Still just a matter of days?
Cheers
Hello Matt, I have the problem with the root page disappearing from the UK searches of Google.co..uk. All of my other pages are cached. The root page appeared yesterday for a few hours, but is gone again today. I realise I’m not the only one who faces this issue, but is there anything that can be done to rectify the situation?
Thanks Matt.
hi Matt,
more then 40 DC IPs show the old Pagerank from july2006 PR update.
Example:
64.233.163. /19 /44 /80 /83 / 84/ /99 /104 /107 /115 /133 /184 /189
So now we have DC IPs with
PR from july
PR from october
PR from december
I have got an example of a .com domain hosted in the UK with the homepage not showing up in the http://www.example.com search for “pages from the UK” search.
The site is http://www.columbusdt.com
I also have two more .com domains with the same problem. Any help or input would be much appreciated 🙂
Sorry I meant site:www.example.com search!
RE: sem4u Said,
January 15, 2007 @ 1:15 am
I have got an example of a .com domain hosted in the UK with the homepage not showing up in the “pages from the UK” search.
The site is http://www.columbusdt.com
The site belongs to a USA-based business with a USA street address… so its pages are not “pages from the UK”. In this case, the UK filter appears to be doing exactly the job that you’d expect it to do.
Well we”re 5 days later and here’s another exemple of a .com with a root missing in local search.
http://www.madeira.vakantie.com is hosted in Belgium.
A site: search on any worldwide google (.com , .be, .nl etc.) will show 48 pages.
Switching to google.be and turning on ‘sites from Belgium only’ will show 47 pages with the index page being the one that’s missing.
oops, should be http://www.madeira-vakantie.com
The assumption on missing homepages in uk only searches is that this happens for sites hosted outside the uk. Believe me this is not the only case. I have a 5 year old .com site hosted in the uk that up until a few months ago ranked very well for all key search terms in “uk only ” pages. Every 2 months or so the home page disappears from Google indexing for “uk results only”. Still there in the .co.uk index but not in uk only, its out again now after the 11 Jan cache.
I waited 5 days like advised but its just disappeared again- doesn’t look like the fix is working does it?
Its not just non UK hosted sites – so whats causing it?
Sorry, the site url is http://www.natco-online.com
*** if you still see the behavior after 4-5 days ***
Yes, for http://www.warburtech.com/ the root page is missing in “pages from the UK” searches; also about 35 pages from the site completely dropped out of the index many months ago, and another dozen more dropped out several months ago. All alternative domains, and non-www URLs, have had a site-wide 301 redirect to the .com version, for several years.
>> This page is showing a 0 pr at the moment
If the page is, say, domain.com/index.html then DO check to see what PR each of these alternative URLs have:
domain.com/
http://www.domain.com/
http://www.domain.com/index.html
It is possible that Google favours a DIFFERENT URL to the one that you tried to get listed.
A 301 redirect to the canonical URL from each and all of the alternatives will clear the problem up in just a few months.
I usually pick http://www.domain.com/ as the one to be listed, and for folders, they would be http://www.domain.com/foldername/ instead (additionally notice the trailing “/” on the URL.
g1smd – sorry were you talking about my site? PR is 5 not 0 ?
Hi G1 🙂
Long time no see.
It’s not a canonical problem this time.
In my case, i’ve a 301 redirect in place from non-www to www version, and all links point to the root without index.html
Everything else is all white hat as usual 🙂
The site is too young to have any PR so i’m not stressing over this 🙂
Matt suggested to give it 4-5 days for Google to fix this bug and asked exemples if we could still see this behaviour after that period.
I added mine just to show that non-English sites are affected too.
Roy Dale – OK bad example there but I have two sites that don’t have contact details, as they are information sites, which suffer the same ‘no homepage showing up’ problem.
Just an update – the 3 .com sites I talked about 3 days ago where the homepage was not showing up in the “pages from the UK” search.” The homepages are still not showing up so it would appear there is still a problem.
Hi Matt,
http://www.masainternational.com seems to be having the issue where the index page does not show up in the Pages from the UK site: search.
Hello Matt,
Thanks for this very informative topic. I was just checking my http://www.webmasterslookup.com (hosted in the Netherlands) at http://www.google.co.uk. I think te problem of “pages from the UK” is still there. Also I stay puzzled by the PageRank concept. Except for the IP-ranges you mentioned the PR of Webmasterslookup is 5. Sometimes however (mostly in the mornings ECT) it is 4 in the Google bar….
Is there separated server that serves the Google bar??
Thanks again for this Topic,
Wim
Hi Matt, any idea what percentage of google.co.uk users use the pages from the UK feature??
Cheers
Nial
Hello my first ever blog!
http://www.floorheatingonline.com/ is hosted in the UK and whilst it does appears in “Search the web” is nowhere when search pages from the UK”. Have you any information as to when this issue will be fixed and/or do I need to do anything?
Thankyou.
Matt,
Seems that your RSS feed doesn’t work anymore. Can you help my reader please? 🙂
Hey Matt,
Hope your enjoying your holidays. Get stuck into some of those books that you never get a chance to read 🙂
I think I might take a holiday from google until you get back as well..seems to be a good time of year for it.
cheers mate 🙂
Hi Matt, i run the SEO for http://www.mobileshop.com, its a uk based and targeted web site, i have been hit by the latest update, and do not appear when i search the UK, the .com search is fine, can you have a look and see if you can get an update for me please
Matt,
I am in the same boat as hotelmotelnow.com (Socks Manly).
My site has been under a penalty for close to a year, also the same phenomenon that my domain search places me on pages 4+ along with all previously well ranked keyword searches. I have posted on the google boards and did get the “tough love”. I implemented all suggestions. Yes I have affiliate links however my site includes reviews, articles, multimedia downloads, etc. and more.
What do you recommend?
Thanks for reading this and taking the time.
Nate
CORRECTION.. I meant to write my site is the same as Jim Clouse
NOT SOCKS MANLY
Hi Matt
Just wondering if the udate is finished or still ongoing as i have a couple of sites that were previously PR0 but now all the internal pages have been given PR3 but home page on both is still 0. Is this normal. Whats going on?
I went to a couple of the forums seeking answers and quite a few people are experiencing the same thing!
Please help
Many thanks
Google PR is a great tool, but as I have seen before it is not the answer all to being ranked. It seems to be one of our last focuses when kicking our competitor’s asses, we grind out the rankings and when the dust clears check the PR. Are we the only ones kicking ass and taking names when we get around to it? Or is PR a real hook for all you SEO’ers out there.
update:
it looks like the fix is starting to show.
The homepage from the site i mentioned earlier is back in the site: searches for local pages at google.be
Thank you Google, and thank you Matt.
It was a fun thread 🙂
And most likely in the next few days we may see remarks from people who may think that their site is hosted in a specific country while it’s in fact hosted in another due to nameservers with tld’s belonging to another.
eg. .com sites hosted on a uk based server which uses .de nameservers
Matt, maybe you should open a thread someday about how the country of a site is determined.
Still i wonder what caused this temporary bug. Preparing for the upcoming google.eu ? Any ideas/suggestions on how this one will deal with local searches ?
Matt,
strange things happening here: I am the webmaster of a multi-lingual website by a friend of mine. The Site itself is available in English, German, French, Spanish and Italian. In October I reconstructed the whole site. The Spanish and Italian translations were not ready at that time, so the site went on-line (around the time of the september PR-export) in English, German and French, with a structure like:
http://www.domain.com/en/...
http://www.domain.com/de/...
http://www.domain.com/fr/...
Two months later the Spanish and Italian versions got published too:
http://www.domain.com/it/...
http://www.domain.com/es/..
The root-page (www.domain.com) had pagerank 4, all the other pages had one of 0 as they didn`t exist before.
Now, after the recent pagerank export the root kept its pagerank of 4. All the subsequently updated pages in /it/ and /es/ now have a pagerank of 4 too. All others (English, French, German) remained with their 0, though almost all inbound links point to the English version or the root.
Now, I am more than confused. Could you please try to explain how something like this can happen?
Thanks in advance!
Matt I know you are a busy man but if you have a second to comment at http://forums.searchenginewatch.com/showthread.php?t=15834 it would be appreciated. 🙂
Hi Matt,
It is very important you should clarify one point we have a long discussion is going on in WMW that there s -950 or end of result penalty. Even those sites were perfectly legal did not do any kind things other than what normal visitor would need even then they got this penalty. Is there any way we can come out from there.
Regards
good research cutts. some of these bastards want to charge 10 grand for simple keyword typing so im doing it my dam self. Later.
Matt, Thanks for the updates, love your blog. Quick question about the most recent infrastructure updates. I have noticed my backlinks (www.sharkhandicapping.com) go from 131 in August, to 35 in October to 131 in Nov to 86 in Dec and now 36 in january. During this time, we have been adding several back links via advertising and have eliminated non-relevant outgoing links. Woudl the recent updates that have affected PR also affect the back links? Also, I have noticed the links that were removed were primarily reciprocal links. Are reciprocals penalized? Thanks.
I just realized our sites have been experiencing an increase in traffic from simply practicing “white hat” techniques all along. We have a few clients who ask me the same “Google” questions that I ask myself everyday. After watching a video where someone interviewed someone from Google who (albeit kinda being cocky) tried to remind us that the goal is to promote (ethically) to get traffic, I had a revelation. 1) Do “good things” to promote your site. 2) Don’t try to “compete” with Google, etc. 3) repeat number 1. 4) repeat number 1 again. And guess what? If you’re really a legit site, trying to promote info, entertainment or e-commerce, if you do it with “good intentions,” and WORK HARD, you have nothing to worry about. The same practices have held-up (for the most part) for the last zillion years. If you have good intentions, that benefit the majority, you’ll be rewarded. If you try to “trick” your customers, Google, your lenders, vendors, or employees, you’ll eventually be “discovered,” and left in the dust. (That is, unless, of course, you live in a “bad neighborhood governing country etc.)
Anyway, thanks for your time,
Chris Anderson
A “Tired of trying to figure-out what your’re doing entrepreneur.
Matt,
I’ve got of UK hosted dot.coms that have been excluded from the Google.co.uk “pages from the UK” index.
First is http://www.simplyclicks.com . This was excluded today.
Second is the http://www.thegeorgeinrye.com . This has been excluded since early December 2006.
Both sites are very much UK businesses. And both hosted in UK. I would be very grateful ff you could help get them back in the index please.
Great entry and thanks for the update. Over the past week or two I’m seeing pages slip in and out of the supplemental index which I don’t understand – can you shed any light?
Both the sites mentioned by David Burdon – simplyclicks.com and thegeorgeinrye.com ARE listed in “pages from the UK” results at google.co.uk. The same is true for mobileshop.com, which Steve Ash asked about and floorheatingonline.com, which “Iaragood” asked about, although there is still the mystery about why the Home Page/default page or ‘root page’, which appears in “the web” results should be missing from the “pages from the UK” results.
Meanwhile, nothing has changed for the UK sites: autodriveoptions.com and vance-miller.com; they appear in “the web” results, but not at all in “pages from the UK” results. And the two UK sites: acrlimited.com and matcreations.com still have many more pages listed in “the web” results than they have in “pages from the UK” results.
Matt
As mentioned above, a site: search on http://www.natco-online.com = 351 results for “the web” and 350 for ” pages from the UK” – guess whats missing? yep, the home page – for the 3rd time in 4 months.
Its a UK site hosted on a UK server – can someone there please look into this?
Thanks for the post Matt and the comments from others that are in the same position as me – have a .com UK based site that shows nothing in UK only results (it is hosted in Germany with 1and1 btw).
I also own the .co.uk version of the site http://www.mobilechecker.com and 301 this as well as the non www versions of my domains and have got nothing in the UK only index.
Should I submit a reinclusion request for the domain if that helps / I don’t hear anything?
Hi Matt,
I am also experiencing the same problem, my internal pages PR boost from 0 to 5 but home page PR is still 0.
Please update when will the update finish.
Hi Matt,
First time here, it’s a great, rich-of-information blog from the right person!, salute!
I have question about my site that’s hosted in Singapore with .com domain. It’s nowhere displayed in google.com nor google.com.sg (both ‘all’ and ‘only page from singapore’) for site: command, and it’s been around 2 weeks, HOWEVER, the site is well indexed, i can see my pages are cached when i submit allinurl: command.
The caches are constantly and frequently updated.
I am perplexed, why doesn’t the site: command works for my site? I believe it’s affecting search result for my target visitor (Singapore).
Thanks in advance.
PS: Happy Anniversary!
Matt – would be interested in your thoughts on my previous comment about trusting the address verification process involved in Google Maps as proof of a UK-resident homepage.
At the risk of throwing further confusion onto the issues, I also have a further question for you; why does “pages from the UK” use two operators in results (cr=countryUK and cr=countryGB). Why not just one? The UK operator seems to present mainly .co.uk results and is cleaner. What determines if a site ranks well on GB? UK?
A guide for UK Webmasters would be great. Or Belgian or German or Canadian – we all get confused!
Rgds,
D.
Well I went away for a while but it is back again, was there a rollback?
The homepage for http://www.newbuildinspections.com has gone from google.co.uk with the uk filter applied. Site hosted in the UK.
I have other examples if it helps
Matt that looks a pretty good response from this side of the pond with the same old problem. I really think you need to document and publish a method for setting a primary geographic target, then let everyone compete equally within “the web”. Maybe it could be as simple as using the lang= attribute at page level which is a pretty clean specification and copes with language variations within countries as well. It is widely used already and would also fix the issue you guys have with not be able to set location on a page and directory basis, as opposed to domain level. I always get people to set it properly anyway in case it is already one of your 100+ indicators but would help many people if you gave some clear guidance 🙂
Matt that looks like a pretty good response from this side of the pond with the same old problem. I really think you need to document and publish a method for setting a primary geographic target, then let everyone compete equally within “the web”. Maybe it could be as simple as using the lang= attribute at page level which is a pretty clean specification and copes with language variations within countries as well. It is widely used already and would also fix the issue you guys have with not be able to set location on a page and directory basis, as opposed to domain level. I always get people to set it properly anyway in case it is already one of your 100+ indicators but would help many people if you gave some clear guidance 🙂
Apologies hit the button twice, please delete this and one of the previous posts.
It looks like this issue has now been fixed: .com homepages reappear for Google UK searches
It’s been more than 5 days since your post, so I presume that the final fix you mention must be in place.
Yet, since yesterday I’ve seen 2 new domains displaying this error.
1 of them is http://www.w1office.com
Matt
We have had a continual problem with the index page of our site: http://www.business-sale.com not appearing in the UK serch of Google.co.uk
HAs been over a week now. I thought this issue was fixed?
en-uk
en-us
etc
Meant en-gb
ISO 639-1 and 639-2 language codes:
http://www.blackwellpublishing.com/xml/dtds/4-0/help/isolanguagecodes.htm
ISO 3166 country codes:
http://www.davros.org/misc/iso3166.html
Matt,
Just to confirm this .co.uk issue is still happening. I have 6 domains here suffering with lack of presence of pages from .com domains when searched with ” pages from the UK”.
Regards
pages from the UK”, the root page does not appear but the rest of your pages do, then this paragraph applies to you. I’d wait 4-5 days to let this second change percolate completely into our index, and if you still see the behavior after 4-5 days, please leave a comment with the name of your site
http://www.hotel-assist.com
Matt Cutts [url=http://www.mattcutts.com/blog/]said[/url] :
** Jon Goldberg, that’s not really my area. I know that when the ODP was down for a bit, that kept us from downloading a fresh copy.
As far as I know http://rdf.dmoz.org/rdf/ hasn’t been down in years, and new archive copies were flowing at least every few weeks at http://rdf.dmoz.org/rdf/archive/ until the main server crashed a few months ago….
My home page went missing from google.com.au (Pages from Australia) on Wednesday.
http://www.diamondsandopals.com
Hello Matt,
I have a question. Once an export has begun how long does the full process take to finalize? Most of the sites I am working with have had no change but I have noticed big changes in “future pagerank” information when checking. My newest personal site was said to be at pagerank 3 after the update but has ended up at 0 and is now said to be at pagerank 1 for the next update.
After seeing this I checked several sites with 2 different “future pagerank” tools and noticed a large shift in results. Some were higher some were lower. Can you give some clarification as to how these tools work in relation to actual Google pagerank?
Thanks for being so helpful
Mich D Feinx
Hi Matt
My .com site http://www.bombaybedspread.com is UK hosted and the homepage does not appear with the “pages from UK only” filter applied.
thanks
Raj
Keniki, actually the lang attribute copes perfectly well with all of the possibilities you describe, because country/region is seperated from language, and the domain extension where it is defined as non regionally specific .com .net .org (gTLDs) becomes irrelevent. Where the domain extension is regional it could possibly override the lang attribute region value or visa versa when they conflict, but how the search engines decide to weight that in their algorithms is another debate.
Additionally language specific indexes are separate from regional indexes, so as long as the language code is set properly you’ll get the richest search experience possible for all the languages Google caters to.
http://www.rfc-editor.org/rfc/rfc4646.txt
Hi Matt,
For a few days the issue went away, but it seems to be back again…
good news………..
http://www.portraits4pets.com‘s root page has been back on UK listings for 7 days, fantastic
bad news………….
I daren’t update my root page…incase the changes trigger the ”problem” and the index page is dropped yet again. (seems to happen after I’ve updated links to recent work?) Am I paranoid?
by the way another webmaster on the google webmaster forum has come to the same conclusion today.
Sally
I have come to the same conclusion too.
Hi Matt –
I appreciate that you’ve made yourself so available to the public.
“I’ve got pages from my site in the supplemental results, for example.” It’s great to know that a Google god like you has pages in “supplemental hell.”
Perhaps this is not the relevant place to post this, but this issue appears to be the most pressing of all, and it’s happening quite a bit, from what I can tell, so perhaps you’ll do a blog about it?
One of my sites – with original and helpful content – was completely “lost” from Google’s index yesterday, and I’ve been discussing this disconcerting issue in the Google Webmasters group as well as on an SEO forum. Some at the group think I’ve made a “grey hat” error, while the SEO forum believes it’s just a glitch that has happened to many. My Google webmaster central account cites no violation that I can find. Google does admit that sites fall out of the index completely at times, does it not, when it remarks: “If your site is well-linked from others on the web, it’s likely that we’ll add it again during our next crawl.” I think this is a VERY important issue, because people all over the place are having panic attacks based on their sites mysteriously disappearing with no discernible reason. Is it possible to have a clearer and most comprehensive statement than just that one simple sentence above? I found that here, among remarks concerning just falling down in the SERPS, but this one sentence appears to be discussing being removed completely from the index.
http://www.google.com/support/webmasters/bin/answer.py?answer=34443&hl=en
I would also like to find out more about this possible “grey hat” technique I was using, as I firmly believe it is providing a service to people with disabilities. MSN and Yahoo make no prejudice against it; in fact, it would appear to complement their procedures. Any chance I could get someone’s ear at Google about this issue?
Thank you again for making yourself available. I am certain that you are exhausted from the millions of “adoring fans” who are jockeying for your attention.
Jamie
Hi Matt,
re the problem you mentioned about having an index page missing in google.co.uk for .com domains hosted outside US, I recently get exactly those symptoms for http://www.royalty-free.tv.
All the best,
Gavin
Hi,
Just some feedback about your Pages in the UK Post. We have a domain name http://www.thechildrensfurniturecompany.com. for “children’s furniture” on google.co.uk All of the web it is top but then disappears for Pages from the UK. Is this an example of the same problem?
Thanks for your help,
Simon
Matt, what you describe about the lose of the home page of .com sites in Google UK pages, while still being shown correctly in the WW results is still happening to me. Site is http://www.companypartners.com .
Site is hosted and registered in the UK.
It appears and disappears with regularity and has been doing so for a year. The fix you mention above may have helped last week – because it was there – but this week it has gone again.
Please help while I have some hair left.
Hi Matt,
just saw your comment about the dissapearing .com root page problem on local (in this case .fr) searches.
The site is hosted in France, but the root page doesnt appear here :
http://www.google.fr/search?hl=fr&q=site%3Aoubouffer.com&btnG=Rechercher&meta=cr%3DcountryFR
Even though you can see it here:
http://www.google.fr/search?hl=fr&q=site%3Aoubouffer.com&btnG=Rechercher&meta=lr%3Dlang_fr
Is this a problem on your side or am I doing something wrong ?
thks
Gareth
Matt
Index page gone again today on “UK search” filter after reappearing around the time of your post on the subject. The site: theweddingsinger.uk.com.
Still a fair few affected around in the forums, any updates you can give us would be much appreciated.
Thanks
John
Hi Matt
I have at least one UK .com with the homepage problem. Would rather not post my clients URL, but would like to pass it on to your team. Could you drop me an email.
Thanks
Hi Matt,
Still not on the UK search.
Spoke with Adam in the forum a couple of weeks ago and then was back on uk search for 10 days and then gone!.
Not showing on UK google search as of now 22.1.07.
My other site http://www.awkspc.com does not show up either unless you search from the WWW.
If you visit the Google webmaster forum you will find people spending there life trying to work out what the phenomenum is all about.Hope you can solve this so we can all get back to work!
Thanks
Awk
a list
of UK .com sites with missing root pages on google UK (searchUK)
please add your sites to:
http://groups.google.com/group/Google_Webmaster_Help-Indexing/t/6afd76fc860df703
Searching for demonkar in google.co.uk results obtained from site when searched the web but only links or where the site is mentioned obtained when searching for pages in uk.Is this because site is .com and not .co.uk?The site is hosted by an American company.Thankyou.
Keniki: are you using HTML characters such as < and > ? That might be why it’s getting blocked.
(Either that, or Tompkins made a swear.)
Hi Matt
Just returning to this thread again and thansk for the above note stating:
— Matt Said (January 12, 2007 @ 1:48 am)
“Vincent Marcello, thanks for mentioning it. If it’s still happening in a few days, this will be a good datapoint to help make sure we get the issue squashed.” —
Yes, unfortunately the ‘glitch’ is still there and the sites root index for (southbourne.com) is still not appearing in the ‘Pages from the UK’ I have checked this daily for the last 11 days and still no appearance and came back to report it.
Thanks
Vince
Hi Matt
The root pages problem is happening on two of my sites, both .coms hosted in the UK:
http://www.sportsinjurybulletin.com
http://www.teachingexpertise.com
Traffic has plummeted on Sports Injury Bulletin over the last week; Teaching Expertise is brand new and traffic is stable when it should be growing.
Greetings Matt – OK we were one of the ‘guilty parties’ on the UK issue – here’s where the post stands at the moment. If anyone cares to join in please do.
http://www.webproworld.com/viewtopic.php?t=71505
Any chance of some famous Matt Cutts input?
Many thanks as always David – ctabuk
How many times Supplemental Index will be updated in a year ?
Is there any chance for a supplemental page to be moved into regular index ?
Hi Matt
our site suffers from the home page not being shown when you do a search for pagesa from UK. I have been waiting patiently hoping the changes being made will resolve it but no luck so far. Website address http://www.keyzone.com (try seraching for kvm switch or server cabinet)
Appreciate you help
Hi Keniki
Please can you limit this to a condensed version? I only ask because there appears now to be a lot of noise in this thread and this subject has now become a serious issue for some of us. If you limit it to the issue at hand rather than multiple postings every time you have a thought, this thread may be actioned faster.
It’s not my blog and my business to tell advise you, but past experience here has taught me that if there is a modicum of exercised patience then we get this subject dealt with and some good feedback.
I for one am very concerned about the issue of the supplemental pages and how they are dealing with our .com extension in the UK, I would rather see answers and updates to this subject than lot’s of ‘cyber-space junk’
Matt,
http://www.thegeorgeinrye.com has re-entered the Google.co.uk “pages from the UK” index. Many thanks.
You’ll see a tongue in cheek post on the issue at :
http://www.webproworld.com/viewtopic.php?t=71639
As you’ll notice, my own site comes up as UK based on Google maps but not at Google.co.uk.
Thanks for the info, Matt
Where can I fine past google PR updated dates .
My site suddenly jumped to PR2 to PR3 , but don’t know when it happens.?
Also is there a place to find next date of google PR update happens?
Help! All I know is my site http://www.drinksgear.com is registered for worldwide results, but not the uk. Having a major detrimental effect on us so any advice would be gratefully recieved…..have emailed google a couple of times, but no response (is this normal?). I’d appreciate if anyone could point me in the direction of a working email address?
Matt
Google.co.uk (pages from the UK) search: [ Taj Mahal ]
Our site “Explore the Taj Mahal” at home page ( wwwtaj-mahalnet ) has been switching between showing our home page, and one of the inner pages. 95% the inner page shows over the past few weeks… All other national Googles that I’ve tested are showing our home page.
Note: This is for the .NET tld, not the .COM
All our other sites are showing the correct home pages, just the Taj site is wrong.
All our sites are hosted in the UK.
Regards,
William Donelson
Hi Matt
The root page syndrome has also struck http://www.laybrook.com , a client’s website. The home page went awol on UK-specific searches around the time of your recent infrastructure status post.
Could it be because we now have some backlinks on US-based directories?
Anyway, thanks for inviting comments on the issue. I certainly feel like a valued Google customer just by writing this post.
Diarmuid
*** if you still see the behavior after 4-5 days ***
Yes, it is now some 14 days later, and http://www.warburtech.com/ still has the root page missing in “pages from the UK” searches.
I previously mentioned that about 35 pages from the site completely dropped out of the index many months ago, and another dozen more dropped out several months ago. All alternative domains, and non-www URLs, have had a site-wide 301 redirect to the .com version, for several years.
I see that 3 or 4 other pages have reappeared in the index in the last few weeks, but for “Pages from the UK” searches, the root page is still not there.
hi guys,
does this problem affect other google rankings on the normal search results? one of my sites (www.dinner2go.co.uk) used to rank between 1-4 (page 1 for most keywords/phrases), but i noticed it dropped to page 4/page 5 from sometime around late dec/early jan 2007…
i’ve heard rumors about a problem with the new google algorithm, but i’m not sure whether or not that’s true. if there is an issue, what can i do about getting my site back up in the results in the meantime while it’s fixed? i’m losing a lot of business as a result of this drop 🙁
Sohail A. Rahim
Hi Matt
Just had a look through the industry I’m working in and found 3 sites that have this problem (searching on google UK and checking the UK only box – all these are UK sites)
http://www.easycar.com
http://www.carhire3000.com
http://www.1car1.com
this was just the first few I looked at so I think the problem is quite widespread
I am still a lot of issues with .com index pages dropping out of the UK specific results through the index page not showing. What is the latest Matt as I have held on for two weeks following your instructions not to worry for 4/5 days but nothing appears to be happening.
Is their any time schedule of when this problem is going to be fixed once and for all? I can only assume it is proving more of a problem then you guys first imagined and that there are knock-on effects to the rest of the crawl.
What is the current status? Help!
Matt – any news on the status of —
“Google.co.uk (pages from the UK) search: [ Taj Mahal ]”
not listing home page for Explore the Taj Mahal
Thanks
William
My website home page is still not showing up in the UK listings for Google. Do you know when this problem is going to be solved?
Hi Matt
Still no movement in the pages from the UK. Any chance we could have an update as to when it will be fixed, is it going to be fixed, is it maybe a shift in policy or other?
In the past I have pretty much left it alone hoping it will work itself out, but still checking on a daily basis to see if it has ‘been fixed’ but nothing yet for (southbourne.com).
This week I decided to get a bit more pro-active and started going over the past history of the domain to see if I could identify any visible (or non-visible) issues that could be affecting the domain and came up with nothing of importance. I compared several other domains I noticed missing in the pages from .co.uk index to see if there was a common factor or thread – again I found nothing.
Looking at incoming links for anything that may have been deemed ‘dodgy’ and found nothing. We looked at content, structure, coding, duplication and did not find anything again (well, nothing that warranted disappearance).
The conclusion I have drawn and reading back through these posts that it is a Google issue. A few months ago I had blogged about this in which I had a reply from Greywolf that I was mistaken and my root index was in pages from the UK. The day he replied it must have been fixed.
A couple of weeks later it had disappeared again. What happened there? was that original fix some kind of rollback? or something else? or was it a proper fix that couldn’t be sustained because maybe it had detrimental effects in other areas?
My above comments are not written in stone just musings and suggestions while we try and make head or tale of this
For those who’s waiting for answer, it may be wise if you read this:
http://www.mattcutts.com/blog/climbing-mt-email/
cheers!
Matt
Just to add my weight to the above comments, my UK based and hosted .com site is still suffering the missing root page penalty from the UK only search.
This is a bug that can only be addressed by your colleagues, any assistance or advice would be appreciated by us all.
thanks and regards
adrian
Matt,
We appreciate you are very busy, but when is the .com issue going to be fixed?
There are some seriously disgruntled UK business owners that believe that Google has no interest in the UK Market. What compounds this more is the the fact that you have gone very quiet on this subject.
It would be great if there was something tangible that we can throw back to clients.
I am seeing more sites dropping out. One business has called me this morning as they have dropped out and with Valentines coming up he is very concerned this will effect business dramatically. When his site dropped out in late November, it seriously affected sales.
An update would be most welcome.
Al the best
Paul
The UK results issue has been going on for many months, there is no regular pattern or link between pages dropping and re-appearing. I have found that updating content can trigger a reinclusion but it is not always the case and so is something to try but not depend on.
I have found no common denominator with being dropped in the first place, some sites that are dropped have had regular updates some are quite stale and have the same issue.
Some come back by themselves but many come back after new content, the problem is that the sites will re-appear and the randomly drop again.
I think that anyone under pressure from their clients should send them a circular mail explaining the issue with links to the subject, there are some excellent threads on this both here, in Webworkshop and in VirtualPromote, this should show them that it is beyond your control.
In the meantime Matt knows this is an issue and I see little point in hammering him on it each day.
Shaun,
I disagree, if Matt is not informed of more instances of sites disappearing then what chance do we have of making sure a fix is put in place sooner rather than later.
I appreciate that Matt is doing all he can to get the issue rectified, but here we are, already in February 2007. How much longer are UK businesses going to suffer?
I know of business owners that have written to Google, but as yet have not received a response back. Matt’s blog is a way of communicating with someone at Google who can provide answers.
Matt himself said to bug him should there be issues following the last fix. If we maintain silence then it is going to take even longer to fix.
Business owners are worried, they only see the black and white, for many their sites have disappeared from a section of Google that would usually provide healthy conversion. My own stats say as much as 50% of Google users in the UK use the filter. What would occur if Google were to suffer a 50% drop in online revenue through no fault of their own?
Hi Paul,
I understand your frustration but lets face it Matt is a clever guy, he knows what the issue is and he knows the extent of it.
That is completely different to what his response can be, this has to be inline with Google protocol.
Matt isnt going to fix the bug himeself and he will rely on techie feedback to give you a response, in the meantime badgering him may reduce the feedback we get.
If my boat had a hole in it then telling me there is a hole in my boat every few minutes would not get it fixed any quicker.
The way I look at it if I was given a hard time after delivering information in good faith from my internal systems I would be reluctant to go on record again for some time.
Hi all,
Just thought I’d add my name to the pertition.
Anything to offer clients would be appreciated, since it’s easier said than done to explain the problem to people who may not be web savvy.
In the last 6 months I’ve had about 5 sites (of about 10/20 UK based ‘.coms’) suffering this problem to varying extents, so it is a significant problem.
Afterthought:
As far as I have noticed so far, only 1 of my sites is still suffering the problem since it was allegedly fixed (in Jan).
Cheers
Hi Matt,
Just letting you know that http://www.awkspcrepairs.com as of today 3/2/07 is showing up now in uk only searches.
Thanks if you had a hand in this. My other site however is not.
Just keeping you updated
Thanks
Awk
Matt,
Our local UK SERPs have now (I hope) stabilised with our homepage for our Explore the Taj Mahal website.
Thanks!
Dear Matt
I made a comment on 11th January in respect to one of our sites which we 301 diverted a number of domains that could have been classed as duplication. The site is still for some unknown reason being classed as Supplemental Results and we have page rankings on of 6 on a number of Google servers but most a rank of 0. I thought that this may have been resolved over the past month but see we are still in the same issue. Would you be able to advice on any reasons why this may still be the case?
Many Thanks
Stuart
Hi Matt,
Just a note to say some client sites have reappeared in Google UK Specific, but others have now disappeared 🙁
Is this coincidence? Is there a crawl underway at this time?
It’s a bit like the Ying/Yang effect.
I know us Brits keep harping on at you, but you love it really!
Well I am not holding my breath but we have have now finally re-appeared (www. southbourne.com) back in the local results. Thanks for all help in this blog and hopefully the current fix is here to stay.
Hi keniki
If you have a issues with me, please feel welcome to PM me and we’ll sort it out. But I am not taking it public here on another blog.
Hi Keniki
You are welcome 😉 at the same time I don’t want to take this thread up with a tete-a-tete between us. But feel welcome to drop by for a chat at the other place. 🙂
*** if you still see the behavior after 4-5 days ***
Just to update you Matt,
I confirm this problem still exists on 3 sites I deal with as of today.
Thanks
I still see .co.uk with root domain problems in UK searches, but today a lot of sites are showing extra pages indexed.
Additionally, the recent “1-2 of 120 pages” problem seems to be fixed – but with a twist. With filter=1 selected I now see “1 to 30 of 150” but all results after the first are supplemental. With filter=0, I get 150 results but they are not supplemental.
Thingsa are in the works. I’ll look again in a few days and see how it settles.
*** if you still see the behavior after 4-5 days ***
Hello again Matt,
In case you are not aware, the problem still exists and its now 4-5 weeks. Is there any chance of an update on this problem?
Thanks.
No change. Just looked again at a .com site that is UK-based (UK hosted) and belongs to a UK company (location, street address etc in plain view text on the site) but is poorly represented in ‘pages from the UK’ results and has its default index.html page excluded from the UK results – but is well represented in “the web” results, where its default page is included. Also, the site gets four times as many pages listed in “the web” results compared with “the web” results.
Correction: Also, the site gets four times as many pages listed in “the web” results compared with “pages from the UK” results.
Maybe this thread is not being followed anymore – but the problem of ‘main page disappeared from “Pages from the UK” ‘ still persisits. And its been 7 weeks now, not the 4-5 days first postulated.
Is there any way to get a response from Google on this issue which is affecting huge numbers of UK sites?
Thanks
Matt,
I disagree with your opinion that supplemental results are not something to be concerned with. My site (http://www.projectresponder.com) was hit by the supplemental results “bug” back when Big Daddy was released and sending a note to sesnyc06@gmail.com seemed to get the site back on track. Now, for whatever reason, one year later, almost to the day, my site is back in supplemental pergatory. My site has always ranked well, not always number one, but in the top five for a lot of searches because of the content, popularity and age of the site. Now, most all of my pages are supplemental and all the rankings are gone. So as far as I’m concerned, being stuck in supplemental is the same as not being in the index. If a page is unable to be found, it’s no different than the page not existing.
I believe that supplemental = nowhere to be found, we don’t like you
Regards,
Michael
I have to agree with Michael here. Being in the Supplemental index is not something that we shouldn’t worry about. One of my websites has almost all pages in the supplemental index which has pretty much killed it. Previously ranking in the top 10 for some pretty good keywords and now it is nowhere to be seen.
“If you used to have pages in our main web index and now they’re in the supplemental results, a good hypothesis is that we might not be counting links to your pages with the same weight as we have in the past. The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit).”
This doesn’t really work for a lot of sites. Apart from my other websites I also run a jewellery business and the website for it is purely a product site. Who is going to link to a product page on my site? Do I have to write multitudes of content for each product just in the hope that someone will link to it because of the quality content?
Hi Matt –
If I have understood this issue correctly then I have a website that is still falling foul of this scenario.
I have entered site:thedomainname.com into google and done a www search, at which point the index page is listed, however, when you flick to the “pages from the UK” option it is not there.
The site is a UK site hosted in the UK on a UK Server and IP but is hosted on the .com.
Is this one of the instances you are refering to in this post?
Thanks.
i’ve so many supplemental results in my main site, I dont know what exactly happens, my page rank becomes dropped.
Any of you guys know when is google gonna update the PR, I had a wild guess… maybe around the end of the month.. but now i dont know
What is the Google command to see ONLY the pages in the supplemental index for a specific web site?
Today is may 7, did anyone notice a dramitic change in ranking starting yesterday? is there an update in progress?
Is there an official website that posts when google updates algorithms? thanks
@Generic, yes, there appears to have been an update in progress over the last week. I’ve noticed some slight changes on my websites and client websites but nothing really dramatic. I’ve noticed a lot of backlinks have been cleaned up (ie: links from sites that no longer exist are being removed) and almost like this update was a purging more than anything. I lost a little rank on one of my most popular sites main page, but I noticed a huge increase in interior pages getting indexed and receiving pagerank, so I’m quite happy with the results.
@zenegra – the best way to find out if there’s an update going on is to do a Google Blog Search and see if people are blogging about it. Chances are, if the bloggers are posting about it, then there’s something happening.
Hello Matt,
I’ve noticed that PageRanks dropped for most PR 10 websites like apple.com and ibm.com. Is this due to the infrastructure updates of the Google algorithm?
I’ve written about this on my blog (which has the same theme as yours 🙂 ) so please check it out: http://pufone.org/blog/google-pagerank-drops-for-major-websites/
Probably anoying stuff, please feel free to ignore or remove: Could you please write about what has happened with the PageRanks of those websites?
Regards,
Mihai Pufone.
Hi Matt,
How do you know when a PR update had taken place?
Do you monitor some websites and see a PR change or what?
I’ll be happy to know.
Hello Matt,
I am wondering if there is any specific time range for PR update. How long does it take for PR to update and how often does PR of toolbar update. Also is it necessary for SERPS to change when PR updates occur. Thanks in advance for any help
I am runing with domian name mentioned above.
actually my backlinks droped by huge amount in recent google update held last month.
What can be the possibilities.
Any suggestion would be appreciated
I have exactly the same issue with th eweb site I manage
http://www.forex-stream.com
with google.co.il
The command
site:www.forex-stream.com at www search returns all site’s pages including it’s root page.
When the option “Search Hebrew Pages” the root page is ommited.
The site is hosted in Israel and has no duplicate content.
Any help will be appreciated.
————————————————–
However, if you 1) have a .com that is hosted outside the US, 2) searching on (say) google.co.uk for [site:yourdomain.com] returns your root page and all your pages for “Search the web”, 3) if you switch to (say) “pages from the UK”, the root page does not appear but the rest of your pages do, then this paragraph applies to you. I’d wait 4-5 days to let this second change percolate completely into our index, and if you still see the behavior after 4-5 days, please leave a comment with the name of your site.
James,
are the backlinks found with the Links: command on google?
if so, it’s normal that not all of the backlinks are written there, it could drop by a huge amount, I don’t think it will affect the pagerank of your site.
I ahve two domains I’m looking at right now:
http://www.3bweb.com
http://www.andrewnebbett.com
Both are UK companies, hosted on the same server(s) located in USA.
Searching all of the web through google.co.uk for site:mydomain.com gives me all the pages, but ZERO for “pages from the UK”
Obviously this is a huge problem – any ideas?
As many of the website are hosted on shared ip hosting plans, can this effect the page rank? if so any solution for this.
Is there an official website that posts when google updates algorithms? thanks
To my understanding I was bit confused by the factual data. Well when i searched for supplemental result , i went through many theories and conclusions. And this is where I got certain relaxation . As you clearly stated that the SR are the result of confusion by the Google Bot , which thinks that the page is not so much relevant but its worth keeping.
Could you suggest any specific way by which we could avoid it?
My site was offline when update happened and because of this seems to not had any page rank.Is this because it was offline. Hopefully on next update things will change
I can’t figure out why all the web pages I make (other than the main pages of subdomains) get thrown in supplemental index? could it be something wrong with my linking structures… or is it some form of initiation for new webpages?
Hey Matt,
Firstly, thank you for your, always interesting, blog posts and information updates.
I have a question that is semi-related to your post. It relates specifically to regional index listings. I understand that at this point the main ways to be listed in a regional index is to have a country specific TLD or be hosting the website in the region and the IP address clearly resolving as being hosted in that region.
My question; In the future do you see any integration with maps.google.com such that a business physically located and confirmed in maps.google as being within a specific region is then to be indexed and present in the regional specific SERP’s without need for the regional host or regional TLD?
Is there an official website that posts when google updates algorithms? thanks
I was interested in your comments of .com sites hosted oustide the US.
I wonder..I have a .com site, hosted outside the US which is wholly related to the UK. The site does not appear in UK searches only the rest of the web..Are there any plans to use anything other than location of host to determine how a site is indexed?
this is not good for SEOs. because labeling of supplemental result help us to improve our site’s contents. today i got happy when i found all pages of my website http://www.3mobilephonedeal.co.uk/ out of supplemental result.
My question is: What is causing the page (url) to become a supplemental result when having all listed above done?
i want to ask that my site for e.g http://www.abc.com have more then 500 back links before a week , but now suddenly google webmaster tools shows that there are no backlinks to http://www.abc.com and also my http://www.abc.com is out of index from google , but abc.com/ is indexed in the google and i am worried that y google remove my default page from the index and add abc.com/ to its indexing and also it drops all of backlinks against http://www.abc.com , can u help me ?
In the last update my website was off line and i got the site back online after the update and because of this my site never seemed to get a change in the PR. Also all my internal pages show no PR at all will this change or will i have to wait for next update
PR is not important..Important is content and links…I dont believ more on Pagerank 🙁
Terry, you will have to wait for the next PR update.
I’d also like to know why backlinks are getting dropped.
I have a very large site, over 5 years old, that appears to have gotten the -50 penalty. Over 90% of google traffic gone. It keeps trying to struggle up and getting pushed down it seems.
5 years of complete white hat. Of course I have been searching hig and low for problems and I filed for reinclusion. One problem I did find is that someone (not me!) spammed the sitename and some gobbleygook, and there is another sitename in that spam too, not mine, all over the place in blogs. I can only assume a competitor did this.
I have always had the (I see now simplistic) view that if I just provide info, label it exactly for what it is, and let google sort the rest I will have exactly what vivitors want and all will be happy. Worked great for years, the site worked itself up in the ranks slowly but surely.
Now this, and I am clueless.
Many people say PR is not important, well I beg to differ. Going from a PR of 5 to a PR of 0 will drastically affect your search engine traffic, and your revenues consequently. The problem is that Google has been punishing sites with no clue whatsoever to their reasons. I’ve seen sites which went from 2 to 0, from 4 to 0, even from 6 to 2. Why? There’s really no pattern, it seems random, since I only work white hat.
I personally would like to search google.com for worldwide results and use country specific versions to search that country’s result.
Great Article
Content and links and shame on pagerank
So when is the next Page Rank update?
My site is myob.com and is hosted in Melbourne, Australia. On that domain, I have six sub sites, each defined by site=en_AU or UK or NZ etc. Any site-specific searches outside of Australia doesn’t show the site for that country.
I do get hits for myob.com/us which is a site that should have disapeared in May.
Any ideas on how to get my other countries to show would be much appreciated.
Hi Matt,
Thank you for sharing this helpful information, it’s really appreciated and prove that Pagerank value is still important for ranking pages and staying in the main Google index. Thanks again
As of now there has not been a Toolbar PR export for 138 days. The longest time between toolbar PR exports recorded in the below Page Rank Export List was 122 days (Oct 19/2005 to Feb 18/2006). What’s going on?
Hello,
Thanks for the post, as a google fan, I am happy to have employees or insider post such useful information.
Please post more, so that we can benefit from it!
Your fan from Japan!
Anton
Don’t your guys think that we should be due for a google PR UPdate in September ? I’ve been waiting for 4 months and most of my websites have been stuck at their old PRs. I’ve also seen a huge drop in the backlinks for all websites that I own.
Has PR lost it’s importance ? Are Backlinks not important anymore, like they used to be ?
Still looking for answers…
Of course backlinks and Google PR is not lost their importance.
Only Google lost desire to share that information with us.
At last.
Google page rank has been upadated.
My website http://www.visitsouthall.co.uk/ came in as PR3.
page title is almost irrlevent these days it seems
Is it true that pagerank may soon be gone?
matt,
thanks for clearing up the issue about supplemental results. you know it’s disappointing when you see your blood and sweat put into supplemental pages. but at least now this is quite a good news.
Search engine ranking is more important than PR because every webmaster wants visitors
No dissrespect to Matt, he is good at articulating things so people can understand them, but I don’t think he has it accurate .. google is still sandboxing and has some sort of supplimental results that you can get stuck into from duplicate content, etc. PR is still important because it gives people trust.. like a name brand
I hate the supplemental results, I wish Google would instead show random results that are a bit different each time rather than punish sites in that other index which seems to still be in place 1-1-08
Maybe the future of search will be more niche orientated instead of banishing sites into the dark supplemental pit to rot for 8 months with zero traffic… very unfair indeed.
When should we expect to see a new PR update? It seems that when my PR goes up, so do my rankings… in this, it is not clear to me how PR is an ongoing cycle and should not create a new Google position. Any word on that?
I have come across many websites and blogs whose page rank is 5 or even 6 but their content is not so good comparatively, while I have also seen many blogs and websites whose page rank is low but has great content. I also don’t believe in page rank.
Thanks i would like to share about infrastructure development services nowadays all the IT companies are jumped into infrastructure development because the customers should need proper delivery from the services.
thnxxx
Cheers. xx
im having a similar issue on my site show-plates.com , i was wondering if you could look it over .
Hi Matt
We been struggling to break through to page 1 we are currently siting on page 3 our main key words are cheap mobile phones cheap iphones and blackberry we are new to SEO but have done all the work ourselves our home page is talkalot
I know the market we are trying to break into is massive but i am sure with a bit of direction from an expert we would make more progress.
Cheers
Chris Orchard
Is there a fast / easy way to get notified / to find on your blog about your ‘musings’ on infrastructure status reports?