Infrastructure status, January 2007
Okay, it’s been a while since my last infrastructure status report, so I’ll briefly cover the things that I know are going on. The executive summary is that things are relatively quiet.
The quarterly-ish PageRank export is underway. As always, don’t expect traffic or rankings to dramatically change, because these PageRank values are already incorporated into our scoring. The same quarterly-ish data push that updates PageRank in the toolbar also updates the data for related:, link: and info: (remember that operator?). You can read more about PageRank from this previous post if you swing that way. Also remember that the link: operator only shows a subsample of the links to a page that we know of. I’ve mentioned before that some data centers (I believe 64.233.183.xx and 72.14.203.xx) continue to show PageRank values from a slightly older infrastructure. Not a big deal, but I wanted to mention it for the hard-core data center watchers so that they don’t get confused.
There were some situations where site: would show supplemental results ahead of regular results. I believe we’ve changed that so that regular results will usually show ahead of supplemental results for site: queries.
As a reminder, supplemental results aren’t something to be afraid of; I’ve got pages from my site in the supplemental results, for example. A complete software rewrite of the infrastructure for supplemental results launched in Summer o’ 2005, and the supplemental results continue to get fresher. Having urls in the supplemental results doesn’t mean that you have some sort of penalty at all; the main determinant of whether a url is in our main web index or in the supplemental index is PageRank. If you used to have pages in our main web index and now they’re in the supplemental results, a good hypothesis is that we might not be counting links to your pages with the same weight as we have in the past. The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit).
I think going forward, you’ll continue to see the supplemental results get even fresher, and website owners may see more traffic from their supplemental results pages. To check out the current freshness of the supplemental results, I grabbed 20 supplemental pages from my site and checked out their crawl date using the “cache:” command and looking in the cached page header. The oldest supplemental results page that I saw was from September 7th, 2006 (and I only saw 2-3 pages from September; most were from December or November). The most recent of the 20 pages was from January 7, 2007, which shows that supplemental results can be quite fresh at this point.
Let’s see, what else? I think we’re going to change the “filetype:” operator so that it doesn’t require an additional query word, so that you could do filetype:doc or filetype:pdf or whatever. That isn’t live at this point, but I believe it will be down the road.
I’ve mentioned this before, but one of our data pushes that used to happen every 3-4 weeks is now happening more like every 1-2 days. Regular searchers won’t really notice this, but if you see more variance in your rankings, I believe it’s probably due to that data push happening more frequently.
An SEO or two has been holding my feet to the fire about root pages of .com’s that are hosted outside the US. Barry has talked about the issue a little bit here. In some (pretty rare) circumstances, you’ll see the root page when you search site:domain.com on google.co.uk for the “search the web” option but you won’t see the root page when you switch to “search pages from the UK”. I thought we’d nailed this issue in December, but we found another way that this can happen. I believe a fix has been submitted and is percolating its way through the system. Of the ~7 examples that I know of, I believe all but one is working now (and the remaining site is doing a chain of like five 302 redirects to weird/long/deep urls). However, if you 1) have a .com that is hosted outside the US, 2) searching on (say) google.co.uk for [site:yourdomain.com] returns your root page and all your pages for “Search the web”, 3) if you switch to (say) “pages from the UK”, the root page does not appear but the rest of your pages do, then this paragraph applies to you. I’d wait 4-5 days to let this second change percolate completely into our index, and if you still see the behavior after 4-5 days, please leave a comment with the name of your site.
Right now I’m not expecting any major infrastructure-related upheavals to our rankings. Should that change in the future, I’ll be here to talk about it then. 
Update: A few people were seeing PageRank 0 for their site. There was a small auxiliary push that needed to happen to complement the PageRank push, and that push happened a few hours ago (i.e. Jan 11, 2007). If you were getting stressed, you might want to re-check now. If you never even noticed, well, good for you.
JLH Said,
January 10, 2007 @ 9:38 pm
Thanks for the comprehensive update, now if you could only stroll over to http://groups.google.com/group/Google_Webmaster_Help and log in so that you get a ‘Green G’ by your name and post this (Or maybe Adam is reading this, hint-hint, wink-wink). Then sticky it to the top. It would really help clear up a lot of noise (imagine noise crossed out and replaced with FAQ). You’ve re-re-re-reconfirmed some answers to very common questions; TBPR, link:, .com/co.uk, supplementals, etc.
Once again thanks again for a most authoritative informative post in an unofficial-your-personal-blog kind of way.
Matt Cutts Said,
January 10, 2007 @ 10:02 pm
Happy to help, JLH. I’ll ask Adam if he can point it out over on the webmaster group.
Nice_guy Said,
January 10, 2007 @ 10:08 pm
Hi Matt,
Thank you for your posting. It’s very timely, because we’ve been noticing a sharp drop in our site’s PR from 7 to 0 spreading across Google Datacenters all of the last few hours.
We’ve also noticed that some other established sites are also having the same problem. Is this something you think we should be worry about, or just a glitch that will come to pass?
Thank you
Harith Said,
January 10, 2007 @ 10:11 pm
Matt
Thanks for detailed weather report. A theoritical question
How can we see the difference between changes in rankings as a result of those data pushes that happening like every 1-2 and the traditional Everflux. Can we assume that from now on rankings of sites wouldn’t be as stable as before?
Thanks.
db Said,
January 10, 2007 @ 10:38 pm
Much thanks for the supplemental update.
I can see users avoiding visits to anything that says supplemental when they do a search. It kind of screams second class.
Best to avoid it in the first place but when you have some new blog entries they likely won’t have any links so they’ll end up in supplemental. Its just the nature of blogging that people might link to a site but not the actual post so once the post moves off the front page it’ll be supplemtal no matter how good the content.
JLH Said,
January 10, 2007 @ 10:52 pm
Thanks again Matt.
Philipp Lenssen reveals how page rank really works.
http://blog.outer-court.com/archive/2007-01-11-n25.html
I did that calc that reveals you are ranking 3778 pages per second, very impressive. So I’d guess the 90 seconds I cost you to read my comment and send a note off to Adam means 340,020 pages will not get updated next time. To those affected, I apologize.
Hagrin Said,
January 10, 2007 @ 11:02 pm
Matt -
Thank you yet again for another EXTREMELY informative post. I think you answered every question I had from noticing SERP changes recently.
Kirby Said,
January 10, 2007 @ 11:04 pm
To follow up on SEW’s post, I heard your interview with Mike Grehan. You mentioned you understand more about what its like to be a webmaster now that you are a blogger. You also mentioned how many SEOs are getting creative in obtaining links. However, they represent a small segment of the web.
Try being just mom and pop who dont know how to play the Digg game, dont blog endlessly and dont go to SES or PubCon. They wont attract many editorially given links without a little help.
Matt Cutts Said,
January 10, 2007 @ 11:06 pm
Harith, that data push is on a slightly different cycle (not pushed as part of the normal daily-ish index-data pushes), but it will be harder to distinguish between everflux and the more frequent data push.
S.E.W., too much bold and italics. Pruning.
JLH, you saw the comment to Philipp’s post that it was really Marissa doing all the PageRank? Floating point math hurts my head/CPU, anyway.
Hagrin, glad it helps. I’m taking next week off, so I wanted to throw at least one in-depth post up before I left.
Kirby, there are steps that we take to try to help those Mom/Pop sites as well.
Big Daddy Said,
January 10, 2007 @ 11:29 pm
Matt-
Referring to your comments
“The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links”
Can multiple links coming into a site from another to different pages hurt the site that is receiving the links? Example: State related linking to state related pages. If so, how can one get the penalty lifted and back in good graces with Google? We have a site that was ranking alright and just dropped of the map. Is it a problem to get such links that actually drive traffic?
Feed back would be greatly appreciatet!!
Aloha
Tom Churm Said,
January 11, 2007 @ 12:04 am
Strange thing about Supplemental Results:
I have several domains which have been parked and redirecting via 301s to a main domain for a long time, but, even with the 301 redirects, these parked domains still appear in the Supplemental Index.
I thought 301 redirecting a domain was a way of telling Google not to Index it at all, but instead to index the domain they’re redirecting to?
Michael Lopez Said,
January 11, 2007 @ 12:52 am
Ok, this kinda explains things - I think. I was just surprised to see that my PR dropped from PR4 to PR0. Is this normal? The site is http://www.mikelopez.info.
Michael Lopez Said,
January 11, 2007 @ 12:54 am
Follow up to above post, a friend in the US (I’m in the Philippines) says that he sees PR5 on my site so I guess it will be PR5 after all.
Paul Carpenter Said,
January 11, 2007 @ 1:19 am
Hi Matt
Great news about the UK, .com thing finally getting some resolution - if you think you’ve had your feet held to the fire, you should try explaining it to people who think you’ve done something to get their site banned!
Do you know if the .com sites are likely to regain the rankings that they previously had on “pages from the UK” or will the preference still be given to .co.uk TLDs even after the update?
Cheers for the info anyway!
Socks Manly Said,
January 11, 2007 @ 1:35 am
An issue I’d love adressed is relevant link trading. My site has reviews of adult sites, and we now link with other quality sites like ours where we have reviews in common, review to review. My site got obliterated in the rankings in the last days of June 2006, and I think this was why. Too many links too fast looked spammy.
I’ve heard a lot of other people suggest google is not liking these kind of link trades, but they’re the most relevant results I can provide to my customers!! I’m giving them a second opinion for free, sending people to my competition. These links have always been designed to be a great feature for our users more than to drive traffic.
Great to see things are happening over there quickly in 2007, my page was actually not in the index on Christmas Eve, that was scary!! Then it went all supplemental, and now it’s back to about 10% supplemental, and completely fairly so, some old articles that deserve little to no attention.
Would love to get some tips, we have no idea about search engines, doing our best! socks @ if someone is interested.
Stephen Newton Said,
January 11, 2007 @ 1:46 am
What would be really useful, would be a more intelligent way of telling which country is website is based in. I’m in the UK and recently moved to a host in Utah, where I got a good deal. Now Google’s taken me out of the UK search results. Okay, the server’s in the USA, but that’s not useful information to my visitors.
How about making this information something we could provide via Sitemaps?
Maurice Said,
January 11, 2007 @ 1:56 am
Oooh
Thanks for that (though I have to rewrite my general comments for January that goes in all my reports)
Jon McMullan Said,
January 11, 2007 @ 2:06 am
Time: 09:43am GMT - 11-01-07 - My pagerank score in the toolbar has just dropped from 4 to 3 - is this the toolbar update????? None of my other sites have changed, but I visit iwebtool.com alot and I’m sure they used to be PR5 and they are now 0
Chris Hunt Said,
January 11, 2007 @ 2:12 am
So, Matt, can you confirm that the only effect of being in the supplemental index is that the page gets crawled less often, or does it have an impact on ranking?
If it’s only to do with link frequency it seems curious to base it on PR - since the critical factor is surely how often a page is updated rather than how popular it is. I can think of several very highly linked, presumably high PR, articles which *never* change. Surely, brand new pages that nobody even knows about, let alone links to, are more likely to be changed frequently by their authors?
An unequivocal statement of what the effect of being in the supplemental index is would be a good idea. The webmaster help page talks vaguely about sites in supplemental having “fewer restraints”, but doesn’t make it clear what the end result is going to be.
If crawl speed is the only effect, maybe you should remove the “Supplemental Result” thing from search results altogether? I can’t see what it adds for the average searcher (I doubt if they even notice it), and it would remove half the noise from SEO forums.
Mike (Germany) Said,
January 11, 2007 @ 2:53 am
@michael lopez
Many people has droped the PR by this update export.
Andy Said,
January 11, 2007 @ 3:05 am
Hi Matt,
I wonder if this new update has caused the rather bizarre event that has happened to me. Almost all my new pages have been hit by a penalty since last August. In December I requested re inclusion. On 3rd January all my rankings had returned (a truly euphoric feeling). Checked again later that week and they were still there. Checked this week and they have been bombed again (a truly un euphoric feeling). I would beg you not to do this to anyone else. Second time around it feels twice as bad. It is very tough to take having my new pages bombed again. I would love any form of explanation and I would be delighted to put right whatever I am doing wrong.
Jean Manco Said,
January 11, 2007 @ 3:32 am
Michael Lopez - My site has just lost all PR on the toolbar. This has happened before for no apparent reason - though not usually on the whole site. It usually rights itself next time around. I know of no reason why Google would penalise my site, which is non-profit, does not buy links and has links from universities, the BBC and governmental sites, etc., which certainly should be passing PR.
If completely clean sites with no penalties, which have plenty of good (not bought, not spam) incoming links, suddenly go PR0, I’d say it’s just a little fault in the export of PR values to the toolbar display.
Dave (Original) Said,
January 11, 2007 @ 3:38 am
RE: “Not a big deal, but I wanted to mention it for the hard-core data center watchers so that they don’t get confused.”
==========================================
Those types are in a perpetual state of confusion. A dog chasing it’s tail is more productive than them
Praveen Said,
January 11, 2007 @ 4:02 am
Hello Matt
The site:www.naukri.com does not show up when searched on Google.co.in with option Pages in India. Is this because the IP address of the site is in US.
Tom Churm Said,
January 11, 2007 @ 4:44 am
also @michael lopez: check out my site at http://rss2pdf.com
it dropped from PR 7 to PR 0.
…and i don’t even have any ads on this site - just links to three other websites that i also built. to me, this is pretty evil :/
Juan Said,
January 11, 2007 @ 4:49 am
Could anyone please confirm is there is any fault exporting the PR data into the toolbar? I’ve seen a number of websites with high PR going to PR0. Big SEO company websites like seo-guy or highrankings have PR0 at the moment, from PR6. Could please someone show us the light?
Socks Manly Said,
January 11, 2007 @ 4:56 am
If you guys took a moment to read matt’s comments, you’d realize the toolbar isn’t showing accurate information at the moment. Matt said that they were doing PR updates, so clearly this is the reason! He said to wait a few days and see what happens. This page is showing a 0 pr at the moment, as is mine.
If your site: operator is looking good, I don’t think there’s need for alarm.
mscha Said,
January 11, 2007 @ 4:56 am
I’m from The Netherlands but have a site, mscha.net, mscha.net, which is actually hosted in the US (cheaper hosting with better specs).
Therefore, it (properly) doesn’t show up when I search google.nl with “pages from The Netherlands” enabled.
Is there a way around that? Something like putting the country in a META tag or a header, perhaps?
Thanks,
- Michael
Bob Said,
January 11, 2007 @ 5:15 am
Matt, I’ve watched the video you posted regarding this topic with a few suggestions on what webmasters might do who are impacted. I’ve also read a few of your posts regarding these data refreshes.
Are there any other hints you can provide without spilling the secret sauce? I oversee a couple of sites which are nearly identical in operation and layout with the exception of the topic/industry. One of them is very frequently experiencing significant changes in ranking (up, down, up, down, up, down) and we have been banging our heads against our desks since June trying to figure out what might be unique about that site or make it look different in the eyes of Google.
Again, any additional tips or suggestions would be helpful. Thanks again for all of the great information you do provide.
Tony Ruscoe Said,
January 11, 2007 @ 6:31 am
Thanks for the info, Matt. I have a question relating to updates and data centers.
Is the crawl date (as shown in green next to the page size of some results in the SERP) a good indication as to whether the data center is up to date? Let me explain why I’m asking…
This morning, it seemed that the home page of a PR8 site I manage dropped out of the index completely (it’s not even listed when you use the site: operator and there’s nothing in Google Webmaster Tools to suggest there’s been a problem with our site) meaning a competitor’s site was in top position for our most popular keyword. The date next to their result was shown as “9 Jan 2007″. When I checked some other data centers, I saw they had our site back at the top for the same keyword with the dates for both sites showing as “10 Jan 2007″.
So, does a more recent date mean a specific data center is using more recent index data - or is the fact that the site’s home page is missing be something I should be concerned about?
shpyo Said,
January 11, 2007 @ 7:30 am
Well, current mess with google updates it’s really weird.
Thanks for explain some things!
WilliamC Said,
January 11, 2007 @ 8:12 am
TBPR showing odd numbers during an update is not uncommon, as others have said, do not worry much about it until the update has finished rolling out.
Matt: The new data pushes have been showing interesting variations in rankings, but do we have any idea on what variables change and how so over the course of say 4-5 different pushes? In other words, what measures are being modified continually, and for what purposes?
These answers may give some a better understanding of the flux that they are seeing every few days.
Manca Said,
January 11, 2007 @ 8:29 am
@Socks Manly:
Check your Yahoo mail bro….
@Matt:
Excellent post man. Very informative and quality! However I would like to ask one question regarding wordpress (or any other) posts’ feeds. I mean, I got all feed pages of my blog with around 200 entires into supplemental. What’s the reason of putting feeds into supp. index and why even index those when you know the content is exactly the same as on the original page, just with different schema.
Should I be concerned about this and block all feed/ pages in robots.txt or just leave them as they are?
Another quick question….do pages go to supp. index if they are not being crawled for a long time (what’s is causing that?) ? From what I can read pages hit the supp. index even though they’re not contain duplicate content. Am I right?
Thanks a lot for your time!
Manca
Keith Ort Said,
January 11, 2007 @ 8:47 am
Thanks for the detailed information, Matt. I run my client’s reports over the weekend so we’ll see just how drastic the changes affect them. But atleast I’ll be able to explain to them what happened and why.
David Dalka Said,
January 11, 2007 @ 9:01 am
Nice post!
How can one tell which index a site and/or page is in?
Thanks.
Matt Cutts Said,
January 11, 2007 @ 9:05 am
Big Daddy, I don’t see any special reason why that would hurt off-hand. Bear in mind that lots of cross-linking all over the place may look bad though. In general, if you think you may have had a penalty and you think the site is clean now, I’d follow the directions here:
http://www.google.com/support/webmasters/bin/answer.py?answer=35843
Tom Churm, sometimes it takes a while for the supplemental results to refresh and register that 301, but it sounds like you’re doing everything right. It doesn’t hurt for those pages to show up (if I user clicks on them, they’ll get the 301 to the final page), so I wouldn’t worry about it much.
Michael Lopez, we have correct PageRank stored for you internally, so my advice is not to worry about this. One thing you can do if you’re concerned is to help us pick the right preferred host (www vs. non-www) for your domain. There’s a setting for that in our webmaster console.
I will give you a heads-up (since you mentioned your site) that in a completely unrelated matter, your site is selling text links and has participated in various SEO contests (an SEO contest where you’re required to place the anchortext “online kasino”? C’mon, Marc.). That doesn’t affect the PageRank shown in the toolbar, but it does affect your site’s ability to flow PageRank and the trust in your site.
Socks Manly, “relevant link trading” can definitely be done to excess. I wouldn’t rely on that as your sole method of gaining links.
Chris Hunt, a page that’s a supplemental result now may well get more traffic in the future than it gets right now. Personally, I am a fan of dropping the label “supplemental results” but there are some valid reasons for it, including debugging for example.
Andy, if your site is over-optimized in the sense that users aren’t getting as much value-add, it may be that our algorithms are repeatedly registering your site as not as helpful as other sites. The best recommendation I can give is the solid white-hat treatment (things like great, compelling original content that users love). Without the specific site, that’s the best advice I can give.
Jean Manco, that tends to happen because we’re storing the toolbar PageRank value under a different url, e.g. http://www.example.com instead of example.com.
Bob, there’s not much more I could offer other than the tips that I gave to Andy a couple paragraphs up.
Tony, a more recent date typically means that that data center has the more recent copy of the page. So I wouldn’t worry about it given what you’ve told me so far.
Manca, I wouldn’t worry about the /feed urls showing up. It’s not hurting anything, so I wouldn’t block them. I don’t believe pages go into the supplemental results just because they haven’t been crawled for a long time. I believe that it’s a separate crawl for the supplemental results.
Keith Ort, there won’t be any traffic changes due to the toolbar PageRank changes. If the traffic is different, it’s likely to be normal everflux or other stuff.
David Dalka, all the supplemental results say “Supplemental result” at the end of the snippet for that url.
Adam Said,
January 11, 2007 @ 9:10 am
Ok, but I have following problem:
1. .com site outside the US (hosted in UK)
2, Google.co.uk - was ok for “Search the Web”, but there was nothing in “Pages from the UK”.
3. I sent “reinclusion request” and described situation.
4. After a week my site was added to the “Pages from the UK” index and was placed on 1st positon for my chosen keyword. - SUPERB!
5. Unfortunately, after about 3 weeks the main(root) page disappeared from “Pages from the UK” index. I sent “reinclusion request” and described what happened again.
6. The main page came back after few days. - SUPERB!
7. Unfortunately again - about 5 days ago main page disappeared from “Pages from the UK” again.
8. I sent “reinclusin request” again.
9. Today main page also disappeared from “Search the Web” index!! :-((
What can I do about that?:-( Why it is happen so often?
Pittbug Said,
January 11, 2007 @ 9:11 am
Thanks for the update Matt.
When a PageRank export happens, when is the PR snapshot taken? The day of the export, or some prior date?
This has been debated about on various forums and I wonder if you could help us get over this issue.
Michael Martinez Said,
January 11, 2007 @ 9:17 am
You know Matt, I think this may become one of your most linked entries like the BidDaddy review you posted a year ago. Maybe if you did more of these comprehensive overviews, you would feel less pressure from people to perform on the SEO side and you’d have more freedom to dwell on the other topics you like so much.
It would be nice to see a Quaterly Google Happenings update from you on this scale, maybe a little slightly more extensive.
Or, if not from you, then maybe from the Official Google Blog or the Webmaster Central blog.
Thanks!
Bob Sheth Said,
January 11, 2007 @ 9:41 am
Hi Matt,
Thanks for the post and the comments. Myself and a lot of other webmasters in the forums are expressing concern about this recent update. My problem is that I have recieved page rank on about 10% of my internal pages but nothing on the remaining 90% and more importantly 0 on the domain. Is this normal? Or does it take time for page rank to filter through the ranks.
My main concern is that fact that I have recently taken a couple of smaller property websites off line and 301′ed the domains to the relevant areas of my main site. I thought this would be fine.
Any help is much appreciated.
Thanks
Bob
Jon Goldberg Said,
January 11, 2007 @ 10:44 am
Matt,
Can you make any comment about the status or future of google directory (odp)? Google’s copy of it is over a year old.
vimto kid Said,
January 11, 2007 @ 10:46 am
Hi Matt,
nice post,
in your reply to M Lopez, i see you’ve said:
“selling text links… doesn’t affect the PageRank shown in the toolbar, but it does affect your site’s ability to flow PageRank and the trust in your site.”
For a site that has realised the errors of it’s ways, can you clarify what the procedure is to regain it’s ability to “flow PageRank” once the offending links have been removed? Is it to file a re inclusion request?
Big Daddy Said,
January 11, 2007 @ 10:47 am
Thanks Matt!
One more question: If a site has has 92,800 pages indexed and only 700 aren’t supplemental, could that cause rankings to disappear and the site not even rank for its own http://www.domain.com? If not, would this be some sort of penalty?
Enjoy your time off!!
Aloha
Stuart @ Vip Said,
January 11, 2007 @ 11:20 am
We have been found all but 3 links of about 7,5000 links we have are in te Google supplemental database, also our page rank of 6 now shows as 0.
After following the Google Webmaster Rules. We could see that we had a number of domains pointing at the one location which I understand can be classed as page duplication. So after following what we could see we redirected the following domains with a 301 re-direct which is best practice into one domain such as http://www.example.com
example.co.uk
example.com
http://www.example.co.uk
This I feel was our issue and think that http://www.example.com may have been flagged as duplicate content of http://www.example.co.uk so was classed as supplemental results by the new Google supplemental database.
How would we combat this from happening, we have now lost over 10,000 visits a day for the past 3 months, and have always tried to be search engine friendly and good, well written content.
Socks Manly Said,
January 11, 2007 @ 11:38 am
Awesome information, your extra effort here inspires a nation of often confused webmasters trying to do their best, I thank you for your dilligence and patience!!
Very difficult in the adult industry to get a quality link that isn’t reciprocated!! We do get a lot of natural links just because we’re a quality resource, but other big adult sites would AVOID linking to us, because we’re such a great site
People are talking now that one way links are better, so to work on A->B->C trades, meaning A links to B in return for a C to A link. Since I only have one big site (I know, I’m a radical) and never really like playing these Google games, I’m not concerning myself much with it. Am I stupid to not pay attention to this?
We have over 1,350 quality articles, that are generating about a click a day from G at the moment. One of our competitors is 1st paging virtually every term he targets, on the other hand, for similar reviews. Oh, for a return to the glory days
One more thing I don’t understand, when I look at my log analysis from 2002-2005 I see a good deal of traffic from google.de, google.it, .be, .co.uk, etc etc. Now in Analytics I just see “google” and we don’t seem to be getting traffic from these worldly googles. What’s the story there?
Seconds on the Google directory too, a refresh would be lovely!
Elias Kai Said,
January 11, 2007 @ 11:43 am
There is a lot of NEW Data Centers where Google itself has PR 0 and where those new DC are not cached on other DC’s
Any explanation for this ?
I am talking about :
64.233.171.107
DC: 64.233.179.19
DC: 64.233.179.100
DC: 64.233.185.80
72.14.209.19
209.85.129.99 etc
Netmeg Said,
January 11, 2007 @ 11:47 am
Thanks for the info, Matt.
Just last night I found a bunch of cached pages for one of my sites with early August 2005 dates on ‘em. Hadn’t seen any that old for a while, so I was a little surprised.
I have to assume something is going on - on that same site, all but the first five entries are listed as supplemental - not surprisingly; I just redesigned the site and I don’t think it’s been completely indexed yet. The weird thing is that the first four pages are always the same, but the FIFTH page is different every couple hours. The site is database-driven, so all the pages are dynamic, but it just seemed kind of odd.
Is it normal during all this moving and shifting around for things to completely drop out and pop back up a couple days later? I have another site with maybe 7 pages; it’s been in the index a couple years, and the day before yesterday it completely disappeared out of the index - the site command returned no pages. Then yesterday one page showed up - the plaintext sitemap. Last night four pages showed up, two of which were supplemental. Today there’s two pages - but no home page. Meanwhile, webmaster tools doesn’t report any problems with it (although there’s definitely been some issues with webmaster tools)
Just wondering - seems like every time I look, I see something different.
Netmeg Said,
January 11, 2007 @ 11:48 am
(opps, I meant August 2006, not August 2005 on those cache dates)
Jim Clouse Said,
January 11, 2007 @ 12:37 pm
Matt:
Your statement ‘…In general, if you think you may have had a penalty and you think the site is clean now, I’d follow the directions here:
http://www.google.com/support/webmasters/bin/answer.py?answer=35843...”
In particular, what if your site is clean, have filed 3-4 reinclusion requests and your site is still penalized? What do you do then?
I’ve been penalized 56+ weeks now, have removed all the programming work the outside programmers put up that evidently triggered the penalty, and have added new content-rich pages. As far as I know, the site is squeaky clean. Google sure crawls it aggressively but I can’t get my 2005 rankings back.
But a search for hotelmotelnow.com in any of the data centers comes up from #32-#45. That is weird and indicates some form of penalty doesn’t it?
Is that any way to treat a fellow UK Wildcat?
chris Said,
January 11, 2007 @ 12:40 pm
blogger.com
digg.com
myspace.com
All these established sites lost their pagerank do you have any reason for it. Its a penalty or just a glitch??? Similiarly lots of well established sites lost pagerank can you please comment MATT,
Manca Said,
January 11, 2007 @ 1:02 pm
Hey Matt,
Thanks for a good answer, however I am still not clear about the real cause of pages being flagged as supplemental. I sure have many internal links and I believe all my pages are accessible to Gbot (at least there is a sitemap on the site). I am speaking about a blog in this example, and as you know blogs contain archives and categories where posts that belong to certain category are being listed in that category, showed on main page and also put in archives. The post itself has its permanent page where it’s displayed as well. I am not sure how google thread those? Are they marked as supplementals just because we’ve actually got the same content on couple different places?
Having that in mind I did block category and archive pages from being indexed (meta noindex) so they’d be only available on the main page and on the posts’ pages. Did 301 all across the site (I even setup 301 from post to post/ as well) so there shouldn’t be ANY duplicated content at all on the site….. But now, when I checked site: operator it showed bunch of supplementals (besides those feed/ urls) that are just original posts.
My question is: What is causing the page (url) to become a supplemental result when having all listed above done?
Thanks a ton!
Manca
Al Said,
January 11, 2007 @ 1:05 pm
Hi Matt,
Can you comment all those WebmasterWorld threads please (the last thread is herre http://www.webmasterworld.com/google/3216506.htm).
A lot of webmasters lost 50-100% of their Google traffic for no reason in December.
I have website with 7 subdomains. The strange thing is only 5 of 7 subdomains have lost 99.99% of Google traffic, but the traffic of other 2 subdomains is stable. My income crashed. Result of years of building and promotion is 5-10% of my usual income, thinking about finding “real job” now.
Same with some other webmasters who have just few websites.
Looks like a bug for me and for some other webmasters. Different pages have no Google cache, other ones have. When I search http://www.mydomain.com, I can’t see my main page in Google SERPs etc. Strange site: results…
Subdomains are crosslinked (each page has links to all 7 subdomains), have similar 5-10 words signature (duplicate content).
But I think crosslinked pages and signature is no problem, because all subdomains have same structure, same design, same promotion methods etc. Same situation with other webmasters’ websites.
Can you answer to all of us please. Is this temporary bug (some kind of data loss) or it’s some kind of website’s penalty? We just want to understand what’s this.
Thanks.
Andy Said,
January 11, 2007 @ 1:06 pm
Hi Matt,
Thank you for taking the time to give advice. I would like to think my site is truly white hat. For example in the last year I have written detailed and original reviews for around 100 cameras. It is the reviews that appear to have the problem. I have also started to add my own video guides. User feedback is excellent. If you can spare a moment or two the site is http://www.cameras.co.uk. It would be really appreciated.
Jonathan Said,
January 11, 2007 @ 1:33 pm
I’m in the same boat as everyone else. Why did so many sites move from PRx to PR0? I had quite a few PR6 sites that are now showing PR0…is that some sort of delay problem or why is that happening?
elgoogdotnl Said,
January 11, 2007 @ 1:41 pm
Heya Matt,
Feels good to first time post here,
I know the niche i work in is hard and not widely accepted
1.
Why does Google say in wmtools : not indexed, when i suspect it is banned?
Is there any change it will be in the index in the future? If not, why doesn’t Google just say you are penalized? (site:command=zero)
2
.Also 2 sites don’t show the index page when doing a regional search in the Netherlands, i thought maybe you liked to know that’snot only restricted to the US
3.
I would like to contact Google, even when i would have to pay…..
cheers
elgoogdotnl Said,
January 11, 2007 @ 1:45 pm
US=UK
Gigio Said,
January 11, 2007 @ 2:03 pm
New Data Centers where Google itself has PR 0 and where those new DC Any explanation for this ? ummmmmmmmmmmm
Gigio Said,
January 11, 2007 @ 2:06 pm
Thanks for the supplemental update.
Teddie Said,
January 11, 2007 @ 3:44 pm
The geo-targeting stuff is one of the only issues that really irks me because I think mostly you (Google) do a great job. But you know Matt, until there is a ‘US only’ option in the primary .com (sorry an mean that non existent .us search page) so that the US users get treated the same as the rest of the world, and you guys publish clear guidelines on how to differentiate real international content from local, and that includes local as in US only, geo-targeting will always be a bit of a mess.
What would be wonderful is if users defaulted to their local countries results but could choose to select ‘The Web’ and would then actually get results from the web meaning its entirety and glory - whichever site from whatever country in whichever language deserves to be highest and not a consistently biased subset of it
Maybe country targeting options within Sitemaps Vanessa?
Colin Said,
January 11, 2007 @ 4:02 pm
Does this mean sites without PR since April will finally get a little green in that bar?
Of course this is assuming that they have enough links and presence to get a PR1 or higher, but it seems there hasn’t been an update for new sites lacking PR completely in quite a while.
Am I right about that? If not, how long on average does it take for a site to get their initial pagerank? Or does it vary site to site greatly based on page strength?
Thanks.
Vincent Marcello Said,
January 11, 2007 @ 4:11 pm
Hi Matt
I believe I had mentioned this before a little while ago in my blog and welcome the opportunity that you ahve now mentioned it here for actioning.
Our main development site is: southbourne.com and for years now we have had no problem with .co.uk and pages from the UK. In the last few months there appears to be a problem with our root page missing from search results for ‘pages from the UK’ at one point quite recently it made an apearance again, but now it has gone.
We are hosted on UK servers (WIndows) for this particular site.
Thanks
Vince
Multi-Worded Adam Said,
January 11, 2007 @ 7:12 pm
I like Teddie’s idea, but from a slightly different point of view.
Here in The Great White North, registering a .ca is an unnecessarily long process that people get confused by and quite often abandon. (Side note for those of you who have registered .ca domains in the past: I do realize it’s only an extra 6-7 mouse clicks, but to the non-techie it’s like climbing Mount Everest, ESPECIALLY with the CIRA wording of things.) So a lot of companies elect to register a .com as a result, and understandably so.
These same companies also quite often select US-based hosts simply because Canadian hosts aren’t on a par with their US counterparts; the pricing seems to be getting more in line, but the service is about as reliable as a National Enquirer story sometimes. (Anyone who has ever had to get write permissions set on a directory via a Canadian host will know exactly what I’m referring to.)
It would be nice to allow for these webmasters to select their country of origin and be included in SERPs for that country. If you’re worried about people selecting a non-US country of origin just to rank in non-US SERPs, you could use the same IP geotargeting (except in AOL’s case…not really sure what you’d do there.)
Either that, or find the contact addresses listed on each site and use those. You could probably get 99% of them algorithmically just by matching up contact, contact.*, contactus.*, contact-us.*, and any common address patterns.
Anyway, it’s just a thought. Not anything I’d consider to be a stroke of genius that should be implemented 10 minutes ago.
StockTube Said,
January 11, 2007 @ 8:14 pm
can anyone advise on when will a new site be evaluate and assigned a PR? … would new sites get their PR the same time with the old sites? …
will there be difference in PR ranking for a website (www) and a blog (blogger for example) which does not has the “www” initial? …
would the PR be assign or re-assign on quarterly basis? sorry if my queries sounds too 101 …
cheers …
Aaron Pratt Said,
January 11, 2007 @ 9:59 pm
I agree with Michael Martinez, this has to be one of the most useful posts Matt has ever done. Think of all the incorrect speculation about “supplemental results” put to rest with just a couple of words.
The only problem is that these questions will continue to be asked in areas where incorrect speculation still occurs.
New pages on Google that accurately descibe these things would allow people to reference correct data.
Matt Cutts Said,
January 12, 2007 @ 1:48 am
Adam, there’s a pretty rare set of circumstances where this happens. I would give it a few days, and then if it’s still happening, pipe up then and mention your site name on this thread.
Pittbug, it can vary. I think it’s pretty much when this one dude in the crawl/index team decides to run a certain script to export the values. So whenever he presses the enter key, I guess.
Michael Martinez, I do end up doing these every 2-3 months or so, it seems. I would do it on the official webmaster blog, but it’s always late at night, and by the time it’s done I just want to hit submit and go to bed, so it’s easier to do on my blog.
Bob Sheth/Stuart @ Vip/Elias Kai/chris/Jonathan, I added an update to my original post. I’d read that and maybe do a re-check of your PageRank now.
Jon Goldberg, that’s not really my area. I know that when the ODP was down for a bit, that kept us from downloading a fresh copy.
vimto kid, exactly right. I’d do a reinclusion request.
Socks Manly, I wouldn’t bother with triangular links or “all one-way links, all the time!” or whatever the link fad du jour is. If your an original content site in the adult industry, why not stop by Tony Comstock’s site, or get to know Chelsea Girl at http://prettydumbthings.typepad.com/ or Violet Blue. Don’t neglect the educational value of getting closer with some of the people in your industry.
Jim Clouse, I’d take your site over to the Google webmaster help group and ask for advice from other users. Expect some tough love though. They may say, “Hey, your entry for Hyatt Regency Lake Las Vegas Resort appears verbatim on 40+ other sites! In fact, you’re not even handling the quoted bullet points correctly on that page. What value are you adding? Why would someone go to your page instead of the other 40 copies of this hotel page?” But that tough love will also give you guidance on what your site could do better.
Manca, the main reason would be if you don’t have enough PageRank coming into your site, or if your link structure is set up so that not much PageRank is getting to those pages that are showing up as supplemental results.
Al, I believe that would be data push that has become more frequent. The advice that I gave to Jim Clouse isn’t bad; we’re looking for sites that provide value-add (original content, good information, other aspects of a site that look great to visitors, etc.).
elgoogdotnl, I think we know that the root page issue can affect any country right now, even if it’s rare. If you think a domain is penalized, I’d try to figure out why (site hacked? hidden text? doorway pages? etc.). Enlist help on the webmaster group if you want some more eyes. When you feel comfortable that the site is within our quality guidelines, then I’d do a reinclusion request.
Vincent Marcello, thanks for mentioning it. If it’s still happening in a few days, this will be a good datapoint to help make sure we get the issue squashed.
Aaron Pratt, glad if it helps. I’ll try to chat with Vanessa after vacation to see if she wants to fold some of this into our web documentation.
And with that, I’m turning off the computer, going to bed, and planning to get ready for a vacation with many books and few emails.
cracking toast Said,
January 12, 2007 @ 2:07 am
Hi Matt,
we have a site that, from early December, was in the situation you’ve described where site: operator would show supplemental results ahead of regular results. I can see that this has been fixed and that the site: operator is working consistently again but…
this behaviour of the site: operator was accompanied by the drastric ‘up down’ ‘up down’ swings similar to those decribed by Bob in his post (tho’ just from December for us).
Throughout all these swings, the site: operator showed results for the site. The swings seem to have stabilised now but we’re left in the ‘down’ position :o( These aren’t the small variances in rankings that we’d expect from the more frequent data pushes you’ve mentioned. There seem to be other sites affected similarly too - can you give us some pointers as to where we should be focussing?
cheers for the excellent post,
PS.Enjoy your week off next week.. - I’m sure you’ll have an especially good time if you can relax, happy in the knowledge you’ve replied to this post ;o)
Bob Sheth Said,
January 12, 2007 @ 2:17 am
Matt. A big thank you for the time you take to address invidual webmasters. My homepage PR is still showing 0 but I’m not bothered because my site is growing daily, the serps become better and more relevant daily and my incoming links are growing slowly but steadily.
Thanks again. I hope you enjoy your time off and maybe I’ll catch you at SES London.
Gigs Said,
January 12, 2007 @ 2:18 am
Well, I’m having a nice work morning with PR jumping around between 3 and 5 and subpages between 0 and 3. Before the export the page had 4s and 5s on the different datacenters so I expected a transition towards 5. Now it seems to point towards a 3. This is too confusing, I need to get some coffee and relax. And I thought there is no more “Google Dance”. This is “Google Headbanging”
Ah, yes, coffee.
Richard Mather Said,
January 12, 2007 @ 2:22 am
Hi Matt,
Great post.
With regard to .com sites hosted outside the US and the root not showing on goggle.co.uk UK only sites, I think I can make your ~7 cases that you know of into ~8. The root also doesn’t appear in google.com - only google.co.uk all the web.
If you get the chance I’d really appreciate if you could look into http://www.evictstop.com and see if this bug applies.
Many thanks
Richard
DaveW Said,
January 12, 2007 @ 2:30 am
Thanks for update Matt…
Yeah we got all of those signs - was just wondering what was happening - now we know
All seems preety positive to me - just to summarise then - the higher your pagerank (toolbar or real) the less pages will be in supplemental index. Presumably vice versa also applies - the less PR the more pages will be supplemental?
Does this apply to all websites or can well connected/ optimised sites escape SI irrespective of pr? And (finally) does this mean that pages in SI can/will appear in the main SERPS?
Thanks again.
Mark Nunney Said,
January 12, 2007 @ 3:29 am
Let me get this right: you have “got pages from my site in the supplemental results” and to get pages out of ’supplemental’ you recommend “solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit)”.
Er, your site has over 200,000 quality, relevant inbound links! So how many such links might be enough to get your pages out of ’supplemental?
Casey Yandle Said,
January 12, 2007 @ 4:39 am
Matt, I’m an avid reader of your blog; read it every day, but I never comment. I just try to absorb everything you write, well the info regarding SEO. I do have a few questions for you though.
If you have say 10 pages of your website in the supplemental results does that mean your website will not be ranked for keywords that you are targeting?
Another thing is why do pages get put into the supplemental results? If the pages haven’t been updated in say a year does that mean they are more likely to end up in the supplemental results?
Thanks for your informative blog!!!
Webscutest Said,
January 12, 2007 @ 5:02 am
Was wondering why internal pages with no links are getting a higher PR than the front page which has all the external links pointing to it ?
Example :
http://www.webscutest.com = PR 3
http://www.webscutest.com/archive/index.php/ = PR 4
Al Said,
January 12, 2007 @ 6:17 am
Thanks for answer Matt.
There are still few strange things.
1. Not all of my subdomains were affected. Affected and not affected websites have same structure, same promotion. All the same except keywords and website’s theme (one is celebrity photos, another one is car photos). Same PR, same links, everything is the same.
2. Also, top 20 websites in SERPs are websites with dublicate content too. Look at song lyrics websites - they all have duplicate content. Original content is impossible here. Still my website was removed from SERPs (I was #1-#10). Other websites (even few ones who copied my whole DB about year ago) are still there at #1-#10.
3. Some webmasters websites’ have returned their #1-#5 SERPs in few weeks after ‘disaster’. As they say they did not change anything on their websites during those few weeks.
4. Websites with completely unique content, high PR, white hat SEO were removed from index after years of beeing #1-#5.
These are very strange things.
Dara Said,
January 12, 2007 @ 6:25 am
Hi Matt
I may have that bug as described regarding .com and pages from the web etc..
website is http://www.quinn-direct.com, hosted in ireland.
on google.ie when searching site:www.quinn-direct.com pages from ireland, index page doesn’t seem to appear
Dara
Sally Said,
January 12, 2007 @ 6:29 am
Hi Matt
I must have spent hundreds of hours on google in the past 5 months trying to find out why some .com sites have been dropped from search UK listings. So it was great to read your reassuring post the other day. You said ”please leave a comment with the name of your site.” so hope this helps.
So here is that comment!
My site http://www.portraits4pets.com appeared back on UK listings for 2 days on 6-7 Jan then dissappeared and reappeared again on 9-10 Jan. It has now gone yet again.
Until August 27th http://www.portraits4pets.com had consistently ranked at number 1 or 2 on google UK for ”pet portraits”. But round about that time it was moved to a server in the USA, and dissappeared from UK listings. I had it moved straight back when Vanessa Fox kindly suggested that this was the cause of the problem. Since then the site has been intermittantly returning to UK listings and then going again and I definitely have more grey hairs now as a result.
I have been desperately searching for a solution for some time and now wonder could interlinking sites be a problem?
http://www.portraits4pets.com is an gallery of over 1000 of my past portraits, an archive of my work which I can easily update on a daily basis. http://www.sallylogue.co.uk is primarily information about the artist, forms, database, e-commerce etc managed with help from elsewhere. Could it be I’m being penalised for having 2 websites? As far as I can see theres no duplicate content, althought the root pages do now have a similar focus.
Thank you for your time
Sally
Tim Said,
January 12, 2007 @ 7:22 am
Hi Matt,
Just wondering if you could tell us where webmaster tools fits into the pagerank side of things, is the pagerank data there from the toolbar server, from one of the datacentres, or from another snapshot? The reason I ask is that the data in there seldom appears to be the same as in the toolbar, and the page on my site it says has highest pagerank seems to have PR 0 however else I look at it.
I’ve seen a couple of people ask about this around the web before, any chance you or Vanessa could enlighten us?
NetMidWest Said,
January 12, 2007 @ 7:28 am
I am seeing some odd things with the latest data…
For one site, anything not linked to from the home page is losing PR. It uses a strong 301 from non-www to www in the .htaccess. A site: query (either www or not) returns about 5,000 pages, but allinurl:www. only shows 62 (guess how many links on the homepage, win a prize!) while allinurl: (without) shows about 5,000 pages as well. I have not used the canonical url tool on this site, but there does seem to be a real problem with the canonicalization that is beyond my control.
Another thing I picked up on is this:
http://www.google.com/search?q=allinurl:imdb.com/r/
These are 302 redirects that are being crawled and cached… the content is amazon.com’s, however. With many sites going to PR0 in the update (amazon.com seems to be getting fixed with the auxillary push) is this the cause? More on this here…
Yet another problem seems to be with higher PR on inner pages, some pages getting higher PR while there are no incoming links from outside, some with all incoming links pointing to the homepage only. This has been around before, but seemed to resolve itself.
Have there been some changes in the cannonicalization rules, the handling of 30x redirects and such?
netmeg Said,
January 12, 2007 @ 7:59 am
I don’t think Webmaster Tools is terribly current - I have 62 sites in there, and NONE of my query stats have updated in over a month.
Jonathan Said,
January 12, 2007 @ 8:51 am
Thanks for the update Matt, you rock!
Jim Clouse Said,
January 12, 2007 @ 8:54 am
Matt:
Thanks so much for your feedback. I sincerely appreciate it. I will definitely take my site over to the help group. I respect tough love. Hey, I want the best site on the web.
With that said, I am confused by some of your remarks and would like clarification when you have the time. After reading your comments, I wonder if your comments were directed at http://hotelmotelnow.com/ or to hotel sites in general. Here is why I say that.
The hotel site you mentioned (http://hotelmotelnow.com/hotels.aspx/10204612/) is not verbatim to 40 other sites. And there are no bullet points on the page at all—never have been—as you state in your comments.
We designed this hotel page to be different than the millions of other hotel pages out there. We took the XML and formatted the page to be different. Accordingly, our 78,000 hotel’s layout is different than the other hotel sites. We want to be different and value-added.
Our original content is on the front end. Our technology is unique and was developed in 2001. We provide visual searching of hotels in large cities by depicting their exact location on city maps via clickable dots. Sorting can be done by price range, hot rate hotels, or chain name. Start with the home page and you will see. No other website does it this way. This is unique, compelling content. This type of searching adds tremendous value (re visitors are not looking for hotels outside their desired area this way. Therefore, it is a tremendous time saver.)
In summary, we have original content (our visual searching), good information (hotel information presented differently for all our 78,000 hotels) and are improving other aspects of the site that will indeed make it look great to visitors.
In just the past 52 days we have 1) changed the landing page of all 78,000 hotels, 2) deleted all duplicated content for those hotels and, 3) improved content of our remaining 4,000 webpages.
Under construction is the redesign of those remaining 4,000 pages, inclusion of user-friendly options (sorting by amenities, pet friendly, etc.), implementation of other XML-enabled features, and other features that will continue to set us apart from the competition.
We want to be as different from the other hotel websites as is humanly possible. Since we all, more or less, pull from the same database being totally different is impossible. However, our front end visual searching is what sets us apart from the competition and has for 6 years. Our back end is looking different and will continue to do so as we work on the site.
Given the changes Google has seen in http://hotelmotelnow.com/ specifically since mid-November and over the past year, Google should be impressed enough to lift the 56+ week penalty. We have shown good faith. Our best is yet to come!
Josh Said,
January 12, 2007 @ 9:04 am
Wow, great entry, but I wonder if this applies to what has happened to my site. I used to be in the top for most of my keywords, and now i don’t register for any of them. I still have a good PR and backlinks. This is discouraging!
Joeychgo Said,
January 12, 2007 @ 1:19 pm
Well - I guess the update to the update didnt complete or didnt work - even http://www.mattcutts.com/blog/ is showing a PR0 in my toolbar. My homepage at http://www.vbulletin-faq.com/ went from a PR4 to a PR6 in some datacenters, but not others. Whats disappointing is that some newer sites of mine have no PR anywhere, despite having links and being crawled and indexed.
adrian Said,
January 12, 2007 @ 1:31 pm
Matt
Responding to your request for .com / co.uk site delistings of index pages, my site http://www.selfadhesivelabels.com did initially suffer from being delisted in the UK only search, but I thought I solved it with a lang=”en-GB” list in the opening html tag. This has worked for about 6 weeks now, but the new rolloutout has delisted it again…
We are a well established UK site, hosted in the UK, and with a UK focus, and suspect that the new infrastructure isn’t exactly helping us… any comments appreciated.
Thanks
Adrian
Venus Said,
January 12, 2007 @ 3:54 pm
Hi Matt,
Thanks for this detailed report,
I like this country-website based system, nice job.
pittfall Said,
January 12, 2007 @ 4:18 pm
Matt,
I know that this is probably a tired question, but I have heard you speak and write about the fact that SEOs shouldn’t consider PageRank in their optimization efforts, however, here you said:
“the main determinant of whether a url is in our main web index or in the supplemental index is PageRank.”
I can understand this and appreciate it, however, this further reinforces the argument of why doesn’t Google update toolbar PageRank more frequently? (i.e. Pandia’s Xmas Wish)
To which you continued (from quote above), “The approach I’d recommend in that case is to use solid white-hat SEO to get high-quality links (e.g. editorially given by other sites on the basis of merit).”
The building of links goes back to the age old question: “What came first, the chicken or the egg?” I build quality content and make it accessible to my visitors, however, if my pages are in the supplemental results (due to lower PageRank) and an editorial researcher is working on a piece using Google to find information, they won’t find my pages.
I practice only white-hat techniques, but the only true white-hat approach is “build it and they will come.” I would love to hear your thoughts on this and thank you for being such an approachable person, in real-life and across your blog.
pittfall Said,
January 12, 2007 @ 4:58 pm
Matt,
I have a request. You have stated before that SEOs focus too much on PageRank, but the most recent version of the toolbar only gives us backlinks and such if PageRank is enabled, can this be changed?
Thanks again!
Michael Lopez Said,
January 12, 2007 @ 6:34 pm
Just an update… My PR is now back to normal. In fact it increased by 1. Matt is right - of course he is. LOL
van coussens Said,
January 13, 2007 @ 5:47 am
Matt
Great blog!
Thanks for the latest update - if you get a chance please would you have a quick look at our site - http://www.churchill.com - as it may be impacted by the ”pages in the uk’ problem. The rankings have been jumping all over the place, not sure why.
Thanks very much.
Ian Smith Said,
January 13, 2007 @ 6:32 am
Thanks Matt for your time/trouble in clarifying all this for us.
Above you stated:
“Adam, there’s a pretty rare set of circumstances where this happens. I would give it a few days, and then if it’s still happening, pipe up then and mention your site name on this thread.”
This is is respone to the comment you made:
“An SEO or two has been holding my feet to the fire about root pages of .com’s that are hosted outside the US.”
As of today, I know of at least 3 sites where this is still happening.
Should we just ’sit tight’ and be a bit more patient? Or is there still a problem?
Hope you enjoyed your hol.
Andrew Said,
January 13, 2007 @ 9:54 am
Hi Matt.
You say the update is a snap-shot of the live PR, can you tell us when the snapshot was taken? Is it taken on the date the update is pushed or sometime before hand? My site has been online several months and I have a reason number of backlinks to the site, some from some highish PR pages, but I’m still not see a PR for my site on the toolbar.
Thanks.
Andrew.
Steve Browne Said,
January 13, 2007 @ 5:23 pm
We get this issue with http://www.wiisearcher.com
It has sections for UK and USA people looking to find a Wii, so is relevent to both sets of users.
http://www.wiisearcher.com - Hosted in Germany - google.co.uk (UK Only search) thinks it doesn’t exist.
We have the same for a bunch of other similar sites for other products which simply do not show on UK google event though they are half US-centric and half UK-centric. But Wiisearcher.com would be a good start!
Nikhil Jogia Said,
January 13, 2007 @ 11:33 pm
This isn’t my site, in fact it’s just a random site I found whilst looking for more sites affected by the disappearing sites in regional indexes problem.
http://www.google.com.au/search?hl=en&q=site%3Awww.balancebodybenches.com&btnG=Google+Search&meta=
Now select “Pages From Australia” and the root of the domain disappears!
Nilhan Said,
January 14, 2007 @ 2:32 am
Thanks for the update on the geo-location issues. The fix is working for me so far. Any chance of telling us what went wrong?
Roy Dale Said,
January 14, 2007 @ 9:09 am
More examples of UK Web sites appearing in “the web” results but not appearing at all in “pages from the UK” results on google.co.uk are: gsiglasers.com, autodriveoptions.com and vance-miller.com - and an example of a UK site with many pages listed in “the web” results but only two in “pages from the UK” results (which seems the wrong way about if any distinction has to be made at all) is acrlimited.com
David Viney Said,
January 14, 2007 @ 10:51 am
Hi Matt - good to see the focus on Google UK results (particularly for .coms) which myself and other forum members have frequently chewed over at WebProWorld.
I did have a related question concerning this: does Google take into account local listings (e.g. Google Maps in UK) in determining a site’s origin. If not, are there plans to do so? (after all, is based on a PIN code address verification by post so should defeat spammers).
I ask this, as many of my own sites (and some of my clients) are hosted with 1and1.co.uk which is one of the UK’s most popular hosting providers but has servers in Germany (causing Google to recognise them as “Seiten aus Deutschland”).
Roy Dale Said,
January 14, 2007 @ 1:53 pm
Another thing I notice when comparing site:yourdomain.com results on google.co.uk using “the web” and “pages from the UK” is that as well as fewer pages being listed via “pages from the UK”, the default or Home Page of the site (which typically heads the list using “the web”) is usually missing on the much shorter list that you see using “pages from the UK”. The contact page is usually included on the “pages from the UK” list, as if that is a rule of some kind, but the default/Home Page is not. A typical example is matcreations.com. The site belongs to a UK business and it is hosted in the UK.
joe Said,
January 14, 2007 @ 6:58 pm
Matt 2 weeks ago we submitted our 2nd re-inclusion request after having a “minus 30 penalty” since late October 2005. This re-inclusion request and our site fixes worked and we had the minus 30,40,50 penalty lifted last week which was great obviously.
We assume that our discontinuation of text link advertising on our site helped us get out of this mess.
Now as of today we appear to have dissapeared in the SERP’s again for most searches bar a few major ones which aren’t really traffic generating. Should I submit another re-inclusion request or bide my time. I’m concerned about p!$$!ng Google engineers off with uneccessary re-inclusion requests and at the same time am also concerned with what has now happened.
Kaediem Said,
January 14, 2007 @ 7:23 pm
Hi Matt, this certainly has been a most informative post and run of comments - perhaps by the time you return from your vacation this issue will be resolved but I too am wondering about the page rank of home pages being pr0 while the internal pages have pr3/pr4.
I’m talking about new sites, for example http://www.emailhoaxesurbanlegends.com This is surely the puzzle du jour for many webmasters.
Thanks
Lisa
Jacopo Gonzales Said,
January 15, 2007 @ 12:15 am
Hi Matt,
I have a question for you: how come a website like Searchengineland.com (and many other sites) isn’t showing its PR on the toolbar yet?
Still just a matter of days?
Cheers
BigMan Said,
January 15, 2007 @ 12:24 am
Hello Matt, I have the problem with the root page disappearing from the UK searches of Google.co..uk. All of my other pages are cached. The root page appeared yesterday for a few hours, but is gone again today. I realise I’m not the only one who faces this issue, but is there anything that can be done to rectify the situation?
Thanks Matt.
wolfgang Said,
January 15, 2007 @ 1:15 am
hi Matt,
more then 40 DC IPs show the old Pagerank from july2006 PR update.
Example:
64.233.163. /19 /44 /80 /83 / 84/ /99 /104 /107 /115 /133 /184 /189
So now we have DC IPs with
PR from july
PR from october
PR from december
sem4u Said,
January 15, 2007 @ 1:15 am
I have got an example of a .com domain hosted in the UK with the homepage not showing up in the http://www.example.com search for “pages from the UK” search.
The site is http://www.columbusdt.com
I also have two more .com domains with the same problem. Any help or input would be much appreciated
sem4u Said,
January 15, 2007 @ 1:17 am
Sorry I meant site:www.example.com search!
Roy Dale Said,
January 15, 2007 @ 8:06 am
RE: sem4u Said,
January 15, 2007 @ 1:15 am
I have got an example of a .com domain hosted in the UK with the homepage not showing up in the “pages from the UK” search.
The site is http://www.columbusdt.com
The site belongs to a USA-based business with a USA street address… so its pages are not “pages from the UK”. In this case, the UK filter appears to be doing exactly the job that you’d expect it to do.
Danny Said,
January 15, 2007 @ 8:28 am
Well we”re 5 days later and here’s another exemple of a .com with a root missing in local search.
http://www.madeira.vakantie.com is hosted in Belgium.
A site: search on any worldwide google (.com , .be, .nl etc.) will show 48 pages.
Switching to google.be and turning on ’sites from Belgium only’ will show 47 pages with the index page being the one that’s missing.
Danny Said,
January 15, 2007 @ 8:29 am
oops, should be http://www.madeira-vakantie.com
Derek Martin Said,
January 15, 2007 @ 12:44 pm
The assumption on missing homepages in uk only searches is that this happens for sites hosted outside the uk. Believe me this is not the only case. I have a 5 year old .com site hosted in the uk that up until a few months ago ranked very well for all key search terms in “uk only ” pages. Every 2 months or so the home page disappears from Google indexing for “uk results only”. Still there in the .co.uk index but not in uk only, its out again now after the 11 Jan cache.
I waited 5 days like advised but its just disappeared again- doesn’t look like the fix is working does it?
Its not just non UK hosted sites - so whats causing it?
Derek Martin Said,
January 15, 2007 @ 12:47 pm
Sorry, the site url is http://www.natco-online.com
Russell W Said,
January 15, 2007 @ 12:58 pm
*** if you still see the behavior after 4-5 days ***
Yes, for http://www.warburtech.com/ the root page is missing in “pages from the UK” searches; also about 35 pages from the site completely dropped out of the index many months ago, and another dozen more dropped out several months ago. All alternative domains, and non-www URLs, have had a site-wide 301 redirect to the .com version, for several years.
g1smd Said,
January 15, 2007 @ 1:17 pm
>> This page is showing a 0 pr at the moment
If the page is, say, domain.com/index.html then DO check to see what PR each of these alternative URLs have:
domain.com/
http://www.domain.com/
http://www.domain.com/index.html
It is possible that Google favours a DIFFERENT URL to the one that you tried to get listed.
A 301 redirect to the canonical URL from each and all of the alternatives will clear the problem up in just a few months.
I usually pick http://www.domain.com/ as the one to be listed, and for folders, they would be http://www.domain.com/foldername/ instead (additionally notice the trailing “/” on the URL.
Derek Martin Said,
January 15, 2007 @ 1:48 pm
g1smd - sorry were you talking about my site? PR is 5 not 0 ?
Danny Said,
January 15, 2007 @ 2:58 pm
Hi G1
Long time no see.
It’s not a canonical problem this time.
In my case, i’ve a 301 redirect in place from non-www to www version, and all links point to the root without index.html
Everything else is all white hat as usual
The site is too young to have any PR so i’m not stressing over this
Matt suggested to give it 4-5 days for Google to fix this bug and asked exemples if we could still see this behaviour after that period.
I added mine just to show that non-English sites are affected too.
sem4u Said,
January 15, 2007 @ 2:59 pm
Roy Dale - OK bad example there but I have two sites that don’t have contact details, as they are information sites, which suffer the same ‘no homepage showing up’ problem.
Ian Smith Said,
January 15, 2007 @ 4:21 pm
Just an update - the 3 .com sites I talked about 3 days ago where the homepage was not showing up in the “pages from the UK” search.” The homepages are still not showing up so it would appear there is still a problem.
Mike Papageorge Said,
January 16, 2007 @ 1:08 am
Hi Matt,
http://www.masainternational.com seems to be having the issue where the index page does not show up in the Pages from the UK site: search.
Wim Hoogenraad Said,
January 16, 2007 @ 1:31 am
Hello Matt,
Thanks for this very informative topic. I was just checking my http://www.webmasterslookup.com (hosted in the Netherlands) at http://www.google.co.uk. I think te problem of “pages from the UK” is still there. Also I stay puzzled by the PageRank concept. Except for the IP-ranges you mentioned the PR of Webmasterslookup is 5. Sometimes however (mostly in the mornings ECT) it is 4 in the Google bar….
Is there separated server that serves the Google bar??
Thanks again for this Topic,
Wim
Nial Said,
January 16, 2007 @ 2:28 am
Hi Matt, any idea what percentage of google.co.uk users use the pages from the UK feature??
Cheers
Nial
laragood Said,
January 16, 2007 @ 2:42 am
Hello my first ever blog!
http://www.floorheatingonline.com/ is hosted in the UK and whilst it does appears in “Search the web” is nowhere when search pages from the UK”. Have you any information as to when this issue will be fixed and/or do I need to do anything?
Thankyou.
deedee Said,
January 16, 2007 @ 5:06 am
Matt,
Seems that your RSS feed doesn’t work anymore. Can you help my reader please?
askcybersteve Said,
January 16, 2007 @ 6:19 am
Hey Matt,
Hope your enjoying your holidays. Get stuck into some of those books that you never get a chance to read
I think I might take a holiday from google until you get back as well..seems to be a good time of year for it.
cheers mate
Steve Ash Said,
January 16, 2007 @ 6:35 am
Hi Matt, i run the SEO for http://www.mobileshop.com, its a uk based and targeted web site, i have been hit by the latest update, and do not appear when i search the UK, the .com search is fine, can you have a look and see if you can get an update for me please
Nate121 Said,
January 16, 2007 @ 7:10 am
Matt,
I am in the same boat as hotelmotelnow.com (Socks Manly).
My site has been under a penalty for close to a year, also the same phenomenon that my domain search places me on pages 4+ along with all previously well ranked keyword searches. I have posted on the google boards and did get the “tough love”. I implemented all suggestions. Yes I have affiliate links however my site includes reviews, articles, multimedia downloads, etc. and more.
What do you recommend?
Thanks for reading this and taking the time.
Nate
Nate121 Said,
January 16, 2007 @ 7:44 am
CORRECTION.. I meant to write my site is the same as Jim Clouse
NOT SOCKS MANLY
Jason Said,
January 16, 2007 @ 8:37 am
Hi Matt
Just wondering if the udate is finished or still ongoing as i have a couple of sites that were previously PR0 but now all the internal pages have been given PR3 but home page on both is still 0. Is this normal. Whats going on?
I went to a couple of the forums seeking answers and quite a few people are experiencing the same thing!
Please help
Many thanks
Zach Hoffman Said,
January 16, 2007 @ 9:48 am
Google PR is a great tool, but as I have seen before it is not the answer all to being ranked. It seems to be one of our last focuses when kicking our competitor’s asses, we grind out the rankings and when the dust clears check the PR. Are we the only ones kicking ass and taking names when we get around to it? Or is PR a real hook for all you SEO’ers out there.
Danny Said,
January 16, 2007 @ 3:41 pm
update:
it looks like the fix is starting to show.
The homepage from the site i mentioned earlier is back in the site: searches for local pages at google.be
Thank you Google, and thank you Matt.
It was a fun thread
And most likely in the next few days we may see remarks from people who may think that their site is hosted in a specific country while it’s in fact hosted in another due to nameservers with tld’s belonging to another.
eg. .com sites hosted on a uk based server which uses .de nameservers
Matt, maybe you should open a thread someday about how the country of a site is determined.
Still i wonder what caused this temporary bug. Preparing for the upcoming google.eu ? Any ideas/suggestions on how this one will deal with local searches ?
Ramone Said,
January 16, 2007 @ 5:18 pm
Matt,
strange things happening here: I am the webmaster of a multi-lingual website by a friend of mine. The Site itself is available in English, German, French, Spanish and Italian. In October I reconstructed the whole site. The Spanish and Italian translations were not ready at that time, so the site went on-line (around the time of the september PR-export) in English, German and French, with a structure like:
http://www.domain.com/en/...
http://www.domain.com/de/...
http://www.domain.com/fr/...
Two months later the Spanish and Italian versions got published too:
http://www.domain.com/it/...
http://www.domain.com/es/..
The root-page (www.domain.com) had pagerank 4, all the other pages had one of 0 as they didn`t exist before.
Now, after the recent pagerank export the root kept its pagerank of 4. All the subsequently updated pages in /it/ and /es/ now have a pagerank of 4 too. All others (English, French, German) remained with their 0, though almost all inbound links point to the English version or the root.
Now, I am more than confused. Could you please try to explain how something like this can happen?
Thanks in advance!
Chris Boggs Said,
January 16, 2007 @ 6:18 pm
Matt I know you are a busy man but if you have a second to comment at http://forums.searchenginewatch.com/showthread.php?t=15834 it would be appreciated.
Ron Gupta Said,
January 16, 2007 @ 7:44 pm
Hi Matt,
It is very important you should clarify one point we have a long discussion is going on in WMW that there s -950 or end of result penalty. Even those sites were perfectly legal did not do any kind things other than what normal visitor would need even then they got this penalty. Is there any way we can come out from there.
Regards
COLE WEBB Said,
January 16, 2007 @ 8:11 pm
good research cutts. some of these bastards want to charge 10 grand for simple keyword typing so im doing it my dam self. Later.
RobD Said,
January 16, 2007 @ 9:49 pm
Matt, Thanks for the updates, love your blog. Quick question about the most recent infrastructure updates. I have noticed my backlinks (www.sharkhandicapping.com) go from 131 in August, to 35 in October to 131 in Nov to 86 in Dec and now 36 in january. During this time, we have been adding several back links via advertising and have eliminated non-relevant outgoing links. Wou