Comments on: Crawl caching proxy http://www.mattcutts.com/blog/crawl-caching-proxy/ neat fun stuff Wed, 08 Feb 2012 21:30:01 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Ayush Agrawalhttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-840882 Ayush Agrawal Tue, 12 Jul 2011 20:58:08 +0000 http://www.mattcutts.com/blog/?p=282#comment-840882 I don't know if caching and compressions had anything to do with SEO, anyway even if it had, we thought it would only be a positive impact, however we have noticed a significant decline in our SERP rankings (concerned website - <a href="http://www.filmitadka.in" rel="nofollow">FilmiTadka</a>) after we enabled Gzip compression and Apache Keep Alive.Please advice am I even thinking in the right direction? I don’t know if caching and compressions had anything to do with SEO, anyway even if it had, we thought it would only be a positive impact, however we have noticed a significant decline in our SERP rankings (concerned website – FilmiTadka) after we enabled Gzip compression and Apache Keep Alive.

Please advice am I even thinking in the right direction?

]]> By: David Thomas Marquardthttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-825378 David Thomas Marquardt Tue, 28 Jun 2011 17:35:02 +0000 http://www.mattcutts.com/blog/?p=282#comment-825378 I have been looking for this info. Thanks Matt! I have been looking for this info. Thanks Matt!

]]> By: Bambarbiahttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-400112 Bambarbia Mon, 05 Oct 2009 17:18:24 +0000 http://www.mattcutts.com/blog/?p=282#comment-400112 Hi Matt,What is current status of this? I noticed for each incoming User-request Mediapartners-Google crawls a page 2 times - just because I have 2 AdSence units on a page!!!Such an obvious performance bottleneck... I can use HTTP caching to help Mediapartners - but I'll have to disable gzipped output (problems with Apache HTTPD)Additionally, I have some pages restricted from Googlebot, but enabled for Mediapartners-Google. Hi Matt,

What is current status of this? I noticed for each incoming User-request Mediapartners-Google crawls a page 2 times – just because I have 2 AdSence units on a page!!!

Such an obvious performance bottleneck… I can use HTTP caching to help Mediapartners – but I’ll have to disable gzipped output (problems with Apache HTTPD)

Additionally, I have some pages restricted from Googlebot, but enabled for Mediapartners-Google.

]]> By: Used limousineshttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-349157 Used limousines Thu, 18 Jun 2009 04:13:17 +0000 http://www.mattcutts.com/blog/?p=282#comment-349157 Matt’s diagram pretty much matches what we’ve been seeing, although I still think there might be some issues regarding robots.txt, but we’re still collecting some data on that, so I’m going to wait until that’s done before commenting on it.Thanks David Janes Matt’s diagram pretty much matches what we’ve been seeing, although I still think there might be some issues regarding robots.txt, but we’re still collecting some data on that, so I’m going to wait until that’s done before commenting on it.

Thanks
David Janes

]]> By: Electric Golf Trolleyshttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-277687 Electric Golf Trolleys Mon, 16 Mar 2009 11:09:20 +0000 http://www.mattcutts.com/blog/?p=282#comment-277687 As a Qualified Microsoft MCSE NT4 and 2000 server series, I know that the original web server calle Proxy server and especially the later editions called ISA 200# (Internet Security Acceleration ) Server caches many pages for your LAN network. So i guess the server models used by ISPs are doing the same. Like your I.E. local PC application - how many times do you use F5 to refresh? As a Qualified Microsoft MCSE NT4 and 2000 server series, I know that the original web server calle Proxy server and especially the later editions called ISA 200# (Internet Security Acceleration ) Server caches many pages for your LAN network. So i guess the server models used by ISPs are doing the same. Like your I.E. local PC application – how many times do you use F5 to refresh?

]]> By: Svi Poslovihttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-128006 Svi Poslovi Fri, 30 May 2008 07:49:16 +0000 http://www.mattcutts.com/blog/?p=282#comment-128006 I switched to VPS and now my job listings aggregator works like a charm. Well, there was a shift from cPanel to Plesk, as well, but I don't think cPanel was directly responsible. It was probably some flooding protection... I switched to VPS and now my job listings aggregator works like a charm. Well, there was a shift from cPanel to Plesk, as well, but I don’t think cPanel was directly responsible. It was probably some flooding protection…

]]> By: Svi poslovihttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-125574 Svi poslovi Mon, 07 Apr 2008 18:43:38 +0000 http://www.mattcutts.com/blog/?p=282#comment-125574 Interesting post. I still do have problems with Googlebot crawling my pages after I turn on optimization (gzip compression) for: text/html text/plain text/xml text/css application/javascriptYou wouldn't believe it. Nothing is crawled. I look into it and let you know if I find out what's the problem. If anyone here knows about it, yell it out! :-) Interesting post. I still do have problems with Googlebot crawling my pages after I turn on optimization (gzip compression) for:
text/html
text/plain
text/xml
text/css
application/javascript

You wouldn’t believe it. Nothing is crawled. I look into it and let you know if I find out what’s the problem. If anyone here knows about it, yell it out! :-)

]]> By: David Hornehttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-121286 David Horne Thu, 24 Jan 2008 20:10:43 +0000 http://www.mattcutts.com/blog/?p=282#comment-121286 Also just as a side note remember it is extremely important to do a proper robots.txt for your proxy if you don't then your proxy will crawl the web and google will index other peoples websites under your domain. This seems great but its pretty unethical if you ask me and causes a lot more problems then the benefits of fresh content.Here is the proper format for PHProxy 5User-agent: * Disallow : /index.php?Just put that in your robots.txt file and upload it to the root of your server.There are other ones for cgi proxy and glype.If your worried you can't do well in the Search Engines without stealing content check out my site at http://www.proxybolt.com and I guarentee you I am doing very well in the SE's and have a ton of visitors without having to steal. Also just as a side note remember it is extremely important to do a proper robots.txt for your proxy if you don’t then your proxy will crawl the web and google will index other peoples websites under your domain. This seems great but its pretty unethical if you ask me and causes a lot more problems then the benefits of fresh content.

Here is the proper format for PHProxy 5

User-agent: *
Disallow : /index.php?

Just put that in your robots.txt file and upload it to the root of your server.

There are other ones for cgi proxy and glype.

If your worried you can’t do well in the Search Engines without stealing content check out my site at http://www.proxybolt.com and I guarentee you I am doing very well in the SE’s and have a ton of visitors without having to steal.

]]> By: Bishanhttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-114152 Bishan Wed, 10 Oct 2007 05:13:15 +0000 http://www.mattcutts.com/blog/?p=282#comment-114152 Hi Matt,I have a site_map.htm page on my site which contains links to all of the pages on my site. google has indexed the page successfully but when I look at the cached page it is months out of date. google states that the page is a snapshot from a crawl on December 27th 2005 but this cannot be the case as the version shown is months old. i.e. I have updated the site map with several new pages but they aren't appearing on the cached version.Any ideas on what the problem is?Thanks Bishan Hi Matt,

I have a site_map.htm page on my site which contains links to all of the pages on my site. google has indexed the page successfully but when I look at the cached page it is months out of date. google states that the page is a snapshot from a crawl on December 27th 2005 but this cannot be the case as the version shown is months old. i.e. I have updated the site map with several new pages but they aren’t appearing on the cached version.

Any ideas on what the problem is?

Thanks
Bishan

]]> By: Yuriyhttp://www.mattcutts.com/blog/crawl-caching-proxy/#comment-113526 Yuriy Wed, 26 Sep 2007 23:08:07 +0000 http://www.mattcutts.com/blog/?p=282#comment-113526 At last I found this information. Thanks a lot. It is a part of my diploma now !! :-) Any progress in "Bigdaddy" field ? There is not too much information in Internet about this. Thanks again. At last I found this information. Thanks a lot. It is a part of my diploma now !! :-) Any progress in “Bigdaddy” field ? There is not too much information in Internet about this. Thanks again.

]]>