Comments on: Crawl caching proxy http://www.mattcutts.com/blog/crawl-caching-proxy/ neat fun stuff Fri, 06 Nov 2009 18:35:07 -0800 http://wordpress.org/?v=2.8.5 hourly 1 By: Bambarbia http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-400112 Bambarbia Mon, 05 Oct 2009 17:18:24 +0000 http://www.mattcutts.com/blog/?p=282#comment-400112 Hi Matt, What is current status of this? I noticed for each incoming User-request Mediapartners-Google crawls a page 2 times - just because I have 2 AdSence units on a page!!! Such an obvious performance bottleneck... I can use HTTP caching to help Mediapartners - but I'll have to disable gzipped output (problems with Apache HTTPD) Additionally, I have some pages restricted from Googlebot, but enabled for Mediapartners-Google. Hi Matt,

What is current status of this? I noticed for each incoming User-request Mediapartners-Google crawls a page 2 times – just because I have 2 AdSence units on a page!!!

Such an obvious performance bottleneck… I can use HTTP caching to help Mediapartners – but I’ll have to disable gzipped output (problems with Apache HTTPD)

Additionally, I have some pages restricted from Googlebot, but enabled for Mediapartners-Google.

]]> By: Used limousines http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-349157 Used limousines Thu, 18 Jun 2009 04:13:17 +0000 http://www.mattcutts.com/blog/?p=282#comment-349157 Matt’s diagram pretty much matches what we’ve been seeing, although I still think there might be some issues regarding robots.txt, but we’re still collecting some data on that, so I’m going to wait until that’s done before commenting on it. Thanks David Janes Matt’s diagram pretty much matches what we’ve been seeing, although I still think there might be some issues regarding robots.txt, but we’re still collecting some data on that, so I’m going to wait until that’s done before commenting on it.

Thanks
David Janes

]]> By: Electric Golf Trolleys http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-277687 Electric Golf Trolleys Mon, 16 Mar 2009 11:09:20 +0000 http://www.mattcutts.com/blog/?p=282#comment-277687 As a Qualified Microsoft MCSE NT4 and 2000 server series, I know that the original web server calle Proxy server and especially the later editions called ISA 200# (Internet Security Acceleration ) Server caches many pages for your LAN network. So i guess the server models used by ISPs are doing the same. Like your I.E. local PC application - how many times do you use F5 to refresh? As a Qualified Microsoft MCSE NT4 and 2000 server series, I know that the original web server calle Proxy server and especially the later editions called ISA 200# (Internet Security Acceleration ) Server caches many pages for your LAN network. So i guess the server models used by ISPs are doing the same. Like your I.E. local PC application – how many times do you use F5 to refresh?

]]> By: Svi Poslovi http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-128006 Svi Poslovi Fri, 30 May 2008 07:49:16 +0000 http://www.mattcutts.com/blog/?p=282#comment-128006 I switched to VPS and now my job listings aggregator works like a charm. Well, there was a shift from cPanel to Plesk, as well, but I don't think cPanel was directly responsible. It was probably some flooding protection... I switched to VPS and now my job listings aggregator works like a charm. Well, there was a shift from cPanel to Plesk, as well, but I don’t think cPanel was directly responsible. It was probably some flooding protection…

]]> By: Svi poslovi http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-125574 Svi poslovi Mon, 07 Apr 2008 18:43:38 +0000 http://www.mattcutts.com/blog/?p=282#comment-125574 Interesting post. I still do have problems with Googlebot crawling my pages after I turn on optimization (gzip compression) for: text/html text/plain text/xml text/css application/javascript You wouldn't believe it. Nothing is crawled. I look into it and let you know if I find out what's the problem. If anyone here knows about it, yell it out! :-) Interesting post. I still do have problems with Googlebot crawling my pages after I turn on optimization (gzip compression) for:
text/html
text/plain
text/xml
text/css
application/javascript

You wouldn’t believe it. Nothing is crawled. I look into it and let you know if I find out what’s the problem. If anyone here knows about it, yell it out! :-)

]]> By: David Horne http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-121286 David Horne Thu, 24 Jan 2008 20:10:43 +0000 http://www.mattcutts.com/blog/?p=282#comment-121286 Also just as a side note remember it is extremely important to do a proper robots.txt for your proxy if you don't then your proxy will crawl the web and google will index other peoples websites under your domain. This seems great but its pretty unethical if you ask me and causes a lot more problems then the benefits of fresh content. Here is the proper format for PHProxy 5 User-agent: * Disallow : /index.php? Just put that in your robots.txt file and upload it to the root of your server. There are other ones for cgi proxy and glype. If your worried you can't do well in the Search Engines without stealing content check out my site at http://www.proxybolt.com and I guarentee you I am doing very well in the SE's and have a ton of visitors without having to steal. Also just as a side note remember it is extremely important to do a proper robots.txt for your proxy if you don’t then your proxy will crawl the web and google will index other peoples websites under your domain. This seems great but its pretty unethical if you ask me and causes a lot more problems then the benefits of fresh content.

Here is the proper format for PHProxy 5

User-agent: *
Disallow : /index.php?

Just put that in your robots.txt file and upload it to the root of your server.

There are other ones for cgi proxy and glype.

If your worried you can’t do well in the Search Engines without stealing content check out my site at http://www.proxybolt.com and I guarentee you I am doing very well in the SE’s and have a ton of visitors without having to steal.

]]> By: Bishan http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-114152 Bishan Wed, 10 Oct 2007 05:13:15 +0000 http://www.mattcutts.com/blog/?p=282#comment-114152 Hi Matt, I have a site_map.htm page on my site which contains links to all of the pages on my site. google has indexed the page successfully but when I look at the cached page it is months out of date. google states that the page is a snapshot from a crawl on December 27th 2005 but this cannot be the case as the version shown is months old. i.e. I have updated the site map with several new pages but they aren't appearing on the cached version. Any ideas on what the problem is? Thanks Bishan Hi Matt,

I have a site_map.htm page on my site which contains links to all of the pages on my site. google has indexed the page successfully but when I look at the cached page it is months out of date. google states that the page is a snapshot from a crawl on December 27th 2005 but this cannot be the case as the version shown is months old. i.e. I have updated the site map with several new pages but they aren’t appearing on the cached version.

Any ideas on what the problem is?

Thanks
Bishan

]]> By: Yuriy http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-113526 Yuriy Wed, 26 Sep 2007 23:08:07 +0000 http://www.mattcutts.com/blog/?p=282#comment-113526 At last I found this information. Thanks a lot. It is a part of my diploma now !! :-) Any progress in "Bigdaddy" field ? There is not too much information in Internet about this. Thanks again. At last I found this information. Thanks a lot. It is a part of my diploma now !! :-) Any progress in “Bigdaddy” field ? There is not too much information in Internet about this. Thanks again.

]]> By: Prem Kasera http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-112454 Prem Kasera Mon, 10 Sep 2007 12:46:19 +0000 http://www.mattcutts.com/blog/?p=282#comment-112454 Hi Matt I have purchased a expired domain before few weeks.put a web pages that shows "Under Maintenance" this has been live over a month. url is http://www.scryypy.com whenever i am checking the google cached version it shows me cache of website http://designerpad.org/ (http://209.85.135.104/search?sourceid=navclient&ie=UTF-8&rlz=1T4SKPB_enIN235IN236&q=cache:http%3A%2F%2Fwww.scryypy.com%2F) Can you help me why it is and how can i resolve this prob. Thanks Hi Matt
I have purchased a expired domain before few weeks.put a web pages that shows “Under Maintenance” this has been live over a month. url is http://www.scryypy.com whenever i am checking the google cached version it shows me cache of website http://designerpad.org/ (http://209.85.135.104/search?sourceid=navclient&ie=UTF-8&rlz=1T4SKPB_enIN235IN236&q=cache:http%3A%2F%2Fwww.scryypy.com%2F)

Can you help me why it is and how can i resolve this prob.

Thanks

]]> By: SEO Expert Dubai http://www.mattcutts.com/blog/crawl-caching-proxy/#comment-111765 SEO Expert Dubai Tue, 28 Aug 2007 11:23:39 +0000 http://www.mattcutts.com/blog/?p=282#comment-111765 Hi Matt, Thank you for clearing this issues witch was really hard to understand from other webmaster's point of view. any how it was clear and detailed information. Thanks again. Hi Matt,

Thank you for clearing this issues witch was really hard to understand from other webmaster’s point of view. any how it was clear and detailed information.

Thanks again.

]]>