Comments on: How to fetch a url with curl or wget silently http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/ neat fun stuff Sun, 12 Oct 2008 10:58:00 +0000 http://wordpress.org/?v=2.6.2 By: دروس http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-105017 دروس Thu, 24 May 2007 22:26:01 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-105017 Oh…for a min there I though wget –spider http://example.com would give me all the links spidered right from the default site page. Something like Google’s site:http://example.com cgiproxy guy Said, January 21, 2007 @ 7:06 am wget is a program with incredible untapped potential for most people I personally like wget –delete-after http://website.com it deletes the output after the execution, not as nice as quite or /dev/null 2>&1 but very powerful still. also wget will work with tor, just a question of having tor proxy set up right on your server and digging for the additional commands. Jeff Huckaby Said, April 10, 2007 @ 7:21 pm Verbosity can be good. Wget or curl with their respective “quiet” options will silence some output from those scripts but not all. They will still likely show critical errors, which is why you may want the redirects to /dev/null. However, we often see cases where you need some errors but not others. wget has a -nv flag that is not verbose but not quiet. You can also use /etc/cron.d/filename on most linux systems to fine-tune your cron. You can specify a mail address within the file you place in this directory. This can be useful to alert someone in case of a problem. Also, don’t overlook security. Run your crons with a user with as few privileges as possible. If you simply need to wget a file, then a normal user with no login privileges will often suffice. Lastly, don’t forget –tries=number option. This will have wget retry in case of a failure. Note the default is 20 retries unless a failure occurs. There is also a –retry-connrefused which will retry even when a connection is refused, useful for overloaded URLs. Lastly, there is the –timeout option. Always use this option if you are fetching URLs frequently. The default read timeout is 900 seconds. That’s 15 minutes! I’ve seen many servers with dozens of crons piled up because they are polling every 5 minutes but the server is slow, so they are waiting 10 minutes or so to get the data. The problem quickly snowballs out of control. In brief, we recommend: 1. use the least privileged user as possible for the user running the cron. 2. explicitly set timeouts to work with your application 3. decide what level of error reporting you need and use -q -nv and/or /etc/cron.d as required. These tips are mostly for wget but curl has many of the same options. Lastly, one more security tip. We often create a “wgetforuser” which is wget with permission that users can use. We then set the main wget to only be used by root. This helps (does not prevent) some attacks where a wget command is passed into an insecure web application. http://www.ihsac.com Oh…for a min there I though wget –spider http://example.com would give me all the links spidered right from the default site page. Something like Google’s site:http://example.com

cgiproxy guy Said,
January 21, 2007 @ 7:06 am

wget is a program with incredible untapped potential for most people

I personally like wget –delete-after http://website.com

it deletes the output after the execution, not as nice as quite or /dev/null 2>&1 but very powerful still.

also wget will work with tor, just a question of having tor proxy set up right on your server and digging for the additional commands.

Jeff Huckaby Said,
April 10, 2007 @ 7:21 pm

Verbosity can be good. Wget or curl with their respective “quiet” options will silence some output from those scripts but not all. They will still likely show critical errors, which is why you may want the redirects to /dev/null. However, we often see cases where you need some errors but not others. wget has a -nv flag that is not verbose but not quiet. You can also use /etc/cron.d/filename on most linux systems to fine-tune your cron. You can specify a mail address within the file you place in this directory. This can be useful to alert someone in case of a problem.

Also, don’t overlook security. Run your crons with a user with as few privileges as possible. If you simply need to wget a file, then a normal user with no login privileges will often suffice.

Lastly, don’t forget –tries=number option. This will have wget retry in case of a failure. Note the default is 20 retries unless a failure occurs. There is also a –retry-connrefused which will retry even when a connection is refused, useful for overloaded URLs.

Lastly, there is the –timeout option. Always use this option if you are fetching URLs frequently. The default read timeout is 900 seconds. That’s 15 minutes! I’ve seen many servers with dozens of crons piled up because they are polling every 5 minutes but the server is slow, so they are waiting 10 minutes or so to get the data. The problem quickly snowballs out of control.

In brief, we recommend:
1. use the least privileged user as possible for the user running the cron.
2. explicitly set timeouts to work with your application
3. decide what level of error reporting you need and use -q -nv and/or /etc/cron.d as required.

These tips are mostly for wget but curl has many of the same options.

Lastly, one more security tip. We often create a “wgetforuser” which is wget with permission that users can use. We then set the main wget to only be used by root. This helps (does not prevent) some attacks where a wget command is passed into an insecure web application.

http://www.ihsac.com

]]> By: Jeff Huckaby http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-101338 Jeff Huckaby Wed, 11 Apr 2007 02:21:31 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-101338 Verbosity can be good. Wget or curl with their respective "quiet" options will silence some output from those scripts but not all. They will still likely show critical errors, which is why you may want the redirects to /dev/null. However, we often see cases where you need some errors but not others. wget has a -nv flag that is not verbose but not quiet. You can also use /etc/cron.d/filename on most linux systems to fine-tune your cron. You can specify a mail address within the file you place in this directory. This can be useful to alert someone in case of a problem. Also, don't overlook security. Run your crons with a user with as few privileges as possible. If you simply need to wget a file, then a normal user with no login privileges will often suffice. Lastly, don't forget --tries=number option. This will have wget retry in case of a failure. Note the default is 20 retries unless a failure occurs. There is also a --retry-connrefused which will retry even when a connection is refused, useful for overloaded URLs. Lastly, there is the --timeout option. Always use this option if you are fetching URLs frequently. The default read timeout is 900 seconds. That's 15 minutes! I've seen many servers with dozens of crons piled up because they are polling every 5 minutes but the server is slow, so they are waiting 10 minutes or so to get the data. The problem quickly snowballs out of control. In brief, we recommend: 1. use the least privileged user as possible for the user running the cron. 2. explicitly set timeouts to work with your application 3. decide what level of error reporting you need and use -q -nv and/or /etc/cron.d as required. These tips are mostly for wget but curl has many of the same options. Lastly, one more security tip. We often create a "wgetforuser" which is wget with permission that users can use. We then set the main wget to only be used by root. This helps (does not prevent) some attacks where a wget command is passed into an insecure web application. Verbosity can be good. Wget or curl with their respective “quiet” options will silence some output from those scripts but not all. They will still likely show critical errors, which is why you may want the redirects to /dev/null. However, we often see cases where you need some errors but not others. wget has a -nv flag that is not verbose but not quiet. You can also use /etc/cron.d/filename on most linux systems to fine-tune your cron. You can specify a mail address within the file you place in this directory. This can be useful to alert someone in case of a problem.

Also, don’t overlook security. Run your crons with a user with as few privileges as possible. If you simply need to wget a file, then a normal user with no login privileges will often suffice.

Lastly, don’t forget –tries=number option. This will have wget retry in case of a failure. Note the default is 20 retries unless a failure occurs. There is also a –retry-connrefused which will retry even when a connection is refused, useful for overloaded URLs.

Lastly, there is the –timeout option. Always use this option if you are fetching URLs frequently. The default read timeout is 900 seconds. That’s 15 minutes! I’ve seen many servers with dozens of crons piled up because they are polling every 5 minutes but the server is slow, so they are waiting 10 minutes or so to get the data. The problem quickly snowballs out of control.

In brief, we recommend:
1. use the least privileged user as possible for the user running the cron.
2. explicitly set timeouts to work with your application
3. decide what level of error reporting you need and use -q -nv and/or /etc/cron.d as required.

These tips are mostly for wget but curl has many of the same options.

Lastly, one more security tip. We often create a “wgetforuser” which is wget with permission that users can use. We then set the main wget to only be used by root. This helps (does not prevent) some attacks where a wget command is passed into an insecure web application.

]]> By: cgiproxy guy http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-94078 cgiproxy guy Sun, 21 Jan 2007 15:06:52 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-94078 wget is a program with incredible untapped potential for most people I personally like wget --delete-after http://website.com it deletes the output after the execution, not as nice as quite or /dev/null 2>&1 but very powerful still. also wget will work with tor, just a question of having tor proxy set up right on your server and digging for the additional commands. wget is a program with incredible untapped potential for most people

I personally like wget –delete-after http://website.com

it deletes the output after the execution, not as nice as quite or /dev/null 2>&1 but very powerful still.

also wget will work with tor, just a question of having tor proxy set up right on your server and digging for the additional commands.

]]> By: Anjanesh http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93320 Anjanesh Tue, 09 Jan 2007 02:22:52 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93320 Oh...for a min there I though <b>wget --spider http://example.com</b> would give me all the links spidered right from the default site page. Something like Google's site:http://example.com Oh…for a min there I though wget –spider http://example.com would give me all the links spidered right from the default site page. Something like Google’s site:http://example.com

]]> By: Nick http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93283 Nick Sun, 07 Jan 2007 21:05:47 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93283 I had another problem. I had restricted access to curl, wget and every other suspicious bot to my site, so the only way to achive a cronjob like this was by using the -A directive, which sends an agent header. eg. 10 * * * * curl -A Firefox http://....... > /dev/null 2>&1 I had another problem. I had restricted access to curl, wget and every other suspicious bot to my site, so the only way to achive a cronjob like this was by using the -A directive, which sends an agent header.

eg.

10 * * * * curl -A Firefox http://……. > /dev/null 2>&1

]]> By: Matt http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93244 Matt Fri, 05 Jan 2007 22:47:57 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93244 Another useful one to know is: wget --spider I have some protected pages that are inside my framework that need to be run at intervals, --spider makes wget behave as a web spider (it won't download any pages, it'll just check to see if they are there). You can also disable output by passing everything to /dev/null * * * * * wget --spider http://www.example.com >/dev/null 2>&1 Another useful one to know is: wget –spider

I have some protected pages that are inside my framework that need to be run at intervals, –spider makes wget behave as a web spider (it won’t download any pages, it’ll just check to see if they are there).

You can also disable output by passing everything to /dev/null

* * * * * wget –spider http://www.example.com >/dev/null 2>&1

]]> By: Matt Sandy http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93236 Matt Sandy Fri, 05 Jan 2007 19:51:00 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93236 feddy, as far as I know you can't use a proxy with that function, but if you really need more functionality then go about it the curl way. feddy, as far as I know you can’t use a proxy with that function, but if you really need more functionality then go about it the curl way.

]]> By: feddy http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93233 feddy Fri, 05 Jan 2007 17:47:29 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93233 Matt Sandy, Do you know how you could add proxy/tor support with file_get_contents() Matt Sandy, Do you know how you could add proxy/tor support with file_get_contents()

]]> By: Steve http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93196 Steve Thu, 04 Jan 2007 18:19:20 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93196 feddy: 2>&1 redirects stderr to stdout so that everything ends up in stdout and therefore to /dev/null... feddy: 2>&1 redirects stderr to stdout so that everything ends up in stdout and therefore to /dev/null…

]]> By: Matt Sandy http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93187 Matt Sandy Thu, 04 Jan 2007 16:30:09 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93187 I simple use the file_get_contentents() for everything GET, I save curl for POST. I simple use the file_get_contentents() for everything GET, I save curl for POST.

]]>