Comments on: How to fetch a url with curl or wget silently http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/ neat fun stuff Thu, 18 Mar 2010 06:22:18 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Jame http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-460687 Jame Sun, 31 Jan 2010 07:44:05 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-460687 Hi , I have a problem with wget command. I want to get the file just only 1 time but the command can 't do that. wget --tries=1 http://xxx.com It 's mean retry for 1 time wget --tries=0 http://xxx.com It 's mean retry indefinite times Could you tell me how can I use wget command without retry? Thanks Hi , I have a problem with wget command. I want to get the file just only 1 time but the command can ‘t do that.

wget –tries=1 http://xxx.com
It ’s mean retry for 1 time

wget –tries=0 http://xxx.com
It ’s mean retry indefinite times

Could you tell me how can I use wget command without retry?

Thanks

]]> By: name http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-444755 name Sun, 27 Dec 2009 17:56:37 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-444755 --output-document=/dev/null –output-document=/dev/null

]]> By: sohbet http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-436657 sohbet Fri, 11 Dec 2009 14:56:18 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-436657 thnxxx thnxxx

]]> By: Robert http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-398567 Robert Thu, 01 Oct 2009 07:38:33 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-398567 @David To get into a .htpasswd protected folder, use: --http-user=xxxxx --http-passwd=xxxxx Cheers, Robert @David

To get into a .htpasswd protected folder, use:
–http-user=xxxxx –http-passwd=xxxxx

Cheers,
Robert

]]> By: Aneem http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-375463 Aneem Thu, 13 Aug 2009 15:30:21 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-375463 How to fetch a url and save it in another folder rather than existing/root folder? How to fetch a url and save it in another folder rather than existing/root folder?

]]> By: David Spector http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-336639 David Spector Thu, 28 May 2009 23:55:58 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-336639 I'm hoping I can use "wget http://...?arg=value&pwd=password" in crontab to call a server I wrote to get a particular action done (it sends me the results in an email). I’m hoping I can use “wget http://…?arg=value&pwd=password” in crontab to call a server I wrote to get a particular action done (it sends me the results in an email).

]]> By: دروس http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-105017 دروس Thu, 24 May 2007 22:26:01 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-105017 Oh…for a min there I though wget –spider http://example.com would give me all the links spidered right from the default site page. Something like Google’s site:http://example.com cgiproxy guy Said, January 21, 2007 @ 7:06 am wget is a program with incredible untapped potential for most people I personally like wget –delete-after http://website.com it deletes the output after the execution, not as nice as quite or /dev/null 2>&1 but very powerful still. also wget will work with tor, just a question of having tor proxy set up right on your server and digging for the additional commands. Jeff Huckaby Said, April 10, 2007 @ 7:21 pm Verbosity can be good. Wget or curl with their respective “quiet” options will silence some output from those scripts but not all. They will still likely show critical errors, which is why you may want the redirects to /dev/null. However, we often see cases where you need some errors but not others. wget has a -nv flag that is not verbose but not quiet. You can also use /etc/cron.d/filename on most linux systems to fine-tune your cron. You can specify a mail address within the file you place in this directory. This can be useful to alert someone in case of a problem. Also, don’t overlook security. Run your crons with a user with as few privileges as possible. If you simply need to wget a file, then a normal user with no login privileges will often suffice. Lastly, don’t forget –tries=number option. This will have wget retry in case of a failure. Note the default is 20 retries unless a failure occurs. There is also a –retry-connrefused which will retry even when a connection is refused, useful for overloaded URLs. Lastly, there is the –timeout option. Always use this option if you are fetching URLs frequently. The default read timeout is 900 seconds. That’s 15 minutes! I’ve seen many servers with dozens of crons piled up because they are polling every 5 minutes but the server is slow, so they are waiting 10 minutes or so to get the data. The problem quickly snowballs out of control. In brief, we recommend: 1. use the least privileged user as possible for the user running the cron. 2. explicitly set timeouts to work with your application 3. decide what level of error reporting you need and use -q -nv and/or /etc/cron.d as required. These tips are mostly for wget but curl has many of the same options. Lastly, one more security tip. We often create a “wgetforuser” which is wget with permission that users can use. We then set the main wget to only be used by root. This helps (does not prevent) some attacks where a wget command is passed into an insecure web application. http://www.ihsac.com Oh…for a min there I though wget –spider http://example.com would give me all the links spidered right from the default site page. Something like Google’s site:http://example.com

cgiproxy guy Said,
January 21, 2007 @ 7:06 am

wget is a program with incredible untapped potential for most people

I personally like wget –delete-after http://website.com

it deletes the output after the execution, not as nice as quite or /dev/null 2>&1 but very powerful still.

also wget will work with tor, just a question of having tor proxy set up right on your server and digging for the additional commands.

Jeff Huckaby Said,
April 10, 2007 @ 7:21 pm

Verbosity can be good. Wget or curl with their respective “quiet” options will silence some output from those scripts but not all. They will still likely show critical errors, which is why you may want the redirects to /dev/null. However, we often see cases where you need some errors but not others. wget has a -nv flag that is not verbose but not quiet. You can also use /etc/cron.d/filename on most linux systems to fine-tune your cron. You can specify a mail address within the file you place in this directory. This can be useful to alert someone in case of a problem.

Also, don’t overlook security. Run your crons with a user with as few privileges as possible. If you simply need to wget a file, then a normal user with no login privileges will often suffice.

Lastly, don’t forget –tries=number option. This will have wget retry in case of a failure. Note the default is 20 retries unless a failure occurs. There is also a –retry-connrefused which will retry even when a connection is refused, useful for overloaded URLs.

Lastly, there is the –timeout option. Always use this option if you are fetching URLs frequently. The default read timeout is 900 seconds. That’s 15 minutes! I’ve seen many servers with dozens of crons piled up because they are polling every 5 minutes but the server is slow, so they are waiting 10 minutes or so to get the data. The problem quickly snowballs out of control.

In brief, we recommend:
1. use the least privileged user as possible for the user running the cron.
2. explicitly set timeouts to work with your application
3. decide what level of error reporting you need and use -q -nv and/or /etc/cron.d as required.

These tips are mostly for wget but curl has many of the same options.

Lastly, one more security tip. We often create a “wgetforuser” which is wget with permission that users can use. We then set the main wget to only be used by root. This helps (does not prevent) some attacks where a wget command is passed into an insecure web application.

http://www.ihsac.com

]]> By: Jeff Huckaby http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-101338 Jeff Huckaby Wed, 11 Apr 2007 02:21:31 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-101338 Verbosity can be good. Wget or curl with their respective "quiet" options will silence some output from those scripts but not all. They will still likely show critical errors, which is why you may want the redirects to /dev/null. However, we often see cases where you need some errors but not others. wget has a -nv flag that is not verbose but not quiet. You can also use /etc/cron.d/filename on most linux systems to fine-tune your cron. You can specify a mail address within the file you place in this directory. This can be useful to alert someone in case of a problem. Also, don't overlook security. Run your crons with a user with as few privileges as possible. If you simply need to wget a file, then a normal user with no login privileges will often suffice. Lastly, don't forget --tries=number option. This will have wget retry in case of a failure. Note the default is 20 retries unless a failure occurs. There is also a --retry-connrefused which will retry even when a connection is refused, useful for overloaded URLs. Lastly, there is the --timeout option. Always use this option if you are fetching URLs frequently. The default read timeout is 900 seconds. That's 15 minutes! I've seen many servers with dozens of crons piled up because they are polling every 5 minutes but the server is slow, so they are waiting 10 minutes or so to get the data. The problem quickly snowballs out of control. In brief, we recommend: 1. use the least privileged user as possible for the user running the cron. 2. explicitly set timeouts to work with your application 3. decide what level of error reporting you need and use -q -nv and/or /etc/cron.d as required. These tips are mostly for wget but curl has many of the same options. Lastly, one more security tip. We often create a "wgetforuser" which is wget with permission that users can use. We then set the main wget to only be used by root. This helps (does not prevent) some attacks where a wget command is passed into an insecure web application. Verbosity can be good. Wget or curl with their respective “quiet” options will silence some output from those scripts but not all. They will still likely show critical errors, which is why you may want the redirects to /dev/null. However, we often see cases where you need some errors but not others. wget has a -nv flag that is not verbose but not quiet. You can also use /etc/cron.d/filename on most linux systems to fine-tune your cron. You can specify a mail address within the file you place in this directory. This can be useful to alert someone in case of a problem.

Also, don’t overlook security. Run your crons with a user with as few privileges as possible. If you simply need to wget a file, then a normal user with no login privileges will often suffice.

Lastly, don’t forget –tries=number option. This will have wget retry in case of a failure. Note the default is 20 retries unless a failure occurs. There is also a –retry-connrefused which will retry even when a connection is refused, useful for overloaded URLs.

Lastly, there is the –timeout option. Always use this option if you are fetching URLs frequently. The default read timeout is 900 seconds. That’s 15 minutes! I’ve seen many servers with dozens of crons piled up because they are polling every 5 minutes but the server is slow, so they are waiting 10 minutes or so to get the data. The problem quickly snowballs out of control.

In brief, we recommend:
1. use the least privileged user as possible for the user running the cron.
2. explicitly set timeouts to work with your application
3. decide what level of error reporting you need and use -q -nv and/or /etc/cron.d as required.

These tips are mostly for wget but curl has many of the same options.

Lastly, one more security tip. We often create a “wgetforuser” which is wget with permission that users can use. We then set the main wget to only be used by root. This helps (does not prevent) some attacks where a wget command is passed into an insecure web application.

]]> By: cgiproxy guy http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-94078 cgiproxy guy Sun, 21 Jan 2007 15:06:52 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-94078 wget is a program with incredible untapped potential for most people I personally like wget --delete-after http://website.com it deletes the output after the execution, not as nice as quite or /dev/null 2>&1 but very powerful still. also wget will work with tor, just a question of having tor proxy set up right on your server and digging for the additional commands. wget is a program with incredible untapped potential for most people

I personally like wget –delete-after http://website.com

it deletes the output after the execution, not as nice as quite or /dev/null 2>&1 but very powerful still.

also wget will work with tor, just a question of having tor proxy set up right on your server and digging for the additional commands.

]]> By: Anjanesh http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93320 Anjanesh Tue, 09 Jan 2007 02:22:52 +0000 http://www.mattcutts.com/blog/how-to-fetch-a-url-with-curl-or-wget-silently/#comment-93320 Oh...for a min there I though <b>wget --spider http://example.com</b> would give me all the links spidered right from the default site page. Something like Google's site:http://example.com Oh…for a min there I though wget –spider http://example.com would give me all the links spidered right from the default site page. Something like Google’s site:http://example.com

]]>