Re: Wget exit codes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 R Kimber wrote: > On Sun, 09 Dec 2007 13:19:51 -0800 > Micah Cowan wrote: > > >> I believe I already answered this: it is because a non-zero exit >> status always means "something's wrong". Myriad scripts invoke >> utilities in ways similar to: >> >> if ! wget http://foo.com/ >> then >> echo "Something went wrong with the download." >> fi >> >> If Wget starts using non-zero to mean a "special" kind of success, >> scripting suddenly becomes much more complicated (and Wget suddenly >> ceases to be "Unixy"). > > 'Something wrong' seems to be a somewhat semantic issue, and depends on > your outlook. > > When grep doesn't return a zero exit code, it doesn't mean there's > something wrong. Just that it didn't find the string. If you are > trying to ensure the string isn't there, the non zero exit code is > telling you that there's something right. Yes... utilities like grep and test work a little differently, returning true or false, rather than success or failure. I don't see Wget in the same utility space as those. It's much more analagous to cp or mv, to me. Actually, the specific case of timestamping is much more similar to rsync, to me, than anything else. Having Rsync return nonzero when it decided against copying something over because it's not new would be wrong; having Wget return nonzero in the same situation is just as wrong. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHXs627M8hyUobTrERAo7xAJ904nOmCFgn5ZF6fynRXlISKKG/1gCfXV78 3qWXT+i2R08i9iGb7sOQPOw= =gymS -END PGP SIGNATURE-
Re: Wget exit codes
On Sun, 09 Dec 2007 13:19:51 -0800 Micah Cowan wrote: > I believe I already answered this: it is because a non-zero exit > status always means "something's wrong". Myriad scripts invoke > utilities in ways similar to: > > if ! wget http://foo.com/ > then > echo "Something went wrong with the download." > fi > > If Wget starts using non-zero to mean a "special" kind of success, > scripting suddenly becomes much more complicated (and Wget suddenly > ceases to be "Unixy"). 'Something wrong' seems to be a somewhat semantic issue, and depends on your outlook. When grep doesn't return a zero exit code, it doesn't mean there's something wrong. Just that it didn't find the string. If you are trying to ensure the string isn't there, the non zero exit code is telling you that there's something right. - Richard. -- Richard Kimber http://www.psr.keele.ac.uk/
Re: Wget exit codes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gerard wrote: >> On December 09, 2007 at 07:03PM Stuart Moore wrote: > >> Could the exit code used be determined by a flag? E.g. by default it >> uses unix convention, 0 for any success; with an >> --extended_error_codes flag or similar then it uses extra error codes >> depending on the type of success (but for sanity uses the same codes >> for failure with or without the flag) >> >> That should allow both of you to use it for scripting. > > Curl has a "-w" option that takes several different variables. I presently use > the 'http_code' and 'size_download' variables to gather information I need to > determine what exactly transpired. They are also great for debugging a script; > however, that is another matter. Presently, I direct this info to a file and > then grep the file to gather the information I need. It works faster than > doing multiple file comparisons, especially when the files are compressed. > > If wget were to implement something along this line, possibly even setting a > specific variable indicating whether or not a file was downloaded, it would > make scripting a lot easier and less prone to breakage. This is the sort of thing I think I was talking about when I referred to a "mapping" file. I was also thinking it might not hurt to have a minor tool to aid in reading information back out from such a file, to avoid having everyone "reinvent the wheel" scores of times to do the same thing, if nothing else. The concept also ties in rather well with the "Wget 2.0" concept of having a "metadatabase", containing information about mappings between original URLs and local filenames, MIME types, portions downloaded, etc. And, I still like this idea better than having an option to switch between "exit status" modes. > Curl presently has one of the best exit code implementations available. > Studying their model would seem like a worth while venture. Will do. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHXTZl7M8hyUobTrERAq3mAJ0Q9k2s8VGitMjrub6t1kpYyWwtVACgjChR NCZpEEj4k0uNJPwee0dGvzY= =nTih -END PGP SIGNATURE-
Re: Wget exit codes
> On December 09, 2007 at 07:03PM Stuart Moore wrote: > Could the exit code used be determined by a flag? E.g. by default it > uses unix convention, 0 for any success; with an > --extended_error_codes flag or similar then it uses extra error codes > depending on the type of success (but for sanity uses the same codes > for failure with or without the flag) > > That should allow both of you to use it for scripting. Curl has a "-w" option that takes several different variables. I presently use the 'http_code' and 'size_download' variables to gather information I need to determine what exactly transpired. They are also great for debugging a script; however, that is another matter. Presently, I direct this info to a file and then grep the file to gather the information I need. It works faster than doing multiple file comparisons, especially when the files are compressed. If wget were to implement something along this line, possibly even setting a specific variable indicating whether or not a file was downloaded, it would make scripting a lot easier and less prone to breakage. Curl presently has one of the best exit code implementations available. Studying their model would seem like a worth while venture. -- Gerard
Re: Wget exit codes
On Dec 9, 2007 7:03 PM, Stuart Moore <[EMAIL PROTECTED]> wrote: > Could the exit code used be determined by a flag? E.g. by default it > uses unix convention, 0 for any success; with an > --extended_error_codes flag or similar then it uses extra error codes > depending on the type of success (but for sanity uses the same codes > for failure with or without the flag) > > That should allow both of you to use it for scripting. I like this idea. Like Micah said, there should _NOT_ be any return value non-zero in which the command was successful, even if it didn't download anything (if that's what we asked it to do, then it was successful). I think it would behoove us to have multiple return values for different *errors*, but not for different cases of success. I think this would be a very simple and helpful patch. (Well, "simple" may be an understatement because we'd have to go through every possible point of failure to create a return value.. mah.)
Re: Wget exit codes
On 09/12/2007, Micah Cowan <[EMAIL PROTECTED]> wrote: > Non-zero exit statuses indicate failure, partial or complete. I am not > going to try to use them to mean something else. Exit codes are, plain > and simple, the wrong way to communicate these differences. Perhaps if, > in the early days, the designers of Unix had had the foresight to > partition exit codes a bit more equally between success and failure > conditions, it would be different. As it is, such is not the case, and I > am _not_ going to fudge things so that a few special cases are simpler, > while complicating the more general cases. > > Now that we've established, rather clearly at this point, I think, what > is _not_ going to be done, can we please leave this topic and continue > onto the topic of what we _will_ do? > Could the exit code used be determined by a flag? E.g. by default it uses unix convention, 0 for any success; with an --extended_error_codes flag or similar then it uses extra error codes depending on the type of success (but for sanity uses the same codes for failure with or without the flag) That should allow both of you to use it for scripting. Stuart
Re: Wget exit codes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gerard wrote: >> On December 09, 2007 at 04:19PM Micah Cowan wrote: > >> I believe I already answered this: it is because a non-zero exit status >> always means "something's wrong". Myriad scripts invoke utilities in >> ways similar to: >> >> if ! wget http://foo.com/ >> then >> echo "Something went wrong with the download." >> fi > > > > wget http://foo.com > EC=$? > case $EC in > [0]) > bla bla bla > ;; > [1]) > bla bla bla > ;; > *) > more bla > ;; > esac > > A very simple way to handle an exit code. Place the entire code into a > function and it can be used anywhere in the script. Fine. What about Hrvoje's more salient examples? "sh -e" or "wget -N foo || ..." The point is, just because some people may want to use your special if construct above, doesn't mean everyone should be made to. >> If Wget starts using non-zero to mean a "special" kind of success, >> scripting suddenly becomes much more complicated (and Wget suddenly >> ceases to be "Unixy"). > > The terminology here is incorrect. This is not a special kind of success. It > is simply exiting with a specific code defining what has transpired. This > makes scripting far easier. not more difficult. The condition is one of success, regardless of what code is issued. Issuing codes that indicate something other than success, when wget has done what you've asked, is wrong. And there is absolutely no way to interpret your "code 1" example (from another message) as anything other than success, in any case, so you're really not making your point very well. Non-zero exit statuses indicate failure, partial or complete. I am not going to try to use them to mean something else. Exit codes are, plain and simple, the wrong way to communicate these differences. Perhaps if, in the early days, the designers of Unix had had the foresight to partition exit codes a bit more equally between success and failure conditions, it would be different. As it is, such is not the case, and I am _not_ going to fudge things so that a few special cases are simpler, while complicating the more general cases. Now that we've established, rather clearly at this point, I think, what is _not_ going to be done, can we please leave this topic and continue onto the topic of what we _will_ do? - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHXGzh7M8hyUobTrERAitqAKCI7BSJrmirsmIAxzrXIvijpFHmHgCghVU6 qtEizoBNm6d/NBB/kGIVCnM= =Y26j -END PGP SIGNATURE-
Re: Wget exit codes
> On December 09, 2007 at 04:19PM Micah Cowan wrote: > I believe I already answered this: it is because a non-zero exit status > always means "something's wrong". Myriad scripts invoke utilities in > ways similar to: > > if ! wget http://foo.com/ > then > echo "Something went wrong with the download." > fi wget http://foo.com EC=$? case $EC in [0]) bla bla bla ;; [1]) bla bla bla ;; *) more bla ;; esac A very simple way to handle an exit code. Place the entire code into a function and it can be used anywhere in the script. > If Wget starts using non-zero to mean a "special" kind of success, > scripting suddenly becomes much more complicated (and Wget suddenly > ceases to be "Unixy"). The terminology here is incorrect. This is not a special kind of success. It is simply exiting with a specific code defining what has transpired. This makes scripting far easier. not more difficult. -- Gerard
Re: Wget exit codes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Gerard wrote: > On Sunday December 09, 2007 at 03:13:32 (PM) Hrvoje Niksic wrote: > >> Gerard <[EMAIL PROTECTED]> writes: >> In particular, if Wget chooses not to download a file because the local timestamp is still current, or because its size corresponds to that of the remote file, these should result in an exit status of zero. >>> I disagree. If wget has not downloaded a file, exiting with zero >>> could lead the end user to believe that it had. >> Specifying `-n' means "download if needed". There is no reason to >> report a non-zero exit status if there was no need to perform the >> download. It is simply not an error condition, it is one of the two >> success conditions (the other being download of the new contents). > > I am fully aware of what the '-n' flag is designed to do. The problem is that > there is no elequent way from within a script to determine whether or not wget > did in fact download a newer version of a file. I fail to see what is so hard > to understand regarding this concept. Presently, I am forced to keep mutilple > copies of files in different directories and then using various scripting > techniques, determine if wget downloaded a newer version. There is nothing hard to understand: we fully appreciate that it is desirable to determine whether or not Wget downloaded a file. The problem is that the exit status is entirely the wrong mechanism by which to communicate this. A command-line programs should only issue a non-zero exit status in the event that there was a "problem". By giving -N, you specifically asked Wget not to download the file if the local copy's still fresh. Wget did what you asked: therefore, it's success. >>> I disagree again. If wget did not download a file, no matter what >>> the reason, then it should not exit with zero. I have written >>> several scripts that utilize wget to download files. Because wget >>> fails to issue a useful code upon completion, I am forced to use >>> hacks to find out what actually transpired. Curl utilizes certain >>> error codes, # 73 for instance, that are quite useful. >> I agree that Wget should allow the caller to find out what happened, >> but I don't think exit codes can be of much use there. For one, they >> don't allow distinction between different "successful" conditions, >> which is a problem in many cases. Also, their meaning is much harder >> to define in presence of multiple downloads (wget URL1 URL2...). > > Again, I disagree. Example: > > 0 All files downloaded successfully > 1 File newer on server - file downloaded > 2 No newer file on server - no file downloaded > > Obviously the error codes would (should) be much more detailed than that. See > the 'curl' exit codes as an example. I have made suggestions regarding just > this matter on their forum and have received positive feedback. 1 and 2 have no useful meaning in the realm we're supposedly discussing now, which is multiple downloads. And, as previously mentioned, they indicate to the parent process that it failed to do its job, which is especially deceptive for your 1 code above. > In the case of multiple downloads, wget should issue a separate code for each > item. That is absolutely impossible. One exit status per process. This is why I suggested a "mapping" file. Another benefit of a mapping file is that we could use our own coding system, including delineation between different types of "success". - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHXF557M8hyUobTrERApfrAJ9QxVdlNylIcDJF99vHALqweCPUSACZAaqI txJVY6ZwhAkuuKMiWIUxyTg= =Z1ky -END PGP SIGNATURE-
Re: Wget exit codes
R Kimber <[EMAIL PROTECTED]> writes: >> I agree that Wget should allow the caller to find out what >> happened, but I don't think exit codes can be of much use there. >> For one, they don't allow distinction between different >> "successful" conditions, which is a problem in many cases. > > I'm not sure I understand this. Why is it that there cannot be > different exit codes for different 'successful' conditions? Because by Unix convention success is indicated by exit status 0. When a process exits with 0, scripts started with `sh -e' or tests such as `wget URL || exit $?' won't fail. Exiting with any non-zero exit status on success would cause spurious failures to be reported.
Re: Wget exit codes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 R Kimber wrote: > On Sun, 09 Dec 2007 21:13:32 +0100 > Hrvoje Niksic wrote: > >>> I disagree again. If wget did not download a file, no matter what >>> the reason, then it should not exit with zero. I have written >>> several scripts that utilize wget to download files. Because wget >>> fails to issue a useful code upon completion, I am forced to use >>> hacks to find out what actually transpired. Curl utilizes certain >>> error codes, # 73 for instance, that are quite useful. >> I agree that Wget should allow the caller to find out what happened, >> but I don't think exit codes can be of much use there. For one, they >> don't allow distinction between different "successful" conditions, >> which is a problem in many cases. > > I'm not sure I understand this. Why is it that there cannot be > different exit codes for different 'successful' conditions? > > A program can exit under various conditions, some of which might be > situations in which errors occurred and some might be situations in > which no errors occurred but different things happened. Having exit > codes to indicate what actually happened seems eminently desirable, > and it's unclear to me why it is not possible. I believe I already answered this: it is because a non-zero exit status always means "something's wrong". Myriad scripts invoke utilities in ways similar to: if ! wget http://foo.com/ then echo "Something went wrong with the download." fi If Wget starts using non-zero to mean a "special" kind of success, scripting suddenly becomes much more complicated (and Wget suddenly ceases to be "Unixy"). - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer... http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHXFv37M8hyUobTrERAjBMAJ9UB/zKsUt2rynPbYVu4MGjD9J55wCdGofh XxPdOAuW9Gq80V2Fhpygod8= =xsCk -END PGP SIGNATURE-
Re: Wget exit codes
On Sunday December 09, 2007 at 03:13:32 (PM) Hrvoje Niksic wrote: > Gerard <[EMAIL PROTECTED]> writes: > > >> In particular, if Wget chooses not to download a file because the > >> local timestamp is still current, or because its size corresponds > >> to that of the remote file, these should result in an exit status > >> of zero. > > > > I disagree. If wget has not downloaded a file, exiting with zero > > could lead the end user to believe that it had. > > Specifying `-n' means "download if needed". There is no reason to > report a non-zero exit status if there was no need to perform the > download. It is simply not an error condition, it is one of the two > success conditions (the other being download of the new contents). I am fully aware of what the '-n' flag is designed to do. The problem is that there is no elequent way from within a script to determine whether or not wget did in fact download a newer version of a file. I fail to see what is so hard to understand regarding this concept. Presently, I am forced to keep mutilple copies of files in different directories and then using various scripting techniques, determine if wget downloaded a newer version. > > > I disagree again. If wget did not download a file, no matter what > > the reason, then it should not exit with zero. I have written > > several scripts that utilize wget to download files. Because wget > > fails to issue a useful code upon completion, I am forced to use > > hacks to find out what actually transpired. Curl utilizes certain > > error codes, # 73 for instance, that are quite useful. > > I agree that Wget should allow the caller to find out what happened, > but I don't think exit codes can be of much use there. For one, they > don't allow distinction between different "successful" conditions, > which is a problem in many cases. Also, their meaning is much harder > to define in presence of multiple downloads (wget URL1 URL2...). Again, I disagree. Example: 0 All files downloaded successfully 1 File newer on server - file downloaded 2 No newer file on server - no file downloaded Obviously the error codes would (should) be much more detailed than that. See the 'curl' exit codes as an example. I have made suggestions regarding just this matter on their forum and have received positive feedback. In the case of multiple downloads, wget should issue a separate code for each item. I fail to see the problem with that approach. If this was in a script, I would supply wget with each item individually for finer grain control; however, I believe it is still doable from the command line.If not, then disabling exit codes except for 'zero' = all success and '99' = one or more files failed, might be a possible solution. -- Gerard
Re: Wget exit codes
On Sun, 09 Dec 2007 21:13:32 +0100 Hrvoje Niksic wrote: > > I disagree again. If wget did not download a file, no matter what > > the reason, then it should not exit with zero. I have written > > several scripts that utilize wget to download files. Because wget > > fails to issue a useful code upon completion, I am forced to use > > hacks to find out what actually transpired. Curl utilizes certain > > error codes, # 73 for instance, that are quite useful. > > I agree that Wget should allow the caller to find out what happened, > but I don't think exit codes can be of much use there. For one, they > don't allow distinction between different "successful" conditions, > which is a problem in many cases. I'm not sure I understand this. Why is it that there cannot be different exit codes for different 'successful' conditions? A program can exit under various conditions, some of which might be situations in which errors occurred and some might be situations in which no errors occurred but different things happened. Having exit codes to indicate what actually happened seems eminently desirable, and it's unclear to me why it is not possible. - Richard. -- Richard Kimber http://www.psr.keele.ac.uk/
Re: Wget exit codes
Gerard <[EMAIL PROTECTED]> writes: >> In particular, if Wget chooses not to download a file because the >> local timestamp is still current, or because its size corresponds >> to that of the remote file, these should result in an exit status >> of zero. > > I disagree. If wget has not downloaded a file, exiting with zero > could lead the end user to believe that it had. Specifying `-n' means "download if needed". There is no reason to report a non-zero exit status if there was no need to perform the download. It is simply not an error condition, it is one of the two success conditions (the other being download of the new contents). > I disagree again. If wget did not download a file, no matter what > the reason, then it should not exit with zero. I have written > several scripts that utilize wget to download files. Because wget > fails to issue a useful code upon completion, I am forced to use > hacks to find out what actually transpired. Curl utilizes certain > error codes, # 73 for instance, that are quite useful. I agree that Wget should allow the caller to find out what happened, but I don't think exit codes can be of much use there. For one, they don't allow distinction between different "successful" conditions, which is a problem in many cases. Also, their meaning is much harder to define in presence of multiple downloads (wget URL1 URL2...).
Re: Wget exit codes
On Saturday December 08, 2007 at 09:18:08 (PM) Micah Cowan wrote: > So, perhaps its time to come back to the question of differentiated exit > codes from Wget. This may be a 1.12 question, or perhaps a 1.13 > question, but at any rate, with 1.11 ready to go out the day, we perhaps > have a little more time to discuss it in depth. > > I think the easiest question to ask is how Wget should handle failures > when only one file has been specified, and recursive-descent isn't > specified. Deciding how Wget should handle multiple arguments, or > recursiveness, or both together, is somewhat less straightforward. > > One thing I do feel strongly, is that if Wget has successfully done its > job, and an error condition has not occurred, Wget should return zero. I agree. However, we are assuming that wget has indeed downloaded a file correctly. > In particular, if Wget chooses not to download a file because the local > timestamp is still current, or because its size corresponds to that of > the remote file, these should result in an exit status of zero. I disagree. If wget has not downloaded a file, exiting with zero could lead the end user to believe that it had. > I agree that differentiating between these conditions could be very useful > from a scripting standpoint, but in Unix (and most of the rest of the world, > for that matter), anything that's not 0 is interpreted as an error. I disagree again. If wget did not download a file, no matter what the reason, then it should not exit with zero. I have written several scripts that utilize wget to download files. Because wget fails to issue a useful code upon completion, I am forced to use hacks to find out what actually transpired. Curl utilizes certain error codes, # 73 for instance, that are quite useful. > There is a fair amount of room for delineation between various error > statuses; for better or worse, there is no such room for success > statuses, and it won't be productive to try to work around it by using a > few codes from the error space as "other" success conditions. I believe that the problem is in defining the word 'success'. Properly indicating what transpired by issuing a factually correct exit code is essential. In any case, it would make my job of writing scripts that use wget immensely easier and less prone to breakage. Jus my 2ยข. -- Gerard
