handling of test == by BASH's POSIX mode
Is there a reason why bash doesn't treat == as an illegal test operator when running in POSIX mode? This is problematic because use of test == in scripts that should be POSIX isn't getting caught when I run them under bash's POSIX mode. The scripts then fail when run under dash which seems to be stricter about this. I have reconfigured my system's default /bin/sh back to /bin/dash to ensure better POSIX compliance, but it would be nice if I didn't have to do that. I am running Ubuntu's distribution of bash, per: jseymour@ubuntu:~/tracked/git$ uname -a Linux ubuntu 2.6.32-41-generic #88-Ubuntu SMP Thu Mar 29 13:10:32 UTC 2012 x86_64 GNU/Linux jseymour@ubuntu:~/tracked/git$ bash --version GNU bash, version 4.1.5(1)-release (x86_64-pc-linux-gnu) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. jon.
Re: handling of test == by BASH's POSIX mode
2012/5/27 Jon Seymour jon.seym...@gmail.com: Is there a reason why bash doesn't treat == as an illegal test operator when running in POSIX mode? POSIX does not say == is not allowed. POSIX tells you what the shell should at least be able to do. A POSIX compliant shell can have whatever other features it likes, as long as the POSIX features are covered. This is problematic because use of test == in scripts that should be POSIX isn't getting caught when I run them under bash's POSIX mode. The scripts then fail when run under dash which seems to be stricter about this. Don't use non-POSIX features in a POSIX script, and you'll be fine. http://www.opengroup.org/onlinepubs/9699919799/utilities/contents.html -- Geir Hauge
Re: handling of test == by BASH's POSIX mode
On 27/05/2012, at 17:39, Geir Hauge geir.ha...@gmail.com wrote: 2012/5/27 Jon Seymour jon.seym...@gmail.com: Is there a reason why bash doesn't treat == as an illegal test operator when running in POSIX mode? POSIX does not say == is not allowed. POSIX tells you what the shell should at least be able to do. A POSIX compliant shell can have whatever other features it likes, as long as the POSIX features are covered. I guess the question is better phrased thus: what use case is usefully served by having bash's POSIX mode support a superset of test operators than other compliant POSIX shells? As it stands, I can't use bash's POSIX mode to verify the validity or otherwise of a POSIX script because bash won't report these kinds of errors - even when running in POSIX mode. There is an --enable-strict-posix (?) configuration option. Will this do what I expect? This is problematic because use of test == in scripts that should be POSIX isn't getting caught when I run them under bash's POSIX mode. The scripts then fail when run under dash which seems to be stricter about this. Don't use non-POSIX features in a POSIX script, and you'll be fine. http://www.opengroup.org/onlinepubs/9699919799/utilities/contents.html Which is the exactly the point. Practically speaking when I write scripts I expect an interpreter that claims to be running in POSIX mode to give me some help to flag usage of non POSIX idioms. Yes, I can second guess the interpreter by reading the spec, but is this really the most efficient way to catch these kinds of errors? Jon.
Re: handling of test == by BASH's POSIX mode
On Sunday, May 27, 2012 08:45:46 PM Jon Seymour wrote: On 27/05/2012, at 17:39, Geir Hauge geir.ha...@gmail.com wrote: 2012/5/27 Jon Seymour jon.seym...@gmail.com: Is there a reason why bash doesn't treat == as an illegal test operator when running in POSIX mode? POSIX does not say == is not allowed. POSIX tells you what the shell should at least be able to do. A POSIX compliant shell can have whatever other features it likes, as long as the POSIX features are covered. I guess the question is better phrased thus: what use case is usefully served by having bash's POSIX mode support a superset of test operators than other compliant POSIX shells? As it stands, I can't use bash's POSIX mode to verify the validity or otherwise of a POSIX script because bash won't report these kinds of errors - even when running in POSIX mode. There is an --enable-strict-posix (?) configuration option. Will this do what I expect? This is problematic because use of test == in scripts that should be POSIX isn't getting caught when I run them under bash's POSIX mode. The scripts then fail when run under dash which seems to be stricter about this. Don't use non-POSIX features in a POSIX script, and you'll be fine. http://www.opengroup.org/onlinepubs/9699919799/utilities/contents.html Which is the exactly the point. Practically speaking when I write scripts I expect an interpreter that claims to be running in POSIX mode to give me some help to flag usage of non POSIX idioms. Yes, I can second guess the interpreter by reading the spec, but is this really the most efficient way to catch these kinds of errors? Jon. There are no shells in existence that can do what you want. All major shells claiming to be POSIX compatible include some superset that can't be disabled. The only shell I have installed not supporting == in [ is dash, and there are so many scripts in the wild using == with [ it would be a miracle if your system didn't break because of it. Even the coreutils /usr/bin/[ supports ==. Performing that kind of checking, rigorously, in a shell, would be impossible to do statically anyway. Any such lint tool is limited to lexical analysis which makes it not very useful for testing unless your script is perfectly free of side-efffects. And who writes side-effect free shell scripts? How would the shell check for the correctness of: $(rm -rf somepath; echo '[') x == x ] -- Dan Douglas signature.asc Description: This is a digitally signed message part.
Re: handling of test == by BASH's POSIX mode
Jon Seymour jon.seym...@gmail.com writes: As it stands, I can't use bash's POSIX mode to verify the validity or otherwise of a POSIX script because bash won't report these kinds of errors - even when running in POSIX mode. You can't do that anyway: POSIX mode does not disable proper extensions to POSIX, only those that conflict with POSIX. Specifically, in the case of the test utility, POSIX makes the behaviour unspecified when a three argument invocation does not match the POSIX-defined binary operators. There is an --enable-strict-posix (?) configuration option. Will this do what I expect? That just switches the default for POSIX mode. Andreas. -- Andreas Schwab, sch...@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 And now for something completely different.
Re: handling of test == by BASH's POSIX mode
On Sun, May 27, 2012 at 9:24 PM, Dan Douglas orm...@gmail.com wrote: On Sunday, May 27, 2012 08:45:46 PM Jon Seymour wrote: On 27/05/2012, at 17:39, Geir Hauge geir.ha...@gmail.com wrote: I guess the question is better phrased thus: what use case is usefully served by having bash's POSIX mode support a superset of test operators than other compliant POSIX shells? As it stands, I can't use bash's POSIX mode to verify the validity or otherwise of a POSIX script because bash won't report these kinds of errors - even when running in POSIX mode. There are no shells in existence that can do what you want. All major shells claiming to be POSIX compatible include some superset that can't be disabled. The only shell I have installed not supporting == in [ is dash, and there are so many scripts in the wild using == with [ it would be a miracle if your system didn't break because of it. Even the coreutils /usr/bin/[ supports ==. I take your point that this isn't really a bash problem, but a POSIX problem - POSIX hasn't provided a way to validate whether a script only uses features that are required to be supported by POSIX compliant interpreters. An example of this is the failure to prohibit support for additional test operators which ultimately results in the creation of interoperability problems between bash and dash - surely not a good thing. However, I think my question still remains unanswered - even if bash is technically compliant with POSIX, what use case is usefully served by having bash support a superset of the POSIX test operators while executing in POSIX mode? Yes, doing allows bash to execute an ill-defined superset of valid POSIX scripts, but why is that useful - surely POSIX compliant mode should only be required to execute strictly compliant POSIX scripts and if we want to execute a superset, then we should explicitly specify which superset we want to use, not hope that the POSIX compliant interpreter installed on a machine happens to support that superset. You point about scripts failing when running under dash is a good one. This is exactly my problem: I replaced /bin/sh - dash with /bin/sh - bash because a 3rd party product installation script failed when dash was the POSIX shell. Which is good - I fixed my installation problem. Better would have been if the original developer had never released the non-POSIX script in the first place, something that might not have happened if the bash POSIX implementation was more conservative. The fact that I didn't switch back to dash after I installed the product eventually caused me to inject a non-POSIX test == operator into some scripts I was working on for the git project - again, something that would not have happened if bash's POSIX mode was more conservative. Performing that kind of checking, rigorously, in a shell, would be impossible to do statically anyway. Any such lint tool is limited to lexical analysis which makes it not very useful for testing unless your script is perfectly free of side-efffects. And who writes side-effect free shell scripts? How would the shell check for the correctness of: $(rm -rf somepath; echo '[') x == x ] I wasn't claiming that static checking would be viable. In fact, the impossibility of static checking is precisely why it would be useful to have real POSIX compliant interpreters that were as conservative as possible in the syntax and commands they accepted at least in their so-called POSIX mode. Currently, for example, if I want to test git sh scripts, I need to ensure that /bin/sh is pointing at dash, because if I use bash, errors of this kind can slip through. There really isn't a more effective way for me to fix address this problem than by abandoning bash as my POSIX shell and using dash instead since it's POSIX mode does seem to be more conservative. Given your example: jseymour@ubuntu:~/tracked/git$ bash --posix -c '$(rm -rf /tmp/foo; echo [) x == x ] echo yes' yes jseymour@ubuntu:~/tracked/git$ dash -c '$(rm -rf /tmp/foo; echo [) x == x ] echo yes' [: 1: x: unexpected operator bash --posix doesn't give me any clue that == isn't required to be supported POSIX interpreters, but at least dash does. jon.
Re: handling of test == by BASH's POSIX mode
On Sun, May 27, 2012 at 9:31 PM, Andreas Schwab sch...@linux-m68k.org wrote: Jon Seymour jon.seym...@gmail.com writes: As it stands, I can't use bash's POSIX mode to verify the validity or otherwise of a POSIX script because bash won't report these kinds of errors - even when running in POSIX mode. You can't do that anyway: POSIX mode does not disable proper extensions to POSIX, only those that conflict with POSIX. Specifically, in the case of the test utility, POSIX makes the behaviour unspecified when a three argument invocation does not match the POSIX-defined binary operators. I understand that the behaviour is unspecitied by POSIX - I didn't know that before, but I know that now - thanks. That said, from the point of view of promoting interoperable scripts, my view is that it (in an ideal world**) would be better if bash chose to fail, while executing in POSIX mode, in this case. Yes, it would be annoying to the script writer who has to replace == with =, but at least the script eco-system as a whole would benefit from a more conservative understanding of what it means to be a valid POSIX script. As it stands now, it is well nigh impossible to avoid undefined behaviour when the interpreter chooses to gracefully glosses over the fact that the script is utiising. ** I guess I can except that current bash behaviour is, on balance, the correct pragmatic decision since there would no doubt be widespread carnage in the scripting universe if bash was suddenly to become pickier about how it supports POSIX mode. Is there a case, I wonder, for enabling more conservative interpretation of test operators via a shell option of some kind? There is an --enable-strict-posix (?) configuration option. Will this do what I expect? That just switches the default for POSIX mode. Thanks. jon.
Re: handling of test == by BASH's POSIX mode
On Sun, May 27, 2012 at 11:09 PM, Jon Seymour jon.seym...@gmail.com wrote: On Sun, May 27, 2012 at 9:31 PM, Andreas Schwab sch...@linux-m68k.org wrote: Jon Seymour jon.seymour@gm ** I guess I can except that current bash behaviour is, on balance, except - accept
Re: handling of test == by BASH's POSIX mode
POSIX hasn't provided a way to validate whether a script only uses features that are required to be supported by POSIX compliant interpreters. I believe that was someone else's point, but yes that would be a problem for anyone who wanted to implement compliance check warnings. even if bash is technically compliant with POSIX, what use case is usefully served by having bash support a superset of the POSIX test operators while executing in POSIX mode? It's a matter of practicality and the fact that nobody has written it yet. If you wanted to implement conformance checks without making Bash even bigger and slower, and harder to maintain, I don't think there would be objections. Bash just modifies conflicting features to the minimal extent necessary to bring it into compliance, which seems to be the path of least resistance. This would be a big job, I think, and not quite at the top of my wish-list. Right now you can increase the number of things that fail by explicitly disabling non-POSIX built-ins using the Bash enable builtin. I wasn't claiming that static checking would be viable. In fact, the impossibility of static checking is precisely why it would be useful to have real POSIX compliant interpreters that were as conservative as possible in the syntax and commands they accepted at least in their so-called POSIX mode. Dash is useful for testing. The Bash answer is [[, which CAN do a lot of special error handling on due to it being a compound command. I wrote a bit about this here: http://mywiki.wooledge.org/BashFAQ/031/#Theory In reality, [[ is one of the very most portable non-POSIX features available. Most people shouldn't have to worry about avoiding it. On Sunday, May 27, 2012 11:09:03 PM Jon Seymour wrote: That said, from the point of view of promoting interoperable scripts, my view is that it (in an ideal world**) would be better if bash chose to fail, while executing in POSIX mode, in this case. In an ideal world, POSIX would define [[, Dash wouldn't exist, and we would have some resource other than POSIX that specifies what's ACTUALLY portable between modern shells, so that the only people who have to worry are those targeting Busybox and Solaris Heirloom, or stubborn curmudgeons who insist on Dash... some of these things feel like supporting IE6 - it's better to just not. This is exactly my problem: I replaced /bin/sh - dash with /bin/sh - bash because a 3rd party product installation script failed when dash was the POSIX shell. If you need a fast small shell, use mksh. It supports some of the more essential features of both Bash and Ksh (arrays, ((, [[), and some of its own, without all the draconian restrictions of dash. (Note this is querying my package manager, including symbols and source files) # equery s dash * app-shells/dash-0.5.7.1 Total files : 73 Total size : 1.14 MiB # equery s mksh * app-shells/mksh- Total files : 39 Total size : 1.61 MiB # equery s bash * app-shells/bash-4.2_p28 Total files : 464 Total size : 6.33 MiB -- Dan Douglas signature.asc Description: This is a digitally signed message part.
Re: handling of test == by BASH's POSIX mode
On Mon, May 28, 2012 at 2:08 AM, Dan Douglas orm...@gmail.com wrote: ... Bash just modifies conflicting features to the minimal extent necessary to bring it into compliance, which seems to be the path of least resistance. Sure. I understand that this is a reasonable philosophy given that aiming for complete avoidance of unspecified behaviour in the POSIX spec would probably lead to a completely unusable shell. That said, it is a shame from the point of view of interoperable scripts. Perhaps I should just accept that dash, being more minimal, does a better job of being that conformance testing shell? This would be a big job, I think, and not quite at the top of my wish-list. Right now you can increase the number of things that fail by explicitly disabling non-POSIX built-ins using the Bash enable builtin. Thanks for that tip. Dash is useful for testing. The Bash answer is [[, which CAN do a lot of special error handling on due to it being a compound command. I wrote a bit about this here: http://mywiki.wooledge.org/BashFAQ/031/#Theory In reality, [[ is one of the very most portable non-POSIX features available. Most people shouldn't have to worry about avoiding it. Thanks, I'll have a read. jon.
