Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 15/03/2023 00:17, Chet Ramey wrote: On 3/14/23 4:58 PM, Harald van Dijk wrote: On 14/03/2023 20:41, Chet Ramey wrote: On 3/12/23 10:19 PM, Harald van Dijk via austin-group-l at The Open Group wrote: bash appears to disables the reading of .profile in POSIX mode entirely. This isn't quite correct. By default, a login shell named `sh' or `-sh' reads /etc/profile and ~/.profile. You can compile bash for `strict posix' conformance, or invoke it with POSIXLY_CORRECT or POSIX_PEDANTIC in the environment, and it won't. Isn't it? The mode bash gets into when invoked as sh is described in the manpage (looking at the 5.2.15 manpage) as: If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well. [...] When invoked as sh, bash enters posix mode after the startup files are read. The mode bash gets into when POSIXLY_CORRECT is set, the mode that can also be obtained with --posix, is described in the manpage as: When bash is started in posix mode, as with the --posix command line option, it follows the POSIX standard for startup files. Right. When you force posix mode immediately, as I said above, bash won't read the startup files. A login shell named sh or -sh reads the historical startup fles, then enters posix mode. We are clear on how bash behaves, I believe. What is not clear to me is how I should have worded my message so that you would not have taken issue with it. As far as I can tell, you are saying the same thing I was. I wrote that bash appears to disables the reading of .profile in POSIX mode entirely. It does. I do not see any contradiction between that and what you wrote: while a clarification that this does not apply to bash being invoked as sh or -sh is useful, I made no comment about what happens when bash is invoked as sh or -sh. My comment was about POSIX mode, and when invoked as sh or -sh, per the manpage and your own message, the startup file processing happens while bash is still in non-POSIX mode. So what should I have written instead? If I had written bash appears to disable the reading of .profile in POSIX mode (i.e. bash --posix) entirely. would that have been sufficiently clear to avoid this discussion? If so I will keep that in mind. Cheers, Harald van Dijk
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 3/14/23 4:58 PM, Harald van Dijk wrote: On 14/03/2023 20:41, Chet Ramey wrote: On 3/12/23 10:19 PM, Harald van Dijk via austin-group-l at The Open Group wrote: bash appears to disables the reading of .profile in POSIX mode entirely. This isn't quite correct. By default, a login shell named `sh' or `-sh' reads /etc/profile and ~/.profile. You can compile bash for `strict posix' conformance, or invoke it with POSIXLY_CORRECT or POSIX_PEDANTIC in the environment, and it won't. Isn't it? The mode bash gets into when invoked as sh is described in the manpage (looking at the 5.2.15 manpage) as: If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well. [...] When invoked as sh, bash enters posix mode after the startup files are read. The mode bash gets into when POSIXLY_CORRECT is set, the mode that can also be obtained with --posix, is described in the manpage as: When bash is started in posix mode, as with the --posix command line option, it follows the POSIX standard for startup files. Right. When you force posix mode immediately, as I said above, bash won't read the startup files. A login shell named sh or -sh reads the historical startup fles, then enters posix mode. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 14/03/2023 20:41, Chet Ramey wrote: On 3/12/23 10:19 PM, Harald van Dijk via austin-group-l at The Open Group wrote: bash appears to disables the reading of .profile in POSIX mode entirely. This isn't quite correct. By default, a login shell named `sh' or `-sh' reads /etc/profile and ~/.profile. You can compile bash for `strict posix' conformance, or invoke it with POSIXLY_CORRECT or POSIX_PEDANTIC in the environment, and it won't. Isn't it? The mode bash gets into when invoked as sh is described in the manpage (looking at the 5.2.15 manpage) as: If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well. [...] When invoked as sh, bash enters posix mode after the startup files are read. The mode bash gets into when POSIXLY_CORRECT is set, the mode that can also be obtained with --posix, is described in the manpage as: When bash is started in posix mode, as with the --posix command line option, it follows the POSIX standard for startup files. My wording appears to be consistent with how bash describes itself in its manpage: there is POSIX mode during startup file processing, and POSIX mode after startup file processing. Only the POSIXLY_CORRECT / --posix behaviour causes startup file processing to happen in POSIX mode, and results in the reading of .profile being disabled. But if I am misunderstanding how you intend for the different modes of bash to be referred to, if you can clarify I can take care to be more careful with my wording in the future. Cheers, Harald van Dijk
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 3/12/23 10:19 PM, Harald van Dijk via austin-group-l at The Open Group wrote: bash appears to disables the reading of .profile in POSIX mode entirely. This isn't quite correct. By default, a login shell named `sh' or `-sh' reads /etc/profile and ~/.profile. You can compile bash for `strict posix' conformance, or invoke it with POSIXLY_CORRECT or POSIX_PEDANTIC in the environment, and it won't. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 12/03/2023 19:10, Robert Elz wrote:
Date:Fri, 10 Mar 2023 23:40:18 +
From:"Harald van Dijk via austin-group-l at The Open Group"
Message-ID:
| Based on past experiences, I am assuming the e-mail this is a reply to
| was meant to be sent to the list and I am quoting it in full and
| replying on the list for that reason.
Thanks, and yes, it was - my MUA absolutely believes in the one true
meaning of Reply-To (where the author of the message to which the reply
is being sent requests that replies be sent -- to addresses in that field,
and no others). I need to manually override it when I choose to ignore
that request and send to different addresses (which is allowed, but in
general, done only with proper consideration of why). This list always
directs that all replies go only to the author of the message, and never
to the list itself. Irritating...
I know. Even more frustrating is the reasoning we have been given as to
the reasoning for it ("to better handle DMARC email authentication for
messages" -- no, DMARC absolutely does not require that).
| Sourcing arbitrary script fragments and having assurance that they do
| not exit the shell is not reasonable, as the arbitrary script fragment
| could contain an 'exit' command.
Of course, deliberate exits aren't the issue, only accidental ones.
| Beyond shell options and variable assignments not persisting in the
| parent shell, are there any other issues you see with running them in a
| subshell?
The whole point of many . scripts is to alter the shell's environment,
if they were just arbitrary commands, not intended to affect the current
shell, they'd just be sh scripts, and run the normal way. The very act
of using the '.' command more or less means "must run in the current shell).
"(. file)" is silly, "file" would accomplish the same thing (if executable,
otherwise "sh < file" after finding the path to file) in a more obvious way.
This does not result in the same behaviour. (. file) can be used to
inherit unexported shell variables, shell functions, aliases, etc. from
the parent shell that would not be available to an explicitly launched
sh subprocess.
I agree that it is likely that a . script would be sourced in order to
change the current shell environment, but I do not agree that it is the
only legitimate use of it, and I had assumed based on your "without
risking the shell exiting" that you were considering one of those other
uses.
Apart from options and variables, . files often define functions, change
the umask and perhaps ulimit, and may alter the current directory, set exit
(or other) traps, ... anything in fact.
Sure, they are in the same category as options and variables.
As an example, consider what you might put in your .profile or $ENV
file - those are run in more or less the same way as a '.' file (just
without the PATH search to locate the file). XRAT C.2.5.3 says almost
exactly that about ENV.
Indeed. Worth explicitly pointing out here is that they are only used by
interactive shells, so in this comparison, only the behaviour of the '.'
command in interactive shells is relevant.
(Strangely though, even though .profile is
mentioned several times as a place where things can be set, it doesn't
appear in the standard (as something that shells process) at all - which
is kind of odd really, since it is considerably older then ENV, and as best
I can tell, supported by everything. The closest that we get is a mention
in XRAT that "some shells" run it at startup of a login shell. Which are
the other shells? That is, the ones that don't run .profile? And I don't
mean in situations like bash, which prefers .bash_profile if it exists.
bash appears to disables the reading of .profile in POSIX mode entirely.
yash only ever appears to source .yash_profile, it does not fall back to
.profile if no .yash_profile exists.
I doubt that you'd want those scripts run in a subshell environment,
I also doubt that you want the shell to exit if there's an error in
one of them. How would you ever be able to log in (and start a shell)
if it exited before you ever had a chance to run a command? If you can't
log in, because your shell won't start, how would you ever fix the problem?
As best I can tell (I have done very limited testing of this) shells tend
to simply abort processing one of those scripts upon encountering an error
(like a syntax error, etc - not executing "exit" - that should exit) and
just go on to the next step of initialising the shell. They don't just
exit because there's a syntax error - most shells report the error (not all),
but I couldn't find one which exits.
This is indeed a problem. My own testing supports your conclusion,
shells are in agreement that a syntax error here does not result in the
shell terminating, and does result in the remainder of the envfile being
ignored. An
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
Date:Fri, 10 Mar 2023 23:40:18 + From:"Harald van Dijk via austin-group-l at The Open Group" Message-ID: | Based on past experiences, I am assuming the e-mail this is a reply to | was meant to be sent to the list and I am quoting it in full and | replying on the list for that reason. Thanks, and yes, it was - my MUA absolutely believes in the one true meaning of Reply-To (where the author of the message to which the reply is being sent requests that replies be sent -- to addresses in that field, and no others). I need to manually override it when I choose to ignore that request and send to different addresses (which is allowed, but in general, done only with proper consideration of why). This list always directs that all replies go only to the author of the message, and never to the list itself. Irritating... | Sourcing arbitrary script fragments and having assurance that they do | not exit the shell is not reasonable, as the arbitrary script fragment | could contain an 'exit' command. Of course, deliberate exits aren't the issue, only accidental ones. | Beyond shell options and variable assignments not persisting in the | parent shell, are there any other issues you see with running them in a | subshell? The whole point of many . scripts is to alter the shell's environment, if they were just arbitrary commands, not intended to affect the current shell, they'd just be sh scripts, and run the normal way. The very act of using the '.' command more or less means "must run in the current shell). "(. file)" is silly, "file" would accomplish the same thing (if executable, otherwise "sh < file" after finding the path to file) in a more obvious way. Apart from options and variables, . files often define functions, change the umask and perhaps ulimit, and may alter the current directory, set exit (or other) traps, ... anything in fact. As an example, consider what you might put in your .profile or $ENV file - those are run in more or less the same way as a '.' file (just without the PATH search to locate the file). XRAT C.2.5.3 says almost exactly that about ENV. (Strangely though, even though .profile is mentioned several times as a place where things can be set, it doesn't appear in the standard (as something that shells process) at all - which is kind of odd really, since it is considerably older then ENV, and as best I can tell, supported by everything. The closest that we get is a mention in XRAT that "some shells" run it at startup of a login shell. Which are the other shells? That is, the ones that don't run .profile? And I don't mean in situations like bash, which prefers .bash_profile if it exists. I doubt that you'd want those scripts run in a subshell environment, I also doubt that you want the shell to exit if there's an error in one of them. How would you ever be able to log in (and start a shell) if it exited before you ever had a chance to run a command? If you can't log in, because your shell won't start, how would you ever fix the problem? As best I can tell (I have done very limited testing of this) shells tend to simply abort processing one of those scripts upon encountering an error (like a syntax error, etc - not executing "exit" - that should exit) and just go on to the next step of initialising the shell. They don't just exit because there's a syntax error - most shells report the error (not all), but I couldn't find one which exits. | You have left out bash 4 here. For the same reason I didn't include ancient versions of all the other shells either. That's obsolete, not going to change in the future, and has been replaced. [And because I happen not to have a binary of it at the minute - I could make one, I do have sources, just don't really see the need.] | I do not expect bosh to have a large user base (even if it will be wider | than mine), but as I am sure J�rg would have pointed out, the shell has | historical significance in that it is a descendant of the Bourne shell | from which POSIX shell language is also derived. So is/was ksh88, and then ksh93 ... they were just modified more. | (Although I wouldn't be | opposed to a change to POSIX to *allow* something different.) As I hinted in the note in bugid:1629 which spawned this discussion (bugnote:6200) I expect this part might need to move to "may exit" rather than "shall not exit" (away from "shall exit" which it is now, in the cases in question, not all) for a release cycle (or two) - but then again given the number, and popularity, of the shells which already don't exit in these circumstances, perhaps that won't be needed. That should be discussed further. The reason that read errors are different in this regard (at least in the main script, not in . files -- not sure it is possible to have an equivalent to a read error in "eval" - perhaps an EILSEQ (bad char encoding) in the string might count? -- and that
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
Based on past experiences, I am assuming the e-mail this is a reply to was meant to be sent to the list and I am quoting it in full and replying on the list for that reason. On 10/03/2023 21:24, Robert Elz wrote: Date:Fri, 10 Mar 2023 18:13:00 + From:"Harald van Dijk via austin-group-l at The Open Group" Message-ID: | Other shells that exit are bosh, yash, and my own. It's both what POSIX | currently requires (contrary to what kre wrote on the bug) That's not how I intended what I wrote to be interpreted, I meant exactly what you said - when I wrote "most shells are doing what shells always have, and what the standard requires", I meant "exiting". I'm glad that that is what you meant. I wrote my mail before reading your mail where you said you tested something different, so when I read that you wrote that what most shells are doing is what the standard requires, it could only be interpreted to mean the opposite of what you intended. But as I wrote in my previous message, I was actually testing "command eval" rather than "command ." which I would normally expect to work about the same way in this regard, but it turns out that not shells all do. Further, and subsequent to when I sent that last message, I went and looked at my tests again - the way I do these is by (for tests like this one) composing a command line as input to one shell (each has its own xterm in my shell testing root window page - they're all tiled), then pasting it into the windows for all the others - then I can see the results from all of them, at the same time, and easily compare what happened (the command always starts $SHELL kind of like the example Geoff showed, except I do not quote that, because sometimes SHELL needs to be "bash -o posix" or similar, and I want that field split, not treated as a quoted word. For this, I tested both without, and with, "command" present ... but it turns out that somehow, for some of the shells, instead of running both tests, I managed to paste the wrong command, and ran the one without "command" twice, without noticing. That even included the NetBSD sh test, which contrary to what I said before, turns out does do the same thing for "." and "eval" in both cases (exit without command, not exit with it) which is what I had expected, before I saw the results of the incorrect test - before I noticed it was incorrect. | and what I think is probably the right thing for shells to do. I don't. I want to be able to source arbitrary script fragments, and eval arbitrary strings (there are no security issues here, the fragments and strings, are all provided by the user running the shell - anything that could be done buried one of those other ways, could simply be done as a command without subterfuge) without risking the shell exiting. Sometimes running them in a subshell works, but only sometimes. Sourcing arbitrary script fragments and having assurance that they do not exit the shell is not reasonable, as the arbitrary script fragment could contain an 'exit' command. In order to be able to do this, a subshell is needed. That will mean that variable assignments etc. do not persist in the parent shell, but that is unavoidable. Either the sourced script affects the parent shell, in which case it can cause the parent shell to exit, or it does not affect the parent shell. Beyond shell options and variable assignments not persisting in the parent shell, are there any other issues you see with running them in a subshell? | Whether bug 1629 should introduce a significant shell consistency issue | is not separate from bug 1629. Perhaps that one, and some new one, yet to be submitted, should be considered together, but resolving 1629 the right way should not be held hostage by other ancient weirdness that might not be so easy to alter. But perhaps after all, it might be - if it is only yash, bosh and your shell not already continuing after "command . file" fails because of a syntax error, then those might not matter, and those, plus, I think, mksh and ancient pdksh (and consequently, probably ksh88 as well) for "command eval 'gibberish<;)'" failing the same way then I'd guess mksh can get changed, and the others also no longer really matter. You have left out bash 4 here. For 'command eval', that version behaves the same way as for 'command .' in that it causes the shell to exit on syntax errors. | Bug 1629 started as trying to see what | shell authors are willing to implement. No, it started because read errors were not being handled in a rational way. A proposed solution depended upon what shell authors are willing to implement. True, I should have said "the discussion on bug 1629 started" rather than "bug 1629 started". | and I know bosh sadly isn't going to see an update anyway, Really? I thought some group of people had taken over Schilling's stuff. Whether they consider bosh worth continuing
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
Date:Fri, 10 Mar 2023 18:13:00 + From:"Harald van Dijk via austin-group-l at The Open Group" Message-ID: | Other shells that exit are bosh, yash, and my own. It's both what POSIX | currently requires (contrary to what kre wrote on the bug) That's not how I intended what I wrote to be interpreted, I meant exactly what you said - when I wrote "most shells are doing what shells always have, and what the standard requires", I meant "exiting". But as I wrote in my previous message, I was actually testing "command eval" rather than "command ." which I would normally expect to work about the same way in this regard, but it turns out that not shells all do. Further, and subsequent to when I sent that last message, I went and looked at my tests again - the way I do these is by (for tests like this one) composing a command line as input to one shell (each has its own xterm in my shell testing root window page - they're all tiled), then pasting it into the windows for all the others - then I can see the results from all of them, at the same time, and easily compare what happened (the command always starts $SHELL kind of like the example Geoff showed, except I do not quote that, because sometimes SHELL needs to be "bash -o posix" or similar, and I want that field split, not treated as a quoted word. For this, I tested both without, and with, "command" present ... but it turns out that somehow, for some of the shells, instead of running both tests, I managed to paste the wrong command, and ran the one without "command" twice, without noticing. That even included the NetBSD sh test, which contrary to what I said before, turns out does do the same thing for "." and "eval" in both cases (exit without command, not exit with it) which is what I had expected, before I saw the results of the incorrect test - before I noticed it was incorrect. | and what I think is probably the right thing for shells to do. I don't. I want to be able to source arbitrary script fragments, and eval arbitrary strings (there are no security issues here, the fragments and strings, are all provided by the user running the shell - anything that could be done buried one of those other ways, could simply be done as a command without subterfuge) without risking the shell exiting. Sometimes running them in a subshell works, but only sometimes. | Whether bug 1629 should introduce a significant shell consistency issue | is not separate from bug 1629. Perhaps that one, and some new one, yet to be submitted, should be considered together, but resolving 1629 the right way should not be held hostage by other ancient weirdness that might not be so easy to alter. But perhaps after all, it might be - if it is only yash, bosh and your shell not already continuing after "command . file" fails because of a syntax error, then those might not matter, and those, plus, I think, mksh and ancient pdksh (and consequently, probably ksh88 as well) for "command eval 'gibberish<;)'" failing the same way then I'd guess mksh can get changed, and the others also no longer really matter. | Bug 1629 started as trying to see what | shell authors are willing to implement. No, it started because read errors were not being handled in a rational way. A proposed solution depended upon what shell authors are willing to implement. | and I know bosh sadly isn't going to see an update anyway, Really? I thought some group of people had taken over Schilling's stuff. Whether they consider bosh worth continuing with I am not sure (it still has more important issues than this remaining in it, and I don't believe is used much, if at all). | but I would hope that authors | of the other shells also have the good sense to implement something that | makes sense to them and keep it internally consistent, There is so much in the shell already which is not internally consistent, that one more thing (particularly in an area rarely seen) would hardly be noticed, but I very much doubt that there will not be at least an attempt to alter the "what happens when there's an error which is "shall exit" detected when running a special built-in as a sub-command of "command". Syntax errors aren't the only one, command eval 'shift 0 >/' is another (redirection errors are also "shall exit" when used with a special built-in as is being done here). I expect that will probably succeed, even if we all need to make some more changes to almost never encountered parts of the shell, and most probably it won't be "we all" in any case. kre
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
On 10/03/2023 17:12, Geoff Clare via austin-group-l at The Open Group wrote: -- (0006200) kre (reporter) - 2023-03-10 14:52 https://www.austingroupbugs.net/view.php?id=1629#c6200 -- Re https://www.austingroupbugs.net/view.php?id=1629#c6197 https://www.austingroupbugs.net/view.php?id=1629#c6198 https://www.austingroupbugs.net/view.php?id=1629#c6199 : I agree it is probably a problem that shells exit on syntax errors after "command ." (also "command eval") but it is a problem in the sense that it would be better if that isn't what happened - and it isn't just "some shells" that exit, it is most of them. None of the shells I tried exited. I used a file containing: foo < and ran: "$shell" -c 'command . ./file ; echo $?' All of bash, ksh88, ksh93, dash and mksh reported the syntax error and then executed the echo command (which output a non-zero number). For bash, it depends on the version. Version 4 will exit. Other shells that exit are bosh, yash, and my own. It's both what POSIX currently requires (contrary to what kre wrote on the bug) and what I think is probably the right thing for shells to do. In any case, this isn't a problem with the proposed resolution of this issue (read error behaviour) - a new bug needs to be filed and processed to get the behaviour of "command ." and "command eval" altered to be "not exit" (and for this, it may need to start out just being "may exit" rather than "shall not exit" with a future directions indicating that the stronger form will apply in a later version. I agree this should be handled as a separate issue, hence my replying by email instead of a note in the bug. Whether bug 1629 should introduce a significant shell consistency issue is not separate from bug 1629. Bug 1629 started as trying to see what shell authors are willing to implement. In that case, let me be clear, as it stands this is not something I am going to implement as written. I know you have no reason to care what I do in my shell, and I know bosh sadly isn't going to see an update anyway, but I would hope that authors of the other shells also have the good sense to implement something that makes sense to them and keep it internally consistent, whether that is by ignoring what 1629 specifies, or by ignoring what is required for syntax errors. There are more issues that make me doubt this is going to be implemented as written in other shells too, but that's for the other authors to raise. Cheers, Harald van Dijk
Re: Syntax error with "command . file" (was: [1003.1(2016/18)/Issue7+TC2 0001629]: Shell vs. read(2) errors on the script)
Date:Fri, 10 Mar 2023 17:12:50 + From:"Geoff Clare via austin-group-l at The Open Group" Message-ID: | All of bash, ksh88, ksh93, dash and mksh reported the syntax error and | then executed the echo command (which output a non-zero number). yash and bosh don't, they simply exit. But you caught me ... the tests I did yesterday were of "command eval" which I assumed would be treated the same (I see no reason why there should be a difference), but apparently isn't in many shells (including the NetBSD one). kre
