Re: shell `var=value function`
On 6/10/19 11:38 AM, enh wrote:
> the toybox project is adding a shell, and came across this issue that
> i (Android native tools/libraries maintainer) have hit before because
> mksh does exactly what POSIX says and bash does what POSIX probably
> intended:
> http://lists.landley.net/pipermail/toybox-landley.net/2019-June/010530.html
>
> repeated here, lightly edited and extended, because i know there are
> plenty of shell folks here:
>
> I've reached the part of the posix shell stuff (section 2.9.1: simple
> commands)
> that specifies this behavior, and posix doesn't match bash:
>
> If no command name results, or if the command name is a special built-in or
> function, variable assignments shall affect the current execution
> environment.
> Otherwise, the variable assignments shall be exported for the execution
> environment of the command and shall not affect the current execution
> environment except as a side-effect of the expansions performed in step 4.
This has been changed in the current published version of POSIX, which
makes the variable persistance behavior unspecified:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01
"If the command name is a function that is not a standard utility
implemented as a function, variable assignments shall affect the current
execution environment during the execution of the function. It is unspecified:
Whether or not the variable assignments persist after the completion of
the function
Whether or not the variables gain the export attribute during the
execution of the function
Whether or not export attributes gained as a result of the variable
assignments persist after the completion of the function (if variable
assignments persist after the completion of the function)"
>
> A) This is not what bash does, or has ever done:
>
> $ hello() { echo boing=$BOING; }
> $ BOING=123 hello
> $ echo $BOING
While this is true for bash in its default mode, bash implements the
previously-mandated POSIX behavior in POSIX mode:
$ cat x3
set -o posix
hello() { echo boing=$BOING; }
BOING=123 hello
echo after: $BOING
$ ../bash-5.0-patched/bash x3
boing=123
after: 123
I agree that the default bash behavior is more useful and often what is
desired (that's why it's the default), and the next version of bash will
probably make that the default in POSIX mode as well.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: shell `var=value function`
On Mon, Jun 10, 2019 at 8:56 AM Chet Ramey wrote:
>
> On 6/10/19 11:38 AM, enh wrote:
> > the toybox project is adding a shell, and came across this issue that
> > i (Android native tools/libraries maintainer) have hit before because
> > mksh does exactly what POSIX says and bash does what POSIX probably
> > intended:
> > http://lists.landley.net/pipermail/toybox-landley.net/2019-June/010530.html
> >
> > repeated here, lightly edited and extended, because i know there are
> > plenty of shell folks here:
> >
> > I've reached the part of the posix shell stuff (section 2.9.1: simple
> > commands)
> > that specifies this behavior, and posix doesn't match bash:
> >
> > If no command name results, or if the command name is a special built-in
> > or
> > function, variable assignments shall affect the current execution
> > environment.
> > Otherwise, the variable assignments shall be exported for the execution
> > environment of the command and shall not affect the current execution
> > environment except as a side-effect of the expansions performed in step 4.
>
> This has been changed in the current published version of POSIX, which
> makes the variable persistance behavior unspecified:
>
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01
excellent!
> "If the command name is a function that is not a standard utility
> implemented as a function, variable assignments shall affect the current
> execution environment during the execution of the function. It is unspecified:
>
> Whether or not the variable assignments persist after the completion of
> the function
>
> Whether or not the variables gain the export attribute during the
> execution of the function
>
> Whether or not export attributes gained as a result of the variable
> assignments persist after the completion of the function (if variable
> assignments persist after the completion of the function)"
>
>
>
> >
> > A) This is not what bash does, or has ever done:
> >
> > $ hello() { echo boing=$BOING; }
> > $ BOING=123 hello
> > $ echo $BOING
>
> While this is true for bash in its default mode, bash implements the
> previously-mandated POSIX behavior in POSIX mode:
>
> $ cat x3
> set -o posix
> hello() { echo boing=$BOING; }
> BOING=123 hello
> echo after: $BOING
> $ ../bash-5.0-patched/bash x3
> boing=123
> after: 123
>
> I agree that the default bash behavior is more useful and often what is
> desired (that's why it's the default), and the next version of bash will
> probably make that the default in POSIX mode as well.
yeah, it also seems like the least surprising behavior... when i first
saw this with folks porting from bash to mksh to run on Android, i was
convinced it was a mksh bug because it didn't make any sense that
you'd get different behavior for a function than for an external
command!
> Chet
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: shell `var=value function`
Date:Mon, 10 Jun 2019 09:23:38 -0700 From:enh Message-ID: | yeah, it also seems like the least surprising behavior... when i first | saw this with folks porting from bash to mksh to run on Android, i was | convinced it was a mksh bug because it didn't make any sense that | you'd get different behavior for a function than for an external | command! It is (was) purely ancient Bourne shell behaviour, which did the exact same assignment in both cases, but in one had forked already, but not yet done the exec of course - when the exec happens (or exit if that fails) the effect of the assignment was lost. When there was no fork, the assignment persisted. This distinction (rather than anything related to what makes sense) is what underlies many of the weirder rules. Some of this was cleaned up in early ksh, and so is also cleaned up in POSIX, other stuff wasn't, and so persists (not just in this area.) kre
Re: shell `var=value function`
On Mon, Jun 10, 2019 at 12:06 PM Robert Elz wrote: > > Date:Mon, 10 Jun 2019 09:23:38 -0700 > From:enh > Message-ID: > > > | yeah, it also seems like the least surprising behavior... when i first > | saw this with folks porting from bash to mksh to run on Android, i was > | convinced it was a mksh bug because it didn't make any sense that > | you'd get different behavior for a function than for an external > | command! > > It is (was) purely ancient Bourne shell behaviour, which did the exact > same assignment in both cases, but in one had forked already, but not > yet done the exec of course - when the exec happens (or exit if that > fails) the effect of the assignment was lost. When there was no fork, > the assignment persisted. > > This distinction (rather than anything related to what makes sense) is > what underlies many of the weirder rules. > > Some of this was cleaned up in early ksh, and so is also cleaned up in > POSIX, other stuff wasn't, and so persists (not just in this area.) reminds me of Bruce Tognazzini's story: "There's an urban myth about a clothing distributor who sent a sample of a sports coat to an offshore manufacturer asking for 10,000 copies. Well, the copies were perfect ... right down to a cigarette burn on the left sleeve of the original." (i read it in his blue book, but the only version i see on the web is https://www.wired.com/1995/09/tognazzini/) > kre >
Re: shell `var=value function`
To me, it is one of the underspecified responsibilities of a standard utility
implemented as a shell function to use an internal procedure that makes the
current local state as if a subshell was forked to exec a utility on media,
including with temporary exports, etc.
When the shell function finishes execution, I read it as local variables
created by the function persist because they add to the current variable state,
but modified variables get restored to the values hidden by that internal
routine, as non-export values a utilty environment wouldn't have access to,
plus the previous values of exported variables get restored from the save; by
use of a separate pseudo-unfork routine. Whether local or exported, temporary
exports from assigning a variable as part of the invoke should be reverted;
either unset or reacquire previous value.
A nice utility implementation will do an explicit unset on variables it creates
before calling the unfork, so the namespace of the environment isn't polluted
by these as artifacts, but the standard doesn't require this. As a result the
wording change cited has to leave things unspecified, with the consequence any
local variable modifications may persist, even though this is nominally a bug
for any utility other than a special builtin.
On Monday, June 10, 2019 enh wrote:
On Mon, Jun 10, 2019 at 8:56 AM Chet Ramey wrote:
>
> On 6/10/19 11:38 AM, enh wrote:
> > the toybox project is adding a shell, and came across this issue that
> > i (Android native tools/libraries maintainer) have hit before because
> > mksh does exactly what POSIX says and bash does what POSIX probably
> > intended:
> > http://lists.landley.net/pipermail/toybox-landley.net/2019-June/010530.html
> >
> > repeated here, lightly edited and extended, because i know there are
> > plenty of shell folks here:
> >
> > I've reached the part of the posix shell stuff (section 2.9.1: simple
> > commands)
> > that specifies this behavior, and posix doesn't match bash:
> >
> > If no command name results, or if the command name is a special built-in or
> > function, variable assignments shall affect the current execution
> >environment.
> > Otherwise, the variable assignments shall be exported for the execution
> > environment of the command and shall not affect the current execution
> > environment except as a side-effect of the expansions performed in step 4.
>
> This has been changed in the current published version of POSIX, which
> makes the variable persistance behavior unspecified:
>
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01
excellent!
> "If the command name is a function that is not a standard utility
> implemented as a function, variable assignments shall affect the current
> execution environment during the execution of the function. It is unspecified:
>
> Whether or not the variable assignments persist after the completion of
> the function
>
> Whether or not the variables gain the export attribute during the
> execution of the function
>
> Whether or not export attributes gained as a result of the variable
> assignments persist after the completion of the function (if variable
> assignments persist after the completion of the function)"
>
>
>
> >
> > A) This is not what bash does, or has ever done:
> >
> > $ hello() { echo boing=$BOING; }
> > $ BOING=123 hello
> > $ echo $BOING
>
> While this is true for bash in its default mode, bash implements the
> previously-mandated POSIX behavior in POSIX mode:
>
> $ cat x3
> set -o posix
> hello() { echo boing=$BOING; }
> BOING=123 hello
> echo after: $BOING
> $ ../bash-5.0-patched/bash x3
> boing=123
> after: 123
>
> I agree that the default bash behavior is more useful and often what is
> desired (that's why it's the default), and the next version of bash will
> probably make that the default in POSIX mode as well.
yeah, it also seems like the least surprising behavior... when i first
saw this with folks porting from bash to mksh to run on Android, i was
convinced it was a mksh bug because it didn't make any sense that
you'd get different behavior for a function than for an external
command!
> Chet
>
> --
> ``The lyf so short, the craft so long to lerne.'' - Chaucer
> ``Ars longa, vita brevis'' - Hippocrates
> Chet Ramey, UTech, CWRU c...@case.edu http://tiswww.cwru.edu/~chet/
Re: shell `var=value function`
Date:Mon, 10 Jun 2019 19:55:05 + (UTC)
From:Shware Systems
Message-ID: <1277876767.1002266.1560196505...@mail.yahoo.com>
| To me, it is one of the underspecified responsibilities of a standard
| utility implemented as a shell function
No-one is talking (specifically) about standard utilities implemnented
as shell functions - just shell functions
| to use an internal procedure that makes the current local state
| as if a subshell was forked to exec a utility on media, including
| with temporary exports, etc.
Sure, such a function needs to beware of what it might do which would
otherwise affect the environment.But that has nothing do do with
anything here.
Consider such a function (say someone decided to implement "c99" as a
shell function)
c99() {
# do whatever run the passes, etc, all very carefully
}
Now I run
FOOBAR=hello c99 ...
where neither the real (external) c99, nor the function, does anything
with a variable called FOOBAR.
For the real one, FOOBAR will be present in its environment, where it
will be ignored, and affect nothing at all. After the command comletes
all knowledge of the assignment of "hello" to FOOBAR is forgotten (it
remains in whatever state it was before that command).
For the function - the way the spec used to be writtem (but fortunately
no more) and the way that old shells used to work (and some still do)
the FOOBAR=hello will persist in the calling shell after the c99() function
finished. There is nothing whatever the function implementation can do
about that, if the shell implements var-assigns before functions that way.
| When the shell function finishes execution, I read it as local variables
| created by the function persist
That's certainly true, if a function deliberately alters the environment
it stays altered.
| but modified variables get restored to the values hidden by that
| internal routine,
but that isn't - anything the function does which alters the environment
alters the environment (except of course if the function creates a sub-shell
environment one way or another, and modfies things inside that).
If you're considering variables like FOOBAR above, and wondering what
happens if the function happened to modify FOOBAR internally, then yes,
that's a good question, and one for which I don't think there is a
current answer (I don't even know if shells are consistent - though I
would expect those that allow such env vars to persist would simply
keep the function modified value - my guess is that most others effectively
create a local variable FOOBAR which means that any pre-existing
value in the calling environment cannot be altered by the function.
But I don't think that is specified anywhere.
Sorry, I couldn't follow what you meant when you started talking about
pseudo unfork routines ...
| A nice utility implementation will do an explicit unset on variables
| it creates before calling the unfork,
What is an unfork - and how do I do that?
And yes, some way to deal with local vars that the function does not
want expported is important, but for now that is a very hard problem - at
the very least the shell nbeeds to, for every variable it creates, keep
track of the state of that variable from outside, so they can later be
restored. It is possible, but the coding is tricky and messy, as until
vars are saved the only place the function can modify (to remember old
values) is in the positional params, so doing this means pushing new ones
to remember at least one var's set/unset state and (if set) value, so that
var can then be used to remember the state of others, so they can all be
restored later. Ugly code.
This is why just about everyone agrees that we need real local vars for
functions. What we haven't been able to agree upon, is what their exact
semantics should be - and since everyone has implemented something different
than everyone else, there is little common ground upon which to base a
standard for them,
| so the namespace of the environment isn't polluted by these as
| artifacts, but the standard doesn't require this. As a result
| the wording change cited has to leave things unspecified,
It leaves things unspecified (for now) because it used to require the
exact wrong behaviour, and penalising shells for doing what the standard
said they must do would be grossly unfair, if it were to be changed to
require the opposite.So instead, for now, it is just unspecified,
which at least allows those shells that used to implement the standard to
cange to implement something rational instead.
| with the consequence any local variable modifications may persist,
local var modifications will always persist. No-one is even suggesting
altering that, and if they were, I suspect almost everyone here would
object.Being able to write functions which modify variables is one
of their more useful characteristics.
kre
Re: shell `var=value function`
On 2019-06-10 08:38:52 -0700, enh wrote:
> but here's mksh:
>
> ~$ mksh
> $ hello() { echo boing=$BOING; }
> $ BOING=123 hello
> boing=123
> $ echo $BOING
> 123
> $
>
> and ksh, since that seems to come up on this list:
>
> $ ksh
> ~ [1]$ hello() { echo boing=$BOING; }
> ~ [2]$ BOING=123 hello
> boing=123
> ~ [3]$ echo $BOING
> 123
> ~ [4]$
>
> zsh also agrees. so it seems like bash is the exception (but bash is
> also the most useful?).
No, zsh doesn't agree:
zira% hello() { echo boing=$BOING; }
zira% BOING=123 hello
boing=123
zira% echo $BOING
zira%
But with some builtins and some variables, these variables remain set.
Very ugly.
--
Vincent Lefèvre - Web:
