"command" behaviour
The "command" builtin is supposed to disable the "special" properties of "special" builtins, which, as far as I can tell, simply means it should stop them from exiting the shell on error as they normally do. In practice this is a bit hit and miss, but it works for many builtins on many shells. However, the following: command exec /dev/null/nonexistent/file || echo oops instantly exits all the shells I can find, never getting to the 'echo'. So is it that no shell is compliant, or am I missing something and is this actually compliant behaviour? The thing is that test -x /some/file && exec /some/file has a race condition between the "test" and the "exec"; if the other method worked, it would be atomic. Thanks, - M.
Re: "command" behaviour
Op 19-10-16 om 16:39 schreef Joerg Schilling: > Martijn Dekker wrote: > >> The "command" builtin is supposed to disable the "special" properties of >> "special" builtins, which, as far as I can tell, simply means it should >> stop them from exiting the shell on error as they normally do. >> >> In practice this is a bit hit and miss, but it works for many builtins >> on many shells. However, the following: >> >> command exec /dev/null/nonexistent/file || echo oops >> >> instantly exits all the shells I can find, never getting to the 'echo'. > > The exec*() syscall family always starts with destroing the previous address > space of the calling command. There is no other way... OK, maybe I misspoke when I said shells "instantly" exit: in fact they produce a diagnostic error message, then instantly exit. All the shells seem to manage to stick around that long after the 'exec' command fails. (Bash even manages to speak a little Dutch.) By my logic, if the shell can stick around to produce an error message, it could also decide not to exit. $ cat >test.sh < command exec /dev/null/nonexistent/file || echo oops > EOF $ bash test.sh test.sh: regel 1: /dev/null/nonexistent/file: Not a directory test.sh: regel 1: exec: Kan /dev/null/nonexistent/file niet uitvoeren: Not a directory $ ksh test.sh test.sh[1]: exec: /dev/null/nonexistent/file: cannot execute [Not a directory] $ mksh test.sh test.sh[1]: /dev/null/nonexistent/file: not found $ yash test.sh exec: cannot execute command `/dev/null/nonexistent/file': Not a directory $ zsh -o posixbuiltins test.sh test.sh:1: not a directory: /dev/null/nonexistent/file $ dash test.sh test.sh: 1: exec: /dev/null/nonexistent/file: not found - M.
Re: "command" behaviour
Op 19-10-16 om 16:30 schreef Chet Ramey: > On 10/19/16 10:15 AM, Martijn Dekker wrote: >> The "command" builtin is supposed to disable the "special" properties of >> "special" builtins, which, as far as I can tell, simply means it should >> stop them from exiting the shell on error as they normally do. >> >> In practice this is a bit hit and miss, but it works for many builtins >> on many shells. However, the following: >> >> command exec /dev/null/nonexistent/file || echo oops >> >> instantly exits all the shells I can find, never getting to the 'echo'. > > It doesn't matter whether the `command' prefix is supplied; the description > of `exec' says (here, command refers to the argument to exec): > > "If command is specified, exec shall not return to the shell" Maybe that needs changing then. I can see no good reason why this is mandated, making atomic checking and error handling impossible. In any case I think the full description is multi-interpretable at best: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_20 | If /command/ is specified, exec shall not return to the shell; rather, | the exit status of the process shall be the exit status of the | program implementing /command/, which overlaid the shell. But, in the situation under discussion, there is no such command, because it was not found; therefore it cannot possibly have any exit status, and the shell was not overlaid (as proven by the fact that it produces a diagnostic error message before exiting); so, as far as I can tell, it is logically impossible for this entire sentence to have any relevance to this situation at all. | If command is not found, the exit status shall be 127. If command is | found, but it is not an executable utility, the exit status shall be | 126. If a redirection error occurs (see Consequences of Shell | Errors), the shell shall exit with a value in the range 1-125. | Otherwise, exec shall return a zero exit status. None of this, given the preceding, provides any reason to treat 'exec' differently than any other special builtin. Thanks, - M.
Re: Should aliases expand after 'command'?
Op 24-10-16 om 02:39 schreef David Korn: > In ksh-88 on which the standard was based, the man page says that if an > alias ends in a space then the next word is checked for an alias. This > seems to be omitted in the posix standard without any comment in the > rationale as to why. No, it's there. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_03_01 | If the value of the alias replacing the word ends in a , the | shell shall check the next command word for alias substitution; this | process shall continue until a word is found that is not a valid | alias or an alias value does not end in a . > Anyway in ksh93, command is both a builtin command and an alias for > 'command ' which causes the word after command to be expanded. But why? Thanks, - Martijn
Re: "command" behaviour
Op 28-10-16 om 14:58 schreef Vincent Lefevre: > And after the > exec is handled, the shell is no longer involved, so that 2.8.1 > cannot apply in any case. But that is just the logic I was questioning in my replies to Jörg and Chet. How can the exec be "handled" if the file to exec doesn't exist? How does a shell that is "no longer involved" inform the user that the file doesn't exist before exiting? > This would be a change of the existing behavior and might be a > security issue as code after the exec could be run This is a valid argument. - M.
Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C
Op 01-11-16 om 15:58 schreef Robert Elz: > Depends what you believe "this function" really is - but it certainly is > not avoiding race conditions when two processes are attempting to create > the file at the same time. Says who? > What -C is (or should be) really for is avoiding accidentally writing > on top of (long existing) stable files (particularly ones that are hard > to re-create). That's not what the spec says, though. It simply says: | -C (Uppercase C.) Prevent existing files from being overwritten by | the shell's '>' redirection operator (see Redirecting Output); This says nothing about how long the files are supposed to have existed or how stable they are supposed to be before -C prevents them from being overwritten. As written, the spec inherently requires atomicity. AFAIK, there also is no other POSIX shell mechanism that could create files both atomically and non-destructively. If 'mktemp' were a POSIX standard utility, this wouldn't be an issue, but it's not, so it is. - M.
Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C
Op 02-11-16 om 10:51 schreef Stephane Chazelas: > At the moment, there's no way (that I know) to create a temp file > reliably with POSIX utilities (though it may be possible to create a > temp dir reliably, which is generally a better idea than creating a temp > file in a world writable directory anyway). Hmmm. Interesting. (Just for clarity: below I'm still talking about creating temporary files for any use as in 'mktemp', and *not* about locking.) If both 'mkdir' and 'ln' operate atomically, there could be a safe workaround for creating a regular file directly under /tmp. It would involve creating a (very) temporary directory under /tmp using 'mkdir -m700', then creating the file inside there, setting the mode, etc. with no need for atomicity, then attempting to 'ln' that file back to /tmp until we've got an available name. Do you think this could work? The lack of a good POSIX shell way to create a temporary file has irked me for years. So, as part of my cross-platform POSIX shell library "modernish", I've created a shell implementation of 'mktemp' which is a superset of currently available 'mktemp' implementations. To create regular files, it currently uses the 'set -C'-plus-output-redirection technique discussed earlier. At the time of writing it I was assuming it must be safe, but it isn't, and (given the discussions here) it seems uncertain if it will ever be safe in the future[*]. If the workaround described above is good, I will have to implement it. It also has an -F flag to create a temporary FIFO. Should 'mkfifo' be considered atomic? Thanks, - Martijn [*] I don't get all the objections. Is there any good reason for *not* ensuring that noclobber is atomic? It's not as if modern operating systems make this hard to implement. Why not err on the side of safety?
Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C
Op 02-11-16 om 13:32 schreef Martijn Dekker: > If both 'mkdir' and 'ln' operate atomically, there could be a safe > workaround for creating a regular file directly under /tmp. It would > involve creating a (very) temporary directory under /tmp using 'mkdir > -m700', then creating the file inside there, setting the mode, etc. with > no need for atomicity, then attempting to 'ln' that file back to /tmp > until we've got an available name. Do you think this could work? No one replied to poke holes in this, so I went ahead and implemented this workaround in the modernish shell library implementation of 'mktemp'. <https://github.com/modernish/modernish> Just one thing still worries me a bit. Though 'ln' without the -f option is never supposed to overwrite files, the spec also states: | If the last operand specifies an existing file of a type not | specified by the System Interfaces volume of POSIX.1-2008, the | behavior is implementation-defined. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ln.html Does this mean I cannot actually rely on 'ln' not overwriting a file or otherwise behaving unexpectedly? Thanks, - M.
Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C
Op 07-11-16 om 03:55 schreef Shware Systems:
> To last question, yes, but the effects are supposed to be documented so
> generic guard code that may invoke platform specific pre-ln attempt
> handling can be written. This is a compromise to disqualifying a system
> that defines additional file types from being considered conforming at
> all. In a script this might look like:
> if [ -e /app/$platform/linkhandler ] ;
> then { . .../linkhandler }
> else { do ln directly }; fi
Thanks for the reply. Where can I find more info about this? Is there a
standardised /app directory structure? I don't find it on any actual
system I've access to.
> To some extent it's also the operator's responsibility to sandbox use of
> non-standard file types outside directories the standard says portable
> applications need access to, such as $TMP, to avoid issues.
That makes sense. Many things would break if /tmp does not operate portably.
> A platform
> aware application might create such a file in a $TMP/appsubdir directory
> but shouldn't link it into /tmp after, iow, but to an ~/app/files type
> directory instead. That is more a training issue to me, not something
> the standard can reasonably address or make a requirement.
Unfortunately, by far the most common use case of 'mktemp' is to create
a temporary file in /tmp, so my cross-platform shell implementation of
it will have to be able to do that.
For the time being, unless anyone has concrete evidence or convincing
arguments to the contrary, I will assume that any issues with 'ln'
atomicity into /tmp are theoretical.
- M.
Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C
Op 07-11-16 om 13:45 schreef Stephane Chazelas: > 2016-11-02 13:32:44 +0000, Martijn Dekker: > [...] >> > If both 'mkdir' and 'ln' operate atomically, there could be a safe >> > workaround for creating a regular file directly under /tmp. It would >> > involve creating a (very) temporary directory under /tmp using 'mkdir >> > -m700', then creating the file inside there, setting the mode, etc. with >> > no need for atomicity, then attempting to 'ln' that file back to /tmp >> > until we've got an available name. Do you think this could work? > [...] > > I don't think you can use ln here. > > ln "$tempdir/file" "$tempfile" > > would create a "$tempfile/file" link if "$tempfile" existed and > was of type directory or a symlink eventually resolving to a > directory. ...and it's not as if I didn't know that either. Sorry for generating useless noise here. I let the details blind me to the obvious. > You could use "link" (Unix, not POSIX), or "ln -T" (GNU, not > POSIX) or "mv -Tn" (GNU) instead. "link" seems good. It's on AIX (according to ibm.com), on anything using GNU coreutils, on the Mac, on FreeBSD, NetBSD, and Solaris 13. And I've confirmed for all of these but AIX that it can be invoked by a regular user. But it's not on OpenBSD. Drat. :/ GNU "ln -T" or GNU "mv -Tn" won't help me as "link" and "ln" are both part of the same GNU coreutils package. The current OpenBSD release version has m4 with mkstemp() (thanks for pointing that out earlier, Geoff -- I hadn't discovered that yet), so between those and "link" it should hopefully be covered. I could even fall back on OS implementations of "mktemp" which are also fairly ubiquitous. This is going to need a feature test. Since m4 with mkstemp() is POSIX, it should be preferred. Well, back to work. Thanks, - M.
Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C
Op 07-11-16 om 18:08 schreef Martijn Dekker: > Op 07-11-16 om 13:45 schreef Stephane Chazelas: >> You could use "link" (Unix, not POSIX), or "ln -T" (GNU, not >> POSIX) or "mv -Tn" (GNU) instead. > > "link" seems good. It's on AIX (according to ibm.com), on anything using > GNU coreutils, on the Mac, on FreeBSD, NetBSD, and Solaris 13. And I've > confirmed for all of these but AIX that it can be invoked by a regular user. > > But it's not on OpenBSD. Drat. :/ Also, on Mac OS X and FreeBSD, 'link' turns out to act like 'ln': if a directory or a symlink to it exists, the file is created inside it. This is contrary to what the man page says, by the way. But in any case it means I can't use it. - M.
Clarify ${var-$*}, ${1+$*}, etc. [was: [1003.1(2013)/Issue7+TC1 0000888]: Clarify expansion of '@' and '*' special parameters]
On 05-Nov-2014 at 18:00, Geoff Clare wrote via bug 888 on Austin Group
Bug Tracker:
> to:
>
> The following examples illustrate some of the ways in which '*' and '@'
> can be expanded:
>
> set "abc" "def ghi" "jkl"
[...]
> IFS='' # null
[...]
> unset var
> printf '%s\n' ${var-$*}
> abcdef ghijkl
This example has made it in the published standard by now.
Unfortunately, following discussions on the bug-bash list[*], I believe
this particular example is incorrect. This refers to C165.pdf, section
C.2.5, page 3724, lines 127689-127690.
According to the specification (2.5.2 Special Parameters), given null
IFS and unset 'var', the output of
printf '%s\n' ${var-$*}
should be:
abc
def ghi
jkl
and not
abcdef ghijkl
Reasoning:
First, given an unset 'var', unquoted $* and unquoted ${var-$*} (and
unquoted ${1+$*}, etc) should act identically. If they don't, that's a
bug either way.
Second, the cited example is contrary to the specification, which says:
"Expands to the positional parameters, starting from one, initially
producing one field for each positional parameter that is set. When the
expansion occurs in a context where field splitting will be performed,
any empty fields may be discarded and each of the non-empty fields shall
be further split as described in Field Splitting. [...]"
Well, the expansion occurs "in a context where field splitting will be
performed" because it is unquoted (the fact that IFS happens to be null
is neither here nor there; its value or lack thereof has no bearing on
the lexical context). So the non-empty fields, having been generated,
"shall be further split as described in Field Splitting", which, given
that IFS is null, is a no-op.
In other words, quoted "$*" should join the fields into one, but (given
null IFS) unquoted $* should leave the fields alone altogether.
In other words again, unquoted $* should act identically to unquoted $@,
hence, unquoted ${var-$*}, ${1+$*}, etc. should act identically to
unquoted ${var-$@}, ${1+$@}, etc.
(Note that the case of a default assignment using a parameter
substitution, e.g. ${var=$*} given unset 'var', is different; 2.6.2
Parameter Expansion explicitly states for ${parameter:=[word]} that "in
all cases, the final value of /parameter/ shall be substituted". So the
example in lines 127695-127696 is correct.)
Situation in the wild:
For ${var-$*}, ${1+$*}, etc., every current POSIX shell except bash (as
of 3.x) acts like the specification. Bash 3.x (2007) and up act like the
buggy example (which makes me wonder if the buggy example was
accidentally inspired by bash's behaviour).
Given the test script:
IFS=
set "abc" "def ghi" "jkl"
printf '$*: '
printf '|%s| ' $*
printf ' ${1+$*}: '
printf '|%s| ' ${1+$*}
test results for some current shells are as follows:
bash: $*: |abc| |def ghi| |jkl| ${1+$*}: |abcdef ghijkl|
dash: $*: |abc| |def ghi| |jkl| ${1+$*}: |abc| |def ghi| |jkl|
yash: $*: |abc| |def ghi| |jkl| ${1+$*}: |abc| |def ghi| |jkl|
zsh: $*: |abc| |def ghi| |jkl| ${1+$*}: |abc| |def ghi| |jkl|
ksh93: $*: |abc| |def ghi| |jkl| ${1+$*}: |abc| |def ghi| |jkl|
mksh: $*: |abc| |def ghi| |jkl| ${1+$*}: |abc| |def ghi| |jkl|
oksh: $*: |abc| |def ghi| |jkl| ${1+$*}: |abc| |def ghi| |jkl|
Thanks to Chet Ramey for giving me a heads-up on this on the bug-bash
list. There, he also wrote (on 27-Feb-2017 at 21:03):
> You might also prepare a counter to the argument that at the time the
> $* on the right side of the parameter expansion is expanded, the rules
> in force are detailed in 2.6.2 ("word shall be subjected to tilde
> expansion, parameter expansion, command substitution, and arithmetic
> expansion"), and word splitting isn't listed. I think that came up before.
I've got three counterpoints to that:
1. This is irrelevant, because the bug does not concern splitting
fields, but instead failure to generate fields as specified.
2. Even if it were relevant, in the case under discussion, IFS is null,
so field splitting is a no-op anyway.
3. In the general case, field splitting not being listed there simply
means that field splitting is not performed for the word on the right
side /within/ an expansion such as ${var-$*} (in this case, that word
being $*). If neither the ${var-$*} expansion as a whole nor the /word/
within the expansion (in this case, $*) are quoted, then field splitting
is performed as normal for the expansion as a whole (as is pathname
expansion/globbing), as detailed in 2.6.
Thanks,
- M.
Rethinking pre/post-increment operators [was: [1003.1(2013)/Issue7+TC1 0001128]: Where is the ',' (comma) operator ?]
(0003623) stephane (reporter) - 2017-03-17 10:32
http://austingroupbugs.net/view.php?id=1128#c3623
> That's for instance why the spec says it's unspecified (or implementation
> defined, I don't remember) whether ++ and -- are supported or not, to tell
> applications not to assume $((--a)) is the same as $((- - a)) (as they
> could if they had left that bit out).
The exact wording is that these are "not required" to be supported.
Speaking of which, I don't understand why this is. The
{pre,post}-{in,de}crement operators are a very frequently used feature
of C-style arithmetics, so users expect them. Implementing these surely
is rather trivial.
So it seems like an unnecessary annoyance that you cannot rely on these
in a cross-platform shell script -- particularly since (as Stéphane
points out above) this creates an avoidable ambiguity: $((--a)) and
$((++a)) do not have the same meaning on every shell.
Are there any reasons for that decision, other than "ksh88 didn't
support them", that I may not be aware of?
If ksh88 is the reason, then maybe it's time to revisit this.
The only current POSIX shells I'm aware of that *don't* support them are
Almquist derivatives dash and FreeBSD sh[*]; even Busybox ash added them
(as did ksh88 clone pdksh, back in 1993).
(Dash silently accepts $((--a)) and $((++a)) as equivalent to $((a)), as
did ksh88; FreeBSD sh rejects them.)
- M.
[*] as well as another Almquist derivative, NetBSD sh, but it is far too
obsolete to be considered POSIX in 2017.
Re: Clarify ${var-$*}, ${1+$*}, etc. [was: [1003.1(2013)/Issue7+TC1 0000888]: Clarify expansion of '@' and '*' special parameters]
Op 21-03-17 om 10:42 schreef Geoff Clare:
> Martijn Dekker wrote, on 18 Mar 2017:
>> 1. This is irrelevant, because the bug does not concern splitting
>> fields, but instead failure to generate fields as specified.
>>
>> 2. Even if it were relevant, in the case under discussion, IFS is null,
>> so field splitting is a no-op anyway.
>>
>> 3. In the general case, field splitting not being listed there simply
>> means that field splitting is not performed for the word on the right
>> side /within/ an expansion such as ${var-$*} (in this case, that word
>> being $*). If neither the ${var-$*} expansion as a whole nor the /word/
>> within the expansion (in this case, $*) are quoted, then field splitting
>> is performed as normal for the expansion as a whole (as is pathname
>> expansion/globbing), as detailed in 2.6.
>
> Points 1 and 2 are wrong, but 3 is right and that's the one that matters.
>
> (The reason 1 and 2 are wrong is because of the part of the $* description
> you omitted above: "When the expansion occurs in a context where field
> splitting will not be performed, the initial fields shall be joined to
> form a single field ...")
Sorry to be awkward, but this makes no sense to me and I'd like to
understand the intention of the standard.
Since the expansion is unquoted in this example, how are we in a
"context where field splitting will not be performed"? Aren't we
lexically in a field splitting context, but field splitting happens to
be a no-op because IFS is null?
What am I missing? Am I perhaps misinterpreting the word "context" and
it means some other kind of context than lexical context?
Actually, if I'm wrong in 1 and 2 and you are right, then doesn't the
standard say that "the initial fields shall be joined to form a single
field", so then isn't my whole bug report wrong and the original example
correct?
(But if that's the case, no existing shells would be compliant... not
even bash, which would only be compliant for unquoted ${var-$*}, ${1+$*}
etc. and not for simple unquoted $*.)
> Please could you submit a new Mantis bug to request the change to the
> example output.
Done.
- M.
Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C
Op 29-03-17 om 18:30 schreef shwares...@aol.com: > I agree something along these lines should be added. As worded now the > assumption looks to be more the system is a single user one, or > sandboxed to that effect, not a multi-user server. Yes. > It doesn't even have to be a different user; a separate process that > happens to use the same tmpnam() template for a FIFO creation might > cause a race condition. This is true, but I figured a user who has nefarious processes running under their own account has bigger problems to worry about than -o noclobber. > On some systems stdin and stdout may be implemented as FIFOs, so a > shell could be expected to access them along with regular files in > redirects. I'd have it be process private directories, not simply > user private, How does one go about making these in POSIX? > therefore, or that non-regular files like FIFOs are required to be > atomically unlinked before attempting the regular file creation. I think "atomically" and "before ..." are mutually exclusive. - M.
Capturing standard error using command substitution and redirection
On every POSIX shell I can test[*] except bash, ksh93 and FreeBSD sh, it
is possible to capture a command's output to standard error in a
variable like so:
$ ls_err=$(set +x; ls -l /dev/null/nonexistent /bin/sh 2>&1 1>&3) 3>&1
/bin/sh
$ echo "$ls_err"
ls: /dev/null/nonexistent: Not a directory
On bash and FreeBSD sh, this results in an error:
$ ls_err=$(set +x; ls -l /dev/null/nonexistent /bin/sh 2>&1 1>&3) 3>&1
$ echo "$ls_err"
-bash: 3: Bad file descriptor
On ksh93, standard output is discarded but not standard error:
$ ls_err=$(set +x; ls -l /dev/null/nonexistent /bin/sh 2>&1 1>&3) 3>&1
$ echo "$ls_err"
ls: /dev/null/nonexistent: Not a directory
It does work on both bash, ksh93 and FreeBSD sh when you use curly
braces as a workaround:
$ { ls_err=$(set +x; ls -l /dev/null/nonexistent /bin/sh 2>&1 1>&3); } 3>&1
/bin/sh
$ echo "$ls_err"
ls: /dev/null/nonexistent: Not a directory
Should the necessity of these curly braces be considered a bug?
- M.
[*] Confirmed to work without the curly braces on dash, Busybox ash,
yash, pdksh, mksh/lksh, and zsh
Re: Capturing standard error using command substitution and redirection
Op 03-07-17 om 15:18 schreef Joerg Schilling:
> In our case, it is doubtful whether something like:
>
> var=value
>
> allows IO redirection at all, as there is no command. Note that we recently
> discovered that var=$(cmd) does not set $? because there is no regular
> command.
>
> { var=value; }
>
> converts the pure assignement into something tha can be seen as a construct
> that
> forms a command and thus allows redirection.
That makes sense, thanks. I think I found the exact answer in 2.9.1
"Simple Commands"[*]:
[...]
| 3. Redirections shall be performed as described in Redirection.
|
| 4. Each variable assignment shall be expanded for tilde expansion,
|parameter expansion, command substitution, arithmetic expansion,
|and quote removal prior to assigning the value.
|
| In the preceding list, the order of steps 3 and 4 may be reversed if
| no command name results from step 2 or if the command name matches
| the name of a special built-in utility; see Special Built-In
| Utilities.
There is no command name, so apparently bash, ksh93 and FreeBSD sh
reverse the order of steps 3 and 4, as allowed by the last paragraph
quoted above. Using the braces forces them to evaluate the redirections
in the order I want, as per 2.9.4 Compound Commands.
- M.
[*]
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01
Should "exec" run a shell function?
A test script:
#! /bin/sh
true() {
echo "exec runs function"
}
exec true
On zsh, pdksh and mksh, 'exec' runs the shell function. On all other
POSIX shells (at least bash, dash, yash, ksh93, FreeBSD sh), it runs the
external 'true' utility.
Which behaviour is correct?
On the one hand, exec "shall replace the shell with /command/ without
creating a new process". A shell function is included in the definition
of "command" (ref: 2.9.1 Simple Commands) and so this ought to work with
'exec', in which case every shell except zsh and pdksh/mksh is broken.
On the other hand, can 'exec' legitimately run a shell function from the
current shell (or a builtin, for that matter) when it is supposed to
*replace* the shell with the command in question? Logically, you can't
run something that's part of a shell process that you're supposed to
have replaced.
- M.
Re: Should "exec" run a shell function?
Op 17-07-17 om 12:18 schreef Geoff Clare: > Joerg Schilling wrote, on 17 Jul 2017: >> >> Geoff Clare wrote: >> >>> I thought initially that all uses of "command" should change to "utility", >>> but I think it is common for exec to be used before a pipeline. For this >>> to work correctly, I suspect that exec needs to be a reserved word. >> >> Do you believe the parser should know about "exec"? > > Having played a bit with how exec works in pipelines, it seems my > suspicion was incorrect. We'll need to be careful how we update the > wording, to ensure it doesn't have incorrect implications for the > pipeline case. Looks like your initial thought to use "utility" works. How about builtins, though? Since the utility is expected to overlay the current shell process, can it be said that (exec true) is a POSIXly correct way of guaranteeing the execution of the external bin/true utility rather than the shell builtin? - M.
Re: Should "exec" run a shell function?
Op 20-07-17 om 10:30 schreef Geoff Clare: > I just noticed something not touched on in the previous discussion. > > The EXIT STATUS section for exec says: > > If command is specified, exec shall not return to the shell; > rather, the exit status of the process shall be the exit status of > the program implementing command, which overlaid the shell. > > The use of "program" and "overlaid the shell" here means that the > standard clearly does not allow the execution of built-in utilities and > functions. I had noticed that and that's how I interpreted it as well, but the EXIT STATUS section seems like a strange place to specify that builtins and functions cannot be executed by 'exec'. It looks like an afterthought. So I wasn't sure if that should be considered normative, particularly since - two long-standing shells (pdksh/mksh and zsh) do not act accordingly, even in their POSIX modes; - there seems to be a widespread notion that POSIX does not provide for a way to guarantee the execution of an external command (except by doing a path search manually, as extern() in modernish does). So it's good to have you clarify this; now I can use this thread as a source when I report this as a bug. :) > Of course, we could still choose to treat this as a defect in the > standard. The text does need clarifying, IMHO. - M.
Should 'exec' export assignments to external commands?
Consider something like LC_ALL=C exec ls According to POSIX, should LC_ALL be exported to 'ls'? Relevant here is that 'exec' is a special builtin (unlike 'command'). 2.9.1 Simple Commands seems to suggest[*1] that it is unspecified whether LC_ALL will be exported, but all current POSIX shells do[*2], and the opposite seems very counterintuitive. - M. [*1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01 | * If the command name is a special built-in utility, variable | assignments shall affect the current execution environment. Unless | the set -a option is on (see set), it is unspecified: | - Whether or not the variables gain the export attribute during | the execution of the special built-in utility | - Whether or not export attributes gained as a result of the | variable assignments persist after the completion of the special | built-in utility [*2] bosh (shilytools sh) didn't export assignments preceding 'exec' but recently changed its behaviour to match the other POSIX shells.
FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant
I've been in correspondence with a few people who still seem to believe
that AT&T ksh88 is POSIX compliant because POSIX was originally based on
ksh88. On Solaris, to this day, /usr/xpg4/bin/sh is ksh88 and is
considered "the POSIX compliant shell".
While developing modernish (my cross-platform shell language extension
and feature testing library), I've not only found that POSIX
intentionally deviates from ksh88 in at least three important ways (see
FTL_NOFNOVER, FTL_PARONEARG and FTL_UPP below), but ksh88 also has many
serious bugs, some of which have fatal effects. It is also no longer
being developed, and closed source, so these bugs will never be fixed. I
believe the following shows that ksh88 should not actually be considered
compliant.
/usr/xpg4/bin/sh on Solaris 11.3 has at least the following bugs. The
list is not exhaustive; this is just what I've discovered while
developing modernish. The IDs here are those from modernish's shell
bug/quirk/feature testing framework. FTL_* are considered "fatal" shell
bugs that will normally cause modernish to refuse to initialise.
* FTL_ASGNBIERR: Variable assignments preceding regular builtin commands
should not persist after the command exits, but with this bug they do if
the builtin exits with an error. This may break scripts in obscure ways.
For example, 'LC_ALL=C test a -wrong b' causes the LC_ALL assignment to
persist, so your locale settings are gone. Even more havoc may be
wreaked with a similar temporary assignment to PATH...
* FTL_BRACSQBR: a bracket glob pattern containing a quoted closing
square bracket ']' is never matched, not even if that character is
escaped or passed from a quoted variable. For example,
case ] in [a\]b] ) echo ok;; * ) echo bug;; esac
case a in [a\]b] ) echo ok;; * ) echo bug;; esac
case b in [a\]b] ) echo ok;; * ) echo bug;; esac
all output 'bug'.
* FTL_COMMAND2P: double parsing of parameter expansion when using
'command' with an external command. For example,
a=\$x
x=bug
command printf '%s\n' "$a"
prints 'bug'. Clearly, this makes the 'command' builtin unusable.
* FTL_NOFNOVER: ksh88 does not allow you to override a regular shell
builtins with a shell function. POSIX specifies that only special
builtins can't be overridden with shell functions, and all other shells
allow overriding regular builtins.
* FTL_PARONEARG: When IFS is empty (i.e. field splitting is off), "$@"
is counted as a single argument instead of each positional parameter as
separate arguments. In other words, "$@" does not *generate* fields when
field *splitting* is disabled (which is of course illogical and POSIX
has rightly fixed this).
* FTL_UNSETFAIL: the 'unset' command sets a non-zero exit status if the
variable to unset was not already set, whereas POSIX says: "Unsetting a
variable or function that was not previously set shall not be considered
an error [...]"
This can cause spurious non-zero exit statuses for functions that end in
an 'unset' command to clean up internal variables. (In itself this is
easy to work around; modernish only considers this bug fatal because all
existing shells that have this bug have other fatal errors as well.)
* FTL_UPP: Cannot access "$@" or "$*" if set -u (-o nounset) is active
and there are no positional parameters. If that option is set, ksh88
errors out on accessing "$@" and "$*". This makes 'set -u' impractical
to use, so POSIX changed this from ksh88, rendering "$@" and "$*" exempt
from 'set -u'.
* BUG_ARITHINIT: Using unset or empty variables in arithmetic
expressions causes the shell to error out with a "bad number" error.
Instead, according to POSIX, it should take them as a value of zero.
* BUG_CMDPV: More 'command' builtin breakage: the -p and -v options
cannot be combined, so you can't query the location of the command
within the default system PATH.
* BUG_HDPARQUOT: Quotes within parameter substitutions in Here-Documents
aren't removed. For instance, if 'var' is set, ${var+"x"} in a here-
document erroneously yields "x", not x.
* BUG_PP_08: When IFS is null, unquoted $* within a substitution (e.g.
${1+$*} or ${var-$*}) does not generate one field for each positional
parameter as expected, but instead joins them into a single field.
* BUG_PSUBBKSL: A backslash-escaped character within a quoted parameter
substitution is not unescaped, e.g.
echo "${foo-\x}"
outputs '\x' instead of 'x'.
That's all I have. Maybe some of you know some other ksh88 bugs...
- Martijn
P.S. Not that it's very relevant anymore, but for completeness's sake:
/usr/bin/ksh on Solaris 10.1 (non-sh ksh88, which was replaced by ksh93
on Solaris 11) has the following bugs as well:
* FTL_NOARITH: incomplete and buggy support for POSIX shell arithmetic.
Particularly, octal numbers by leading zero such as $((010 == 8)) are
not supported. There are other bugs as well, like:
x=42
echo $(( (y=x)==0x2A ))
results in "0x2A: bad number", which is clearly wrong, becaus
Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant
Op 29-09-17 om 14:16 schreef Stephane Chazelas:
> 2017-09-29 11:59:52 +0200, Martijn Dekker:
> [...]
>
> Ouch, quite of lot of nasty bugs. Have you reported them to
> Solaris?
No, because I thought ksh88 is no longer being developed. Is that incorrect?
Where/how would I report Solaris bugs?
>> * FTL_BRACSQBR: a bracket glob pattern containing a quoted closing
>> square bracket ']' is never matched, not even if that character is
>> escaped or passed from a quoted variable. For example,
>> case ] in [a\]b] ) echo ok;; * ) echo bug;; esac
>> case a in [a\]b] ) echo ok;; * ) echo bug;; esac
>> case b in [a\]b] ) echo ok;; * ) echo bug;; esac
>> all output 'bug'.
>
> Is that specified by POSIX? I have that notion that "]" can only
> be included in a set if used as the first character ([]ab]).
>
> I understand that quotes remove the special meaning of glob
> operators, but that's not clear to me what should happen if
> you're quoting only part of that [...] operator.
My understanding is that a backslash should remove the special meaning
of the following character, in this case the first ']'. But note that
quoting all of it doesn't work either on ksh88:
case ] in ['a]b'] ) echo ok;; * ) echo bug;; esac
var=a]b; case ] in ["$var"] ) echo ok ;; * ) echo bug ;; esac
both still output "bug".
In fact, so does
var=]ab; case ] in ["$var"] ) echo ok ;; * ) echo bug ;; esac
Even moving ']' to the start of a variable will not make it work.
2010 versions of ksh93 have this bug as well, and they are still seeing
some considerable use. But it's fixed in 2012 versions.
> ksh88's [a\]b] matches on [a]b].
I'm not sure what you mean there.
> older versions of zsh don't match "-" with [a\-b]
Older versions of zsh (< 5.0.7) have lots of other fatal bugs as well.
> ksh93 matches "\" with ["^"c]
Remember the negator in POSIX glob bracket patterns is !, not ^. But
yes, that ksh93 bug has modernish ID BUG_BRACQUOT. Even if a '!' or '^'
is passed in a quoted variable it's still interpreted as a negator. (I
discovered that bug myself some time ago because it requires a
workaround for the trim() function in the var/string modernish module.)
> ksh93 and zsh match "b" with [[:alpha\:]] (mksh matches neither
> "a]" nor "b" nor "[[:alpha:]]")
mksh does not have character classes at all (BUG_NOCHCLASS); this is a
design decision on Thorsten's part.
> [...]
>> * FTL_PARONEARG: When IFS is empty (i.e. field splitting is off), "$@"
>> is counted as a single argument instead of each positional parameter as
>> separate arguments. In other words, "$@" does not *generate* fields when
>> field *splitting* is disabled (which is of course illogical and POSIX
>> has rightly fixed this).
>
> Same bug as in the Bourne shell. I hadn't realised it was also
> in ksh88.
It's in early ksh93 versions as well.
> [...]
>> * BUG_HDPARQUOT: Quotes within parameter substitutions in Here-Documents
>> aren't removed. For instance, if 'var' is set, ${var+"x"} in a here-
>> document erroneously yields "x", not x.
>
> Is that specified?
As far as I know, is not specified that they should act any differently
just because they're in a here-document, so I think by default that
should be considered a bug. (The only other shell I've found that acts
like this is FreeBSD sh, which is otherwise nearly bug-free.)
>> * BUG_PP_08: When IFS is null, unquoted $* within a substitution (e.g.
>> ${1+$*} or ${var-$*}) does not generate one field for each positional
>> parameter as expected, but instead joins them into a single field.
>
> IIRC that was clarified as "unspecified" recently.
I see. Do you (or does anyone) have a link?
>> * BUG_PSUBBKSL: A backslash-escaped character within a quoted parameter
>> substitution is not unescaped, e.g.
>> echo "${foo-\x}"
>> outputs '\x' instead of 'x'.
> [...]
>
> Not sure about that one, there have been extensive discussions
> on similar matter in the past few years.
Hmm. Yeah, this one is probably wrong. Most shells treat \} differently
from other backslash-quoted characters.
unset -v foo; printf '%s ' "${foo-\}}" "${foo-\x}" "${foo-\\}"
bash 3.2, dash:
\} \x
bash 4.4, ksh93, mksh, zsh, FreeBSD sh, dash (Debian patched):
} \x
yash, oksh:
} x
As far as I can tell, POSIX only says this about it:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02
| Any '}' escaped by a or within a quoted string, and
| characters in embedded arithmetic expansions, command substitutions,
| and variable expansions, shall not be examined in determining the
| matching '}'."
I will have to do a search on those discussions. Thanks.
- M.
Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant
Op 30-09-17 om 15:01 schreef Joerg Schilling:
> Stephane Chazelas wrote:
>
>> I knew that was the case for ksh93's functions defined with the
>> function f {...;} syntax (though using static scoping), but
>> hadn't realised ksh88 also did local (dynamic) scoping for
>> options and traps even with the f(){...;} syntax.
>>
>> So it looks like that's another non-compliance.
>>
>> If I read the spec correctly, one should expect
>>
>> noglob() { set -f; }
>> glob() { set +f; }
>>
>> noglob; echo *; glob
>>
>> to output *, but that doesn't work with /usr/xpg4/bin/sh on
>> Solaris.
[...]
> This is an interesting behavior, but it is not related to local/dynamic
> scoping, since this is a test whether a function is run in a sub-shell.
That's easy to disprove. I just did the the following on my Solaris 11.3
VM using /usr/xpg4/bin/sh:
$ noglob() { set -f; g=no; }
$ glob() { set +f; g=yes; }
$ noglob; echo "$- $g"
ism no
$ glob; echo "$- $g"
ism yes
So the shell options don't change outside of the function, but the
variable does. Which means that the function is not run in a subshell,
and that shell options have a mandatory local scope, so /usr/xpg4/bin/sh
functions are not POSIX compliant.
> I am not sure whether this is covered by POSIX at all.
It is. See:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05
This says that part of the function definition is a Compound Command:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04
which says that a compound command like ( some commands here ) is
executed in a subshell whereas one like { some commands here; } is not.
Hence, according to POSIX, a function defined as
fn() ( some commands here )
is executed in a subshell, whereas a function defined as
fn() { some commands here; }
is not.
- Martijn
Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant
Op 01-10-17 om 12:58 schreef Joerg Schilling:
> Martijn Dekker wrote:
>> So the shell options don't change outside of the function, but the
>> variable does. Which means that the function is not run in a subshell,
>> and that shell options have a mandatory local scope, so /usr/xpg4/bin/sh
>> functions are not POSIX compliant.
>>
>>> I am not sure whether this is covered by POSIX at all.
>>
>> It is. See:
>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_05
>
> My impression is that POSIX would need to define that a function has to be
> executed completely inside the environment of the calling shell process.
>
> Such a text is however missing in POSIX.
It is not missing. As previously shown (and you removed from the quoted
text), the POSIX definition of a "function" incorporates the POSIX
definition of a "Compound Command". And that says:
{ compound-list ; }
Execute compound-list in the current process environment.
(see link given previously).
"In the current process environment" seems pretty clear to me.
- M.
Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant
Op 01-10-17 om 14:43 schreef Joerg Schilling:
> The standard does not claim that a function has to be executed the same way
> as
> a compound ist would have been executed, or do you see such a claim in the
> standard?
>
> The way I read the standard just mentions that the function body in a
> function
> definition is parsed like a compund list.
2.9.5 Function Definition Command
[...]
fname ( ) compound-command [io-redirect ...]
[...]
"The argument /compound-command/ represents a compound command, as
described in Compound Commands."
It "represents" a compound command. This does not mean it is only parsed
as a compound command; it means it is to be treated as a compound
command in every way described under "Compound Commands".
2.9.4 Compound Commands
[...]
{ compound-list ; }
Execute compound-list in the current process environment.
This means any compound-command in that form (which therefore includes
any such compound-command that is the body of a function definition), is
executed in the current process environment.
I honestly don't see how your reading of it makes sense.
- Martijn
Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant
Op 01-10-17 om 15:26 schreef Jilles Tjoelker:
> This text clearly defines things like
>
> cat < $(ls -ld -- "$v") `ls -ld -- "$v"` ${w#"*"}
> EOF
>
> the same way
>
> a="$(ls -ld -- "$v") `ls -ld -- "$v"` ${w#"*"}"
>
> are defined.
>
> However, things like "${a-"*"}" are unspecified (in practice, the
> asterisk may either be literal or generate pathnames, depending on the
> shell).
Wow. Indeed.
According to a quick test I just did, the currently maintained shells
that generate pathnames on "${a-"*"}" are FreeBSD sh, AT&T ksh93,
/usr/xpg4/bin/sh (the ksh88 fork on Solaris), and bosh (schily's Bourne
shell fork).
> With
>
> cat < ${a-"*"}
> EOF
>
> the same problem occurs. It is unspecified what the interpretation of
> the characters between the double-quotes is. For example, ksh93 will do
> tilde expansion within a here-document when ${a-"${a-~}"} is used, the
> same way it will do when "${a-"${a-~}"}" is used normally. In other
> cases, special characters within the unquoted part lead to strange
> syntax errors.
This now all makes sense to me.
Here are some more test results.
Test script:
s=1234; unset -v u
cat < Given that and the behaviour of the Heirloom Bourne shell (which also
> keeps the quotes), I think it is valid to keep the quotes.
Yes. It is now clear to me that the modernish BUG_HDPARQUOT bug test is
wrong. My error was considering the quotes in ${foo-"bar"} as being the
same as the quotes in ${foo#"bar"} or ${foo%"bar"}. There is no bug here
(except in Solaris /usr/xpg4/bin/sh).
However, in currently maintained shells, the FreeBSD sh behaviour with
keeping the quotes in ${foo-"bar"} in here-documents appears to be in a
minority shared only with bosh, so it's very easily missed, and it may
bite people writing cross-platform scripts (as it did me).
So I'll rename it to a quirk (QRK_HDPARQUOT) instead of removing it, and
will update the modernish documentation accordingly. A quirk is not to
denote that the behaviour is wrong, just that it's an uncommon behaviour
that cross-shell script authors should be aware of.
- Martijn
ksh93 compliance [was: FYI: ksh88 (/usr/xpg4/bin/sh) ...]
Op 05-10-17 om 18:35 schreef stephane.chaze...@gmail.com:
> Note that I'm not under the impression that ksh93 aimed for
> POSIX compliant. For instance, it did explicitely break POSIX
> compliance (and backward compatibility) when the <> redirection
> operator changed from being short for 0<> to short for 1<> (I
> suppose to be consistent with the new <>; operator (and that one
> more generally wants to use it on stdout than stdin))
Interesting. I wasn't aware of that one -- probably because I've not
ever had a reason to use <>.
Funny how ksh93 deviates on at least this aspect, apparently
intentionally, and seems hyper-compliant in other aspects, like
disabling local variables with 'typeset' in POSIX functions.
This is another thing cross-platform script authors may need to be aware
of, so this is now going into modernish as BUG_REDIRIOO. (Deliberate or
not, it's clearly a bug in terms of POSIX compliance.)
Do you (or does anyone) know of any other aspects where ksh93
deliberately deviates from POSIX? I'm aware of the bugs below, but they
are clearly bugs and not deliberate deviations.
FYI, on the current ksh93 release version, Version AJM 93u+ 2012-08-01,
modernish as of this moment detects (and where necessary works around)
the following bugs:
BUG_BRACQUOT: shell quoting within bracket patterns has no effect (zsh <
5.3; ksh93) This bug means the - retains it special meaning of
'character range', and an initial ! (or ^) retains the meaning of
negation, even in quoted strings within bracket patterns, including
quoted variables.
for var in 'a-c' '!a'; do
case b in
( ["$var"] )echo 'bug' ;;
( [$var] ) echo 'no bug' ;;
esac
done
Output: bug (twice)
[Is this actually a bug, or just a quirk/minority behaviour? ksh93 and
older zsh are the only ones to act this way, but I can't seem to find
the relevant POSIX text now. But in any case, quoting '*' and '?' in
'case' glob patterns always works to disable their behaviour as
wildcards, so to me it smells like a bug for shell quoting to be ignored
in glob bracket patterns but not other glob patterns.]
BUG_CMDSPASGN: preceding a special builtin by 'command' does not stop
preceding invocation-local variable assignments from becoming global.
For instance:
foo=abc command eval "some command using \"\$foo\""
'eval' is a special builtin, and 'command' apparently fails to
completely disable its 'special' properties, so the 'foo' assignment
wrongly persists beyond the command.
This bug appears to be fixed in the unreleased beta.
BUG_FNSUBSH: This is a bad bug and it's been in ksh93 since 1993!
Function definitions within non-forked subshells (including command
substitutions) are silently ignored if a function by the same name
exists in the main shell, so the *wrong* code is executed. unset -f is
also silently ignored. Test script:
foo() { echo WRONG; }
(foo() { echo ok; } && foo && unset -f foo && foo)
Output:
WRONG
WRONG
BUG_IFSISSET: ${IFS+s} always yields 's' even if IFS is unset. This
applies to IFS only. [[ -v IFS ]] does not work either; it always
returns true. So is not possible to test in any normal way if IFS is set
or unset. The workaround I've found is to test field splitting behaviour:
case ${IFS:+n} in # non-empty: it is set
( '' ) set -- 'a b' # empty: test for default field splitting
set -- $1
[ "$#" -eq 1 ] # no field splitting: it is empty and set
esac
(This workaround is automatically incorporated into the modernish
isset() function if it detects this bug, so modernish 'isset IFS' will
always work correctly.)
This bug appears to be fixed in the unreleased beta.
BUG_ISSETLOOP: Expansions like ${var+set} and ${var+:nonempty) remain
static when used within a for, while or until loop; the expansions don't
change along with the state of the variable, so they cannot be used to
check whether a variable is set and/or empty within a loop if the state
of that variable may change in the course of the loop.
unset -v foo
r=
for i in 1 2 3 4; do
case ${foo+s} in
( s ) r=${r}s; unset -v foo ;;
( '' ) r=${r}u; foo= ;;
esac
done
printf '%s\n' "$r"
Output: (expected: usus)
It is an effective workaround to use a shell function (such as modernish
isset()) that tests if the variable is set, even if that function uses
the same ${foo+s} sort of parameter expansion.
BUG_KUNSETIFS: Can't unset IFS under very specific circumstances. "unset
-v IFS" is a known POSIX shell idiom to activate default field
splitting. With this bug, the unset builtin silently fails to unset IFS
(i.e. fails to activate field splitting) if we're executing an eval or a
trap and a number of specific conditions are met. See BUG_KUNSETIFS.t
for more information.
https://github.com/modernish/modernish/blob/maste
Re: ksh93 compliance [was: FYI: ksh88 (/usr/xpg4/bin/sh) ...]
Op 06-10-17 om 15:26 schreef Joerg Schilling:
> Martijn Dekker wrote:
>
>> Funny how ksh93 deviates on at least this aspect, apparently
>> intentionally, and seems hyper-compliant in other aspects, like
>> disabling local variables with 'typeset' in POSIX functions.
>
> The fact that typeset does not work in POSIX function to create local
> variables
> does not look like something that causes POSIX compliance.
Agreed.
>> Do you (or does anyone) know of any other aspects where ksh93
>> deliberately deviates from POSIX? I'm aware of the bugs below, but they
>> are clearly bugs and not deliberate deviations.
>
> I discovered aprox. 10 deviations, but I am sorry that I did not write them
> down, so I currently only remember this one:
>
> the builtin command "times" does not create POSIX compliant output.
Ah yes, I had forgotten about that. In fact, ksh93 does not have a
'times' builtin at all -- it defaults to an alias:
$ type times
times is an alias for '{ { time;} 2>&1;}'
Using the 'time' command without an argument is an extension to POSIX so
of course the format can be whatever. But aliasing 'times' to this by
default does create a non-compliant situation.
I'm not sure if this actually affects any real-world scripts. Has anyone
ever heard of a script that parses the output of 'times'?
>> BUG_FNSUBSH: This is a bad bug and it's been in ksh93 since 1993!
>> Function definitions within non-forked subshells (including command
>> substitutions) are silently ignored if a function by the same name
>> exists in the main shell, so the *wrong* code is executed. unset -f is
>> also silently ignored. Test script:
>> foo() { echo WRONG; }
>> (foo() { echo ok; } && foo && unset -f foo && foo)
>> Output:
>> WRONG
>> WRONG
>
> This also works as expected in ksh88.
That's not surprising because ksh88 forks its subshells the classical
way; this is ostensibly a bug introduced along with non-forking
subshells in ksh93 (as evidenced by the fact that forcing the subshell
to be forked is an effective workaround).
By the way, I currently know of two ways to force a subshell to be
forked in ksh93: one is pretty obvious, turn it into a backgroud job:
([[ $(ksh -c 'echo $PPID') != $$ ]] && echo forked) & wait "$!"
The other one only works with a command substitution subshell. If you
add a dummy command that redirects (or even closes) standard output, the
command substitution subshell will be forked.
echo $(: 1>&-; [[ $(ksh -c 'echo $PPID') != $$ ]] && echo forked)
I discovered this accidentally and I am not sure if the behaviour is
intentional, but I suppose it sort of makes sense: ksh needs to be sure
it can capture stdout from the command substitution and I figure the
problem was found too hard to solve for a non-forking subshell.
Hmm. Playing with this some more, I'm finding it's actually smart enough
to fork a subshell mid-execution:
echo $([[ $(ksh -c 'echo $PPID') != $$ ]] && echo forked1; \
: 1>&-; [[ $(ksh -c 'echo $PPID') != $$ ]] && echo forked2)
outputs just "forked2". This certainly looks intentional.
>> BUG_TESTERR1A: test/[ exits with a non-error 'false' status (1) if an
>> invalid argument is given to an operator, instead of a status > 1 as
>> required by POSIX.
>
> Do you have an example?
>
> "test 1 -gt aa" results in $? == 0 with ksh93
That's because 'test' arguments in ksh93 are not integers but arithmetic
expressions, and 'aa' is a valid arithmetic expression which defaults to
0 if the variable 'aa' is unset or empty. Try:
test 1 -gt 1gg
Result is a 'ksh: test: 1gg: arithmetic syntax error' and $? == 1.
- M.
Re: FYI: ksh88 (/usr/xpg4/bin/sh) is not actually POSIX compliant
Op 30-09-17 om 17:35 schreef Alan Coopersmith: > It is true that ksh88 is no longer being developed, but the Solaris > /usr/xpg4/bin/sh is a fork of ksh88, modified for standards compliance, > that is still being maintained, and bugs fixed as necessary. That's good to know! >> Where/how would I report Solaris bugs? > > Customers with support contracts can report bugs via Oracle Support. I was afraid it might be something like that. I'm no corporate customer, I'm just a slightly obsessed guy with a weird love for that quirky shell language, trying to prove the idea that a good solid shell library could (a) really enhance the language, as well as get rid of some of its main annoyances, and (b) make cross-platform scripting downright feasible. https://github.com/modernish/modernish Shell bugs are a major obstacle to that goal. As part of that, I've been on a bit of a crusade to get all the POSIX-related bugs fixed in all the shells. My library experiments with things in shell that I don't believe have really been tried before, so I keep running into all sorts of bugs in various shells. Some of those bugs went unreported for decades. All of which is to say, I'm not about to take out a support contract just to report bugs, but I think it might still be in your interest to get my bug reports for /usr/xpg4/bin/sh. > The people responsible for running the standards conformance test suites > on Solaris are also on this mailing list, and can file bugs if they see > issues we need to fix. I hope one of them can help by verifying and filing the bugs that I posted to this list. Is there anyone in particular I could cc in hopes of getting it noticed? Thanks, - Martijn
Re: ksh93 compliance [was: FYI: ksh88 (/usr/xpg4/bin/sh) ...]
Op 06-10-17 om 18:05 schreef Joerg Schilling:
> Martijn Dekker wrote:
[...]
>> That's not surprising because ksh88 forks its subshells the classical
>> way; this is ostensibly a bug introduced along with non-forking
>> subshells in ksh93 (as evidenced by the fact that forcing the subshell
>> to be forked is an effective workaround).
>
> I cannot see such a correlation.
My tests show it, though. If you force a subshell to fork, suddenly you
can unset or change a function set in the main shell.
With a non-forking subshell:
foo() { echo WRONG; }
(foo() { echo ok; } && foo && unset -f foo && foo)
Output:
WRONG
WRONG
With a forking subshell:
foo() { echo WRONG; }
(foo() { echo ok; } && foo && unset -f foo && foo) & wait
Output:
ok
ksh: foo: not found
or, to use the trick with the output redirection in a command
substitution to force the subshell to fork:
foo() { echo WRONG; }
: $(: 1>&-; foo() { echo ok 1>&2; } && foo && unset -f foo && foo)
Output:
ok
ksh: foo: not found
> Non-forking subshells are a feature mainly introduced to permit a native port
> to Win-DOS, where fork() is unknown. Note that David created "uwin" which is
> much faster than Cygwin because of this trick.
Interesting. I've always seen it described as primarily a performance
enhancement.
uwin is good to know about, thanks.
- M.
Testing if two files are on the same file system
Is there a way, using POSIX shell and utilities, to reliably test if two files are on the same file system? It seems like a basic feature that the shell should have access to, so I'd like to add it to the modernish shell library. The only POSIX command I've found so far that identifies the file system that a file is on, is 'df -P', so I've experimented with parsing its output. Given two file arguments, it normally outputs their file system names in the first field of the second and third lines. The fields are separated by one or more spaces. Unfortunately, on the Mac, some of the file system names contain spaces (on my system: "map -hosts" and "map auto_home"). POSIX doesn't seem to prohibit this: file system names are considered implementation-defined[*]. Given that there is a fixed number of fields, it would be possible to cope with this if no other fields can contain spaces, but the last field is a directory name that may also contain spaces -- or even newlines. So the output of 'df -P' is unparseable. GNU stat (or gstat) makes this trivial: 'stat --file-system --format=%i /file/one /file/two' provides two hexadecimal file system identifiers to compare. But GNU stat is not standard and nowhere near ubiquitous. BSD 'stat' does not have any such feature. Is there anything else POSIX or de-facto standard that could be (ab)used to determine if two files are on the same file system? Thanks, - M. [*] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/df.html#tag_20_33_10
Re: Testing if two files are on the same file system
Op 18-10-17 om 16:11 schreef Geoff Clare:
> After the filesystem name there are four numeric fields with a trailing '%'
> on the fourth. Treating this as the terminator for the filesystem name
> ought to be good enough in practice. It would only not work in the
> extremely unlikely event that a filesystem name has within it four numeric
> fields with a trailing '%' on the fourth, or if two different filesystem
> names are the same when trailing blanks are removed.
>
> I tried it with the following command and it seems to work:
>
> df -P file1 file2 |
> sed '1d;s/\([[:blank:]]\{1,\}[[:digit:]]\{1,\}\)\{4\}%[[:blank:]].*//'
That's very clever, and does seem to be fairly bullet-proof. Thanks!
There is another, less unlikely problem, though: a file system could be
mounted on a directory with a name containing a newline, which would
break line-based parsing. These days, with things like FUSE, you don't
even need to be root to perform a mount. So a cleverly crafted mount
could conceivably be abused to manipulate the behaviour of a script
using this technique.
But modernish can at least detect this condition and reduce it to a
simple denial of service: if the parsed 'df -P' output doesn't contain
exactly two lines, things have gone pear-shaped and the program should
halt immediately[*] to avoid potential disaster.
I'll also have to make it detect GNU 'stat' and use 'gstat --file-system
--format=%i' instead where possible, because that's the only properly
robust way I know of and it's fairly widespread.
- Martijn
[*]
https://github.com/modernish/modernish#user-content-reliable-emergency-halt
Re: Testing if two files are on the same file system
Op 19-10-17 om 13:12 schreef Joerg Schilling: > No, the first field usually is the background storage, it may be the filesytem > name in some cases. Whatever it technically may be, the POSIX spec calls it a file system name, so I'm using that term to be consistent with it. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/df.html#tag_20_33_10 > You need to look at the 2nd line and you may use the '%' sign to synchronize > fields and you may even do this: > > /usr/xpg4/bin/df -P /etc/passwd /etc/hosts > Dateisystem 512-Blöcke Belegt Verfügbar Kapazität Eingehängt > auf > /dev/dsk/c1t0d0s3 2420396023115442 84648097%/ > /dev/dsk/c1t0d0s3 2420396023115442 84648097%/ > > and just compare the last two lines. What if this is done while something else is writing to a file? As far as I can tell, there's nothing to say the number of free blocks can't change between printing the first and second lines. As for the '%' sign method, see Geoff's message and my reply to it. - M.
Re: Testing if two files are on the same file system
Op 19-10-17 om 15:14 schreef Joerg Wunsch:
> As Martijn Dekker wrote:
>
>> There is another, less unlikely problem, though: a file system could be
>> mounted on a directory with a name containing a newline, which would
>> break line-based parsing.
>
> Pipe the output of df -P through "tail -n +2 | head -1". :-)
>
> That way, you don't have to care whether the mountpoint adds rubbish
> after a newline. As I understand it, you are going to apply df -P to
> a single file at a time only, so it's sure its filesystem name will
> always at the beginning of line 2.
Well no, the idea was to apply 'df -P' to two files at once. Once you've
got two lines in a variable, it's fast and easy to use POSIX parameter
substitutions to split and compare them.
You're right, I could avoid the whole problem by using two separate 'df
-P' invocations and parsing them separately. But that would double the
overhead. This feature is especially useful in directory traversal
loops, so performance is important.
An actual case of a file system being mounted on a directory with a name
containing a newline is a sure sign of something dodgy going on, so I
don't really mind killing a modernish program outright if that condition
is detected.
To give people a more concerete idea of what I'm talking about, below is
my function that now performs this feature on systems without GNU
'stat'. It's an internal function, meant to be invoked indirectly via
'is onsamefs file1 file2' (don't follow symlinks) or 'is -L onsamefs
file1 file2' (follow symlinks).
The bulk of the overhead (i.e.: the subshell forks) is in the command
substitution at the beginning. This function performs the feature using
just two or three forks, depending on the shell.
Notes:
- $CCn is a newline;
- DEFPATH=$(getconf PATH);
- die() performs a reliable emergency program halt on scripts (the '||
return' is only ever reached on an interactive shell that traps and
ignores SIGINT).
_Msh_testOnSameFs() {
_Msh_is=$(export LC_ALL=C POSIXLY_CORRECT=Y "PATH=$DEFPATH"
unset -f df sed # QRK_EXECFNBI compat
exec df -P -- "$1" "$2" | exec sed "1d;
s/\([[:blank:]]\{1,\}[[:digit:]]\{1,\}\)\{4\}%[[:blank:]].*//")
# Sanity check: verify that exactly 2 lines were produced.
case ${_Msh_is} in
(*"$CCn"*"$CCn"*) die "is onsamefs: internal error 1" || return ;;
(*"$CCn"*) ;;
(*) die "is onsamefs: internal error 2" || return ;;
esac
# If the two lines are identical, the files are on the same filesystem.
case ${_Msh_is#*"$CCn"} in
("${_Msh_is%%"$CCn"*}") unset -v _Msh_is ;;
(*) ! unset -v _Msh_is ;;
esac
}
The more robust and faster version using GNU 'stat' (just one fork) is
identical, except for the command substitution which looks like:
_Msh_is=$(export LC_ALL=C POSIXLY_CORRECT=Y
exec /opt/local/bin/gstat --file-system --format=%i,%t -- "$1" "$2")
(The presence of GNU 'stat' and its specific path are of course detected
at init time.)
I'll push the new feature to github tonight after some more testing, so
people can try to break it. :)
- Martijn
Re: Testing if two files are on the same file system
Op 19-10-17 om 15:06 schreef Martijn Dekker:
> Op 18-10-17 om 16:11 schreef Geoff Clare:
>> After the filesystem name there are four numeric fields with a trailing '%'
>> on the fourth. Treating this as the terminator for the filesystem name
>> ought to be good enough in practice. It would only not work in the
>> extremely unlikely event that a filesystem name has within it four numeric
>> fields with a trailing '%' on the fourth, or if two different filesystem
>> names are the same when trailing blanks are removed.
>>
>> I tried it with the following command and it seems to work:
>>
>> df -P file1 file2 |
>> sed '1d;s/\([[:blank:]]\{1,\}[[:digit:]]\{1,\}\)\{4\}%[[:blank:]].*//'
>
> That's very clever, and does seem to be fairly bullet-proof. Thanks!
Hmm... on Linux, there can be several tmpfs mounts and they all have the
same file system name in the first column. Example:
tmpfs 33039212 0 33039212 0% /dev/shm
tmpfs 33039212 3277180 29762032 10% /run
tmpfs 5120 0 5120 0% /run/lock
tmpfs 33039212 0 33039212 0% /sys/fs/cgroup
Looks like we also need the mount point to uniquely identify a file
system. A little tweak to the sed incantation accomplishes that:
df -P /dev/shm /run |
sed '1d;
s/\([[:blank:]]\{1,\}[[:digit:]]\{1,\}\)\{4\}%[[:blank:]]\{1,\}/,/'
gives
tmpfs,/dev/shm
tmpfs,/run
But then, wouldn't it be sufficient simply to use the mount point only?
Can two file systems ever be on the same mount point?
- Martijn
Re: Testing if two files are on the same file system
Op 23-10-17 om 17:36 schreef Vincent Lefevre: > The initial question was actually not clear. First, you should define > what a file system is. If this is not what is identified by st_dev[*], > what is it? That's a good point. What I mean is an instance of a file system on a particular device or partition, mounted on a particular mount point. A concrete working definition is: if I can create a hard link from a file on one directory into another directory, those directories are on the same file system. I'd like to be able to do something like: if is onsamefs "$file1" "$file2"; then ln "$file2" "$file1"# hard links are possible else ln -s "$file2" "$file1" fi I found that, on Linux, when a file system is mounted twice on different mount points using mount --bind, 'ln' will refuse to create a hard link between the different mount points, even though they are on the same device (with the same device ID). So including the mount point in the definition turns out to be essential. It looks like neither GNU 'stat' nor BSD 'stat' will allow this. (I know in the concrete example above I could just do ln "$file2" "$file1" 2>/dev/null || ln -s "$file2" "$file1" but there are other use cases. Besides, doing a proper test first is actually better, as you can distinguish between failure modes.) Thanks, - M.
Re: Testing if two files are on the same file system
Op 24-10-17 om 11:22 schreef Stephane Chazelas: > using sed on the output of df > # > > the output "df -P" is only specified in the POSIX locale. That's now how I read the spec at http://pubs.opengroup.org/onlinepubs/9699919799/utilities/df.html#tag_20_33_10 It says the first line (the header) is only specified in the POSIX loicale, but it makes no such statement about any subsequent lines. We're ignoring the first line, so no problem. > I'm not sure what the output of "LC_ALL=C df -P" should be for a > mount point like a UTF-8 /home/stéphane That's a good point. I had actually added LC_ALL=C to my 'df' invocation, more out of habit than anything. I should remove it. > symlinks and device files > # > > Depending on the stat implementation, for symlinks, you'll get > the st_dev of the symlink or the file it points to, while > AFAICT, df gives you the information for the target of the > symlink In modernish I deal with that: - for 'is onsamefs', by stripping the last element from each given path if it's not a directory; - for 'is -L onsamefs' (-L = follow symlinks), by stripping the last element from each given path if it's not a directory or a non-broken symlink. An extra complication for 'is -L onsamefs' is that 'df -P' on Mac and BSD does not like device files, e.g. 'df -P /dev/null' gives an error: "Raw devices not supported". Is that situation POSIX compliant? I suspect not, but in any case I have to deal with it. It's fine for 'is onsamefs /dev/null /foo/bar' as the final /null is stripped anyway, but for 'is -L on samefs symlink_to_dev_null /foo/bar', a valid symlink to a device still gives the error. The only way to deal with that is to read the the symlink's target, then strip the last element off the target. Reading the target of a symlink using POSIX shell and utilities is worth an entirely new thread (unfortunately 'readlink' is not universal and implementations differ in behaviour), but I am pretty sure I found a robust way... - Martijn
Symlink behaviour [was; Testing if two files are on the same file system]
Op 25-10-17 om 14:40 schreef Stephane Chazelas: > The behaviour of df on symlinks is unspecified AFAICT Hmmm... Base Definitions, 3.381 Symbolic Link: | A type of file with the property that when the file is encountered | during pathname resolution, a string stored by the file is used to | modify the pathname resolution. Which references Base Definitions, 4.13 Pathname Resolution: | If a symbolic link is encountered during pathname resolution, the | behavior shall depend on whether the pathname component is at the end | of the pathname and on the function being performed. If all of the | following are true, then pathname resolution is complete: | 1. This is the last pathname component of the pathname. | 2. The pathname has no trailing . | 3. The function is required to act on the symbolic link itself, or |certain arguments direct that the function act on the symbolic |link itself. | In all other cases, the system shall prefix the remaining pathname, | if any, with the contents of the symbolic link, except that if the | contents of the symbolic link is the empty string, then either | pathname resolution shall fail with functions reporting an [ENOENT] | error and utilities writing an equivalent diagnostic message, or | the pathname of the directory containing the symbolic link shall be | used in place of the contents of the symbolic link. [...] I think this means 'df' and all other utilities (and even basic libc functions like fopen()) are required to follow symlinks given as arguments, unless explicitly specified otherwise either by the spec or (if a relevant option exists) by the user/programmer. - Martijn
Preserving standard input for background commands in scripts
2.9.3 List states under "Asynchronous Lists":
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_03_02
| If job control is disabled (see set, -m), the standard input for an
| asynchronous list, before any explicit redirections are performed,
| shall be considered to be assigned to a file that has the same
| properties as /dev/null. This shall not happen if job control is
| enabled. In all cases, explicit redirection of standard input shall
| override this activity.
Question: should the last sentence quoted above also apply if standard
input is explicitly redirected to itself? E.g., when running the
following with job control disabled (e.g. as a script) and standard
input on a terminal:
if [ -t 0 ]; then echo yes; else echo no; fi <&0 & wait
then bash and yash outout "yes", whereas bosh, (d)ash, ksh93,
pdksh/mksh/lksh, zsh, and Solaris /usr/xpg4/bin/sh all output "no".
Which is right? The way I read the text above, it seems like bash and
yash are, but they are in the minority. Does this need clarification?
If this worked more widely, it would have been a convenient way to
preserve standard input for a background job, which is useful for
commands like stty.
It does work on all shells to redirect standard input to a copy of
itself, like
{ if [ -t 0 ]; then echo yes; else echo no; fi <&3 & } 3<&0; wait
but that comes at the cost of using another file descriptor for the copy.
- M.
Re: UTF-8 locale & POSIX text model
Op 22-11-17 om 13:58 schreef Danny Niu: > Q1: What is the rationale for not making POSIX an application of ASCII? Actually, it mostly is. POSIX mandates that all supported locales include the "portable character set", which is ASCII minus some control characters. http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html - M.
Re: UTF-8 locale & POSIX text model
Op 22-11-17 om 16:02 schreef Geoff Clare: > Danny Niu wrote, on 22 Nov 2017: >> >> Q1: What is the rationale for not making POSIX an application of ASCII? > > So that systems which use other encodings (specifically EBCDIC) can > be POSIX-conforming. IBM z/OS is certified UNIX 95 and uses EBCDIC. Well, it looks like my previous reply was completely clueless, sorry about that. But then how should I interpret the table in 6.1 Portable Character Set, particularly the UCS column? - Martijn
Should shell quoting within glob bracket patterns be effective?
Re: https://github.com/att/ast/issues/71 Consider this test script: (set -o posix) 2>/dev/null && set -o posix emulate sh 2>/dev/null # for zsh for var in 'a-c' '!a'; do case b in ( ["$var"] ) echo 'quirk' ;; ( [$var] ) echo 'no quirk' ;; esac done Most shells output 'no quirk' for both values of 'var', but AT&T ksh93 outputs 'quirk' for both, as does zsh 5.2 and earlier (zsh-as-sh changed to match the majority in 5.3). Now one of the current ksh93 lead developers says this does not look like a bug. Does POSIX specify anything, either way, regarding the effect of shell quoting within glob bracket patterns? I can't find any relevant text under "2.13 Pattern Matching Notation" or anything it references, so clarification would be appreciated. Thanks, - Martijn
Re: Should shell quoting within glob bracket patterns be effective?
Op 10-04-18 om 21:06 schreef Robert Elz: Date:Tue, 10 Apr 2018 13:41:25 +0100 From:Martijn Dekker Message-ID: | Does POSIX specify anything, either way, regarding the effect of shell | quoting within glob bracket patterns? I would say it is unclear - in general, quoting inside [] does not work (XCU 2.13 char classes are derived from XBD 9.3.5 char classes, and in the latter, quote characters are just characters ["] is a char class containing just a double quote character. However: $ ksh93 -c 'case \" in ["a-z"]) echo match;; *) echo no match;; esac' no match The quotes are not considered part of the bracket expression, but removed by the shell, even on ksh93. Also, 2.13.1 does say: When pattern matching is used where shell quote removal is not performed [...] [this includes case patterns as quote removal is not performed on them] But it is. POSIX explicitly specifies quote removal for 'case' patterns: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04_05 | The conditional construct case shall execute the /compound-list/ | corresponding to the first one of several /patterns/ (see Pattern | Matching Notation) that is matched by the string resulting from the | tilde expansion, parameter expansion, command substitution, | arithmetic expansion, and quote removal of the given word. The NetBSD sh however produces "no quirk" for both - I hope you won't change it to ksh93's counterintuitive behaviour. Your current behaviour is certainly consistent with POSIX (as well as every other current shell except ksh93). - M.
Re: Should shell quoting within glob bracket patterns be effective?
Op 10-04-18 om 21:52 schreef Robert Elz: No, it doesn't. Read that again, with the emphasis I am adding ... |http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04_05 | | The conditional construct case shall execute the/compound-list/ | | corresponding to the first one of several/patterns/ (see Pattern | | Matching Notation) that is matched by the string resulting from the | | tilde expansion, parameter expansion, command substitution, | | arithmetic expansion, and quote removal **of the given word**. That part is talking about the "case $x in" $x is the "given word", that is certainly subject to quote removal. Quite right, I stand corrected. Thanks, - M.
Re: Should shell quoting within glob bracket patterns be effective?
Op 10-04-18 om 15:59 schreef Joerg Schilling: Whom do you call "current ksh93 lead developers"? As far as I can tell from what's going on at the github repo, Siteshwar Vashisht and Kurtis Rader currently appear to be in charge of its development. - M.
Re: Should shell quoting within glob bracket patterns be effective?
Op 10-04-18 om 22:50 schreef Jilles Tjoelker: I prefer "no quirk" twice as output but it is indeed not fully specified. I agree with your preference. Ignoring shell quoting in glob bracket patterns means removing a useful feature: the ability to pass an arbitrary string of characters in a parameter, one of which is to be matched. OTOH, honouring shell quoting in glob bracket patterns does not remove any functionality, as you can simply not quote the expansion (which is safe, as it is not subject to split or glob in that context). So if this is indeed not specified, I think the standard ought to be amended to specify the current majority behaviour (everything but ksh93). - Martijn
Re: Fall-back string matching in "case"
Op 17-04-18 om 16:02 schreef Geoff Clare: (0003970) joerg (reporter) - 2018-04-17 13:46 In the Bourne Shell and in ksh88, the case statement first tries to match the case argument against the pattern using gmatch() and in case this does not result in a match, the quoting on the pattern is removed and a plain string compare is done. [...] Personally I am very strongly opposed to allowing this behaviour in the standard. It breaks applications which perform input validation such as: case $input in [0-9]) ;; *) echo "Input must be a single digit"; exit ;; esac This code is supposed only to allow the script to proceed when $input is a single digit, but with fall-back string matching it would proceed with input='[0-9]'. This behaviour is, frankly, utterly broken and I'm astonished that any shell author would think it was a good idea. Wow... $ case "[0-9]" in [0-9]) echo BUG;; *) echo ok;; esac bash: ok bosh: BUG (including bosh -o posix) Busybox ash: ok dash: ok FreeBSD sh: ok ksh93: BUG NetBSD sh: ok mksh: ok pdksh: ok yash: ok zsh: ok Even ksh93 exhibits this breakage to this day, up to including current git. That is certainly something to be aware of. I'll file a bug report at github.com/att/ast. As for bosh, I understand Jörg intends to preserve historical Bourne shell behaviour by default, but this should certainly be fixed for the POSIX mode. - M.
Re: Fall-back string matching in "case"
Op 17-04-18 om 17:16 schreef Joerg Schilling: Martijn Dekker wrote: Wow... $ case "[0-9]" in [0-9]) echo BUG;; *) echo ok;; esac bash: ok bosh: BUG (including bosh -o posix) /usr/xpg4/bin/sh $ case "[0-9]" in [0-9]) echo BUG;; *) echo ok;; esac BUG So bosh behaves just like the POSIX shell. Yes, I also used to think that ksh88 is "the POSIX shell" on Solaris, until I found a load of bugs in it, and was corrected here on my misconception that ksh88 is still considered a reference implementation. It is not, at least not any more. And as I've shown here before, it still has enough bugs in it that it should really not be considered POSIX compliant at all. The behaviour in question: 1. is undocumented; 2. is not specified by the text of the standard, in fact directly contravenes it; 3. breaks legit and important use cases (as demonstrated by Geoff); 4. requires an ugly workaround that nobody would expect to have to use (see 1 and 2); 5. has 0 real uses that wouldn't be better served by simply quoting the pattern; 6. of current shells, is only exhibited by a minority of bosh (how many regular users?), ksh88 (basically dead and buried) and ksh93. These all seem like good reasons for fixing it to me. - M.
What exit status for an executable that is found but cannot be invoked?
2.8.2 "Exit Status for Commands" specifies: If a command is not found, the exit status shall be 127. If the command name is found, but it is not an executable utility, the exit status shall be 126. This does not say what should happen if the command is found and is an executable utility, but the utility cannot be invoked because of some error or other. Most shells return 126 in such cases, as I would expect, but zsh returns 126 only for EACCES (permission denied) and ENOEXEC (exec format error), returning 127 for other error conditions such as E2BIG (argument list too long). I was going to report this to the zsh developers as a bug, but then found that POSIX doesn't actually specify the exit status in such cases. Seems like an oversight to me. Agreed? - Martijn
Re: What exit status for an executable that is found but cannot be invoked?
Op 20-04-18 om 00:34 schreef Shware Systems: Whatever the reason for the load failure, at that point in time it is "not an executable utility", in other words. That is an interesting interpretation of the words "executable utility". I do not believe that most people would read it this way. Also consider error conditions such as E2BIG. If the utility is found and the argument list is too long, that does not make the utility non-executable even by your definition. There is no load failure; loading the file is never attempted to begin with, because there is not enough space to pass it the list of arguments. POSIX does not define the term "executable utility" exactly but it does define "executable file" under 3.154: | A regular file acceptable as a new process image file by the | equivalent of the exec family of functions, and thus usable as one | form of a utility. This defines "executable" in terms of the file's acceptability to 'exec' and friends, which I take to mean properties such as its format and permissions. It says nothing about errors invoking the file that have nothing to do with the nature of the file, e.g. I/O error, argument list too long, etc. -- so it seems to me that those error conditions do not influence the meaning of "executable". - M.
Re: What exit status for an executable that is found but cannot be invoked?
Op 20-04-18 om 01:29 schreef Don Cragun: On Apr 19, 2018, at 4:04 PM, Martijn Dekker wrote: Op 20-04-18 om 00:34 schreef Shware Systems: Whatever the reason for the load failure, at that point in time it is "not an executable utility", in other words. That is an interesting interpretation of the words "executable utility". I do not believe that most people would read it this way. Interpret them as specified in section 2.9.1.1 of the standard (titled Command Search and Execution in the Shell Command Language part of the Shell and Utilities volume of the standard). It goes into explicit detail about when 126 and 127 are used as exit code when the shell tries to run a utility. If you have the PDF version of the 2016 or 2017 version of the standard, this section can be found on pages 2367-2368 lines 75546-75627. Thanks for the pointer. That section actually seems to take great pains to avoid specifying the status that should be returned in case of error conditions such as "argument list too long" or "I/O error". It merely specifies that the shell *may* return 126 if the file format is neither valid executable nor text (lines 75586-75594, 75613-75617) and *shall* return 127 if the utility cannot be found (75606-75607). Also consider error conditions such as E2BIG. If the utility is found and the argument list is too long, that does not make the utility non-executable even by your definition. There is no load failure; loading the file is never attempted to begin with, because there is not enough space to pass it the list of arguments. It is strange that you believe that. If an attempt to execute the utility fails, that seems to me to be a clear indication that it is not executable (at least with the environment and arguments that were passed to it on this invocation). The environment is not a property of the file. It is illogical to say that a file is non-executable when the attempt to execute it fails due to an error condition that is unrelated to the file. Why would you believe that the exit status should be 127 indicating that the file not found if the file was found, but the exec failed? To be clear, I do not believe that at all. In fact, if the standard says anything clearly about this, it's that 127 is only to be used if the utility was not found. My point is that, though it seems quite obvious that 126 should be used instead, the text of the standard does not actually specify this. It merely specifies 126 if the file is not an executable utility. It does not specify the exit status to use if the file *is* an executable utility but the attempt to exec it fails for some reason unrelated to the nature of the file. In my view, that is a defect in the standard. One meaning of executable is that the file is in a format that can be executed by the underlying system. Another meaning is that the file and the needed resources to run it are available at the time an attempt was made to execute it. As with the environment, the resources needed to run a file are also not a property of the file, so it is equally illogical to say that a file is non-executable if there are insufficient resources to execute it. If such an illogical definition of "executable file" is to be used, at the very least that should be stated very clearly in a section such as XBD 3.154, but it is not. In fact, the definition there is quite logical. According to the standard, both of these meanings are supposed to result in exit code 126. Unfortunately, neither XCU 2.8.2 nor XCU 2.9.1.1 actually say this. Not only that, XCU 2.9.1.1 seems to bend over backwards to avoid saying it. Thanks, - M.
Re: What exit status for an executable that is found but cannot be invoked?
Op 20-04-18 om 07:11 schreef Robert Elz:
I think this has been discussed before.
If anyone has a pointer I'd appreciate it.
At this point, all you are going to get (at best) added to the standard is
that it is unspecified whether 126 or 127 (or I think perhaps even 1 or 2)
is the error that is generated when execution fails for a "less common"
reason - because not all shells handle them the same way.
OK, so I did some testing. You're right that not all do, but I would say
the ones that don't clearly have a bug. Let's trigger E2BIG:
v=foo_bar_baz_quux_lorem_ipsum_dolor_sit_amet
while :; do
v=$v$v$v$v
/usr/bin/env true $v || { echo $?; exit; }
done
Results:
- pdksh and mksh return 1. Clearly, that is awfully broken. For example,
a script has no way to distinguish between a non-match in 'grep' and a
too-long argument list for 'grep'.
- NetBSD sh and dash return 2, which is nearly as broken. Exit status 2
is commonly used for many other things. Depending on the command, a
script may well have no way to distinguish.
- zsh returns 127, which is wrong in any case as it means "not found".
- bash returns 126.
- AT&T ksh93 returns 126.
- FreeBSD sh returns 126.
- schily's bosh returns 126.
- yash returns 126.
Discounting zsh whose behaviour is unambiguously a bug even by the
current norm, there seems to be a clear majority for 126.
I also found that the historical ksh88 and Bourne shells returned 1.
That may be the real reason why this is unspecified. But, frankly, that
behaviour is as broken as the fall-back string matching for 'case'
discussed recently: it breaks command validation based on exit status.
(I'm aware a program or script *could* decide to return 126 or 127 of
its own volition. That doesn't really change anything about the above
though. If you intentionally break things of course you're going to get
the wrong behaviour.)
There is of course also the possibility that execution fails for a non-posix
specified reason - clearly there posix cannot specify what the error
code must be.
[...]
Sure it can. It doesn't need to specify the nature of the error
condition at all. For example:
"If a command is not found, the exit status shall be 127. If the command
name is found, but the command cannot be invoked, the exit status shall
be 126."
This would go with the current majority behaviour while removing clearly
broken behaviour from the standard.
- M.
Re: sh(1): 2.9.5 function: clarification on compound command?
Op 20-04-18 om 14:35 schreef Joerg Schilling: [...]> $ f() echo > /dev/null ksh93: syntax error: `>' unexpected It looks as of all Bourne derived shells behave the same. My testing says that most ash derivatives (dash, NetBSD sh, Busybox ash, but not FreeBSD sh) accept a simple command there, as does zsh, and pdksh and derivatives (including mksh). - Martijn
Re: sh(1): 2.9.5 function: clarification on compound command?
Op 20-04-18 om 15:10 schreef Joerg Schilling: Martijn Dekker wrote: Op 20-04-18 om 14:35 schreef Joerg Schilling: [...]> $ f() echo > /dev/null ksh93: syntax error: `>' unexpected It looks as of all Bourne derived shells behave the same. My testing says that most ash derivatives (dash, NetBSD sh, Busybox ash, but not FreeBSD sh) accept a simple command there, as does zsh, and pdksh and derivatives (including mksh). Note: by "accept a simple command there" I meant, including that redirection. The problem here is not the simple command but the redirection that cannot be assigned to either the function body or the function definition command. Again, my testing says otherwise. Try it yourself if you don't believe me. $ dash -c 'f() echo hi >&2; f' >/dev/null hi $ mksh -c 'f() echo hi >&2; f' >/dev/null hi $ zsh -c 'f() echo hi >&2; f' >/dev/null hi - M.
Re: What exit status for an executable that is found but cannot be invoked?
Op 20-04-18 om 14:34 schreef Robert Elz: [...] This will be fixed in NetBSD-8 (when it is released, soon, we all hope). Testing sh in NetBSD in earlier versions (7.* and before) is just a good way to waste lots of time - there are zillions of (generally unnoticeable, so they don't get fixed and create churn) bugs in those versions. Good news! Thanks for the info. The problem with ... "If a command is not found, the exit status shall be 127. If the command name is found, but the command cannot be invoked, the exit status shall be 126." is that in general, the way the shell finds out if a command name is found, is by attempting to exec it. Then it needs to map the errno value onto one of the 2 error codes. The spec (kind of) says which error codes map to which value but (obviously) cannot say anything about error codes the spec doesn't know about. It could say "all others" but then you'd end up with some "command not found" giving 126 exit status, because the reason the command was not found was because of some error that is non-posix. I just don't see that. POSIX says quite simply that the status is 127 if the command was not found. It does *not* say that the status is 127 only for commands that were not found for reasons recognised by POSIX. As such, the reason the command was not found is irrelevant. If it's not found, it's 127, period. How is that complicated in any way? That's why I think the standard should be simple in a similar way for 126. If the command was found but could not be invoked, the status should be 126. Reason irrelevant. Not complicated. Sometimes it seems like some of you actively look for ways to make things as complicated as possible... :-) - Martijn
Re: What exit status for an executable that is found but cannot be invoked?
Op 20-04-18 om 17:26 schreef Robert Elz:
Date:Fri, 20 Apr 2018 16:16:12 +0200
From:Martijn Dekker
Message-ID: <4f30a2d7-d99d-e04d-b78e-94295999e...@inlv.org>
| That's why I think the standard should be simple in a similar way for
| 126. If the command was found but could not be invoked, the status
| should be 126. Reason irrelevant. Not complicated.
Your shell does
execve("/path/to/something", argv, environ);
The system returns -1, with errno set to EDEADFROGS
Which error status do you set, 126 or 127?
Well, a shell written for that system would know what EDEADFROGS means
on that system, so would know which exit status to set.
A cross-platform shell, on the other hand... OK, I see your point now.
I still think erring on the side of 126 would be best though. Certainly
much better than returning 1 or 2.
I'm also not convinced this scenario is actually valid. Doesn't POSIX
specify ENOENT for "not found"? How would it be valid for a system to
return anything else if a file is not found?
- Martijn
Laundry list
The central POSIX promise of "write once, run anywhere" has never held quite true for the POSIX shell. There are numerous differences, quirks and bugs between both POSIX shells and POSIX utilities. What's more, certain essential utilities (e.g. mktemp) are missing from the standard altogether, so they differ even more or may be absent. As a result, outside of the standards inner circle, the mere suggestion that creating a cross-shell or cross-platform shell script is a good idea usually elicits hollow laughter. Modernish is my ongoing attempt to properly realise "write once, run anywhere" for the shell. Every programming language has a good standard library or two, except shell. So I'm trying to create a cross-platform shell library that knows and works around all the shell bugs and quirks in currently used shell release versions, while implementing important functionality missing from the standard and providing ways to fix the most common shell language annoyances. I've been working intensively at this since late 2015 and while I keep getting closer to my goal, modernish is still not quite robust enough for a first beta release. I keep finding (and reporting) new shell quirks and bugs in current release versions. I think this illustrates the difficulty of the problem I'm trying to solve. In addition, I've found myself forced to abandon my initial goal of POSIX purism and rely on certain features that, while not standardised by POSIX, appear to be universal on all current UNIX-like operating systems and shells, because they are generally seen to be indispensable. These are listed below. And that's the point of this message. I think that (A) certain universally implemented but unstandardised features should be standardised, and (B) standardised features that nobody implements should be removed, especially if they are inherently broken. Before cluttering the Austin group bug system with each of the points below, I'd be interested in your opinions on these. A. Universally implemented but unstandardised features A1. File names longer than 14 bytes. (_POSIX_NAME_MAX == 14) What system still limits file names to 14 bytes? I think that system would not be compatible with much of anything in 2018, so it seems to me that supporting longer file names should not be considered optional. A2. /usr/bin/env This is very, very commonly used in hashbang paths to deal with varying locations of the shell, e.g.: #! /usr/bin/env bash I don't know a system that has 'env' at any other location, so I think /usr/bin/env should join /dev/tty and a few others as a standard path. A3. /dev/urandom Access to cryptographically strong randomness is essential in 2018. Standard access would be even better. Does any POSIX system from the last two decades *not* have /dev/urandom? I don't know of any. Even Microsoft Interix 3.5 on Windows 2000 had it. So I think /dev/urandom should be added to POSIX, possibly along with getentropy() (see bug 1134). Maybe /dev/random as well -- on Linux, this is the blocking variant, but there doesn't seem to be as much point, as most or all non-Linux systems provide this as synonymous to /dev/urandom. A4. Option flag 'i' in $- to see if shell is interactive. For my shell library it is essential to be able to determine if a shell is interactive or not. Thankfully all current shells support the 'i' flag. I think it should be added to the standard that $- shall contain 'i' if the shell is interactive, with it being unspecified whether that option can be changed or not. A5. The -nt, -ot, -ef operators in test/[ All current shells support these. This is the only built-in way shells have to determine if a file is newer, older or the same file as another file. As far as I know, other POSIX utilities do not provide for this possibility. So I think these should be standardised. A6. For $((x)), accepts leading whitespace in the value of x. There is no current shell that considers " 1" to be an invalid number, as C functions like strtol() are specified to accept and skip initial whitespace. So I think that should be specified for valid shell integer values as well. (Most shells accept trailing whitespace as well but that is not universal. Also note that $(($x)) is different: it always accepts leading and trailing whitespace, because $x is expanded before the arithmetic context is entered.) A7. 'var=foo exec somecommand ...' exports var to somecommand This is unspecified in POSIX because 'exec' is a special builtin; the historical Bourne and ksh88 shells didn't export, but all current POSIX shells do. Schily's bosh was the last to change to the common behaviour, sometime last year IIRC. (Note that I'm not counting /usr/xpg4/bin/sh on Solaris which is basically ksh88 and, as I've shown here before, breaks POSIX compliance in several fatal ways, with little prospect of a fix.) B. Standardised features that (almost) no
Re: Laundry list
Op 26-04-18 om 13:32 schreef Robert Elz:
It really is fairly easy - the NetBSD sh used not to allow trailing whitespace,
but does now (since Martijn brought it to my attention.) The only slight
extra hitch is to not allow vars that contain only whitespace (and are not
empty). ie:
var=' '; : $((var))
is not sane, and should continue to be rejected.
I agree, but it might be worth pointing out that ksh88 (which invented
that syntax) accepted an all-whitespace string by defaulting that to
zero, and so currently do bash, AT&T ksh93, pdksh/mksh, and zsh. They
also accept a literal $(( )).
ksh88, ksh93 and bash 4.4+ even accept $((' ')).
| > B3. Elements of a pipeline (except the last) run in current environment.
| >
| This was intentionally not part of the standard.
I think you are misunderstanding the request - it was not to require pipelines
to run in the same shell env, but to forbid it, except for the final element.
[...]
He doesn't believe any are, that's the point, it is only the std that is the
problem.
Thanks, yes, I think that should be prohibited in the standard (except
for the final element) to prevent future incompatibilities.
Huh? What shell, anywhere, exits on command not found ?
[...]
Basically - every command must be run in a sub-shell, just in case
it does not exist - if the script is to continue after.
No-one writes code like that, and no actual shells require it.
Indeed. I'm very, very much for hardening the shell, but this would be
inherently broken given historical and current practice. Instead,
scripts should check for dependencies on initialisation and refuse to
run if any are not found.
However, I think some kind of usable 'command not found' handler would
be very good to have for scripts.
Bash supports command_not_found_handle() and zsh supports
command_not_found_handler(), but both are run in the subshell forked to
exec the command if the command is not found, so they are not useful for
exiting the program (unless you use modernish, in which case die() will
work quite nicely) or otherwise handling the error in the current
environment. Instead, these are designed to enhance 'not found' error
messages on interactive shells.
So here's an entirely new idea. I think an EXECFAIL pseudosignal trap
executed in the current shell environment would be very useful for
scripts to handle 'not found' and other failures to exec commands.
It wouldn't work for 'foo | bar' though, as traps are reset in
subshells. Maybe that could be worked around: if a subshell, including
any element of a pipeline, exits with a status indicating exec failure
(126 or 127), run that trap in the current environment.
Which would only work for the last-executed command in multiple-command
subshells. But then again, you could set 'trap "exit \$?" EXECFAIL' in
that subshell to have the effect propagate back to the current
environment. Maybe that should even be done automatically, although that
would technically violate POSIX...
Do you think such a pseudosignal trap would be useful to have? If so, I
could put in a feature request to all the shell authors in hopes to make
it widespread...
- M.
Re: Laundry list
Op 26-04-18 om 13:48 schreef Joerg Schilling: Robert Elz wrote: | For some commands you need to do that (e.g. if you like to check whether | "set -o posix" is usable. Huh? What shell, anywhere, exits on command not found ? (that is, not counting when the command that was not found was the last, and not when it was run via "exec"). all conforming shells exit with "set -o bla" in a script. This has nothing to do with command not found. The 'set' special built-in command is, in fact, found. This is the rule that special built-ins shall exit on encountering an error, which is a completely different thing, also defined in XCU 2.8.1 Consequences of Shell Errors, second row in the table: "Special built-in utility error": "shall exit". What we are talking about is the last row in that same table: "Command not found": "may exit". Link to that table: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01 - M.
Re: Laundry list
Op 26-04-18 om 13:52 schreef Robert Elz: Date:Thu, 26 Apr 2018 11:23:49 +0100 From:Geoff Clare | > B3. Elements of a pipeline (except the last) run in current environment. | I'm having a hard time imagining what "many existing scripts" could do | that would be broken by having a pipeline element other than the last | be executed in the current environment instead of a subshell. Anything that changes the execution environment. Since shells don't actually run commands in pipelines in the current env, the side effects are hidden. While not very common, such scripts are far more likely I would argue that foo | while read bar; do baz "$bar"; done | quux is likely to be actually quite common indeed. But maybe that reflects my own bias, in that I've done things like this many times. There is a current widespread expectation that the changes to the variable "bar" will not survive that pipeline, so if it does survive, the script may break. It doesn't survive on any current shell, though. That's why I think the standard should specify that only the last element of a pipe may be executed in the current environment -- to prevent future breakage of that type. - Martijn
z/OS shell access? [was: Laundry list]
Op 26-04-18 om 10:35 schreef Alexander Terekhov: Martijn Dekker wrote: [...] > Does anyone care about POSIX shell scripts on z/OS? (Serious question, > not rhetorical.) Do they even run, ... Well, try https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.1.0/com.ibm.zos.v2r1.bpxa400/ish.htm https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.e0za100/ch1unix.htm Thanks for the links. I would really love to try it. Unfortunately I do not have access to a mainframe with z/OS. You post from ibm.com -- are you involved in z/OS shell development? Would it be possible to obtain ssh access to a z/OS POSIX shell with POSIX utilities for testing purposes? My interest would be in supporting z/OS on modernish before the first release. In return I'll email reports of all the bugs or quirks I find to the appropriate party. Just to toot my own horn a bit: while I'm no Stéphane Chazelas, I do have a little bit of a track record by now, with bash, dash, yash, zsh, mksh, NetBSD sh, Schily's bosh, and a few other shells having fixed a total of dozens of bugs I reported since 2015. Thanks, - M.
Re: Laundry list
[Resending to list. I'd accidentally replied to Geoff only.] Op 26-04-18 om 16:19 schreef Geoff Clare: Robert Elz wrote, on 26 Apr 2018: | > B4. Shell may exit if command not found. (XCU 2.8.1, last row in table) | > | > I was only just made aware by kre that POSIX allows this. It's another thing | > that no shell actually does. | | Are you sure? If no shell does it, then it seems odd that it was | allowed in the resolution of bug 882 which was relatively recent (2015). Hmm. OK, time for some testing. For: someshell -c 'foo; echo does not exit' Historical Bourne: - Xenix sh (1988): exits! - Solaris 10.3 sh (2010): does *not* exit. Current Bourne: - schily's bosh: does not exit, including in POSIX mode. AT&T ksh: none of the following exit: - Solaris 10.3 ksh88 (2010) - ksh93, 1993 - ksh93, 2012 pdksh: none of the following exit: - original pdksh 5.2.14 - NetBSD ksh - OpenBSD ksh - mksh bash: does not exit, including in POSIX mode (tested 2.05b [2002] through current git). zsh: does not exit, including in 'emulate sh' mode (tested 4.3.11 through current 5.5). yash: does not exit, including in POSIX mode (tested 2.17 through current 2.47). Almquist: none of the following exit: - NetBSD sh - FreeBSD sh - dash - Busybox ash - Slackware Linux ash (an older ash, not up to POSIX scratch) osh (from oil 0.4.0, http://www.oilshell.org/): does not exit. (This is a new shell written in Python that (cl)aims to have a bash-compatible mode, but a very quick initial test says it parses builtins at the parsing stage instead of the execution stage, so invalid arguments to builtins cause the program to refuse to even start running.) Anyway, so I had to go back to 1988 to find a shell that exits. I think the fact this was fixed in the Solaris Bourne shell is probably evidence that this behaviour was considered broken early on. The original inspiration for POSIX, ksh88, does not exit, and neither does anything since -- unless there is still a shell I haven't discovered. Do you know of any others? kre wrote: Since it is just a "may" it is very hard to argue against - one needs to examine every possible sh and see what it does, and almost no-one has the resources to do that. Nothing is required to exit, so shell authors just ignore this. Script writers (who do have to deal with this) are not well represented, I agree. That's why I'm here. and also tend not to read the std anyway, they just do what works, and "not exit" is what works. One good reason to standardise it, IMHO. Geoff wrote: While working on bug 882 we did try to examine the behaviour of as many shells as possible for the various cases in the table. What's particularly odd is that there are two "may exit" entries that have a superscript 3 to indicate that a future version of this standard may require the shell to not exit, but for some reason we decided not to add a superscript 3 to the command not found case. This makes me wonder whether someone found a shell that does exit. (The alternative would be that we left it alone because it way already that way in the old table - the superscript 3's are on new additions.) I think maybe someone remembered an ancient historical practice. My testing says it's time to consign it to history where it belongs. Thanks, - M.
Re: Laundry list
Op 27-04-18 om 15:11 schreef Martijn Dekker: Op 26-04-18 om 16:19 schreef Geoff Clare: [...] | > B4. Shell may exit if command not found. (XCU 2.8.1, last row in table) | > | > I was only just made aware by kre that POSIX allows this. It's another thing | > that no shell actually does. | | Are you sure? If no shell does it, then it seems odd that it was | allowed in the resolution of bug 882 which was relatively recent (2015). Hmm. OK, time for some testing. [...] The original inspiration for POSIX, ksh88, does not exit, and neither does anything since -- unless there is still a shell I haven't discovered. Do you know of any others? Yes, that would be the z/OS POSIX shell then. Does anyone know what it does? Also, does anyone know whether this shell is an entirely separate entity or a port/derivative of another shell incarnation? Thanks, - M.
Pushing/restoring a file descriptor for a compound command
I need references and opinions about the following, please.
Consider:
{
exec 8Should the effect of the 'exec' persist past the compound command (the
curly braces block)?
My expectation is that the '8<&-' should push file descriptor 8 onto the
shell's internal stack in a closed state, so that it is restored at the
end of the block.
I think this should allow the effect of 'exec' to be local to that
compound command, and I think this construct should be nestable.
According to my testing, nearly all shells do this. However, two very
widely used shells, dash and bash, leave the file descriptor open beyond
the block.
The author of dash, Herbert Xu, said in response to my bug report that
dash is not obligated to push and restore that file descriptor if it is
already closed when entering the compound command -- implying that a
file descriptor should not be pushed if the local state would be
different from the parent state.
He asked me for a POSIX reference proving otherwise. I can't find any.
Without this behaviour, an awkward workaround is required. To guarantee
that the FD will be restored at the end of the block, you'd need to
attempt to push it twice in two different states in two nested compound
command blocks, for instance:
{
{
exec 8/dev/null
Does anyone have pointers to the POSIX text, or other strong evidence,
that this workaround should not be necessary?
Thanks,
- M.
Re: Laundry list
Op 27-04-18 om 18:52 schreef Garrett Wollman: < said: The "#!" should be standardized - at this point, if your system doesn't support it, everyone will consider your system broken. I realize that there's some concern about standardizing pathnames, but standardizing "/usr/bin/env" seems extremely reasonable, as well as bare names like "bash". There's no need to standardize the actual pathname; a script can include installation instructions to add the shebang (with an appropriate, system-specified path), or indeed can come with an installation script that does so automatically by invoking "command -v" to find the path to the desired interpreter. That installation script would need a way to run without editing that path (or explicitly specifying the shell on the command line) in the first place, or it would be a bit pointless. I don't know of any way to accomplish that except by the de-facto standard mechanism of "#! /usr/bin/env sh". There is a long-time and highly widespread expectation that this will work. In addition to shell scripts, the shebang hack is also commonly used with awk and sed scripts (just to name two other POSIX-specified languages). IMO, that's another good reason to standardise the hashbang path plus the location of /usr/bin/env. Thanks for the feedback, - M.
Re: Pushing/restoring a file descriptor for a compound command
Op 27-04-18 om 18:49 schreef Martijn Dekker: The author of dash, Herbert Xu, said in response to my bug report that dash is not obligated to push and restore that file descriptor if it is already closed when entering the compound command -- implying that a file descriptor should not be pushed if the local state would be different from the parent state. Actually, that implication that I understood is not even correct. No shell, not even bash and dash, outputs "oops, closed" for the following: exec 8To rephrase my request: I would welcome opinions on whether the dash/bash behaviour shown in the original message should reasonably considered a bug in standards terms. Thanks, - M.
Re: Pushing/restoring a file descriptor for a compound command
Op 27-04-18 om 23:38 schreef Stephane Chazelas:
2018-04-27 20:28:57 +0200, Martijn Dekker:
[...]
: <&8 || echo "oops, closed"
[...]
Remember ":" is a special builtin, so its failure causes the
shell to exit. Another one of those accidents of implementation
of the Bourne shell that ended up being specified by POSIX, but
which makes little sense (but that other shells ended up
implementing for conformance).
Good point. Ignore the '|| echo "oops, closed"', it's pointless because
a failed direction prints an error message anyway. I should have said
that no shell produces an error there.
That said, do you have any opinion on whether something like
{ ... ; } 3>&-
should push/restore a closed file descriptor if it's already closed, so
that the effect of exec-ing that descriptor within the compound command
is local to that compound command?
- M.
Re: Pushing/restoring a file descriptor for a compound command
Op 28-04-18 om 01:55 schreef Robert Elz:
Date:Sat, 28 Apr 2018 00:23:51 +0200
From:Martijn Dekker
Message-ID: <8800d6d5-67ea-fad4-19c3-dac4bbfd8...@inlv.org>
| That said, do you have any opinion on whether something like
| { ... ; } 3>&-
| should push/restore a closed file descriptor if it's already closed,
I suspect that it is just a bug, the usual way to do this is
saved_fd = fcntl(3, F_DUPFD, BIG_NUM);
close(3);
and later
dup2(saved_fd, 3);
close(saved_fd);
Which works fine, provided fd 3 is open at the beginning, otherwise
all of those fail with EBADF, which tends to just be ignored (and the
dup2() failing leaves fd 3 with whatever it was set to in between.)
It works fine on NetBSD sh though, and on every other ash derivative I
know of except dash -- even for a file descriptor 3 that is closed at
the beginning. How does NetBSD sh handle this?
Fixing it means adding code - as I understand it, in dash, that is
something they avoid unless it is absolutely essential
If it's a bug, surely it would meet that requirement.
- and if no-one
can show where POSIX requires it, probably just won't happen.
The closest thing I've found is the first sentence of POSIX 2.7
Redirection: "Redirection is used to open and close files for the
current shell execution environment (see Shell Execution Environment) or
for any command."
The first part of that sentence refers to exec'ing it, so is irrelevant
here. The relevant bit is "or for any command". This includes compound
commands.
I claim that this implies that a redirection added to a compound command
should always cause the specified descriptor to be saved and restored,
no matter the initial state of it, or the state initialised by the
redirection.
The alternative seems inherently broken: the effect of an 'exec 3within that compound command will leak out of it, though attaching a
3<&- or 3>&- redirection for the same FD creates a reasonable
expectation that it should restore that FD's state when leaving the
compound command.
Moreover, every current POSIX-compliant shell I know of works as I would
expect, except bash and dash. The behaviour of dash appears to be unique
for current ash derivatives, as FreeBSD sh, NetBSD sh, and Busybox ash
also work as I would expect. I think that's further evidence that the
behaviour of bash and dash should be considered a bug.
That's what I've got. Is that a sane interpretation?
It would be nice if there were something more unequivocal in the
standard, but it seems there isn't...
You might have more luck with bash (perhaps.)
Chet, what do you think?
Thanks,
- M.
In defence of compound aliases
I wish I had had the time to respond to the alias comment thread
earlier. Now, I'd like to make my point more coherently by starting anew
instead.
Let me start with a simple real-life example. In the regression test
suite of my shell library under development, modernish, until recently I
had numbered functions in each test script, doTest1(), doTest2(), etc.
This was a hassle. Inserting or removing test functions required awkward
renumbering. A simple pair of aliases solved my problem:
alias TEST='{ testFn() {'
alias ENDT='}; doTest; }'
Now, each regression test function is delimited by TEST ... ENDT instead
of the normal function definition. The ENDT alias calls a handler
function that runs each test function and automatically numbers the
result as the test script is sourced ('.').
I'm frankly astonished that there seems to be a growing consensus to
de-specify this long-standing type of alias usage. Please, reconsider.
The only objections I can see are of a dogmatic nature. There seems to
be a taboo against "redefining shell syntax", similar to the taboo on
using 'eval', even in safe ways. There is no real technical reason that
it's a "BAD IDEA". The above is perfectly handy, productive and robust.
It is also a highly established and portable usage: it works down to the
first Bourne-derived shell that ever implemented aliases -- because
defining new shell blocks was an original use case proposed for aliases
by David Korn for ksh86. See Korn's 1987 memo at:
- roff source:
https://github.com/weiss/original-bsd/blob/master/local/toolchest/ksh/sh.memo
- PDF: https://www.inlv.org/shellstuff/sh.memo.pdf -- note the date on
it is wrong: it's the date I generated it from the roff source. This is
from 1987.
As Korn wrote there: "The combination of aliases and functions can be
used to do things that can’t be done with either of these separately."
Indeed, it's not as if aliases are some sort of strange anomaly in the
programming world. In other languages, similar features are usually
called 'macros'. C library sources are loaded with them. Nobody calls
that a bad idea, or tells library programmers that they should use
functions instead -- because macros/aliases and functions are good at
different things.
Creative use of aliases may be a Bad Idea for casual shell scripters,
and maybe even many advanced ones -- but only in the sort of way that
using 'eval' is a Bad Idea. It should be done by people who know what
they're doing, who are properly familiar with the shell language and its
internals, in a way that can be re-used by others so they don't have to
reinvent the wheel badly. Hence my belief there is a need for a good
library. But in the long run this won't be possible to do portably if
this functionality is removed or de-sanctioned in the standard.
Please, let's not throw out the baby with the bathwater. Fix the spec if
necessary, but keep aliases working as they are.
- Martijn
https://github.com/modernish/modernish
Re: In defence of compound aliases
Op 14-01-19 om 09:02 schreef Ingo Schwarze:
> Hi Martijn,
>
> Martijn Dekker wrote on Mon, Jan 14, 2019 at 06:59:14AM +0100:
>
>> Indeed, it's not as if aliases are some sort of strange anomaly in the
>> programming world. In other languages, similar features are usually
>> called 'macros'. C library sources are loaded with them.
>
> It depends. Some software is indeed riddled with macros.
> Other software does actively avoid them where possible.
>
>> Nobody calls that a bad idea, or tells library programmers that they
>> should use functions instead --
>
> We do exactly that in OpenBSD, and we say exactly that, even with
> a special emphasis, and we have been saying so for a long time.
> Weeding out as much usage of macros as possible has been among the
> most important refactoring techniques employed in LibreSSL, but not
> only there; it is a general consensus diligently applied throughout
> the system.
On my OpenBSD-current VM...
openbsdbox# find /usr/src -path /usr/src/gnu -prune \
-o -name '*.h' -exec grep '^# *define ' {} + | wc -l
132272
That looks like roughly 132272 macros to me in OpenBSD header files,
with GNU sources excluded.
Ah, but my source directory on this VM is about a year old. Let's do a
'cvs up -d' and repeat that command...
...and now they've grown to 144542. (For comparison, /usr/src/gnu has
another 100061.)
What does this consensus you're speaking of entail exactly? Because "we
will not use macros" clearly isn't it.
> Also note that we generally advise against using the shell for any
> kind of serious programming since there a few languages making safe
> programming practices as hard as the shell, even when used in very
> conventional ways without any special creativity.
That is completely true, and it's actually the main thing I intend to
fix with modernish. Safety and robustness should be made the default.
Functionality causing pitfalls, such as split and glob, should be
disabled by default (safe mode) and fixed where at all possible
(--split/--glob operators for safe mode). Argument, bounds and exit
status checking must be paranoid at all times. A program should *never*
be allowed to continue to run if an inconsistent state is detected (this
is built in to modernish and can easily be added to external commands
with harden()). Emergency halt should be reliable, even when triggered
from a subshell.
That's my design philosophy and it's pretty much the polar opposite of
conventional shell scripting design. And it's amazing how many bugs
became very easy to catch simply by doing these things and adopting this
attitude. In general, modernish scripts do *not* go haywire, they die
(including all their processes) the moment a fatal error occurs.
In particular, see:
https://github.com/modernish/modernish#user-content-reliable-emergency-halt
https://github.com/modernish/modernish#user-content-use-safe
https://github.com/modernish/modernish#user-content-use-syscmdharden
In the event you're curious, unfortunately you'll have to try this on
some other shell because OpenBSD ksh is fatally broken for these
purposes. Among other things, disabling global field splitting (IFS='')
breaks "$@" completely, making the positional parameters unusable for
scripts. So much for eliminating split/glob from scripts that don't need
it. This is an ancient, ancient bug that all other shells fixed many
years ago. I reported it to you guys in early 2016 and even backported a
fix for it from mksh with Thorsten Glaser's blessing. This fix was
included in Theo's private binaries for a time, then quietly
disappeared. When I inquired, I got an email reply months later, saying
that it was not possible to audit the fix because the OpenBSD team has
insufficient people who understand the ksh internals.
- Martijn
Re: In defence of compound aliases
Op 14-01-19 om 12:20 schreef Joerg Schilling:
> Martijn Dekker wrote:
[...]
>> alias TEST='{ testFn() {'
>> alias ENDT='}; doTest; }'
>>
>> Now, each regression test function is delimited by TEST ... ENDT instead
>> of the normal function definition. The ENDT alias calls a handler
>> function that runs each test function and automatically numbers the
>> result as the test script is sourced ('.').
>>
>> I'm frankly astonished that there seems to be a growing consensus to
>> de-specify this long-standing type of alias usage. Please, reconsider.
>
> I am not willing to make those things inspecified as long as there is an
> example of an otherwise POSIX compliant shell that is incompatible to that.
Did you mean 'unless there is an example ...'?
I've tried all the shells I can get my hands on and I've found zero
examples of such a shell (except posh, which has removed alias suppport
altogether, but that doesn't count).
> I am also a bit confused as I remember the teleconference where we discussed
> the current text and decided that it was required to call the lexer again
> with
> the repladcement after an alias replacement has been done.
>
> This way, many things are possible and it is up to the user to define and use
> related alias in a useful way.
Yes.
- Martijn
Re: In defence of compound aliases
Op 14-01-19 om 11:59 schreef Robert Elz:
> Date:Mon, 14 Jan 2019 06:59:14 +0100
> From: Martijn Dekker
> Message-ID: <90a5f4e6-64ae-6d08-fd20-9b9addd6d...@inlv.org>
>
> | Let me start with a simple real-life example.
>
> I have a similar issue in tests I have written, and deal with it in
> an automated way without a single alias in sight. That you have
> chosen to invent mangled syntax to deal with this doesn't mean
> by any means that it is the only way, or even a good one.
This "mangled syntax" qualification is gratuitous disapproval and
nothing more. You have still not come up with any actual argument
against it.
And, I repeat, it didn't require any "inventing". As I just showed, it's
David Korn himself who came up with the basic idea sometime in 1987 or
before.
It is also trivial. It's just a pair of shorthand macros for a simple
shell block with a function definition and a function call in it.
Nothing is being "mangled" whatsoever.
And it is useful and productive. It was an easy, straightforward,
functional and efficient way of solving an actual problem.
Where did I claim it's the *only* way? Of course it isn't. But that
doesn't make it bad.
I am willing to bet, though, that using that little pair of aliases is
the most efficient way, both in terms of maintenance and in terms of
execution. In fact it executes the tests even more efficiently than the
old alias-less way, because it eliminates the need for a loop that
checks if each function exists. And converting the regression tests to
this new method was about five minutes of work.
> | I'm frankly astonished that there seems to be a growing consensus to
> | de-specify this long-standing type of alias usage. Please, reconsider.
>
> Ths issue is that the standard documents what actually works in the
> wild - as much of aliases as seems to work everywhere (and is reasonable
> to specify) is being retained. That is not much - if you have time to read
> everything (going back years to when this was first discussed - before my
> time here) you'll see examples of different implementation choices in all
> kinds of areas.
Yes, there are implementation differences. Many other shell
functionality has implementation differences as well.
But defining a shell block using two aliases, e.g. one starting with a
'{' and the other ending with a '}', is very straightforward, and works
consistently on *all* the current shells that support aliases. Fact.
By now, you know better than many that my library regression tests tend
to test the shell (as well as the library) to its limits. So I think I
have a fairly solid basis for claiming this.
(I'm sure you'll agree that triggering segfaults in your NetBSD sh
development version doesn't count as my method not working, and I
suspect aliases have nothing to do with that anyway -- the problem
appeared when my new loop construct started using FIFOs intensively; I
was making use of aliases long before that.)
So, any outright claim that the new draft text "documents what actually
works in the wild" would be false. I think you know this, hence your
added qualifier, "(and is reasonable to specify)".
Somehow I get the feeling you're not going to consider anything that
sanctions an alias usage you disapprove of "reasonable".
The historical practice with defining shell blocks using aliases is
clearly useful, clearly portable, and would be perfectly reasonable to
specify. The standard developers just need to find the accurate wording
for it.
If it is not the scope of the POSIX standard to deprecate a historically
established usage that works consistently on all shells in the wild,
then please don't.
> | The only objections I can see are of a dogmatic nature.
>
> Then you haven't really looked - there are some of us (perhaps
> just me) who believe that aliases are a terrible idea for everything
[...]
This is literally just another way of saying that your objections are of
a dogmatic nature.
> The actual objections that have caused changes (such as it is, the
> definition in the current published standand is gibberish), or perhaps
> that have limited what is proposed to be specified, are all based upon
> real practical problems.
Then, please, show me the problems.
Particularly the real, concrete, practical problems with this:
alias TEST='{ testFn() {'
alias ENDT='}; doTest; }'
doTest() {
function calling testFn and handling result goes here
}
TEST
test code goes here
ENDT# and many more of these
What problems does this cause for the regression test scripts?
How does this not work well, on any POSIXy
Re: Alias implementations being invalidated by proposed new wording?
Op 03-01-19 om 13:39 schreef Robert Elz:
> ps: in ...
>
> alias forever='while :; do' # the kind of thing for which
> # aliases are "useful" in that the
> # same thing can't be done with
> # functions
>
> aside from this being explicitly unspecified in the proposed new
> wording, I dispute that aliases like this are "useful" - they're just "cute".
In principle I don't disagree there; this is a 'cute' one that I have in
modernish. It has two other cute things: 'not' and 'so', with 'not'
being '! ' and 'so' being an equivalent of '[ "$?" -eq 0 ]', so you can
do 'if so' and 'if not so'.
I would not do or recommend this in standard POSIX scripts. This is a
little something I've added exactly to emphasise that modernish
effectively creates a new shell dialect on top of POSIX sh -- and these
make the feel just slightly distinct in a way that I like. So I like a
little cuteness -- sue me. I'm not that strongly invested though, and if
too many other users dislike them during the upcoming alpha testing
(anyone like to host me a mailing list?), I'll probably get rid.
However, for a publicly documented library, the argument that aliases
make the script harder to understand doesn't really apply.
But all of this is not very interesting. Instead, I'd like to invite
readers to take look at some interesting functionality that depends on
compound aliases: an extensible shell loop construct including, for
example, a loop that integrates the POSIX 'find' utility into a shell
loop, with arbitrary file names (even with newlines) processed correctly
by default. This is effectively a recursive 'for' that supports all
'find' expression primaries plus -iterate (akin to -print except it
makes the loop iterate). And there's more:
https://github.com/modernish/modernish#user-content-use-varloop
https://github.com/modernish/modernish/blob/master/lib/modernish/mod/var/loop.mm
https://github.com/modernish/modernish/tree/master/lib/modernish/mod/var/loop
I believe this to be the first actually robust shell loop construct
implemented in shell. Here's what I learned, after throwing away old
experiments and starting over.
To avoid repeated evaluation of loop arguments causing unwanted side
effects, you need three aliases, not one or two. Of course 'do'...'done'
cannot be overloaded. I settled on: LOOP args; DO commands; DONE
To make loop nesting, 'continue' and 'break' work as expected, a
block-local state is indispensable. Keeping the state in variables is
not possible to make work correctly. The one form of block-local state
provided by the POSIX shell is file descriptors, declared local by
adding redirections. So a loop can gain a local state by reserving just
one file descriptor for the purpose, let's say 8. This is added as a
8<&- redirection in the DONE alias, which should create a local state
for FD 8 (some shells have BUG_SCLOSEDFD which means they need
8https://github.com/modernish/modernish/blob/master/lib/modernish/mod/var/loop/repeat.mm
It gets more interesting with 'LOOP for' and 'LOOP select'. These
provide operators for safe splitting and globbing of their arguments for
use when these are globally disabled (safe mode). Split and glob are
processed by the iteration generator process; the main shell never gets
anywhere near it. I take the opportunity to make some things even safer,
e.g. by automatically prepending './' to expanded pathnames that start
with a dash, and removing non-matching patterns (as keeping them in
unexpanded form makes no sense in this context).
Interestingly, this all performs rather well -- as fast as the shell can
run 'read' and 'eval'. Iterations are generated in parallel, so don't
slow down the main shell process on a modern multiprocessing system.
The general idea of the LOOP...DO...DONE aliases is just as simple and
robust as the TEST...ENDT aliases in my regression test suite that I
wrote about in my other recent message: everything is all wrapped in one
enveloping { ... } block, so it's grammatically robust. There's slightly
more code in them, that's all.
None of this would be possible without compound aliases. I do believe
that this is a useful, non-frivolous use case for them. It doesn't even
do anything particularly new or unusual in and of itself -- all it does
is combine some old things (aliases, shell functions, local file
descriptors, FIFOs, background jobs) in a new way.
- Martijn
[*] Actually, the other week, my loop construct did manage to expose a
24 year old race condition with FIFOs in SunOS kernels (Solaris,
illumos)! An Oracle employee saw my thread on that in comp.unix.shell,
filed a bug, and fixed it, about a week ago. He was kind enough to let
me pass significant details on to the illumos bug tracker as well:
https://www.illumos.org/issues/10179
Re: In defence of compound aliases
Op 14-01-19 om 18:54 schreef Joerg Schilling: > Stephane Chazelas wrote: > >> The "alias" and "unalias" builtins were already removed in the >> very first version of posh (0.01 in 2002), I don't know why. >> Maybe the Debian policy at the time was forbidding the use of >> aliases. > > Shure? > > posh > $ type alias > alias is a shell builtin Yes: $ ./posh -c 'alias' posh: alias: not found $ ./posh -c 'type alias' ./posh: type: not found You're on Solaris, right? posh doesn't have 'type' either so you must be running your /usr/xpg4/bin/type external command which reports on your ksh 'alias' command. A good example of how providing builtins as external commands is NOT a good idea if those commands are only relevant to the shell's internal state, no matter what POSIX says. - Martijn
Re: getopts, OPTIND, and grouped options
Op 14-06-19 om 14:41 schreef Michael Greenberg: My question is: are both of these behaviors (the early-incrementing OPTIND=2 at first and the late-incrementing OPTIND=1 at first) allowed? One might read "the `getopts` utility shall place ... the index of the next argument to be processed in the shell variable `OPTIND`" and assume that late-incrementing is the right choice (since we're still processing the first argument after processing `-a`), but it's not 100% clear to me that's the right interpretation. I don't see the text specify anything in particular about "the index of the next argument to be processed" (either value or format), so as far as I can tell, the OPTIND value is only specified at initialisation and at the end of options processing, and shells can do what they want in between. - M.
Re: getopts, OPTIND, and grouped options
Op 14-06-19 om 17:00 schreef Geoff Clare: Michael Greenberg wrote, on 14 Jun 2019: Martijn Dekker writes: Op 14-06-19 om 14:41 schreef Michael Greenberg: My question is: are both of these behaviors (the early-incrementing OPTIND=2 at first and the late-incrementing OPTIND=1 at first) allowed? One might read "the `getopts` utility shall place ... the index of the next argument to be processed in the shell variable `OPTIND`" and assume that late-incrementing is the right choice (since we're still processing the first argument after processing `-a`), but it's not 100% clear to me that's the right interpretation. I don't see the text specify anything in particular about "the index of the next argument to be processed" (either value or format), so as far as I can tell, the OPTIND value is only specified at initialisation and at the end of options processing, and shells can do what they want in between. Fair enough! :) I don't think Martijn's interpretation is valid. Of course it may well not be valid. But Yuki Watanabe makes a big deal of yash being strictly POSIX compliant, yet it uses non-numeric indexes for combined options, so to me it looks like I'm not the first one who interpreted it that way. He seems to be claiming that when the standard says "the index of the next argument to be processed" it is not specifying a value or format, but when it says "the index of the first operand" it is. Given the similarity of those two pieces of text, I don't see how he can reach that conclusion. OK, so I'll explain how I reached that conclusion, without necessarily claiming it's valid. The text only explicitly specifies the value 1 of OPTIND at initialisation time, and it only explicitly requires it to be numeric at the end of option processing. Yes, it would intuitively make sense to assume the value is required to be numeric in between init and end of processing as well, but the text does not state this explicitly. And implementers run into a bit of problem: a purely numeric OPTIND is insufficient to track the state of short options as several options may be combined into a single argument. So that leaves shells with two options: adhere to the intuitive interpretation of the text and maintain an invisible extra state, which is a botch, but that's what the majority does. Or store the extra state in OPTIND as well, rendering it potentially non-numeric until the end of option processing, but otherwise resulting in a cleaner and more transparent implementation. I can't find anything in the text that actually prohibits this. The only ambiguity is how the word "next" should be interpreted. If the only valid interpretation is for OPTIND to be numeric at every stage of option processing (and thus an invisible extra state must necessarily be maintained), then the text is ambiguous, because (unless I'm wrong) it does not actually state this requirement. - M. -- modernish -- harness the shell https://github.com/modernish/modernish
Exporting an unset variable
On FreeBSD sh, when you set the export attribute on an otherwise unset
variable, an empty environment variable is exported, though the variable
continues to be considered unset within the current shell. This is the
only shell I know of that does this.
$ unset -v foo
$ export foo
$ echo ${foo+set}
[empty line]
$ sh -c 'echo ${foo+set}'
set
$ set | grep ^foo=
$ env | grep ^foo=
foo=
Question: is this behaviour a bug in POSIX terms?
As far as I can tell, the POSIX spec doesn't explicitly say what to do
in this case:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_22
| The shell shall give the export attribute to the variables
| corresponding to the specified names, which shall cause them to be in
| the environment of subsequently executed commands.
I could see how this phrasing technically requires the FreeBSD sh
behaviour, but given that no other shell does this, it seems unlikely
this requirement is intended. Could the text do with clarifying this?
Thanks,
- M.
--
modernish -- harness the shell
https://github.com/modernish/modernish
Re: Exporting an unset variable
Op 08-07-19 om 00:59 schreef Robert Elz: [...] No-one is doubting the effect of the export command upon the shell itself. What is in question is what should actually be placed in the environment when an unset, but exported, variable exists in the shell, and some other command is invoked. Exactly right, thank you. After Jilles' reply, it looks like there's a strong consensus for "nothing" among shell implementers. What needs to change is to perhaps add "if set" into the "shall cause them to be exported of" words, making which shall cause them to be, if set, in the environment of subsequently executed commands. IMHO, that's still not clear enough; this could be interpreted to say that it should be added to the environment if it is set at the time the 'export' command is run. What shell implementations actually do is add a variable whose name was previously given the export flag to the environment the moment it acquires a value, whenever that may be. I think the only way to make that really clear is to leave the phrase you quoted as is and add another one, something like If a /name/ is unset and not followed by =/word/, then its /export/ attribute shall be remembered without otherwise affecting its state, and shall take effect if and when the corresponding variable is set. Thanks, - M. -- modernish -- harness the shell https://github.com/modernish/modernish
Re: Bug 1254 gets worse: "Job" definition is wrong
Op 07-08-19 om 09:40 schreef Geoff Clare: It seems that bash has a rather nasty bug here, as can be seen in the following: $ sleep 5 || true && echo foo ^Z [1]+ Stopped(SIGTSTP)sleep 5 foo $ fg sleep 5 When the sleep 5 was stopped, bash treated this as completion of that pipeline (with non-zero "exit" status) and it went ahead and executed the remainder of the AND-OR list at that point, resulting in the "foo" output. Obviously what it should have done is delay processing the remainder of the AND-OR list until after the sleep had terminated (which was a couple of seconds after I typed "fg"). This was tested in bash 4.4.12 - perhaps someone who has bash 5 can check whether it still does the same. Bash 5.0 release does act the same. (Current git doesn't compile for me.) Other test results: AT&T ksh93 acts like you say is correct. bosh (Schily Tools sh) acts like you say is correct. mksh acts like bash, if you use the path to an external 'sleep' -- its 'sleep' builtin cannot be stopped at all. pdksh (/bin/ksh on NetBSD and OpenBSD) acts like bash. zsh acts like bash. yash acts like bash. dash acts like bash. gwsh acts like bash. FreeBSD sh acts like bash. NetBSD sh acts like bash. Busybox ash acts like bash. So it appears that every current shell is broken in the same way, except the two that can claim a direct lineage to the original Bourne shell. Which is interesting because the obsolete Bourne shell on Solaris 10 (/bin/sh) does not seem to support job control at all. - Martijn
Re: Bug 1254 gets worse: "Job" definition is wrong
Op 07-08-19 om 11:09 schreef Joerg Schilling: Martijn Dekker wrote: So it appears that every current shell is broken in the same way, except the two that can claim a direct lineage to the original Bourne shell. Which is interesting because the obsolete Bourne shell on Solaris 10 (/bin/sh) does not seem to support job control at all. This is not correct. The Bourne Shell supports jobcontrol since at least 1988. I guess the code has been written at a similar time as ksh88 got jobcontrol. If you like to use jobcontrol in the Bourne Shell, you however need to either call "set -m" or call "jsh" (a link to sh) in order to get jobcontrol enabled by default - see man page... Thanks for the correction. Sorry, didn't have time to RTFM. After 'set -m' I cannot get /bin/sh on Solaris 10.1 to interrupt the 'sleep' command with ^Z, so I can't easily test whether Geoff's test case works correctly there. In general, I believe that we should not make inventions but rather start a discussion that ends in fixed to various shells and later standardize something that has been verified to be working. FWIW, I agree. - M. -- modernish -- harness the shell https://github.com/modernish/modernish
Exit status 128 [was: exit status for false should be 1-125]
Op 29-01-20 om 16:36 schreef Austin Group Bug Tracker: The exit status of the false utility is simply required to be non-zero. This means applications have no way of distinguishing between a normal exit of this utility and the special exit statuses, greater than or equal to 126, used for "not found", "can't be executed" or "terminated by a signal" conditions. A status that signifies termination by a signal is > 128, isn't it? Doesn't that mean that 128 isn't special itself? - M. -- modernish -- harness the shell https://github.com/modernish/modernish
Re: Exit status 128 [was: exit status for false should be 1-125]
Op 31-01-20 om 10:11 schreef Geoff Clare: Martijn Dekker wrote, on 30 Jan 2020: A status that signifies termination by a signal is > 128, isn't it? Doesn't that mean that 128 isn't special itself? Correct, but see https://austingroupbugs.net/view.php?id=1229#c4269 (specifically the RATIONALE addition, last bullet point). Thanks for the pointer. Does this mean exiting with status 128 is going to be disallowed for conforming shell scripts as well? - Martijn -- modernish -- harness the shell https://github.com/modernish/modernish
awk: FS matching 0 or more characters
Consider:
echo 'one!two!!three!!!end' | awk -v 'FS=!*' \
'{ for (i=NF; i>0; i--) print $i; }'
Onetrueawk, mawk, GNU awk, and Solaris awk all print:
end
three
two
one
However, Busybox awk prints:
d
n
e
e
e
r
h
t
o
w
t
e
n
o
In a way, the Busybox awk behaviour makes more sense. The "!*" ERE
means: match zero or more "!", and that's exactly what it did.
Changing the ERE to '!+' makes all awks behave consistently, so that's
the obvious fix.
But what, if anything, does POSIX have to say about an FS ERE matching
zero or more characters?
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/awk.html#tag_20_06_13_04
I can only find:
1. If FS is a null string, the behavior is unspecified.
That doesn't really apply; FS is a non-null ERE, though one that may
match the null string.
3. [...] Each occurrence of a sequence matching the extended regular
expression shall delimit fields.
Is a null string matching the ERE a "sequence" that matches it?
So at this point I'm not sure whether to report a bug in Busybox awk, or
an area in the standard that needs further specification or
clarification, or neither...
- Martijn
--
modernish -- harness the shell
https://github.com/modernish/modernish
Re: XCU: 'return' from subshell
Op 13-03-20 om 15:17 schreef Chet Ramey:
On 3/13/20 10:14 AM, Harald van Dijk wrote:
[...]
I don't see how you can allow that without also allowing
f() { (return 7; echo no); echo $?; }; f
If that also works in all shells (meaning it doesn't print no, and does
print 7), then by all means standardise it.
I can't find one that doesn't in my quick initial testing, but I don't
have binaries for every shell under the sun.
My testing says it acts identically (outputs '7') on the following
shells: ash variants (Busybox ash, dash, FreeBSD sh, gwsh, NetBSD sh),
bash, bosh, ksh88, ksh93, mksh, pdksh, yash, and zsh.
- M.
--
modernish -- harness the shell
https://github.com/modernish/modernish
Is ksh93's default alias command='command ' POSIX compliant?
I am now the maintainer of what is currently, to the best of my
knowledge, the only actively developed fork of AT&T ksh93. It is based
on the last stable AST version, 93u+ 2012-08-01. Along with a few others
I have been fixing a bunch of bugs. See https://github.com/ksh93/ksh for
the current activity and some history/rationale in the README.md.
One issue is ksh39's default alias
command='command '
which continues alias substitution after 'command', defeating its
ability to bypass aliases.
I think that doing this by default violates POSIX, because 'command' is
specified as a regular builtin, not as an alias. So I have removed it:
https://github.com/ksh93/ksh/commit/61d9bca5
There is some disagreement about that, however -- as you can see in the
comment under that commit (scroll all the way down). And of course it is
possible that I am wrong. So I would like the ask the Austin Group's
opinions. Does this alias, by virtue of being default, violate POSIX?
One issue in particular I would note: on 93u+, 'times' is defined as a
default alias
times='{ { time;} 2>&1;}'
which not only does not produce POSIX compliant output, but also does
not combine with the default 'command' alias: 'command times' is a
syntax error. And that is a perfectly valid POSIX idiom, so at the very
least, that seems like a straight-up standards violation. However, I
have already replaced the 'times' alias by a proper POSIX compliant
builtin on my version. This does demonstrate a problem, however; since
aliases can contain arbitrary shell grammar, continuing alias expansion
after 'command' seems problematic even if it doesn't violate POSIX.
[Possibly, this alias was added to make 'command' work with some
POSIX commands that are also defined as default aliases, i.e. 'fc',
'hash', 'times', and 'type' -- as well as some ksh-specific
scripting-relevant commands: 'autoload', 'compound', 'float',
'functions', 'integer', 'nameref', 'redirect'. I think defining
any of these as aliases is a bug in itself, as 'unalias -a' removes
all of them, so it is impossible for a script to start with a clean
alias slate without losing essential commands. That's why I'm in the
process of converting all of these to proper builtin commands.]
- Martijn
--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh
Re: Is ksh93's default alias command='command ' POSIX compliant?
Op 14-06-20 om 18:03 schreef shwaresyst: The command alias is nominally conforming, I believe, in that recursive alias expansion isn't permitted so looking for a utility named command still occurs. So are you saying that, in order to be POSIXly sure of running a standard command 'foo' given the default shell alias configuration, you have to do "command command foo" or "\command foo"? Surely that can't be the intention...? However, the implementation of various utilities as aliases changes the reporting of 'command -v', or '-V', to that they are aliases and not actual utilities as a user might expect. Such a change may be allowed, but I don't see it as intended for the utilities the standard requires. The definition of times only as an alias, in the manner used, is not conforming; the use of curly braces, being keywords, turns it into a compound command the command utility is not expected to see as an argument list, and defeats the recognition of times as a special built-in per XCU 2.9.1, as time is a regular utility. In fact, that's what all such aliases do, whether they have extra shell grammar or not. If 'unalias -a' deletes it, it's not a regular utility. There still needs to be an implementation that is accessed when "\times", to disable expansion as a possibility, is specified as well, that I see. Yes, there is that too. I believe that logically means that all standard commands should be regular or special builtins, because having a default alias overriding a default builtin only to exhibit the same behaviour is 100% pointless. Thanks for the feedback. - M. -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
Re: Is ksh93's default alias command='command ' POSIX compliant?
Op 16-06-20 om 17:37 schreef Alan Coopersmith: We are maintaining ksh93 packages, but not doing active development work on them. Our packages are still based on the last stable release from AT&T - 2012-08-01 with an unfortunately high number of local patches applied: https://github.com/oracle/solaris-userland/tree/master/components/ksh93 We really don't want our ksh93 to be a separate fork, but got stuck here due to past decisions it's a bit too late to undo now. Maybe not, because 2012-08-01 is precisely the version that we are basing our rebooted development on at the new ksh93 repository. This new project is really picking up steam as we speak. Many bugs have been fixed already and many more fixes are coming. Stability, correctness and POSIX compliance are the immediate goals. Feature development can wait. I would like to invite you (as I do everyone) to submit any of your fixes that are suitable for incorporating upstream to https://github.com/ksh93/ksh as pull requests, documenting in each commit message what it fixes and how it fixes it. And it would be even better if you could periodically check for open issues labelled "help wanted", as we're currently a bit stuck on those and they are bugs in the Solaris version of ksh93 as well. Especially this one: https://github.com/ksh93/ksh/issues/2 Please, have a look at our commit history and let me know if you like what you see. We'd be delighted to have any fixes you are willing to share, and hope we will prove worthy to become the new ksh93 upstream. - Martijn -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
Re: Is ksh93's default alias command='command ' POSIX compliant?
Op 18-06-20 om 14:11 schreef Joerg Schilling: Is my impression correct that you did not use the modifications from Redhat people that introduced non-portability, many bugs and a slowdown? Yes. Be careful when looking at changes from these people, since they e.g. did throw away the ksh93 specific malloc() that is much faster than the system malloc() since it does not need to take care of multi-threading. Also note that ksh93 depends on being able to use a freed()d piece of memory even after the next malloc (if that returned a different start address) as it has been in AT&T malloc() in the time before multi threaded environments. I don't know much about malloc, but what you describe sounds rather dodgy. It evidently works though, so I'm leaving well enough alone. It may turn out to be an advantage for this bugfix project that my primary expertise is in the shell language itself, whereas my C skills are at an intermediate level. I am simply not well-versed enough in C to even think about touching things like malloc. On the other hand, if some change breaks the shell language, I am likely to notice it quickly. I not only have the ksh regression test suite (which I fixed, as it was in a deplorable state), but also the modernish regression tests which have been exceptionally good at breaking shells for a few years now. Even kre has had to admit that much. ;) There's also a policy of documenting in every commit message not only what it fixes, but also how it fixes it. This helps improve everyone's understanding of this very poorly commented codebase, while preserving this documentation for future generations without depending on GitHub in perpetuity. The idea is that 'git blame' will lead you straight to the rationale for every change. But more importantly, this also enables me to know what's going on myself, while forcing contributors to double-check that they understand what they are doing. No pull request is merged until I understand exactly the ins and outs of it. I would like to invite you (as I do everyone) to submit any of your fixes that are suitable for incorporating upstream to https://github.com/ksh93/ksh as pull requests, documenting in each commit message what it fixes and how it fixes it. I pulled the snapshot, compiled and had a quick check for speed. ++ It still uses the mature and highly portable ksh93 build environment. I have no intention of changing the build system, but I would question your praise for it. In my experience, it is currently very brittle. It failed to build on modern Macs before I fixed it for that. It currently fails to build on at least Solaris (see below), NetBSD, as well as AIX and QNX (those latter two as hosted by polarhome.com). And if I restore nmake (which got removed along with the other superfluous non-ksh AST utilities), it fails to build on even more systems, including my Mac. Currently it only works if the build system is forced to fall back to mamake, which uses the Mamfiles machine-generated by nmake and mostly ignores the Makefiles. Of course, the long-term goal is to restore nmake and make the build system actually work with that, so that the Makefiles become editable/usable again. In any case, I don't even know where to begin thinking about changing build systems, and there is community support for keeping the current one. So it is just going to have to be fixed instead... somehow. Anyone who cares about and understands this build sytem, PLEASE send patches or pull requests that make it build on systems it currently fails on. ++ It still compiles out of the box on Solaris That's amazing to me, because for me it consistently fails to compile on the pristine Solaris 11.3 and 11.4 evaluation VMs as downloaded directly from Oracle. The problem is that the 'dll' iffe feature test fails to write the Dllscan_t structure and a few others to a generated header file. The build then fails because Dllscan_t is an unknown identifier. Your build environment must be somehow different from the default Solaris configuration as provided in their VMs. I would be very interested to know how it is different, so that I can build and test it on Solaris myself. BTW, the same type of build failure happens when compiling on NetBSD. - It does not compile/use libshell. Well, it could have fooled me: $ find arch -name libshell\* arch/darwin.i386-64/lib/libshell.a.old arch/darwin.i386-64/lib/libshell.a arch/darwin.i386-64/src/cmd/ksh93/libshell.a I'm not sure how you think ksh could possibly have worked or compiled without libshell... - Using the default compile setup, it is 5.5% slower than the OpenSolaris ksh93. Mesured with the "configure" script from Schilytools. - Using the additional compile options "-O -fast -xspace", it is still 2.5% slower than the OpenSolaris ksh93. It would be great if you could help isolate the commit that caused
Re: LC_CTYPE=UTF-8
Op 25-06-20 om 16:59 schreef Alan Coopersmith: On 6/25/20 6:33 AM, Hans Åberg wrote: Perhaps there should be a default UTF-8 locale: It seems that the current construct does not apply so well to it. If the goal is to standardize existing behavior the standard could define the C.UTF-8 locale (or perhaps a POSIX.UTF-8 locale) that a number of systems already have, which is the standard C/POSIX locale with just the character set changed to UTF-8 instead. Datapoint: AT&T ksh93 has a C.UTF-8 locale built in which it handles internally. I suppose that could be considered a precedent of sorts. - M. -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
Re: LC_CTYPE=UTF-8
Op 25-06-20 om 21:13 schreef Alan Coopersmith: The only thought I had along those lines was that I thought the "C" locale came from the C standard, and might be best left to the C committee to standardize, while this group controls the "POSIX" locale definition. Actually, as far as POSIX is concerned, the two are synonymous. XBD 7.2 "POSIX Locale": https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html#tag_07_02 | Conforming systems shall provide a POSIX locale, also known as the C | locale. In POSIX.1 the requirements for the POSIX locale are more | extensive than the requirements for the C locale as specified in the ISO | C standard. However, in a conforming POSIX implementation, the POSIX | locale and the C locale are identical. -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
Draft suggestion: Job control and subshells
The current POSIX definition (including the current draft) for a subshell environment is: A subshell environment shall be created as a duplicate of the shell environment, except that signal traps that are not being ignored shall be set to the default action. Changes made to the subshell environment shall not affect the shell environment. This does not mention job control. However, I think job control should be disabled in subshells as it makes no sense to do otherwise, because subshells are not interactive and you cannot access a subshell job table from your interactive main shell. Nearly all shells disable job control for all subshells. But for virtual/non-forked subshells, ksh93 currently only disables job control for command substitutions, causing buggy and inconsistent behaviour. So maybe the first sentence should be: "A subshell environment shall be created as a duplicate of the shell environment, except that job control shall be disabled and signal traps that are not being ignored shall be set to the default action." Thoughts? - Martijn -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
Re: Draft suggestion: Job control and subshells
Op 23-07-20 om 14:47 schreef Geoff Clare: Not according to a quick test I just did. The only shell that I tried which won't do job control in a subshell is zsh in its default environment; with "emulate sh" it does. $ for shell in bash dash ksh mksh zsh; do echo $shell ...; $shell -c 'set -m; (sleep 2 & jobs; wait %%)'; done On the vast majority of shells, 'wait %%' works without job control, so that is not a correct test. The only shells I've found where that does *not* work is (1) bash in posix mode, where it complains of no job control and (2) zsh in sh emulation mode, where it is simply ignored. Note also the behaviour of interactive shells changes when launching a background job in a subshell: most do not print job numbers and none print 'Done' notices. Only FreeBSD sh and ksh93 print a job number, but I'm pretty sure that's a bug. The ksh93 code for printing that job number tests for SH_INTERACTIVE (-i) whereas it should be testing for SH_MONITOR (-m). ksh93 job control is also broken in other ways, as you can see from your own "" output. I'm currently trying to fix it. It disables job control in command substitutions but not in non-forked non-comsub subshells, causing several different kinds of misbehaviour. A proper test for job control is whether you can use the 'fg' command; it will generally tell you if job control is disabled. There is no current shell where you can use 'fg' in a subshell: bash: $ (sleep 1 & fg) bash: fg: no job control dash: $ (sleep 1 & fg) dash: 1: fg: job (null) not created under job control FreeBSD sh: $ (sleep 1 & fg) [1] 29348 sh: fg: No job control ksh88: $ (sleep 1 & fg) fg: job not created under job control ksh93: $ (sleep 1 & fg) [1] 17088 /bin/ksh: fg: No job control NetBSD sh: $ (sleep 1 & fg) fg: job not created under job control mksh: $ (sleep 1 & fg) mksh: fg: job control not enabled yash, including yash -o posix: $ (sleep 1 & fg) fg: job control is disabled zsh: % (sleep 1 & fg) fg: no job control in this shell. 'zsh --emulate sh' seems as bit broken but is essentially the same: $ (sleep 1 & fg) fg: no current job So, to me it looks like 100% of shells currently disable job control in subshells, and to me this also still looks like the only sensible thing to do. Unless I'm missing something, my suggestion stands... - Martijn -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
ksh93 job control behaviour [was: Draft suggestion: Job control and subshells]
Op 24-07-20 om 10:47 schreef Geoff Clare: Ksh93 doesn't turn off -m in a subshell, so testing SH_MONITOR wouldn't prevent the job number output. I think it should have a separate flag to indicate when the shell is behaving non-interactively despite the -i flag being set. It depends on how you test it. ksh93 internally keeps track of (different aspects of) job control in three different ways: sh_isoption(SH_MONITOR) // the actual shell option sh_isstate(SH_MONITOR) // internal state flag job.jobcontrol // global state variable I'm not sure what the difference is between the latter two. When ksh tests for job control, sometimes it uses sh_isstate(), sometimes the global variable, and there seems to be no rhyme or reason re which is used when. I do know that ksh93 job control behaviour is broken in a few ways, although it remains to be clarified how exactly. Currently, ksh tests for sh_isstate(SH_INTERACTIVE) to decide whether to output the job number. Adding '&& sh_isstate(SH_MONITOR)' prevents it from outputting the job number in non-forked subshells, even if the -m option is active in them. Bash and dash also don't turn off -m, as reported by set +o, so your claim that most shells "disable" job control is not correct in that regard. At least on ksh93, the state of the -m shell option is not all that relevant for subshells. Most shells (mksh being the exception) don't turn off the -i option in subshells either, yet subshells act non-interactively. ksh93 job control is also broken in other ways, as you can see from your own "" output. I'm currently trying to fix it. It disables job control in command substitutions but not in non-forked non-comsub subshells, causing several different kinds of misbehaviour. It doesn't turn off -m in command substitutions: $ echo $(set +o) | grep monitor set --default --bgnice --braceexpand --monitor --multiline --vi --viraw Obviously that's a good thing since the point of set +o is to be able to obtain the option settings to restore them later. True. However, ksh93 has explicit code for disabling job control in command substitutions. It changes the SH_MONITOR state flag and the job.jobcontrol variable, but not the shell option. A proper test for job control is whether you can use the 'fg' command The fg command is a feature of interactive job control. There is no reason why anyone would want to use it in an (effectively or actually) non-interactive shell environment. Does POSIX specify that job control behaves differently for interactive shells vs. (non-interactive shells or subshells)? If so, where? [...] It is only the interactive aspects of job control that are not available in a subshell. The parts that still work (in particular process group ID assignment and the ability to send a signal to that process group using kill %%) are still available and useful. But ksh 93u+ 2012-08-01 does not do this either (note the extra '; :' is needed to avoid optimising the subshell away when using '-c'): $ /bin/ksh -c 'set -m; (/bin/sleep 2 & ps -o pid,pgid,command); :' PID PGID COMMAND 24570 24570 ksh -c set -m; (/bin/sleep 2 & ps -o pid,pgid,command); : 24571 24570 ksh -c set -m; (/bin/sleep 2 & ps -o pid,pgid,command); : 24572 24570 /bin/sleep 2 However, an "interactive" shell (-i flag, which implies -m) does create a process group: $ /bin/ksh -i -c '(/bin/sleep 2 & ps -o pid,pgid,command); :' [1] 24636 PID PGID COMMAND 24634 24634 /bin/ksh -i -c set -m; (/bin/sleep 2 & ps -o pid,pgid,command); exit 24635 24635 /bin/ksh -i -c set -m; (/bin/sleep 2 & ps -o pid,pgid,command); exit 24636 24635 /bin/sleep 2 So, would you say that ksh93's behaviour with just -m is broken (as in POSIX non-compliant) and needs fixing? At least, ksh93 is the odd one out here. By my testing, other shells (bash, dash, mksh, yash, zsh) do act as you suggest, even when adding the '; :'. - Martijn -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
Re: Bug 1393 ("command" declaration utility) possible solution
Op 23-09-20 om 15:11 schreef Geoff Clare via austin-group-l at The Open
Group:
Playing around with export and readonly in shells that support array
assignments, it's clear that they are treating them as separate tokens
in the grammar in order to accept array assignment syntax:
$ bash -c 'export a=(1 2 3); echo ${a[0]}'
1
$ bash -c 'e=export; $e a=(1 2 3); echo ${a[0]}'
bash: -c: line 0: syntax error near unexpected token `('
$ bash -c 'readonly a=(1 2 3); echo ${a[0]}'
1
$ bash -c 'r=readonly; $r a=(1 2 3); echo ${a[0]}'
bash: -c: line 0: syntax error near unexpected token `('
$ ksh -c 'export a=(1 2 3); echo ${a[0]}'
1
$ ksh -c 'e=export; $e a=(1 2 3); echo ${a[0]}'
ksh: syntax error at line 1: `(' unexpected
$ ksh -c 'readonly a=(1 2 3); echo ${a[0]}'
1
$ ksh -c 'r=readonly; $r a=(1 2 3); echo ${a[0]}'
ksh: syntax error at line 1: `(' unexpected
In order to make "command export ..." and "command readonly ..."
work the same, all that's necessary is for those shells to treat
command as a separate token too.
The standard allows this special treatment of export and readonly
(and command) because it allows shells to extend the syntax such
that they accept input that would, according to the standard, be a
syntax error.
I would question that the currently published standard allows any
regular builtin to override the regular shell grammar with special
syntactic properties. What exactly do you base this on?
Aren't regular builtins supposed to be exactly equivalent to external
commands, with every regular builtin being mandated to have a
corresponding external version? How would you expect an external command
to have special grammatical properties in the shell?
This is also not a majority shell behaviour. For instance, bash does not
currently do this. I wonder if Chet has any plans of changing that now.
The resolution of this bug is particularly problematic for ksh93. The
way the ksh93 parser implements this behaviour for 'command' is by
internally parsing something like
command export foo=bar
as
foo=bar command export
i.e. a preceding assignment. The parser similarly reorders 'command
readonly foo=bar' and 'command typeset foo=bar'. You can check this
using set -o xtrace. And yes, this works even if you literally type the
second variant, and even then the assignment will persist as exported
(which is a clear violation of the standard, but this is how Korn et al
have chosen to do it -- and fundamentally reworking the parser is not on
the cards).
One side effect of this bizarre parser behaviour is that 'command' is
unable to prevent the shell from exiting. Something like 'readonly foo;
command export foo=bar' will exit the shell because 'foo=bar' is
actually executed as an assignment preceding 'command', so the shell
exits before the 'command' builtin is even run.
To fix this for the new POSIX mode of ksh 93u+m, I've disabled this
parsing behaviour so that 'command' is treated as a regular builtin if
the -o posix option is on, and 'typeset' falls back to parsing the
assignment-arguments itself without any special grammatical properties
(the code to do this was still present). This then allows 'command' to
stop the shell from exiting on trying to modify a readonly variable.
But that does mean that 'command' behaves as a regular builtin with
regular shell grammar as I believe the currently published POSIX
standard specifies it, i.e., it disables both "special" and
"declaration" properties of commands it runs.
The bug description claims that the situation for command is different
from export and readonly (it says "A similar argument could be made
for 'export' and 'readonly' themselves, but at least they are already
special built-in utilities"). This is a red herring. Whether or not
a utility is a special built-in does not affect the grammar.
Note that I was careful to say "separate tokens", not "reserved words".
They are not being treated as reserved words, because they are subject
to alias expansion whereas reserved words are not.
(Note that both 'export' and 'readonly' are reserved words on zsh 5.1
and later, including in its POSIX mode. The 'disable -r' command can be
used to disable the reserved words, reverting zsh to the original
builtin commands.)
So I think that the normative text in Issue 8 draft 1 is fine as-is.
It meets the goal of requiring "command export args..." to expand its
arguments the same way as "export args...", and likewise for readonly,
and that remains a worthwhile goal. All that is needed is to add some
explanatory rationale (to command, export, and readonly).
I question that this goal is worthwhile. I think it makes the shell
grammar even more inconsistent than it already is and increases
confusion. It also means that the standard now will not provide for any
way to ensure regular shell grammar parsing for arbitrary commands,
execve(2) optimisation for last command
Most shells 'exec' the last command in -c scripts, e.g.: $ bash -c 'printf "$$ "; sh -c "echo \$\$"' 66303 66303 $ dash -c 'printf "$$ "; sh -c "echo \$\$"' 66304 66304 $ ksh93 -c 'printf "$$ "; sh -c "echo \$\$"' 66305 66305 $ yash -c 'printf "$$ "; sh -c "echo \$\$"' 66309 66309 $ zsh -c 'printf "$$ "; sh -c "echo \$\$"' 66310 66310 However, no shell seems to do this for scripts loaded from a file: $ echo 'printf "$$ "; sh -c "echo \$\$"' >/tmp/test.sh $ chmod +x /tmp/test.sh $ bash /tmp/test.sh 66325 66326 $ dash /tmp/test.sh 66327 66328 $ /bin/ksh /tmp/test.sh 66329 66330 $ yash /tmp/test.sh 66331 66332 $ zsh /tmp/test.sh 66333 66334 My question: why is this? I would have thought that a script is a script and that it should make no essential difference whether a script is taken from a -c argument or loaded from a file. What makes the optimisation appropriate for one but not the other? ksh93 used to use this exec-the-last-command optimisation also for scripts loaded from a regular file, but disabled this (keeping it only for -c scripts) in version 2011-12-24. The corresponding changelog entry is characteristically uninformative, so doesn't help answer my question. Now that a couple of redirection-related bugs have been fixed with that optimisation, we've re-enabled it for regular scripts in ksh 93u+m ahead of releasing the first beta of this reboot (Very Soon Now(tm)), but if there is a good reason that shouldn't be done, we should revert this. -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
Re: execve(2) optimisation for last command
Op 15-04-21 om 22:43 schreef Harald van Dijk: [...] In specific cases, the shell may be able to detect that these issues cannot be a problem. In those specific cases, the optimisation should be perfectly valid and any reasons why it should or should not be performed are not of a technical nature. It makes sense now: instead of stdio, ksh93 uses libast's sfio, which on most systems is capable of peeking ahead on arbitrary files without consuming data. And it doesn't care if the input is a file or a string because both are turned into sfio input streams. Thanks to Chet and Harald for confirming that I've not overlooked some reason why the optimisation shouldn't be re-enabled. -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
Re: what variables are affected by the "allexport" shell option?
Op 14-08-18 om 17:21 schreef Stephane Chazelas:
Hi, the spec for the "allexport" (set -a) shell option says:
POSIX2018> When this option is on, the export attribute shall
POSIX2018> be set for each variable to which an assignment is
POSIX2018> performed; see XBD Variable Assignment. If the
POSIX2018> assignment precedes a utility name in a command,
POSIX2018> the export attribute shall not persist in the
POSIX2018> current execution environment after the utility
POSIX2018> completes, with the exception that preceding one of
POSIX2018> the special built-in utilities causes the export
POSIX2018> attribute to persist after the built-in has
POSIX2018> completed. If the assignment does not precede a
POSIX2018> utility name in the command, or if the assignment
POSIX2018> is a result of the operation of the getopts or read
POSIX2018> utilities, the export attribute shall persist until
POSIX2018> the variable is unset.
the reference to "XBD Variable Assignment" would suggest it's
only meant to be done for variables assigned with an assignment.
But then it goes on about "getopts" and "read", but not "for",
and doesn't say if that applies to variables the shell sets by
itself like $IFS, $LINENO, $PPID, $PWD
[...]
I'd like to rekindle this old discussion. Thread archived at:
https://www.mail-archive.com/austin-group-l@opengroup.org/msg02754.html
Today I received a pull request to make allexport work for ${var:=value}
and ${var=value} on ksh 93u+m. This currently works on bash, dash, *BSD
sh, and zsh, but not on ksh, pdksh/mksh, or yash.
Expansions of that form weren't mentioned in the 2018 discussion.
I need some clarity. Should ${var:=value} and ${var=value} honour
allexport? Is this unspecified in POSIX terms or is support for this
implied/intended by the current text? I think it might be, but shells
differ too much.
--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh
Re: what variables are affected by the "allexport" shell option?
Op 17-01-22 om 22:54 schreef Robert Elz:
Date:Sun, 16 Jan 2022 20:26:00 +
From:"Martijn Dekker via austin-group-l at The Open Group"
Message-ID: <409b0a89-6fe1-d688-23f9-8c918ef89...@inlv.org>
| I need some clarity. Should ${var:=value} and ${var=value} honour
| allexport?
Yes.
Any assignment (in the usual sense of that word) to a variable while -a is
in effect should result in the variable being exported, no matter how
it is accomplished. Further in draft 2.1 it actually says:
When this option is on, a variable that is assigned a value
(see XBD Section 4.23),
[that xref is prehaps unfortunate, it really is not needed here as:]
Yes, that xref is the whole cause of the issue. On ksh, the bug appears
to be historic behaviour inherited from the Bourne shell, but the xref
appears to have mislead the author of the newer shell yash, which
replicated the bug.
But the fact that they're included makes it clear that any assignment
to a variable counts (even the absurd _ variable, which fortunately
is not posix, but is supported by ksh versions, and to varying extents
by most other shells). Assignments performed as a side effect of
word expansions are certainly included.
Then the standard should say that.
Another case worth checking is unset var; set -a; echo $(( var=3 ));
and verify that after that, var is exported ($SHELL -c 'echo $var').
Good point.
The situation with arithmetic assignment seems to be the same: ksh93,
pdksh/mksh, and yash do not export the variable.
It might be worth appending to the "including variables..." clause
above:
and also including side effects of parameter and arithmetic expansions,
Yes.
What follows has nothing to do with -a, but another aspect of ${var=value}
What is less clear about ${var=value} (and the form with the : as
well of course) is whether the assignemnt, when it is performed, ought
to be considered as a XBD 4.23 type variable assignment, with everything
that comes with that (including the rules for how tilde expansion happens).
As far as I can tell, there is currently nothing to indicate that it should.
--
|| modernish -- harness the shell
|| https://github.com/modernish/modernish
||
|| KornShell lives!
|| https://github.com/ksh93/ksh
$? behaviour after comsub in same command
Consider: false || echo $(true) $? dash, mksh and yash print 1. bash, ksh93 and zsh print 0. Which is right? My hunch says dash, mksh and yash are right, as 'true' is executed in a command substitution (a subshell) within the same command, so 'false' should still be the "most recent pipeline" for the purpose of the subsequent $? expansion. Thoughts? My apologies if this has been discussed before; terms like $? are not easily searchable. -- || modernish -- harness the shell || https://github.com/modernish/modernish || || KornShell lives! || https://github.com/ksh93/ksh
