Re: Pushing/restoring a file descriptor for a compound command
On 5/2/18 5:52 AM, Joerg Schilling wrote: > 2)bash ignores the "new" rules. Bash reserves most of these historical accidents for posix mode. -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Pushing/restoring a file descriptor for a compound command
2018-05-02 11:52:09 +0200, Joerg Schilling: > Stephane Chazelaswrote: > > > 2018-04-27 20:28:57 +0200, Martijn Dekker: > > [...] > > > : <&8 || echo "oops, closed" > > [...] > > > > Remember ":" is a special builtin, so its failure causes the > > shell to exit. Another one of those accidents of implementation > > of the Bourne shell that ended up being specified by POSIX, but > > which makes little sense (but that other shells ended up > > implementing for conformance). > > The background seems to be a bit more complex: > > sh -c ' : <&8; echo bla' # Solaris 11 Bourne Shell > bla Here the problem is that the Bourne shell silently ignored the failing of those "dup2()" redirections. $ echo test | sh -c 'cat <&8; echo OK' test OK That bug was fixed in ksh and all other shells. But otherwise, a failing ":" still causes the shell to exit: $ sh -c 'echo < /x; echo OK' sh: /x: cannot open [...] > bash -c ' : <&8; echo bla' > bash: 8: Falsche Dateinummer > bla > > So: > > 1)before POSIX, ":" could not fail > > 2)bash ignores the "new" rules. [...] Not really, it's POSIX again specifying an accident of implementation in the Korn shell (in this case inherited from the Bourne shell, so a "very old" rule). Here, it's a silly requirement. Nobody would expect a failure of the no-op command to exit the shell. That's why bash, zsh and yash only do it in conformance mode (when called as "sh"). $ (exec -a sh bash -c ': <&8; echo X') sh: 8: Bad file descriptor -- Stephane
Re: Pushing/restoring a file descriptor for a compound command
Stephane Chazelaswrote: > 2018-04-27 20:28:57 +0200, Martijn Dekker: > [...] > > : <&8 || echo "oops, closed" > [...] > > Remember ":" is a special builtin, so its failure causes the > shell to exit. Another one of those accidents of implementation > of the Bourne shell that ended up being specified by POSIX, but > which makes little sense (but that other shells ended up > implementing for conformance). The background seems to be a bit more complex: sh -c ' : <&8; echo bla' # Solaris 11 Bourne Shell bla osh -c ' : <&8; echo bla' # Schily Solaris 11 emulation bla ksh -c ' : <&8; echo bla' # ksh88 ksh: 8: bad file unit number bosh -c ' : <&8; echo bla' bosh: 8: Falsche Dateinummer bash -c ' : <&8; echo bla' bash: 8: Falsche Dateinummer bla So: 1) before POSIX, ":" could not fail 2) bash ignores the "new" rules. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: Pushing/restoring a file descriptor for a compound command
Martijn Dekkerwrote: > I need references and opinions about the following, please. > > Consider: > > { > exec 8 ...stuff using file descriptor 8... > } 8<&- > > Should the effect of the 'exec' persist past the compound command (the > curly braces block)? > > My expectation is that the '8<&-' should push file descriptor 8 onto the > shell's internal stack in a closed state, so that it is restored at the > end of the block. The way it is implemented does not matter. > I think this should allow the effect of 'exec' to be local to that > compound command, and I think this construct should be nestable. OK, and the fact that this is supported by the historic Bourne Shell, ksh88 and ksh93 is a strung hint that this should also be in POSIX since it is something that looks a "natural" feature. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: Pushing/restoring a file descriptor for a compound command
On 4/27/18 9:53 PM, Martijn Dekker wrote: > That's what I've got. Is that a sane interpretation? > > It would be nice if there were something more unequivocal in the standard, > but it seems there isn't... > >> You might have more luck with bash (perhaps.) > > Chet, what do you think? There's nothing in Posix that specifies the behavior one way or another, so it's just an incompatibility between shells. I'll take a look. Chet -- ``The lyf so short, the craft so long to lerne.'' - Chaucer ``Ars longa, vita brevis'' - Hippocrates Chet Ramey, UTech, CWRUc...@case.eduhttp://tiswww.cwru.edu/~chet/
Re: Pushing/restoring a file descriptor for a compound command
2018-04-28 00:23:51 +0200, Martijn Dekker: [...] > That said, do you have any opinion on whether something like >{ ... ; } 3>&- > should push/restore a closed file descriptor if it's already closed, so that > the effect of exec-ing that descriptor within the compound command is local > to that compound command? [...] Actually, in my list of "issues" I plan to some day raise to the Austin Group, I had: sh -c '{ exec > a; } > b; echo x' > c which sounds like it's the same. Do we honour the compound command asking for stdout to be redirected, or do we honour the "> b" redirection being temporary and apply only to the command substitution? I'm undecided here, though I'd tend to lean the same way as you do. The problem can also be expressed as: redir_stdout() { exec > "$1" } redir_stdout file1 > file2 Or: close() { eval "exec $1>&-"; } close 1 > file f() { close 1 redir_stdout file echo foo close 1 } f > /dev/null POSIX could arbitrate based on the number of implementations that go one or the other way. Or leave it unspecified (I've not checked what it currently says).. -- Stephane
Re: Pushing/restoring a file descriptor for a compound command
Date:Sat, 28 Apr 2018 03:53:12 +0200 From:Martijn DekkerMessage-ID: <81b93245-e42f-ad62-4005-8ad676733...@inlv.org> | How does NetBSD sh handle this? This isn't really the best place for code samples, but ... "fd" is the file descriptor in question: if ((flags & REDIR_PUSH) && !is_renamed(sv->renamed, fd)) { INTOFF; if (big_sh_fd < 10) find_big_fd(); if ((i = fcntl(fd, F_DUPFD, big_sh_fd)) == -1) { switch (errno) { case EBADF: i = CLOSED; break; case EMFILE: case EINVAL: find_big_fd(); i = fcntl(fd, F_DUPFD, big_sh_fd); if (i >= 0) break; /* FALLTHRU */ default: i = errno; INTON;/* XXX not needed here ? */ error("%d: %s", fd, strerror(i)); /* NOTREACHED */ } } if (i >= 0) (void)fcntl(i, F_SETFD, FD_CLOEXEC); fd_rename(sv, fd, i); CLOSED is < 0 fd_rename() does (aside from bookkeeping) rl->orig = from; rl->into = to; The "sv" arg is the data struct where all this is saved (rl is alloc'd memory, linked to it). and then later, when things are being restored if (rl->into < 0) close(rl->orig); else movefd(rl->into, rl->orig); movefd() ends up translating into dup2() with error, and close-on-exec handling. The FreeBSD sh is similar, but simpler - they don't deal with user fd's >= 10, which makes the data structs needed simpler, and much easier to avoid user fd's stepping all over the shell's internal fds - we allow user fds to be anything the system allows, and the shell makes sure it moves its own fds around if needed to avoid issues. The EINVAL and EMFILE handling is dealing with the consequences of the user/script playing with ulimit, or at least as much as is possible. | If it's a bug, surely it would meet that requirement. Not for me to say, but it it doesn't have to be different (rather than just would be nicer if it was different) then it can be hard to justify making changes. kre
Re: Pushing/restoring a file descriptor for a compound command
Op 28-04-18 om 01:55 schreef Robert Elz: Date:Sat, 28 Apr 2018 00:23:51 +0200 From:Martijn DekkerMessage-ID: <8800d6d5-67ea-fad4-19c3-dac4bbfd8...@inlv.org> | That said, do you have any opinion on whether something like | { ... ; } 3>&- | should push/restore a closed file descriptor if it's already closed, I suspect that it is just a bug, the usual way to do this is saved_fd = fcntl(3, F_DUPFD, BIG_NUM); close(3); and later dup2(saved_fd, 3); close(saved_fd); Which works fine, provided fd 3 is open at the beginning, otherwise all of those fail with EBADF, which tends to just be ignored (and the dup2() failing leaves fd 3 with whatever it was set to in between.) It works fine on NetBSD sh though, and on every other ash derivative I know of except dash -- even for a file descriptor 3 that is closed at the beginning. How does NetBSD sh handle this? Fixing it means adding code - as I understand it, in dash, that is something they avoid unless it is absolutely essential If it's a bug, surely it would meet that requirement. - and if no-one can show where POSIX requires it, probably just won't happen. The closest thing I've found is the first sentence of POSIX 2.7 Redirection: "Redirection is used to open and close files for the current shell execution environment (see Shell Execution Environment) or for any command." The first part of that sentence refers to exec'ing it, so is irrelevant here. The relevant bit is "or for any command". This includes compound commands. I claim that this implies that a redirection added to a compound command should always cause the specified descriptor to be saved and restored, no matter the initial state of it, or the state initialised by the redirection. The alternative seems inherently broken: the effect of an 'exec 3 &- redirection for the same FD creates a reasonable expectation that it should restore that FD's state when leaving the compound command. Moreover, every current POSIX-compliant shell I know of works as I would expect, except bash and dash. The behaviour of dash appears to be unique for current ash derivatives, as FreeBSD sh, NetBSD sh, and Busybox ash also work as I would expect. I think that's further evidence that the behaviour of bash and dash should be considered a bug. That's what I've got. Is that a sane interpretation? It would be nice if there were something more unequivocal in the standard, but it seems there isn't... You might have more luck with bash (perhaps.) Chet, what do you think? Thanks, - M.
Re: Pushing/restoring a file descriptor for a compound command
Date:Sat, 28 Apr 2018 00:23:51 +0200 From:Martijn DekkerMessage-ID: <8800d6d5-67ea-fad4-19c3-dac4bbfd8...@inlv.org> | That said, do you have any opinion on whether something like | { ... ; } 3>&- | should push/restore a closed file descriptor if it's already closed, I suspect that it is just a bug, the usual way to do this is saved_fd = fcntl(3, F_DUPFD, BIG_NUM); close(3); and later dup2(saved_fd, 3); close(saved_fd); Which works fine, provided fd 3 is open at the beginning, otherwise all of those fail with EBADF, which tends to just be ignored (and the dup2() failing leaves fd 3 with whatever it was set to in between.) Fixing it means adding code - as I understand it, in dash, that is something they avoid unless it is absolutely essential - and if no-one can show where POSIX requires it, probably just won't happen. You might have more luck with bash (perhaps.) kre
Re: Pushing/restoring a file descriptor for a compound command
Op 27-04-18 om 23:38 schreef Stephane Chazelas: 2018-04-27 20:28:57 +0200, Martijn Dekker: [...] : <&8 || echo "oops, closed" [...] Remember ":" is a special builtin, so its failure causes the shell to exit. Another one of those accidents of implementation of the Bourne shell that ended up being specified by POSIX, but which makes little sense (but that other shells ended up implementing for conformance). Good point. Ignore the '|| echo "oops, closed"', it's pointless because a failed direction prints an error message anyway. I should have said that no shell produces an error there. That said, do you have any opinion on whether something like { ... ; } 3>&- should push/restore a closed file descriptor if it's already closed, so that the effect of exec-ing that descriptor within the compound command is local to that compound command? - M.
Re: Pushing/restoring a file descriptor for a compound command
2018-04-27 20:28:57 +0200, Martijn Dekker: [...] > : <&8 || echo "oops, closed" [...] Remember ":" is a special builtin, so its failure causes the shell to exit. Another one of those accidents of implementation of the Bourne shell that ended up being specified by POSIX, but which makes little sense (but that other shells ended up implementing for conformance). -- Stephane
Re: Pushing/restoring a file descriptor for a compound command
Op 27-04-18 om 18:49 schreef Martijn Dekker: The author of dash, Herbert Xu, said in response to my bug report that dash is not obligated to push and restore that file descriptor if it is already closed when entering the compound command -- implying that a file descriptor should not be pushed if the local state would be different from the parent state. Actually, that implication that I understood is not even correct. No shell, not even bash and dash, outputs "oops, closed" for the following: exec 8To rephrase my request: I would welcome opinions on whether the dash/bash behaviour shown in the original message should reasonably considered a bug in standards terms. Thanks, - M.
Pushing/restoring a file descriptor for a compound command
I need references and opinions about the following, please. Consider: { exec 8Should the effect of the 'exec' persist past the compound command (the curly braces block)? My expectation is that the '8<&-' should push file descriptor 8 onto the shell's internal stack in a closed state, so that it is restored at the end of the block. I think this should allow the effect of 'exec' to be local to that compound command, and I think this construct should be nestable. According to my testing, nearly all shells do this. However, two very widely used shells, dash and bash, leave the file descriptor open beyond the block. The author of dash, Herbert Xu, said in response to my bug report that dash is not obligated to push and restore that file descriptor if it is already closed when entering the compound command -- implying that a file descriptor should not be pushed if the local state would be different from the parent state. He asked me for a POSIX reference proving otherwise. I can't find any. Without this behaviour, an awkward workaround is required. To guarantee that the FD will be restored at the end of the block, you'd need to attempt to push it twice in two different states in two nested compound command blocks, for instance: { { exec 8/dev/null Does anyone have pointers to the POSIX text, or other strong evidence, that this workaround should not be necessary? Thanks, - M.