Re: How do I get the buffered bytes in a FILE *?

[Jilles Tjoelker via austin-group-l at The Open Group]

On Tue, Apr 12, 2022 at 10:42:02AM +0100, Geoff Clare via austin-group-l at The 
Open Group wrote:
> Rob Landley wrote, on 11 Apr 2022:
> > A bunch of protocols (git, http, mbox, etc) start with lines of data
> > followed by a block of data, so it's natural to want to call
> > getline() and then handle the data block. But getline() takes a FILE
> > * and things like zlib and sendfile() take an integer file
> > descriptor.

> > Posix lets me get the file descriptor out of a FILE * with fileno(),
> > but the point of FILE * is to readahead and buffer. How do I get the
> > buffered data out without reading more from the file descriptor?

> > I can't find a portable way to do this?

> I tried this sequence of calls on a few systems, and it worked in the
> way you would expect:

> fgets(buf, sizeof buf, fp);
> int fd = dup(fileno(fp));
> close(fileno(fp));
> while ((ret = fread(buf, 1, sizeof buf, fp)) > 0) { ... }
> read(fd, buf, sizeof buf);

> It relies on fread() not detecting EBADF until it tries to read more
> data from the underlying fd.

> It has some caveats:

> 1. It needs a file descriptor to be available.

> 2. The close() will remove any fcntl() locks that the calling process
>holds for the file.

> 3. In a multi-threaded process it has the usual problem around fd
>inheritance, but that's addressed in Issue 8 with the addition
>of dup3().

There is another dangerous problem: if another thread or a signal
handler allocates another fd and it is assigned the number fileno(fp),
the while loop might read data from a completely unrelated file. This
could be avoided by dup2/dup3'ing /dev/null onto fileno(fp) instead of
closing it (at the cost of another file descriptor).

> Also, for the standard to require it to work, I think we would need to
> tweak the EBADF error for fgetc() (which fread() references) to say:

> The file descriptor underlying stream is not a valid file
> descriptor open for reading and there is no buffered data
> available to be returned.

Although I don't expect it to break in practice, the close(fileno(fp))
or dup2(..., fileno(fp)) violates the rules about the "active handle" in
XSH 2.5.1 Interaction of File Descriptors and Standard I/O Streams.

I believe the "correct" solution with a stdio implementation that
doesn't offer something like freadhead() is not to use stdio but
implement own buffering.

-- 
Jilles Tjoelker

Re: POSIX gettext() and uselocale()

[Jilles Tjoelker via austin-group-l at The Open Group]

On Tue, May 04, 2021 at 01:07:39AM +0200, Bruno Haible via
austin-group-l at The Open Group wrote:
> https://posix.rhansen.org/p/gettext_split
> says (line 92):

>   "The returned string may be invalidated by a subsequent call to
>bind_textdomain_codeset(), bindtextdomain(), setlocale(),
>textdomain(), or uselocale()."

> While in most programs setlocale(), textdomain(), bindtextdomain(),
> bind_textdomain_codeset() are being called at the beginning of the
> program execution, before any call to gettext(), the situation is
> very different for uselocale().

> 1) uselocale() is meant to have effects ONLY on the thread in which it
>is called.

> 2) uselocale() is a helper function to implement *_l functions where
>the POSIX standard does not specify them or the system does not have
>them.
>For example, when a program wants to have a function to parse
>a number, recognizing only the ASCII digits and only '.' as decimal
>separator, a reliable way to implement such a function is by calling
>uselocale of the "C" locale, strtod(), and then uselocale() again
>to switch the thread back to the previous locale.

>If POSIX did not have uselocale(), it would need to provide many
>more *_l functions.

> If the gettext() result may be invalidated by a uselocale() call (in
> any other thread!), this would mean that

>   ** Programs can use gettext() or uselocale() but not both. **

> and - more or less -

>   ** Multithreaded programs that use libraries (that may use uselocale())
>  cannot use gettext(). **

> I think that specifying gettext() to be so restricted is not useful.
> It would make more sense to allow concurrent uselocale() calls.

> Proposed wording:

>   "The returned string may be invalidated by a subsequent call to
>bind_textdomain_codeset(), bindtextdomain(), setlocale(),
>or textdomain()."

This may be a bit too weak. Now the implementation can never free a
string that was returned by a gettext call on a thread with uselocale()
active, while logically the string may be owned by the locale and could
be freed if that locale is no longer set on any thread and freelocale()
has been called on it as needed.

-- 
Jilles Tjoelker

Re: [1003.1(2016/18)/Issue7+TC2 0001418]: Add options to ulimit to match get/setrlimit()

[Jilles Tjoelker via austin-group-l at The Open Group]

On Tue, Nov 17, 2020 at 03:14:43PM +, Geoff Clare via austin-group-l
at The Open Group wrote:
> Or I could just go with my original suggestion of adding:

> Conforming applications shall specify each option separately; that is,
> grouping option letters (for example, −fH) need not be recognized by
> all implementations.

> to my proposal.

Unfortunately, even that will not be enough. A -H or -S option or the
end-of-options argument "--" may go between the resource option and the
limit value. For example, the following three commands
  ulimit -c -S 1
  ulimit -c -- 1
  ulimit -c -S -- 1
comply to the amended proposed specification, but do not work as
expected in bash (they seem to write the -c limit to stdout and ignore
the operand).

Of course, there is no need for the end-of-options argument, but
consistency with other utilities suggests it should be allowed.

So perhaps additionally:
  Conforming applications shall not use the -- argument to indicate the
  end of options and shall not place the -S and -H options after options
  indicating resources.

These are violations of Guidelines 10 and 11 from 12.2 Utility Syntax
Guidelines.

One might also wonder whether the language of options and operands is
still worth the trouble at this point.

-- 
Jilles Tjoelker

Re: [1003.1(2016/18)/Issue7+TC2 0001418]: Add options to ulimit to match get/setrlimit()

[Jilles Tjoelker via austin-group-l at The Open Group]

On Mon, Nov 09, 2020 at 03:07:43PM +, Geoff Clare via austin-group-l
at The Open Group wrote:
> The ksh and bash behaviour of reporting multiple values seems more
> useful to me, but I wouldn't object if others want to make this
> unspecified.

With bash, reporting multiple values does not work if the options are
grouped into a single argument:

% bash -c 'ulimit -fn' 
bash: line 0: ulimit: n: invalid number
% bash -c 'ulimit -f -n'
file size   (blocks, -f) unlimited
open files  (-n) 231138

With ksh93, both these commands work as expected.

Similarly, commands like  ulimit -fH  do not work in bash. It must be
-Hf, -H -f or -f -H.

I'm testing with
GNU bash, version 5.0.18(3)-release (amd64-portbld-freebsd12.1)
and
  version sh (AT Research) 93u+ 2012-08-01

-- 
Jilles Tjoelker

Re: ksh93 job control behaviour [was: Draft suggestion: Job control and subshells]

[Jilles Tjoelker]

On Wed, Jul 29, 2020 at 09:41:50PM +0700, Robert Elz wrote:
> Date:Wed, 29 Jul 2020 09:45:55 +0100
> From:Geoff Clare 
> Message-ID:  <20200729084555.GA27947@localhost>

>   | It's only easy because (most/all?) shells take the easy option and do
>   | a lexical analysis of the command to be substituted.

> Thanks Harald for the analysis of the way that shells make this work.
> The mechanism they use isn't important to me (nor should it be to
> this group) all that matters is that it is done (because not doing
> so would be absurd).

> The NetBSD shell (and FreeBSD also I believe) don't do lexical analysis
> either, and yet it is still easy to implement.

FreeBSD sh chooses lexical analysis to make things like $(jobs) and
$(trap) work as expected.

How this works internally is that command substitutions containing a
single simple command do not fork, but implement the requirements for a
subshell environment directly (for example by saving and restoring any
saved shell variables). If the simple command turns out to be a builtin
utility, the shell checks whether this invocation (after resolving
'command') can be executed in the same process safely and forks anyway
if not. In the case of trap, invocations like 'trap', 'trap --' and
'command -p trap' are executed without a fork but 'trap "echo hi" EXIT'
is executed in a child process.

The trap builtin itself need not know about this scheme and can simply
manipulate the current process's traps.

> That said:

>   | Applications can't expect the following to work,
>   | but if the feature was implemented "properly", it would:

>   | showalltraps() { trap -p; }
>   | alltraps=$(showalltraps)

> Thanks for pointing that out, while we don't do lexical analysis.
> that doesn't currently work in the NetBSD shell, as I'm being rather
> too aggressive in doing the actual reset - when any command ie executed
> (which isn't trap, eval trap, or  command trap (or eval command trap.
> or eval command eval trap (you get the point)) causes the traps to really
> get reset, even a function call, but when I think about it, that's not
> really necessary, it should be possible to call a function which then
> does "trap" (or trap -p).   (In each of these, if the command does turn
> out to be trap, but it is one of the variants which sets traps, the
> reset happens before the command is performed.   The jobs command
> works just the same way, set +o (and set -o) (with no option name0
> could do the same (it currently isn't needed as we don't change
> any options when creating a subshell ... but as Jilles said, that's
> probably not really the best way.)

There is a slight difference between jobs and trap in that it is
definitely undesirable to execute the parent's trap handlers if the
child receives a signal. So trap shows, in certain cases, the parent's
traps which are really different from the child's. On the other hand, it
is simple to leave the parent's jobs in the data structures until the
child creates the first job of its own. The only additional magic is to
avoid calling waitpid() or similar until that point (to avoid strange
cases with reused process IDs).

-- 
Jilles Tjoelker

Re: ksh93 job control behaviour [was: Draft suggestion: Job control and subshells]

[Jilles Tjoelker]

On Mon, Jul 27, 2020 at 12:01:43PM +0100, Geoff Clare wrote:
> Martijn Dekker  wrote, on 24 Jul 2020:
> > But ksh 93u+ 2012-08-01 does not do this either (note the extra '; :' is
> > needed to avoid optimising the subshell away when using '-c'):

Aside, a case could be made that this optimization is not valid if job
control is enabled (FreeBSD sh also optimizes this case, and then also
does job control in the subshell running in the main shell process,
while it does not do job control in a child process.)

> > $ /bin/ksh -c 'set -m; (/bin/sleep 2 & ps -o pid,pgid,command); :'
> >   PID  PGID COMMAND
> > 24570 24570 ksh -c set -m; (/bin/sleep 2 & ps -o pid,pgid,command); :
> > 24571 24570 ksh -c set -m; (/bin/sleep 2 & ps -o pid,pgid,command); :
> > 24572 24570 /bin/sleep 2

> > However, an "interactive" shell (-i flag, which implies -m) does
> > create a process group:

> > $ /bin/ksh -i -c '(/bin/sleep 2 & ps -o pid,pgid,command); :'
> > [1] 24636
> >   PID  PGID COMMAND
> > 24634 24634 /bin/ksh -i -c set -m; (/bin/sleep 2 & ps -o pid,pgid,command);
> > exit
> > 24635 24635 /bin/ksh -i -c set -m; (/bin/sleep 2 & ps -o pid,pgid,command);
> > exit
> > 24636 24635 /bin/sleep 2

> > So, would you say that ksh93's behaviour with just -m is broken (as
> > in POSIX non-compliant) and needs fixing?

> Yes, it needs fixing.  The description of the set -m option clearly says
> "All jobs shall be run in their own process groups".

Job control is certainly useful in non-interactive shells, so that part
seems good to fix. My envisioned way of using it surrounds some
background commands with set -m ... set +m, so new process groups are
only created when this is expected.

However, for subshells the standard's behaviour may be less useful than
ksh93's here. Since the process group mechanism does not support
nesting, setting the process group ID in the subshell will cause the
grandchild process to leave the child process's job. This can be useful
in specific cases but I do not think it is a good default behaviour,
especially if the processes have a controlling terminal. If the
grandchild process stops, the child process will probably not deal with
it well. Scripts would likely have to 'set +m' in subshells to deal with
it.

A comparable mechanism that does work is starting an interactive job
control shell from another. In this case, the child shell will insist it
is in the foreground (otherwise it will stop itself via a signal until
it is) and will take over the terminal while it is in foreground.

When disabling job control in subshells, it is still possible to leave
the process group by starting a new shell instance and enabling job
control in that.

With the benefit of hindsight, it would probably be better if a subshell
implicitly turned off -m, so it can be turned on again. However,
subshells implicitly modifying shell options seems rather unusual.

-- 
Jilles Tjoelker

Re: Weird possibility with async processes, $!, and long running scripts

[Jilles Tjoelker]

- just only once.

I agree that WNOWAIT is not helpful here because the same child process
may be returned over and over.

What is necessary here is a notification mechanism for process
termination that is not a wait*() function. Such a mechanism does not
seem to exist in POSIX but exists on various operating systems:

* fully queued SIGCHLD with siginfo (works on FreeBSD but not Linux)
* kqueue with EVFILT_PROC (various BSD systems)
* proc connector (Linux)
* whatever pwait(1) uses (Solaris and related systems)

By the way, most of these mechanisms also allow waiting for an unrelated
process to terminate.

> Of course, both of these "solutions" mean keeping zombies in the kernel
> process table - that's the point, as that prevents the kernel from
> re-using the process ID.

Yes, although this keeping is only necessary for the most recent
background process or if $! has been referenced for it.

> Or:

> Every time the shell forks, before running any of the subshell code,
> it could check whether the PID it was assigned is a PID that is still "live"
> in the jobs table, and if so, it simply exits without doing anything.
> Simultaneously the parent is doing the same check using the new child's
> PID.   Since the two are simply forks() of the parent, the data structures
> they see are identical - both child and parent will answer that check the
> same way.   When the check reports "still in use" the child simply exits
> (as mentioned). the parent simply does a waitpid(PID, ...) to clean up
> that child (without ever having entered it into any data structs) and
> then forks again, and the whole process repeats.

> This is the solution I see with most promise, but relies upon the kernel
> not simply assigning the same pid over and over again (even if there happens
> to only be one available unused pid to assign).   To deal with this the
> parent shell would need something like a counter of attempts, and if we
> fail to get a new pid after a few attempts, give up, and signal a fork error.

Reuse can be prevented by delaying the waitpid() on the unwanted
duplicates until a process with a unique PID has been created or a limit
has been reached. In the general case this information can be stored as
a flag in the previous job structure, so it does not allocate unbounded
memory in userspace.

> This looks kind of cumbersome and ugly to me - even though I don't
> currently see any other plausible solution to this, that meets our goals.

> I'd love to hear from anyone who has (or can even imagine, regardless of
> whether it is currently implemented anywhere) a better solution for this
> issue.   Or if for some reason I am not understanding this isn't even a
> potential (certainly it is extremely unlikely) problem, then why.

> ps: note that we don't currently have a problem with the kernel assigning
> the pid of a previously exited process, which is still alive in the
> jobs table, the shell can cope with that - the issue only arises when that
> pid is communicated to the script, and then used by the script.   A similar
> problem would be if the script attempted

>   kill $PID1

> after bg-process-1  has finished (without the script realising that)
> which then ends up signalling $PID2 (the same thing) which is still
> running.   Of course, a similar problem can happen here, without PID2
> being involved - with the script simply signalling some unintended process.
> The only way of avoiding that would be to keep the zombies until the
> script has been made aware that the process is completed, after which it
> is simply a script bug if it tries to kill a process it knows is already
> complete.

A possible fix would be to add a magic variable that returns the most
recent background job's identifier in % form, somewhat like $!
in that referencing it causes the job to be remembered. Scripts would
need to use the new variable instead of $!.

-- 
Jilles Tjoelker

Re: Exporting an unset variable

[Jilles Tjoelker]

On Sun, Jul 07, 2019 at 03:33:14PM +0200, Martijn Dekker wrote:
> On FreeBSD sh, when you set the export attribute on an otherwise unset
> variable, an empty environment variable is exported, though the variable
> continues to be considered unset within the current shell. This is the only
> shell I know of that does this.

> $ unset -v foo
> $ export foo
> $ echo ${foo+set}
> [empty line]
> $ sh -c 'echo ${foo+set}'
> set
> $ set | grep ^foo=
> $ env | grep ^foo=
> foo=

> Question: is this behaviour a bug in POSIX terms?

> As far as I can tell, the POSIX spec doesn't explicitly say what to do in
> this case:
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_22
> | The shell shall give the export attribute to the variables
> | corresponding to the specified names, which shall cause them to be in
> | the environment of subsequently executed commands.

> I could see how this phrasing technically requires the FreeBSD sh behaviour,
> but given that no other shell does this, it seems unlikely this requirement
> is intended. Could the text do with clarifying this?

As FreeBSD sh's maintainer, I decided that this behaviour is a bug and
changed it to be in line with other shells for FreeBSD 13.0:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=233545

Given that this has been broken for so long, all FreeBSD 11.x and 12.x
versions will not receive this fix (unless some circumstance makes it
crucial to have it).

-- 
Jilles Tjoelker

Re: EINTR on 'wait' clarification

[Jilles Tjoelker]

On Thu, Jan 17, 2019 at 12:02:44AM +0200, Volodymyr Boyko wrote:
> According to 
> http://pubs.opengroup.org/onlinepubs/009695399/functions/wait.html:

> > The wait() function shall fail if:
> > [ECHILD]
> > The calling process has no existing unwaited-for child processes.
> > [EINTR]
> > The function was interrupted by a signal. The value of the location
> > pointed to by stat_loc is undefined.

> So from my understanding of above, if we are blocked in 'wait' and
> receiving a signal (SIGCHLD), the call should return with value -1 and
> errno set to EINTR.
> Though the following snippet behaves differently on linux 4.15.0-43
> and freebsd 11.2:

> > #include 
> > #include 
> > #include 
> > #include 
> > #include 
> > #include 
> > #include 
> > #include 
> > static void sigchld_handler(int signo)
> > {
> > const char *const msg = "\nSIGCHLD";
> > (void)signo;
> > write(STDERR_FILENO, msg, strlen(msg));
> > }
> > int main(void)
> > {
> > pid_t wait_ret = 0, pid = 0;
> > int status = 0, ret = -1;
> >
> >
> >
> > sigaction(SIGCHLD, &(struct sigaction){
> >.sa_handler = sigchld_handler,
> > }, NULL);
> >
> >
> >
> > if ((pid = fork()) < 0)
> > {
> >perror("\nfork: ");
> >goto Exit;
> > }
> > if (!pid)
> > {
> >sleep(3);
> >return 0;
> > }
> > if ((wait_ret = wait()) < 0)
> > {
> >perror("\nwait: ");
> >goto Exit;
> > }
> > fprintf(stderr, "\nwait done, pid %d", wait_ret);
> > ret = 0;
> > Exit:
> > fprintf(stderr, "\n");
> > return ret;
> > }

> on freebsd it shows:

> > SIGCHLD
> > wait: Interrupted system call

This is clearly compliant.

> and on linux:

> > SIGCHLD
> > wait done, pid 

I'm assuming that the parent process can execute fast enough so that it
is actually blocked in wait() when the child process terminates.

Taking into account the text in the wait page about deleting pending
SIGCHLD signals, this can only be compliant if _POSIX_REALTIME_SIGNALS
is not defined or if SIGCHLD is not queued. Perhaps you can make Linux
queue SIGCHLD by installing a SA_SIGINFO handler instead of a
traditional one.

If SIGCHLD is not queued, the siginfo_t information is only useful for
debugging.

> So I would like to clarify if the behaviour of code is really
> implementation defined - if no, which behaviour is expected by POSIX.
> To me it seems more like a nonconformity in the linux, taking into
> account linux's man -e 7 signal (which is obviously based on POSIX
> spec):

> > If a signal handler is invoked while a system call or library function call 
> > is blocked, then either:
> >
> >   * the call is automatically restarted after the signal handler 
> > returns; or
> >
> >   * the call fails with the error EINTR.

> > If a blocked call to one of the following interfaces is interrupted by a 
> > signal handler, then the call will be automatically restarted after the 
> > signal  handler  returns  if  the
> >   SA_RESTART flag was used; otherwise the call will fail with the error 
> > EINTR:

> >  * wait(2), wait3(2), wait4(2), waitid(2), and waitpid(2).

I think there is one more compliant result of your test program: only
the "wait done, pid " line, if the wait function succeeds
before noticing the signal; the pending SIGCHLD signal is discarded
("accepted").

-- 
Jilles Tjoelker

Re: [1003.1(2008)/Issue 7 0000789]: Add set -o pipefail

[Jilles Tjoelker]

On Tue, Sep 04, 2018 at 08:36:09PM +, Austin Group Bug Tracker wrote:
> -- 
>  (0004102) kre (reporter) - 2018-09-04 20:36
>  http://austingroupbugs.net/view.php?id=789#c4102 
> -- 
> What has not been considered here is the operation of pipefail
> in an asynchronous pipeline.

> In particular, at what instant is the setting of the pipefail option
> tested.

> I see three possibilities (which all amount to the same thing for
> foreground pipelines, so it does not matter there)

> 1) the pipefail option setting when the pipeline is created (and/or
> perhaps
> the setting for each command in the pipeline when it is started).   This
> allows
>   set -o pipefail
>   my | pipe | line &  P=$!
>   set +o pipefail

> # and then later

>   wait $P

> # and here $? is set to the status of the pipeline, using the pipefail
> option,
> even though it is not enabled when the wait is done.

> 2) the status when the script waits for the status of the pipeline (when
> the
> status is made available to the script).   This allows

>   my | pipe | line &  P=$!

> # and later

>   set -o pipefail; wait $P; X=$?; set +o pipefail

> and X is the status from the pipeline, with the pipefail option set
> (regardless of its value when the pipeline was created, or at any
> intervening time).

> 3) The status of the pipefail option is tested as the shell detects that
> each process of the pipeline is done (has exited), so in

> [snip]

> The first option is the most useful I believe, but I also suspect not
> implemented anywhere - it allows the shell to collect the status as exch
> process in the pipeline exits, and also provides predictable behaviour to
> the application.   Pity that no-one (that I know of) implements that.

I noticed this problem as well. ksh93 (93u+ 2012-08-01 from FreeBSD
ports) implements the first option, and so does my uncommitted version
in FreeBSD sh. Most other shells implement the second option. I think
the first option is the right way because it makes the exit statuses
from  A | B &  and  (A | B) &  match.

> I added pipefail to the NetBSD sh (ash derivative) a while ago, it is
> in the NetBSD 8 release shell - that implementation uses scheme 2
> above for dealing with async pipelines (because being different, and
> implementing (1) did not seem useful, when no-one else does, and (3),
> while simple, is useless.)   We have no arrays (and I hope, never
> will) - there is currently no way to determine which process in the
> pipeline produced the status that is returned when the pipefail option
> is set.   My impression is that most applications which would want to
> use it are content to simply discover "they all worked" and are
> prepared to write the script in a way to avoid inadvertent
> SIGPIPE/EPIPE exit issues (eg: using "sed -e '2,$d'" or "sed -n -e 1p"
> rather than "head -n1" or "sed -e 1q").   Alternatively the script can
> insert another process whose purpose is to insulate an earlier process
> from EPIPE by always reading until EOF, and writing to its stdout,
> (just like cat with no args) but ignoring any write errors (or at
> least, ignoring EPIPE and also ignoring SIGPIPE ... and always doing
> exit 0.)

Another option is to change the SIGPIPE status to 0 in possibly
problematic pipe elements.

> In general I support the pipefail option.   I do not think it is required
> that everything that could possibly be useful as an adjunct be defined at
> the same time - if making the status of the processes in a pipe available
> is useful, and arrays are not to be the solution, then something else
> should be implemented and made available before anyone worries about
> specifying how to do it.   Similarly, we do not need to invent ways to
> deal with pipes closing - let the applications work on it, and if they
> need shell assistance, ask for what they actually need.

The pipefail option is a small bit of code in the shell to make scripts
much simpler, but a similar effect can be obtained using a complicated
function that only needs to be written once.

> But the operation of the option needs to be fully defined (even if that
> includes "is unspecified" for some things.) 

Yes.

-- 
Jilles Tjoelker

Re: Should shell quoting within glob bracket patterns be effective?

[Jilles Tjoelker]

On Tue, Apr 10, 2018 at 01:41:25PM +0100, Martijn Dekker wrote:
> Re: https://github.com/att/ast/issues/71

> Consider this test script:

> (set -o posix) 2>/dev/null && set -o posix
> emulate sh 2>/dev/null  # for zsh
> for var in 'a-c' '!a'; do
>   case b in
>   ( ["$var"] )echo 'quirk' ;;
>   ( [$var] )  echo 'no quirk' ;;
>   esac
> done

> Most shells output 'no quirk' for both values of 'var', but AT ksh93 
> outputs 'quirk' for both, as does zsh 5.2 and earlier (zsh-as-sh changed 
> to match the majority in 5.3). Now one of the current ksh93 lead 
> developers says this does not look like a bug.

> Does POSIX specify anything, either way, regarding the effect of shell 
> quoting within glob bracket patterns? I can't find any relevant text 
> under "2.13 Pattern Matching Notation" or anything it references, so 
> clarification would be appreciated.

The first paragraph of 2.13.1 Patterns Matching a Single Character
contains some confusing or contradictory text about backslashes; this
text was amended for http://austingroupbugs.net/view.php?id=806 but was
confusing or contradictory even before that change. The change was made
for fnmatch() and perhaps the part about backslashes in the first
paragraph was actually meant to handled in the last paragraph in the
part that explicitly says it is only about contexts such as fnmatch()
where shell quote removal is not performed.

The rest of 2.13.1 discusses "quoting" of characters in various
locations. I think it is reasonable to assume that shell quoting is
meant. Only the effect of quoting '!', '-' and ']' in a bracket
expression is not specified (but the effect of quoting '^' is: it makes
the '^' a literal part of the set).

I prefer "no quirk" twice as output but it is indeed not fully
specified.

-- 
Jilles Tjoelker

Re: quoting in shell parameter expressions

[Jilles Tjoelker]

On Tue, Jun 20, 2017 at 04:02:10PM +0200, Joerg Schilling wrote:
> Geoff Clare  wrote:

> > Joerg Schilling  wrote, on 20 Jun 2017:

> > > I would like to get a confirmation on how this expression:

> > >   "${xxx-"a b c"}"

> > > is to be understood.

> > This was discussed when we worked on bug 221, and is made explicitly
> > unspecified by the resolution of that bug (to be applied in Issue 8).

> > See http://austingroupbugs.net/view.php?id=221#c399

> OK, thank you.

> The question came up after I have been send a list of corner cases
> from Robert Elz that lists shell parameter expansions that created
> problems with bosh.

> While most of the corner cases could be correctly identified on how to
> expand them, the four cases he included with quoting, could not easily
> be identified on how to understand them.

> I'll try to ask Glenn Fowler and David Korn whether they could give an
> explanation on why they decided to use the current implementation in
> ksh93.

In FreeBSD sh I have implemented similar behaviour. For the + - = ?
varieties of parameter expansion, this is almost the same as what the
original Almquist shell did. Note that this differs from most other
Almquist derivatives such as dash, busybox ash and NetBSD sh which have
a more bash-like behaviour. DragonflyBSD sh is much like FreeBSD sh.

My reasoning is that this has a degree of consistency and allows doing
some things easily that are otherwise hard (for example, expand a
variable literally if it is set and generate pathnames otherwise using
"${v-"*"}"). With the bash-like behaviour there is almost never any
point in using double-quotes in a quoted parameter expansion of the + -
= ? varieties.

The underlying idea is that a double-quote toggles between quoting
states and the word in ${param+-=?word} inherits the quoting state of
the outer context. Therefore, a quoted parameter substitution can have
unquoted parts. In the original Bourne shell, this allowed postponing
special treatment of $ { } until expansion time. Now that ${param#word}
exists which has its own quoting state for the inner word, that is no
longer possible. This was also taken advantage of to allow things like
your example "${xxx-"a b c"}"; the original Bourne shell will show the
behaviour for cases like "${xxx-"a${s}b${s}c"}" (where s is set to an
IFS character) or "${xxx-"*"}".

A further change is that ksh93 and FreeBSD sh treat a closing brace
literally if the quoting state differs from the outer context, so that
  echo "${x-A"}"B}"
prints A}B and not AB}. This simplifies maintenance of the quoting state
in the expansion phase.

In FreeBSD sh, I also treat a here-document as a separate kind of
quoting that is inherited, so that

cat <

Re: "-" operand to "sh"

[Jilles Tjoelker]

On Wed, Jun 07, 2017 at 01:52:57PM +0200, Joerg Schilling wrote:
> Jilles Tjoelker <jil...@stack.nl> wrote:

> > In interactive mode, job control (-m) is enabled automatically. Some
> > shells, such as FreeBSD sh, dash, mksh and heirloom-sh-050706, allow
> > starting an interactive shell without job control using sh +m, while
> > other shells, such as bash and ksh93, do not (the option is ignored).

> After Stephane pointed to the POSIX standard that includes the same
> text as the ksh man page, I implemented to auto-enable -m in the POSIX
> Bourne Shell (bosh) for interactive shells.

> While doing this, I discovered that the Burne Shell first parses the
> command line options and later decides whether the current shell is
> interactive. For this reason, I do not see that it would be possible
> to use "sh +m" in a useful way.

It is possible to make a distinction between options that are false
because they are explicitly turned off and options that are false
because they have not been mentioned.

> BTW: given that heirloom-sh-050706 is a mostly unmodified Bourne Shell
> from OpenSolaris, it also first parses the options and later decides
> whether it is interactive. So even if it did auto-enable (it does not)
> -m,

>   heirloom-sh-050706 +m

> would be a no-op with respect to the monitor flag.

In my observation it is not a no-op (check the PGID fields):

jilles@lion /home/jilles% jsh 
$ ps jT
USERPID PPID PGID  SID JOBC STAT TT TIME COMMAND
jilles 1501 1499 1501 15010 Ss7  0:00.06 zsh
jilles 1505 1501 1505 15011 S 7  0:00.00 jsh
jilles 1506 1505 1506 15011 R+7  0:00.00 ps jT
$ %
jilles@lion /home/jilles% jsh +m
$ ps jT
USERPID PPID PGID  SID JOBC STAT TT TIME COMMAND
jilles 1501 1499 1501 15010 Ss7  0:00.06 zsh
jilles 1507 1501 1507 15011 S+7  0:00.00 jsh +m
jilles 1508 1507 1507 15011 R+7  0:00.00 ps jT
$ %
jilles@lion /home/jilles%

This is heirloom-sh-050706 from the FreeBSD package, which has only one
source code patch, to turn off shell accounting.

-- 
Jilles Tjoelker

Re: sh(1): is roundtripping of the positional parameter stack possible? (Was: Re: Shell parameter expansions involving '#")

[Jilles Tjoelker]

On Tue, May 16, 2017 at 07:41:43AM +0100, Stephane Chazelas wrote:
> 2017-05-16 10:03:56 +0700, Robert Elz:
> > Date:Mon, 15 May 2017 18:36:58 +0200
> > From:Steffen Nurpmeso <stef...@sdaoden.eu>
> > Message-ID:  <20170515163658.b7ljs%stef...@sdaoden.eu>
> [...]
> > Alternatively, you could implement this as an external #! script, that
> > would be a lot slower, but would at least avoid that issue.

> Or just write it as quote() (...) instead of quote() { ...;}

> > quote() {
> > case "$1" in
> > *\'*)   ;;   # the harder case, we will get to below.
> > *)  printf "'%s'" "$1"; return 0;;
> > esac
> > 
> > _save_IFS="${IFS}" # if possible just make IFS "local"
> > _save_OPTS="$(set +o)"  # quotes there not really needed.
> > IFS=\'
> > set -f
> > set -- $1
> > _result_="${1}"# we know at least $1 and $2 exist, as there
> > shift  # was one quote in the input.
> > 
> > for __arg__
> > do
> > _result_="${_result_}'\\''${__arg__}"
> > done
> > printf "'%s'" "${_result_}"
> > 
> > # now clean up
> > 
> > IFS="${_save_IFS}"  #none of this is needed with a good 
> > "local"...
> > eval "${_save_OPTS}"
> > unset __arg__ _result_ _save_IFS _save_OPTS
> > 
> > return 0;
> > }
> [...]

> That doesn't work properly in POSIX shells. In AT ksh, $IFS is
> Internal Field Delimiter (or Terminator, not "Separator") for
> splitting (as opposed to  for "$*")

> That was fixed in pdksh, zsh and the Almquist shell, but
> unfortunately in the mean time, POSIX specified the AT ksh way
> (later, some variants of ash and pdksh have changed back to the
> POSIX way).

> So in POSIX shells both a''b' and a''b are split into ("a", "",
> "b") so that quote() would quote both to 'a'\'''\''b'

This is easily fixed by adding a quoted empty string:
set -- $1''

If $1 ends with an IFS character, you get a final empty field.
If $1 does not end with an IFS character, the empty string does nothing.

-- 
Jilles Tjoelker

Re: SIGCHLD trap in a shell

[Jilles Tjoelker]

On Tue, Apr 25, 2017 at 03:21:24PM +0200, Vincent Lefevre wrote:
> I wonder how the following script is supposed to behave.

> The POSIX spec seems ambiguous about SIGCHLD trap in a shell. It says:
> "[...] a signal specified using a symbolic name, without the SIG prefix,
> as listed in the tables of signal names in the  header defined
> in XBD Headers"; there, SIGCHLD is listed as "Child process terminated,
> stopped, [XSI] or continued". From this spec, I would assume that the
> trap would be executed for any child process that terminates, stops, or
> continues. But this is not what I observe.

I would not assume that. Since trap actions are not executed while
waiting for a foreground process to exit, it is likely that multiple
instances of SIGCHLD will be coalesced into a single trap action
execution.

Also, SIGCHLD instances may be lost if waitpid() or similar calls
implicitly accept them. This is most likely if the shell calls waitpid()
with SIGCHLD masked but might also happen in a race otherwise.

> Only bash, ksh93 and zsh document what happens concerning SIGCHLD traps.

> Note: For practical reasons, I assume a sleep implementation with
> a floating-point number; this is the case with /bin/sleep under
> Debian (coreutils). It could be any external command with a similar
> behavior.

> 
> #!/bin/sh
> 
> n=${1:-30}
> n=$((n+0))
> 
> c=0
> trap 'c=$((c+1))' CHLD
> 
> for i in `seq $n`; do /bin/sleep 0.1 & done
> 
> while true
> do
>   echo $c
>   /bin/sleep 0.1
> done
> 

> I call this script with:

>

> Under Debian unstable:

> * With ash, dash, mksh and yash, I get:

> 1
> 2
> 3
> 4
> 5
> 6
> [...]

> and each time, $c is increased by 1.

> The +1 comes from the "/bin/sleep 0.1" of the "while" loop ("true"
> and "echo" being built-ins). If I replace them by external commands
> (/bin/true and /bin/echo), I get +3 each time.

> So, I get +1 only for commands that are not executed in background
> (asynchronously).

You can get the SIGCHLD trap action executed for background jobs by
doing slow things that do not fork, such as invoking the read utility.

FreeBSD sh behaves like this as well, with the addition of a check that
prevents nested SIGCHLD traps.

> * With bash and posh, I get:

> 0
> 0
> 0
> 0
> 0
> 0
> [...]

> i.e. no trap executions at all.

> If I add "set -m", then bash behaves like ash, etc., except that it
> no longer reacts to Ctrl-C (the Ctrl-C only interrupts the sleep).
> Anyway, "set -m" is not supposed to be used in shell scripts (see
> rationale for the "set" built-in).

"set -m" can be used usefully in scripts to put jobs into their own
process groups, like
  set -m; something & set +m

This example turns -m mode off immediately, since handling suspension of
foreground jobs is usually undesirable in a script.

Whether this works well depends on the shell implementation.

In any case it seems strange to tie SIGCHLD behaviour to "set -m".

> * With ksh93 and zsh, I get:

> 0
> 30
> 30
> 30
> 30
> 30
> [...]

> (or with a 29 just after the 0 -- this is just a timing reason),
> i.e. a trap execution only for asynchronous lists.

> This corresponds to the behavior of ksh88 as mentioned in the
> rationale for the "set" built-in. Is this the original intent?
> In any case, I think that this is the most useful behavior.

That seems more useful, but there is no text in POSIX that says to
implement it that way.

-- 
Jilles Tjoelker

Re: [1003.1(2013)/Issue7+TC1 0001016]: race condition with set -C

[Jilles Tjoelker]

On Thu, Oct 20, 2016 at 04:40:36PM +, Austin Group Bug Tracker wrote:
> 
> A NOTE has been added to this issue. 
> == 
> http://austingroupbugs.net/view.php?id=1016 
> == 
> [snip]
> -- 
>  (0003446) geoffclare (manager) - 2016-10-20 16:40
>  http://austingroupbugs.net/view.php?id=1016#c3446 
> -- 
> (Response to http://austingroupbugs.net/view.php?id=1016#c2991)

> The FreeBSD sh method described here does not behave correctly in the case
> where the initial stat() fails and then open() with O_EXCL fails, and the
> file that was created in between was not a regular file (e.g. a FIFO).  It
> needs to open the file (without O_CREAT or O_EXCL) and use fstat() to see
> if it is a regular file.

> You then have the problem of what to do if this open() fails. 

I can reproduce this with many shells (FreeBSD sh, bash 4.4.0 in both
non-POSIX and POSIX mode, yash 2.30, ksh93 93u+ 2012-08-01).

My reproducer consists of two processes:
sh -c 'while :; do ln -s /dev/null testf 2>/dev/null; done'
sh -c 'set -C; while true >testf; do rm testf; done'

After a few minutes, the redirection fails with File exists or a similar
error.

With zsh (5.2 in default mode), this command can run for more than half
an hour without failing. This is because zsh does not do an initial
stat(), but first tries an open() with O_EXCL and failing that an open
without O_CREAT or O_EXCL and an fstat() check that it is not a regular
file.

This approach fixes your problem and is slightly simpler, but there is
now a problem when a non-regular file is deleted between the two open()
calls.

NetBSD sh has a slightly different approach, trying without O_CREAT or
O_EXCL first (with an fstat() check that it is not a regular file) and
using O_CREAT | O_EXCL if the initial open fails. This appears to have
the same issue as the FreeBSD sh method.

I think the most important part of this bug report is that > with set -C
will open a regular file exclusively. Almost all shells seem to
implement this part correctly (although they did not 10 years ago).

-- 
Jilles Tjoelker