POSIX character classes (was Re: pipes and sub-shells)

[Thorsten Glaser]

Martijn Dekker dixit:

>* BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
>[expressions] are not supported in glob patterns.

I really really REALLY hate that this will make mksh really big.
We’re talking about 36K .rodata even without titlecase conversion
and BMP-only (16-bit Unicode) here.

Can I get by making them match ASCII only even in UTF-8 mode?

Strictly speaking, POSIX requires only support for the C locale,
and our UTF-8 mode is only close to POSIX anyway, and currently
(though this will change, there have been good points made for
locale tracking) enabled using a mksh-specific set flag.

If I implemented that, we could then say that “lksh -o posix”
is, in the C locale, fully POSIX conformant. (Perhaps — but
certainly a goal to work for, even despite the uselessness
of standards.)

bye,
//mirabilos
-- 
(gnutls can also be used, but if you are compiling lynx for your own use,
there is no reason to consider using that package)
-- Thomas E. Dickey on the Lynx mailing list, about OpenSSL

Re: pipes and sub-shells

[Dan Douglas]

On Mar 23, 2017 10:16 AM, Martijn Dekker  wrote:
> Op 23-03-17 om 15:44 schreef Dan Douglas:
> > On 03/23/2017 04:49 AM, Jean Delvare wrote:
> >> Apparently it requires a more recent version of mksh than we are
> >> shipping:
> >
> > Herestrings are old. Should work in just about any ksh.
>  pdksh (including OpenBSD ksh) doesn't have them.

Ew

> On those (as on POSIX), you need a here-document to get the same effect:
>
> while stuff; do
> stuff
> done < $(cmd)
> EOF
>
> >> $ echo $KSH_VERSION @(#)MIRBSD KSH R50 2014/06/29 openSUSE $
> >> ./test_return.sh ./test_return.sh[10]: syntax error: '('
> >> unexpected
> >>
> >> But indeed works find with the latest upstream version, thanks
> >> for the pointer.
> >
> > It's also not at all equivalent to a pipe. The shell reads the
> > command output from a pipe, stripping trailing newlines, and dumps
> > the result into a temporary file, which then adds a trailing
> > newline, then reads the file back into whatever runs in the loop.
>
> In most use cases, that seems like a distinction without a difference.

It matters when there are zero, or more than one newlines at the end. Besides 
that, it just has no advantage over `out=$(cmd); printf %s -- "$out" | ...', or 
better, `read -rd "" out' to avoid mangling the data. In all cases we can't 
begin processing the loop until we've read the entire possibly large output, so 
I don't suggest that method unless the intention is to take advantage of its 
side-effects.

> > It will also fail if it expands to more than one word due to lack
> > of quotes. (That likely caused your error)
>
> That is true on bash, but not on mksh. I should probably have added
> the quotes anyway though, just to avoid that confusion.

I like that feature. Exclusive to mksh though.

Re: pipes and sub-shells

[Martijn Dekker]

Op 23-03-17 om 10:49 schreef Jean Delvare:
> Apparently it requires a more recent version of mksh than we are
> shipping:
> 
> $ echo $KSH_VERSION
> @(#)MIRBSD KSH R50 2014/06/29 openSUSE

That version is quite ancient, so you should consider upgrading it to
the latest. FYI, modernish 
currently detects the following bugs on it that are relevant for
cross-shell programming. All except BUG_LNNOALIAS, BUG_LNNOEVAL and
BUG_NOCHCLASS have been fixed in the current release. The former two are
fixed in current cvs. The latter is a design decision from Thorsten that
is nonetheless a bug in POSIX terms.

* BUG_CMDPV: 'command -pv' does not find builtins.

* BUG_CMDSPEXIT: preceding a special builtin with 'command' does not
stop it from exiting the shell if the builtin encounters an error.

* BUG_CMDVRESV: 'command -v' does not find reserved words such as "if".

* BUG_LNNOALIAS: $LINENO is always expanded to 0 when used within an alias.

* BUG_LNNOEVAL: $LINENO is always expanded to 0 when used in 'eval'.

* BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
[expressions] are not supported in glob patterns.

* BUG_PP_01: POSIX says that empty "$@" generates zero fields but empty
'' or "" or "$emptyvariable" generates one empty field. This means
concatenating "$@" with one or more other, separately quoted, empty
strings (like "$@""$emptyvariable") should still produce one empty
field. With this bug, this erroneously produces zero fields.

* BUG_PP_02: Like BUG_PP_01, but with unquoted $@ and only with
"$emptyvariable"$@, not $@"$emptyvariable".

* BUG_PP_03: Assigning the positional parameters to a variable using
either var=$* or var="$*" or both doesn't work as expected, using either
default, empty, unset or custom settings of $IFS.

* BUG_PP_04: Like BUG_PP_03, but for a default assignment within a
parameter substitution, i.e. ${var=$*} or ${var="$*"}.

* BUG_SELECTRPL: In a 'select' loop, input that is not a menu item is
not stored in the REPLY variable as it should be.

* BUG_TESTERR0: test/[ exits successfully (exit status 0) if an invalid
argument is given to an operator.

Hope this helps,

- M.

Re: pipes and sub-shells

[Martijn Dekker]

Op 23-03-17 om 15:44 schreef Dan Douglas:
> On 03/23/2017 04:49 AM, Jean Delvare wrote:
>> Apparently it requires a more recent version of mksh than we are 
>> shipping:
> 
> Herestrings are old. Should work in just about any ksh.

They're a ksh93-ism, not a ksh88-ism. pdksh (including OpenBSD ksh)
doesn't have them.

On those (as on POSIX), you need a here-document to get the same effect:

while stuff; do
stuff
done <> $ echo $KSH_VERSION @(#)MIRBSD KSH R50 2014/06/29 openSUSE $
>> ./test_return.sh ./test_return.sh[10]: syntax error: '('
>> unexpected
>> 
>> But indeed works find with the latest upstream version, thanks
>> for the pointer.
> 
> It's also not at all equivalent to a pipe. The shell reads the
> command output from a pipe, stripping trailing newlines, and dumps
> the result into a temporary file, which then adds a trailing
> newline, then reads the file back into whatever runs in the loop.

In most use cases, that seems like a distinction without a difference.

> It will also fail if it expands to more than one word due to lack
> of quotes. (That likely caused your error)

That is true on bash, but not on mksh. I should probably have added
the quotes anyway though, just to avoid that confusion.

- M.

Re: pipes and sub-shells

[Dan Douglas]

On 03/23/2017 04:49 AM, Jean Delvare wrote:
> Apparently it requires a more recent version of mksh than we are
> shipping:

Herestrings are old. Should work in just about any ksh.

> $ echo $KSH_VERSION
> @(#)MIRBSD KSH R50 2014/06/29 openSUSE
> $ ./test_return.sh
> ./test_return.sh[10]: syntax error: '(' unexpected
> 
> But indeed works find with the latest upstream version, thanks for the
> pointer.

It's also not at all equivalent to a pipe. The shell reads the command
output from a pipe, stripping trailing newlines, and dumps the result
into a temporary file, which then adds a trailing newline, then reads
the file back into whatever runs in the loop. It will also fail if it
expands to more than one word due to lack of quotes. (That likely
caused your error)

There are a few legitimate uses for <<<"$(cmd)", which mostly have to
do with wanting that temp file for some reason, but typically don't use
it without good reason.




signature.asc
Description: OpenPGP digital signature

Re: pipes and sub-shells

[Jean Delvare]

Hi Thorsten,

On mer., 2017-03-22 at 19:39 +, Thorsten Glaser wrote:
> Martijn Dekker dixit:
> 
> >Op 22-03-17 om 10:12 schreef Jean Delvare:
> >> Concretely, the customer's code looks like this:
> >>
> >> command | while read line
> >> do
> >>  if 
> >>  then
> >>  exit
> >>  fi
> >>  process $line
> >> done
> >[...]
> >> I was wondering if there is any other trick you can suggest that would 
> >> work in mksh?
> 
> This is actually a FAQ ;-) Use co-processes!
> 
> command |& while read -p line; do …

Looks good, thank you very much :-)

-- 
Jean Delvare
SUSE L3 Support

Re: pipes and sub-shells

[Jean Delvare]

Hi Martijn,

On mer., 2017-03-22 at 10:26 +0100, Martijn Dekker wrote:
> Op 22-03-17 om 10:12 schreef Jean Delvare:
> > 
> > Concretely, the customer's code looks like this:
> > 
> > command | while read line
> > do
> > if 
> > then
> > exit
> > fi
> > process $line
> > done
> [...]
> > 
> > I was wondering if there is any other trick you can suggest that would work 
> > in mksh?
> 
> A here-string with a command substitution:
> 
> while read line
> do
>   if 
>   then
>   exit
>   fi
>   process $line
> done <<<$(command)

Apparently it requires a more recent version of mksh than we are
shipping:

$ echo $KSH_VERSION
@(#)MIRBSD KSH R50 2014/06/29 openSUSE
$ ./test_return.sh
./test_return.sh[10]: syntax error: '(' unexpected

But indeed works find with the latest upstream version, thanks for the
pointer.

Thanks,
-- 
Jean Delvare
SUSE L3 Support