Re: '[[' regression in R59

[Martijn Dekker]

Op 05-05-20 om 22:35 schreef Thorsten Glaser:

Martijn Dekker dixit:


As of mksh-R59 there is a regression in glob pattern matching with '[['.


Thanks, fixed.


Verified. :)


I’m too tired to release R59b tonight, so if you have got any other
bugs/regressions to go into it before I do so tomorrow or so… do tell.
You seem to have more weird-corner-case test coverage than I do.


Nope, all my other tests pass.

- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

'[[' regression in R59

[Martijn Dekker]

As of mksh-R59 there is a regression in glob pattern matching with '[['. 
Backslash-escaping of a * passed from a variable appears to be ignored.


$ mksh-R58 -c 'v=a\\*e; [[ abcde == $v ]]';echo $?
1
$ mksh-R59 -c 'v=a\\*e; [[ abcde == $v ]]';echo $?
0   <== bug

ksh93, bash, yash and zsh all behave like mksh R58 and earlier.

Testing the git repo revealed the regression was introduced by this commit:

| commit 24e61ee166cf50c7c1fcaf58d7393b2e6e557c93
| Author: tg 
| Date:   Tue Apr 7 23:14:44 2020 +
|
| implement full extglob pattern matching on [[ x = $y ]] RHS
|
| eliminates some eval and brings us closer to ksh93

--
modernish -- harness the shell
https://github.com/modernish/modernish

IFS + ${var#foo} bug

[Martijn Dekker]

$ mksh -c 'v="foo!one!two!three"; IFS="!"; printf %s\? ${v#foo}; echo'
??one?two?three?

Expected output (as on all non-mksh shells):
?one?two?three?

So field-splitting ${var#foo} on non-whitespace IFS creates an extra 
initial empty field.


(mksh R49 and earlier had a similar bug with whitespace IFS; as of R50 
it only occurs with non-whitespace IFS.)


--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: [Bug 1857702] Re: " +=" operator does string concatenation for integer variables

[Martijn Dekker]

Op 29-12-19 om 23:55 schreef Thorsten Glaser:

As discussed heavily on IRC, other shells can use ((…)) or let to work
like mksh, and the mksh behaviour is semantically correct.


Your position makes no sense, because in mksh (and ksh93, bash, zsh) you 
can do things like:


$ typeset -i i
$ i=4+4; echo $i
8
$ i=i+1; echo $i
9

...yet, in mksh:

$ i+=1; echo $i
91

That is clearly inconsistent. 'typeset -i' makes shell assignments 
interpret arithmetic expressions (as is documented in the manual page), 
and '+=' is just another form of shell assignment, therefore it should 
be changing the behaviour of '+=' as well as '='.


ksh93, bash and zsh all act correctly here. mksh is a ksh clone and 
should be acting like ksh and all the others that have emulated ksh.


- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

Re: [Bug 1857826] [NEW] mksh ASAN heap-buffer-overflow

[Martijn Dekker]

Op 29-12-19 om 19:20 schreef Fernando Muñoz:

When compiling mksh with ASAN and running [[ -v $XX ]] ($XX being an
undefined environment variable) mksh will crash.


Of course it shouldn't crash, but it's worth noting that the correct 
form is [[ -v XX ]] without the dollar sign (signifying the expansion of 
the variable, not the variable itself).


With the expansion of a nonexistent variable, you're effectively testing 
[[ -v '' ]] which I would guess is probably what triggers the crash.


- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

recent CVS: 'command .' permanently overrides PPs

[Martijn Dekker]

One of the recent commits introduced a new bug: if a dot script is 
sourced with 'command .', the positional parameters are permanently 
overridden by whatever they were at the point of sourcing, and any 
changes to them are ignored.


$ cat >test.mksh
set -- one two
shift
printf '[%s]\n' "$#" "$@"
$ mksh -c 'command . ./test.mksh' dummy oh dear this is not good
[6]
[oh]
[dear]
[this]
[is]
[not]
[good]

--
modernish -- harness the shell
https://github.com/modernish/modernish

[bug] within 'eval', -e/-o errexit appears active, but is inactive

[Martijn Dekker]

I noticed something strange while executing some 'eval'-ed commands: the 
-e/-o errexit appears to become active out of nowhere.


$ mksh -c 'echo $-; eval '\''echo $-'\''; echo $-'
hc
ehc
hc

But it looks like it's appearances only; it's not actually effective.

$ mksh -c 'eval '\''echo $-; false; echo phantom e'\'
ehc
phantom e

I tested mksh versions down to R49 and they all have this bug. However, 
its ancestor pdksh does not have this bug.


However, both pdksh and mksh render 'set -e' ineffective for eval'ed 
commands, which is a bug in itself; no other shell including ksh93 
shares this behaviour. Thus the expected output for the last command 
above is 'hc' and nothing else.


Thanks,

- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish

[PATCH] return status 126 on failure to execute

[Martijn Dekker]

Currently, when a utility fails to execute due to something like E2BIG, 
mksh sets the exit status to 1. That is not satisfactory because there 
is no way to distinguish this error by exit status from a normal 1 for 
false/no match/etc. from utilities such as grep.


Generally accepted convention is to return status 126 in such cases, as 
done by ksh93, bash, yash, dash (current git), FreeBSD sh, and NetBSD 
sh. The attached patch makes that happen.


Test script:

v=foo_bar_baz_quux_lorem_ipsum_dolor_sit_amet
while :; do
v=$v$v
sh -c ":" -- "$v" || { echo "status $?"; exit; }
done

Current output:

longargs.sh[8]: /bin/sh: Argument list too long
status 1

Expected output:

longargs.sh[8]: /bin/sh: Argument list too long
status 126

Thanks,

- M.

--
modernish -- harness the shell
https://github.com/modernish/modernish
diff --git a/exec.c b/exec.c
index 8231b54..01135c4 100644
--- a/exec.c
+++ b/exec.c
@@ -446,7 +446,7 @@ execute(struct op * volatile t,
if (rv == ENOEXEC)
scriptexec(t, (const char **)up);
else
-   errorf(Tf_sD_s, t->str, cstrerror(rv));
+   errorfx(126, Tf_sD_s, t->str, cstrerror(rv));
}
  Break:
exstat = rv & 0xFF;

[Bug] 'command readonly/export' exit on error

[Martijn Dekker]

The 'readonly' and 'export' commands exit the shell on error even if 
they are prefixed with 'command', which should stop that exit from 
happening.


Test cases:

$ mksh -o posix -c 'readonly v; command export v=foo || echo ok'
mksh: read-only: v
$ mksh -o posix -c 'readonly v; command readonly v=foo || echo ok'
mksh: read-only: v

('ok' should be printed in both cases)

Thanks,

- M.

Re: Bug#910276: mksh does never execute an "EXIT trap", if it is created with the "trap" command in a sub shell

[Martijn Dekker]

The bug concerned the EXIT pseudosignal. But, apparently, mksh does not 
execute subshell-specific traps for real signals either:


$ mksh -c '(trap "echo int_subsh" INT; kill -s INT $BASHPID); echo end'
end
$ bash -c '(trap "echo int_subsh" INT; kill -s INT $BASHPID); echo end'
int_subsh
end

Same with other signals, such as TERM.

POSIX specifies that traps are cleared in subshells, but not that new 
traps should not be executed.


(background note for other readers: in both mksh and bash, but in no 
other shell that I know of, $BASHPID denotes the PID of the current 
subshell, as opposed to $$ which always refers to the main shell)


- M.

Re: ${foo#'bar'} problem in here-document

[Martijn Dekker]

Op 09-03-18 om 02:28 schreef Thorsten Glaser:
> Naturally. However, the “bug” seems to only apply to
> trim operations,

Yes, that is expected because trim operations use glob pattern notation
(as in 'case'). Shell quoting is explicitly part of glob pattern
notation, so that you can escape glob characters, etc.

Simple parameter substitutions such as ${foo+'bar'} do not use glob
pattern notation, so the quotes aren't parsed by the substitution.
(Quote removal may still be done *after* processing the substitution,
but that depends on the context and doesn't happen in here-documents.)

> dash (which fails 4 and 5),

Yes, it has the same bug as mksh, in fact it came up on the dash list so
I tested other shells and found it on mksh (and pdksh) as well.

Dash and pdksh are expected to fail 5, though, as they don't support
$'this' quoting style.

- M.

${foo#'bar'} problem in here-document

[Martijn Dekker]

x=notOK
cat <

Re: 'exec' runs shell functions and builtins

[Martijn Dekker]

Op 08-08-17 om 00:49 schreef Thorsten Glaser:
> I completely disagree, especially when combined with other wording.

Well, Geoff Clare did say the existing wording is broken in any case.

If the goal is compatibility with pre-existing practice (in this case,
ksh88) and the majority of other shells which follow it, then the exact
wording is irrelevant anyway.

> But, well, POSIX was never among the more useful standards…

Since you seem to despise it with such a passion, I honestly wonder why
you haven't simply ripped out POSIX compatibility mode a long time ago.
Are there compelling reasons why you have to keep bothering with it?

- M.

Re: 'exec' runs shell functions and builtins

[Martijn Dekker]

Op 26-07-17 om 00:41 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> This behaviour also appears to be contrary to the documentation in
>> mksh(1) ("The command is executed without forking, replacing the shell
>> process"). The test script below demonstrates that neither "exec"
> 
> And this is utter nonsense, the builtin does replace the shell process,

No. On mksh, 'exec builtin' is exactly equivalent to 'builtin; exit'.
The builtin is run in the existing shell process. No
replacing/overlaying of the shell process takes place. It's easy to
verify this in the source code.

- M.

Re: 'exec' runs shell functions and builtins

[Martijn Dekker]

Op 25-07-17 om 23:41 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> This behaviour also appears to be contrary to the documentation in
>> mksh(1) ("The command is executed without forking, replacing the shell
>> process"). The test script below demonstrates that neither "exec"
> 
> And this is utter nonsense, the builtin does replace the shell process,

And under what theory should 'exec' run a shell function?

- M.

[BUG] 'exec' runs shell functions and builtins

[Martijn Dekker]

In mksh (and pdksh), 'exec' looks up shell functions and builtins before
external commands, and if it finds one it appears to do the equivalent
of running the function or builtin followed by 'exit'. This turns out[1]
to be a bug in POSIX terms; 'exec' is supposed to launch a program that
overlays the current shell[2], implying the program launched by 'exec'
is always external to the shell.

This means that
(exec commandname arguments ...)
is a POSIXly correct way of guaranteeing the execution of an external
command without specifying the path.

In pdksh this is definitely a bug as this behaviour differs from ksh88
which pdksh is supposed to be a clone of. The POSIX spec is also based
on ksh88 behaviour.

This behaviour also appears to be contrary to the documentation in
mksh(1) ("The command is executed without forking, replacing the shell
process"). The test script below demonstrates that neither "exec"
replaces the shell process as the first executes a shell function and
the second (within the function) executes a mksh builtin.

testFn() {
exec print "this shell execs both functions and builtins"
}
PATH=/dev/null
exec testFn

Expected output: something like "testFn: not found".
Actual output: "this shell execs" etc.

- M.

[1] https://www.mail-archive.com/austin-group-l@opengroup.org/msg01469.html

[2]
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_20_14

Re: R55: -DMKSH_BINSHREDUCED ineffective

[Martijn Dekker]

Op 16-04-17 om 01:24 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> When building lksh R55, adding -DMKSH_BINSHREDUCED to the CFLAGS no
> 
> CPPFLAGS.

Done that, no difference.

Also, that is not clear from mksh.htm which says to add it to "the
command line".

'truste,

- M.

R55: -DMKSH_BINSHREDUCED ineffective

[Martijn Dekker]

When building lksh R55, adding -DMKSH_BINSHREDUCED to the CFLAGS no
longer seems to build a reduced-size binary. It's identical in size to
mksh now and seems to have the full set of interactive editing features.

(I don't have time to hunt for the cause at the moment so for now I'm
simply reporting it.)

- M.

Re: bug: can't break/continue from within eval

[Martijn Dekker]

Op 08-04-17 om 02:08 schreef Chet Ramey:
> On 4/7/17 7:46 PM, Thorsten Glaser wrote:
>> Chet Ramey dixit:
>>
 'break' and 'continue' are POSIX "special builtins", meaning if they
 fail they should cause the shell to exit, at least in POSIX mode.
>>>
>>> Posix explicitly makes this case unspecified.
>>
>> Indeed, I just found this out as well, thanks.
>>
>> Ib m curiousb & is the 'break' in 'eval break' contained within the
>> compound-list of the do loop? (Ib ll probably make this work as
>> requested nevertheless as I consider it correct, butb & meh, those
>> standardsb &)
> 
> Yeah, I think it is.

Agreed; there is nothing to suggest otherwise. 'eval' executes its
arguments in the current shell environment in place of itself. So after
parsing those arguments, executing them should be equivalent to 'eval'
not being there at all.

- M.

Re: bug: can't break/continue from within eval (plus, status 0 on error)

[Martijn Dekker]

Op 08-04-17 om 00:36 schreef Chet Ramey:
> On 4/7/17 6:56 PM, Martijn Dekker wrote:
> 
>> A second thing:
>>
>> $ mksh -o posix -c 'break; echo $?'
>> mksh: break: can't break
>> 0
>> $ mksh -o posix -c 'continue; echo $?'
>> mksh: continue: can't continue
>> 0
>>
>> 'break' and 'continue' are POSIX "special builtins", meaning if they
>> fail they should cause the shell to exit, at least in POSIX mode.
> 
> Posix explicitly makes this case unspecified.

So it does. My bad.

Remains the question whether the current behaviour is actually sensible.

- M.

bug: can't break/continue from within eval (plus, status 0 on error)

[Martijn Dekker]

$ mksh -o posix -c 'for x in 1 2 3; do eval "break"; done'
mksh: break: can't break
mksh: break: can't break
mksh: break: can't break
$ mksh -o posix -c 'for x in 1 2 3; do eval "continue"; done'
mksh: continue: can't continue
mksh: continue: can't continue
mksh: continue: can't continue

The "break"/"continue" executed within the loop, so this should work (as
it does on every other shell except pdksh).

Actual use case, as requested: one of my shell functions contains a
loop, within which there is a 'case' construct wrapped in 'eval' so it
can use a variable referenced by another variable, and it needs to be
able to conditionally break out of the loop based on its value. Since
mksh won't tolerate the 'break' within the 'eval', I need to make the
code more awkward by moving the 'break' outside of the 'case' within the
'eval' while still keeping it conditional.

A second thing:

$ mksh -o posix -c 'break; echo $?'
mksh: break: can't break
0
$ mksh -o posix -c 'continue; echo $?'
mksh: continue: can't continue
0

'break' and 'continue' are POSIX "special builtins", meaning if they
fail they should cause the shell to exit, at least in POSIX mode.

And in this case I can see good reasons for that. A failing
'break'/'continue' is a strong indicator that the program is in an
inconsistent state where continuing would make no sense and may very
well be harmful.

In non-POSIX mode, it's your decision whether they should exit or not,
but they should at the very least return a nonzero exit status on error.

Thanks,

- M.

Re: alias expansion bug with matching { ... }

[Martijn Dekker]

Op 04-04-17 om 02:26 schreef Thorsten Glaser:
> That’s no excuse… pdksh does simple search for a matching )
> in command substitutions, failing $(case x in y) ;; esac ).

Yeah, I've run into that. It even parses quote marks in comments within
command substitutions, requiring them to be balanced (so an "it's" in a
comment would cause a syntax error pinpointed twelve zillion lines
further on). bash 2 did the same thing. So did ksh88, I'm pretty sure...

> I think “we” agreed somewhere (maybe here, maybe in IRC, maybe
> on the Austin ML) that doing weird things to the shell syntax
> with aliases need not really be supported. So, please, do tell
> me whether you *really* have a *legit* use case for… that.

Define "legit" and "weird" (and explain how they're mutually exclusive). ;-)

What I'm doing is this:
https://github.com/modernish/modernish#use-varsetlocal

So no, this does not really need to be supported in command
substitutions or any other subshells. But I do report every little
oddity I find, sometimes they're indicative of bugs.

- M.

alias expansion bug with matching { ... }

[Martijn Dekker]

Given two aliases:

$ alias OPEN='{' CLOSE='};'

This works fine:

$ { OPEN echo hi; CLOSE }
hi

And this:

$ var=`{ OPEN echo hi; CLOSE }` && echo "$var"
hi

But this does not:

$ var=$({ OPEN echo hi; CLOSE }) && echo "$var"
mksh: syntax error: ')' unexpected

It works on every other shell, even pdksh.

Thanks,

- M.

Re: [preliminary patch] test -v VARNAME

[Martijn Dekker]

Op 26-03-17 om 01:40 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> This patch adds a unary -v operator to test/[/[[ to check if a variable
>> is set, like in ksh93, bash and zsh.
> 
> Eh? Isn’t that the same as [[ -n ${var+x} ]] ?

Yes. Advantage being that -v allows testing a variable name stored in a
variable without using 'eval'. And simpler syntax.

Thanks for the pointers, I'll take a look in the next few days.

- M.

Re: POSIX character classes (was Re: pipes and sub-shells)

[Martijn Dekker]

Op 23-03-17 om 22:02 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> * BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
>> [expressions] are not supported in glob patterns.
> 
> I really really REALLY hate that this will make mksh really big.
> We’re talking about 36K .rodata even without titlecase conversion
> and BMP-only (16-bit Unicode) here.

I sympathise.

Even fnmatch(3) is not compliant on all systems; the BSDs don't seem to
have caught up yet. :(  I don't suppose using that is an option in any
case because of mksh's extended globbing functionality.

Is adding 36k really that much in 2017? On my system, the current
development binary of mksh is 283k after stripping when built with -O2,
235k with -Os. Adding 36k would make it 316k/271k, still quite small.

If that's too much, I guess you should continue to not support them. The
reason modernish detects BUG_NOCHCLASS is not to make some sort of
statement, but to enable programs using the library to easily check for
the presence of the issue and implement alternative methods (such as
falling back to external commands, or just matching ASCII only without
character classes).

> Can I get by making them match ASCII only even in UTF-8 mode?

IMHO, that would defeat their primary purpose, namely locale-dependent
class matching, so no, not really. :)

If Greeks or Russians (or Germans, for that matter) can't count on
[:upper:] matching an upper case letter in their alphabets, then I'd say
for them it would be better to have no support than broken support.

> Strictly speaking, POSIX requires only support for the C locale,
[...]

Yes, but on systems supporting other locales (e.g. UTF-8), it would not
be conforming for character classes to match ASCII only. You either
support UTF-8 or you don't.

- M.

Re: pipes and sub-shells

[Martijn Dekker]

Op 23-03-17 om 10:49 schreef Jean Delvare:
> Apparently it requires a more recent version of mksh than we are
> shipping:
> 
> $ echo $KSH_VERSION
> @(#)MIRBSD KSH R50 2014/06/29 openSUSE

That version is quite ancient, so you should consider upgrading it to
the latest. FYI, modernish 
currently detects the following bugs on it that are relevant for
cross-shell programming. All except BUG_LNNOALIAS, BUG_LNNOEVAL and
BUG_NOCHCLASS have been fixed in the current release. The former two are
fixed in current cvs. The latter is a design decision from Thorsten that
is nonetheless a bug in POSIX terms.

* BUG_CMDPV: 'command -pv' does not find builtins.

* BUG_CMDSPEXIT: preceding a special builtin with 'command' does not
stop it from exiting the shell if the builtin encounters an error.

* BUG_CMDVRESV: 'command -v' does not find reserved words such as "if".

* BUG_LNNOALIAS: $LINENO is always expanded to 0 when used within an alias.

* BUG_LNNOEVAL: $LINENO is always expanded to 0 when used in 'eval'.

* BUG_NOCHCLASS: POSIX-mandated character [:classes:] within bracket
[expressions] are not supported in glob patterns.

* BUG_PP_01: POSIX says that empty "$@" generates zero fields but empty
'' or "" or "$emptyvariable" generates one empty field. This means
concatenating "$@" with one or more other, separately quoted, empty
strings (like "$@""$emptyvariable") should still produce one empty
field. With this bug, this erroneously produces zero fields.

* BUG_PP_02: Like BUG_PP_01, but with unquoted $@ and only with
"$emptyvariable"$@, not $@"$emptyvariable".

* BUG_PP_03: Assigning the positional parameters to a variable using
either var=$* or var="$*" or both doesn't work as expected, using either
default, empty, unset or custom settings of $IFS.

* BUG_PP_04: Like BUG_PP_03, but for a default assignment within a
parameter substitution, i.e. ${var=$*} or ${var="$*"}.

* BUG_SELECTRPL: In a 'select' loop, input that is not a menu item is
not stored in the REPLY variable as it should be.

* BUG_TESTERR0: test/[ exits successfully (exit status 0) if an invalid
argument is given to an operator.

Hope this helps,

- M.

Re: pipes and sub-shells

[Martijn Dekker]

Op 22-03-17 om 10:12 schreef Jean Delvare:
> Concretely, the customer's code looks like this:
> 
> command | while read line
> do
>   if 
>   then
>   exit
>   fi
>   process $line
> done
[...]
> I was wondering if there is any other trick you can suggest that would work 
> in mksh?

A here-string with a command substitution:

while read line
do
if 
then
exit
fi
process $line
done <<<$(command)

Hope this helps,

- M.

Re: LINENO oddities

[Martijn Dekker]

Op 20-03-17 om 05:06 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> LINENO is set to zero during the execution of aliases or commands in
>> 'eval'. This behaviour is different from that of other shells.
> 
> Patches welcome ☺

OK, it turns out that LINENO==0 within alias expansion and LINENO==0
within 'eval' execution are actually two separate problems requiring
separate fixes.

The simple patch attached
- preserves LINENO for alias expansion, and
- makes LINENO within 'eval' act like ksh88, which is good enough for
me. I guess I'd prefer it to act like ksh93, *ash, yash and native zsh,
but I can't figure out how to make that happen.

> (What do you *use* LINENO for? I’ve *never* seen it used *at all*.)

Well, LINENO is POSIX so it ought to work in any case. But, since you
asked...

Modernish <https://github.com/modernish/modernish> includes the
var/setlocal module which provides a stack-based implementation of code
blocks with local variables and shell options. Internally it uses a
temporary shell function. It's basically equivalent to anonymous
functions in zsh, but (like the rest of modernish) works on all POSIX
shells.

It's based on two aliases like this:

alias setlocal='{ _Msh_sL_temp() { _Msh_doSetLocal "${LINENO-}"'

alias endlocal='} && { _Msh_sL_temp "$@"; _Msh_doEndLocal "$?"
"${LINENO-}"; }; }'

LINENO is passed to the internal handling functions for use in error
messages (such as incorrect arguments or stack corruption). On mksh,
those error messages always indicate line 0, because $LINENO is expanded
while expanding an alias.

>> Output on various shells:
>>mksh: 1 2 0 0 0 6
>>lksh: 1 2 0 0 0 6
> 
> Don’t forget “lksh -o posix”, which is likely more important
> to check than just lksh.

Same result.

Thanks,

- M.

Index: funcs.c
===
RCS file: /cvs/src/bin/mksh/funcs.c,v
retrieving revision 1.330
diff -u -r1.330 funcs.c
--- funcs.c	19 Mar 2017 20:59:50 -	1.330
+++ funcs.c	20 Mar 2017 18:06:44 -
@@ -2410,6 +2410,7 @@
 
 	savef = Flag(FERREXIT);
 	Flag(FERREXIT) |= 0x80;
+	s->line = current_lineno;
 	rv = shell(s, false);
 	Flag(FERREXIT) = savef;
 	source = saves;
Index: lex.c
===
RCS file: /cvs/src/bin/mksh/lex.c,v
retrieving revision 1.230
diff -u -r1.230 lex.c
--- lex.c	12 Mar 2017 02:04:39 -	1.230
+++ lex.c	20 Mar 2017 18:06:45 -
@@ -1064,6 +1064,7 @@
 s->u.tblp = p;
 s->flags |= SF_HASALIAS;
 s->next = source;
+s->line = source->line;
 if (source->type == SEOF) {
 	/* prevent infinite recursion at EOS */
 	source->u.tblp = p;

LINENO oddities

[Martijn Dekker]

LINENO is set to zero during the execution of aliases or commands in
'eval'. This behaviour is different from that of other shells.

#! /usr/bin/env mksh
alias showlineno='echo $LINENO'
eval 'echo $LINENO'
showlineno

Actual output:
0
0

Expected output:
3 (or 1)
4

This behaviour seems to be inherited from pdksh, but no other shell has it.

Several shells (dash, yash, zsh in native mode, ksh93) start counting
from 1 when executing an 'eval', as if they considered it a separate
script, so they output 1 for the eval. But all shells output 4 for the
alias.

Another test script:
printf "$LINENO "
printf "$LINENO "
eval '  printf "$LINENO "
printf "$LINENO "
printf "$LINENO " '
printf "$LINENO\n"

Output on various shells:
mksh: 1 2 0 0 0 6
lksh: 1 2 0 0 0 6
   pdksh: 1 2 0 0 0 6
  AT ksh88: 1 2 3 3 3 6
  AT ksh93: 1 2 1 2 3 6
bash: 1 2 5 6 7 6  (?!)
  FreeBSD sh: 1 2 1 2 3 6
dash: 1 2 1 2 3 6
yash: 1 2 1 2 3 6
zsh (native): 1 2 1 2 3 6
zsh (sh): 1 2 3 3 3 6  (like ksh88)

(Looks like I may also have to tell bug-bash that the bash behaviour is
weird, and zsh-workers that zsh's sh mode is not emulating any current sh).

HTH,

- M.

Re: Unsetting local variables

[Martijn Dekker]

Op 17-03-17 om 23:08 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> Op 17-03-17 om 20:53 schreef Thorsten Glaser:
>>> Even mksh’s “global” builtin does not access the global scope.
>>> It’s simply a “typeset” that’s not also “local”.
>>
>> But why the separate builtin (which is unique to mksh, if I'm not
>> mistaken) rather than 'typeset -g' as in zsh, bash 4 and yash?
> 
> Oh. I was not aware of that.
> 
> I think that’s sufficient to deprecate “global” even. Thanks!

Neat. :)

One nitpick with the new man page phrasing though:

| Create parameters in global scope, not in local scope.

That's not quite accurate, as was also explained by the old explanation
for the 'global' command which was patched out: typeset -g "does not
allow a function called from another function to access a parameter at
truly global scope, but only prevents putting an accessed one into local
scope." IOW, it may still inherit the enveloping local scope from a
calling function.

- M.

Re: Unsetting local variables

[Martijn Dekker]

Op 17-03-17 om 20:53 schreef Thorsten Glaser:
> Even mksh’s “global” builtin does not access the global scope.
> It’s simply a “typeset” that’s not also “local”.

But why the separate builtin (which is unique to mksh, if I'm not
mistaken) rather than 'typeset -g' as in zsh, bash 4 and yash?

Please consider adding 'typeset -g' as a synonym for 'global' for those
of us who target multiple shells:

Index: funcs.c
===
RCS file: /cvs/src/bin/mksh/funcs.c,v
retrieving revision 1.327
diff -u -r1.327 funcs.c
--- funcs.c 12 Mar 2017 02:31:26 -  1.327
+++ funcs.c 17 Mar 2017 21:42:39 -
@@ -784,7 +784,7 @@
}

/* see comment below regarding possible opions */
-   opts = istset ? "L#R#UZ#afi#lnprtux" : "p";
+   opts = istset ? "L#R#UZ#agfi#lnprtux" : "p";

builtin_opt.flags |= GF_PLUSOPT;
/*
@@ -829,6 +829,9 @@
case 'f':
func = true;
break;
+   case 'g':
+   localv = false;
+   break;
case 'i':
flag = INTEGER;
basestr = builtin_opt.optarg;
Index: mksh.1
===
RCS file: /cvs/src/bin/mksh/mksh.1,v
retrieving revision 1.427
diff -u -r1.427 mksh.1
--- mksh.1  12 Mar 2017 02:35:19 -  1.427
+++ mksh.1  17 Mar 2017 21:42:41 -
@@ -4803,7 +4803,7 @@
 .Pp
 .It Xo
 .Ic global
-.Oo Op Ic +\-alpnrtUux
+.Oo Op Ic +\-aglnprtUux
 .Op Fl L Ns Op Ar n
 .Op Fl R Ns Op Ar n
 .Op Fl Z Ns Op Ar n
@@ -4815,7 +4815,7 @@
 .Xc
 .It Xo
 .Ic typeset
-.Oo Op Ic +\-alpnrtUux
+.Oo Op Ic +\-aglnprtUux
 .Op Fl LRZ Ns Op Ar n
 .Op Fl i Ns Op Ar n
 .No \*(Ba Fl f Op Fl tux Oc
@@ -4880,6 +4880,11 @@
 .It Fl f
 Function mode.
 Display or set functions and their attributes, instead of parameters.
+.It Fl g
+Disable the local scope for parameters.
+Equivalent to using the
+.Ic global
+command.
 .It Fl i Ns Op Ar n
 Integer attribute.
 .Ar n

Re: [feature] tilde expansion on arbitrary arguments ending in unquoted =~

[Martijn Dekker]

Op 09-03-17 om 16:50 schreef Thorsten Glaser:
> Martijn Dekker dixit:
> 
>> Then the documentation is not consistent with the intended behaviour. If
>> it's intended that any argument ending in an unquoted sequence =~, given
>> to any command, is subject to tilde expansion, the documentation should
>> describe exactly that behaviour.
> 
> Good point. I’ll update the manpage to be more explicit.
> 
> The Korn Shell (although I’ve not recherched whether this also applies
> to AT ksh88 and/or ksh93)

I checked both of these yesterday, and neither of them do this; it
appears to be a pdksh-ism only.

>  has historically enabled tilde expansion
> after an unquoted (at least I hope!) equals sign,

Yes, I verified it only happens if both the = and the ~ are unquoted.

>  and tab completion
> also after an unquoted colon.
> 
> I’ll try to put that into manpage wording RSN.
> 
> Do note that tilde stuff has become more expand-y recently due to a
> change for better POSIX conformance, too. See:
> 
> http://www.mirbsd.org/cvs.cgi/src/bin/mksh/dot.mkshrc.diff?r1=1.102;r2=1.103
> 
> I wonder what POSIX has to say about your tilde issue…

The tilde expansion spec is here:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_01

For assignments (not including arguments to 'typeset' and the like that
may look like assignments but for the purposes of shell grammar are no
such thing), tildes following the '=' and any ':' are expanded.
Presumably this is to allow for things like
PATH=~/bin:~/sbin:...

For non-assignment arguments (which would include arguments to 'export',
'typeset' and the like that may look like assignments but for the
purposes of shell grammar are no such thing), tilde expansion is only
provided for at the beginning of each argument (meaning, it's not
provided for at all in those assignment-like arguments).

- M.

[bug] tilde expansion on arbitrary arguments ending in unquoted =~

[Martijn Dekker]

pdksh and mksh inappropriately do tilde expansion on arbitrary arguments
that happen to end with the two unquoted characters "=~". This should
only happen for real shell assignments (i.e. not even with
assignment-like arguments to "export" and the like).

Command:
echo "some arbitrary stuff "=~

Actual output is like:
some arbitrary stuff =/Users/martijn

Expected output:
some arbitrary stuff =~

- M.

[BUG] ${var=$*} with null IFS is broken

[Martijn Dekker]

Given

set -- one "two three" four
unset -v var
IFS=# null IFS

the unquoted expansion

${var=$*}

generates three words and assigns the last one ("four") to $var.

It should generate one word, consisting of the concatenation of the
positional parameters separated by the first character, without
separation since IFS is null, and assign that concatenated word to the
variable.

Test script:

set -- one "two three" four
unset -v var
IFS=
set -- ${var=$*}
echo "var=$var"
printf '[%s]\n' "$@"

Actual output:

var=four
[one]
[two three]
[four]

Expected output:

var=onetwo threefour
[onetwo threefour]

Thanks,

- M.

typeset -l/-u only support ASCII

[Martijn Dekker]

Quoting mksh.1 under 'typeset':
> -l  Lower case attribute.  All upper case characters in values are 
> converted to lower case.
> -u  Upper case attribute.  All lower case characters in values are 
> converted to upper case.

This description doesn't match mksh's actual behaviour. Only ASCII
characters are converted. The rest is left alone (even if 'set -U' is
activated in a UTF-8 locale).

Either this should be fixed, or the man page should document this
limitation.

Thanks,

- M.

'set -e' with '!' and '&&' POSIX issue

[Martijn Dekker]

There seems to be a POSIX issue with 'set -e' ('set -o errexit') on mksh.

The script:

set -e
! false && false
echo wrong

should not output anything, which is the actual behaviour on every shell
except pdksh and mskh.

This only occurs if the command starts with a '!'. The following script
works correctly on pdksh and mksh:

set -e
true && false
echo wrong

Ref.:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_25
| 2. The -e setting shall be ignored when executing the compound list
| following the while, until, if, or elif reserved word, a pipeline
| beginning with the ! reserved word, or any command of an AND-OR list
| other than the last.

Thanks,

- Martijn

Re: alias VS executable

[Martijn Dekker]

Op 01-10-16 om 20:03 schreef Antoni V.:
> I use a command many times on a day to check what programs are using internet.
> netstat -lantp | grep -i stab | awk -F/ '{print $2 $3}' | sort | uniq
> Then I thought of making a shortcut for it called 'using'.
> I can think of 2 ways.
> 
> 1- add it to .mkshrc as \alias using=
> 2- create /usr/bin/using (chmod +x) and add the one liner to it
> 
> Both will work the exact same way.
> So I think, which one is better? Faster?

Making it an external command is fine; the speed difference is going to
be negligible in comparison to the pipeline of external commands you're
launching.

If you want to use .mkshrc, best use a shell function instead of an
alias. It avoids the shell-grammatical snags you get when you create an
alias out of a compound command. So add this to .mkshrc:

function using {
  netstat -lantp | grep -i stab | awk -F/ '{print $2 $3}' | sort | uniq
}

HTH,

- M.

Re: [NOTABUG] false positive: [ ! -o optname ]

[Martijn Dekker]

Op 25-09-16 om 22:52 schreef Thorsten Glaser:
> Per XSI, -o is a binary primary:
[...]
> Well, $1 is '!' and therefore true. Same for $3, which is
> 'noclobber' and therefore true. That means that the result
> of -o is also true.

Makes complete sense and I should have figured this out myself. Sorry
for the noise and thanks for the references.

I guess this explains why the POSIX spec does not include 'test -o' (or
'test -a' as a synonym for 'test -e'). Not including them avoids this
logical but unintuitive behaviour.

- M.

Re: [PATCH] fix POSIX 'command -pv'

[Martijn Dekker]

Thorsten Glaser schreef op 26-02-16 om 21:57:
> In the end, I decided I don’t like the code and rewrote it entirely,
> separating options processing, simplifying, etc. but thanks anyway!

Yes, that code looks much cleaner. No more opaque mixing of two
different command's option logic.

I also missed something in that patch -- aliases still weren't handled
correctly. But your new code fixes that too.

I also submitted a patch to OpenBSD ksh for this and in the process I
wrote a couple of more robust regression tests, one to make sure
'command' behaves correctly and the other to check 'whence' behaviour
hasn't changed. Here they are as a patch against mksh, latest cvs. (They
pass.)

- M.

cvs diff: Diffing .
Index: check.t
===
RCS file: /cvs/src/bin/mksh/check.t,v
retrieving revision 1.726
diff -u -r1.726 check.t
--- check.t 1 Mar 2016 18:30:25 -   1.726
+++ check.t 5 Mar 2016 03:47:29 -
@@ -12196,13 +12196,75 @@
after   0='swc' 1='二' 2=''
= done
 ---
-name: command-path
+name: command-pvV-posix-priorities
 description:
-   Check 'command -p' is not 'whence -p'
-stdin:
-   command -pv if
+   For POSIX compatibility, command -v should find aliases and reserved
+   words, and command -p[vV] should find aliases, reserved words, and
+   builtins over external commands.
+stdin:
+   PATH=/bin:/usr/bin
+   alias foo="bar baz"
+   bar() { :; }
+   for word in 'if' 'foo' 'bar' 'set' 'true'; do
+   command -v "$word"
+   command -pv "$word"
+   command -V "$word"
+   command -pV "$word"
+   done
+expected-stdout:
+   if
+   if
+   if is a reserved word
+   if is a reserved word
+   alias foo='bar baz'
+   alias foo='bar baz'
+   foo is an alias for 'bar baz'
+   foo is an alias for 'bar baz'
+   bar
+   bar
+   bar is a function
+   bar is a function
+   set
+   set
+   set is a special shell builtin
+   set is a special shell builtin
+   true
+   true
+   true is a shell builtin
+   true is a shell builtin
+---
+name: whence-preserve-tradition
+description:
+   This regression test is to ensure that the POSIX compatibility
+   changes for 'command' (see previous test) do not affect traditional
+   'whence' behaviour.
+stdin:
+   PATH=/bin:/usr/bin
+   alias foo="bar baz"
+   bar() { :; }
+   for word in 'if' 'foo' 'bar' 'set' 'true'; do
+   whence "$word"
+   whence -p "$word"
+   whence -v "$word"
+   whence -pv "$word"
+   done
 expected-stdout:
if
+   if is a reserved word
+   if not found
+   'bar baz'
+   foo is an alias for 'bar baz'
+   foo not found
+   bar
+   bar is a function
+   bar not found
+   set
+   set is a special shell builtin
+   set not found
+   true
+   /usr/bin/true
+   true is a shell builtin
+   true is a tracked alias for /usr/bin/true
 ---
 name: duffs-device
 description:

Re: [PATCH] make 'set +o' useful and POSIX compatible

[Martijn Dekker]

Martijn Dekker schreef op 27-02-16 om 04:44:
> That means it should be possible to do
> 
> save_options=$(set +o)
> 
> then change some options, then later restore the shell options with
> 
> eval "$save_options"

Hmm. "-o interactive" is killing that with an "interactive: bad option"
error.

Yet, misc.c contains this bit of code:

if ((i != (size_t)-1) && (set ? 1U : 0U) == Flag(i))
/*
 * Don't check the context if the flag
 * isn't changing - makes "set -o
interactive"
 * work if you're already interactive.
Needed
 * if the output of "set +o" is to be used.
 */
;

And if I manually experiment with "set -o", then it works as the comment
above says. But there must be some kind of bug in there because evalling
the output of the patched "set +o" outputs the "interactive: bad option"
error.

$ eval "$(set +o)"
../build/mksh: set: interactive: bad option

Ah ha! I see now: the 'interactive' option is turned off in subshells
(including command substitutions), so the output of

set +o

is not the same as that of

(set +o)

Hmm. And the "monitor" (job control) option is also turned off in subshells.

This clearly needs more work. Perhaps those two options should simply be
blacklisted from "set +o" output? Thoughts?

- M.

[PATCH] fix POSIX 'command -pv'

[Martijn Dekker]

I found another minor POSIX bug in mksh 'command': the options -p and -v
don't combine correctly.

The -p flag means that 'command' will search the system default PATH
instead of the current PATH, but does not change that builtins take
precedence over external commands. Hence 'command -p test' will execute
the 'test' builtin and not /bin/test.

'-v' output should reflect what would actually be executed without that
flag. Without the -p flag, this works fine:
$ command -v true
true

However, command -pv always shows the external command, even if command
-p does not execute it:

$ command -pv [
/bin/[
(expected output: '[')

$ command -p [
mksh: [: missing ]
 it's the builtin, as expected

This also means that 'command -pv' should find shell keywords such as
'if' the same way 'command -v' should (as is the behaviour on ksh93,
bash, dash, etc.), so my earlier patch for 'command' of 1 July also
needs to be amended.

$ command -pv if
(nothing, expected output: if)

I believe the following patch fixes it (without affecting 'whence -p').

Thanks,

- M.

diff -ur mksh.orig/funcs.c mksh/funcs.c
--- mksh.orig/funcs.c   2016-01-20 22:34:37.0 +0100
+++ mksh/funcs.c2016-02-14 23:27:54.0 +0100
@@ -548,15 +548,14 @@
 * or whence -pv. This should be considered a feature.
 */
vflag = Vflag;
-   }
-   if (pflag)
+   } else if (pflag)
fcflags &= ~(FC_BI | FC_FUNC);

while ((vflag || rv == 0) && (id = *wp++) != NULL) {
uint32_t h = 0;

tp = NULL;
-   if (!pflag)
+   if (!iam_whence || !pflag)
tp = ktsearch(, id, h = hash(id));
if (!tp && !pflag) {
tp = ktsearch(, id, h ? h : hash(id));

[BUG] Multibyte not enabled with mksh -c command

[Martijn Dekker]

I just noticed something odd.

Within mksh proper:

$ echo $LANG
nl_NL.UTF-8
$ V=bhta
$ echo ${#V}
4

everything is fine; the UTF-8 string is correctly measured to be 4
characters.

But when you do:

$ mksh -c 'echo $LANG; V=bhta; echo ${#V}'
nl_NL.UTF-8
5

five characters (bytes) are measured, even though the locale is still UTF-8.

So it looks like multibyte character support is not activated when
commands are executed with -c.

I've also tested other shells (bash, dash, yash, ATT ksh, zsh) and on
them it makes no difference whether -c is used or not. So it looks like
a bug in mksh.

Thanks,

- M.

Re: [BUG] 'command -v' does not find reserved words

[Martijn Dekker]

Thorsten Glaser schreef op 30-06-15 om 16:55:
 Hrm, why do people use these things so much which I never use in
 any scripts at all? This, and 'type'b

Well, since you asked, I'll explain my use case.

I'm implementing feature testing in a cross-platform POSIX shell library
(inspired by modernizr and jQuery for JavaScript) that extends the POSIX
shell language with new functionality. One aspect of feature testing is
determining if some word is a reserved word or shell built-in in the
current shell. I need to do it in a cross-platform way as much as possible.

My general strategy for determining whether $1 is a shell keyword or
builtin function is to test the exit status of the following subshell block:

( unalias $1
  unset -f $1
  PATH=/dev/null
  command -v $1
) /dev/null

I found that shell keywords such as 'select' are not found like this on
mksh, though they are found on ksh93, bash and zsh. Currently, mksh
needs its own version of the above code, where 'whence' is used in place
of 'command -v'.

By the way, writing such a cross-platform library is a great way to find
all kinds of obscure shell bugs in various shells. And I have to say
that I found virtually none of them in mksh; this here is the first, and
it's pretty minor. The improvement over pdksh is radical (I gave up on
supporting pdksh when I found that $@ does not work correctly if IFS
is empty). mksh is one solid shell; hat off!

 Can you enumerate the bshell reserved wordsb, so I have an idea what
 to include? (Thinking of  and  and so onb) This is pure bloat, as
 it will have to be included in another tableb

 and  are shell grammar operators, not reserved words, and 'command'
is not supposed to find them.

'command -V' (with capital V) is already capable of identifying reserved
words correctly, so (from where I sit) it seems like the table you want
should already exist, and it shouldn't take more than make the '-v'
option query it in the same way that '-V' does. I know nothing about the
mksh code though, so I could be wrong. If you want, I can investigate
and take a stab at creating a patch.

Also, the 'whence' command (without any option) already acts like
'command -v' should, except for aliases where the output format doesn't
include the alias command.

Here is the POSIX reference on reserved words, but (m)ksh, zsh and bash
have additional reserved words, such as 'select'.

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_04

Hope this helps,

- M.

[PATCH] 'command -v' does not find reserved words

[Martijn Dekker]

Hi,

The one-line change in the patch below fixes it, to the best of my
knowledge. 'command' is handled by the c_whence() function. The problem
was in the conversion of 'command' flags to 'whence' behaviour. Shell
keyword (a.k.a. reserved word) search was unjustly limited to only
'whence' or 'command -V', with 'command -v' excluded.

Thanks,

- M.

--- mksh.orig/funcs.c   2015-04-19 21:18:41.0 +0200
+++ mksh/funcs.c2015-06-30 23:59:19.0 +0200
@@ -538,7 +538,7 @@
uint32_t h = 0;

tp = NULL;
-   if ((iam_whence || vflag)  !pflag)
+   if (!pflag)
tp = ktsearch(keywords, id, h = hash(id));
if (!tp  !pflag) {
tp = ktsearch(aliases, id, h ? h : hash(id));