Re: clarification needed: shell 'exec' + function (builtin, ???)

[Thorsten Glaser]

Steffen Nurpmeso dixit:

>  #include 
>  #include 
>  #include 
>  #include 
>  int main(void){
> char inb[16], oub[16], *inbp, *oubp;
> iconv_t id;
> size_t inl, oul;
>
> memcpy(inbp = inb, "a\303\244c", sizeof("a\303\244c"));
> inl = sizeof("a\303\244c") -1;

Not -1 otherwise oub will not be NUL-terminated and end with junk:

$ ./a.out
Converting 4 
GOT 

Without the trailing NUL, stateful conversation may also be
incomplete…

> oul = sizeof oub;
> oubp = oub;
>
> if((id = iconv_open("ascii", "utf8")) == (iconv_t)-1)
>   return 1;

Throws 1 because you need "utf-8", but with it, see above.

> fprintf(stderr, "Converting %lu <%s>\n",(unsigned long)inl, inbp);
> if(iconv(id, , , , ) == (size_t)-1){
>fprintf(stderr, "Fail <%s>\n", strerror(errno));
>return 2;
> }
> fprintf(stderr, "GOT <%s>\n", oub);
> iconv_close(id);
> return 0;
>  }
>
>you should get replacement characters out of the box?

Citrus iconv agrees. Its manpage says:

 If the string pointed to by *src contains a character which is valid
 under the source codeset but can not be converted to the destination
 codeset, the character is replaced by an "invalid character" which
 depends on the destination codeset, e.g., '?', and the conversion is con-
 tinued. iconv() returns the number of such "invalid conversions".

Re: clarification needed: shell 'exec' + function (builtin, …) vs. 'env'

[Thorsten Glaser]

Joerg Schilling via austin-group-l at The Open Group dixit:

>OK, mksh pdksh and posh have te same origin.
>I don't know oksh, loksh

oksh is basically where mksh took off, an intermediate, pdksh without
its compatibility layer and with a small amount of bugfixes; loksh is
a GNU/Linux port of oksh.

>The POSIX text is:
>
>   If exec is specified with command, it shall replace the shell
>   with command without creating a new process. ...
>
>So the main statement in both is that the command is executed in place of

I even referred to that as well, but as I read POSIX it defines
what “command” is to include functions.


Robert Elz via austin-group-l at The Open Group dixit:

>This has been discussed (somewhere) before - but in the context of being
>able to guarantee that the filesystem command is run, and not anything
>else.  Specifying the full path will do that, but to do that portably
>means the script needs to do its own PATH search, and that's ugly.

OK, that is tricky to do _portably_…

>The overall conclusion is that
>   (exec command)
>is the best way to do it.   That requires that exec only ever run filesystem
>commands, which is what it did historically.

Yes, historically, but not universally.

>The standard is simply poorly written (I guess that no-one ever
>imagined anyone being dumb enough to do it differently).

Eh, doing it differently is pretty much what I’d *expect* as user.
I expect that prefixing exec does not change what is actually run.

That being said, the Korn shells have a way to do PATH search, so
"$(whence -p commandname)" args…
implements the problem from above.

>My memory is that this is to be fixed in the next version.

Oh? I’d like to read about this if it’s normative.

(But that’d just mean the current implementation in mksh, where
the legacy ksh88-style way of exec is used if set -o posix is
set, can stay as-is.)

>Note that the sometimes suggested "env command" isn't guaranteed to work,
>env just sets up the environment, for the command, then runs it.  There's
>no reason that env cannot be built into a shell, and if it is, then being
>able to run finctions and builtin commands (particularly with env -i) is
>a very useful ability to have.

AND HOW DOES *THAT* DIFFER FROM exec?

In fact, *env* is the one I’d expect to always run the external
binaries… after all, env can be used to set up the environment in
ways the shell rightfully refuses, and for this it’ll need to run
through an env-changing exec* variant syscall.

By contrast, prepending “exec” in the shell merely makes it so that
the shell afterwards doesn’t continue, where possible optimising it
as exec syscall. (But GNU bash, of course, cannot even get THAT right.)

Interestingly enough, the env documentation speaks of “utility”, not
“command”. I *think* this excludes functions, although it specifically
mentions special built-ins aren’t supported.

Is there any implementation of env that can call functions (and/or
builtins) around? Otherwise I’d argue in favour of making sure that
env can be used to get the external executable always, and allowing
shells to use either for exec. (This is also more right because exec
must always be a builtin, while env is not normally one.)

I always thought the only difficulty in env is that it’s not available
everywhere (e.g. older OpenBSD releases) and that some systems don’t
have it in /usr/bin/ so shebangs are problematic.

bye,
//mirabilos
-- 
22:20⎜ The crazy that persists in his craziness becomes a master
22:21⎜ And the distance between the craziness and geniality is
only measured by the success 18:35⎜ "Psychotics are consistently
inconsistent. The essence of sanity is to be inconsistently inconsistent

mail encoding not-fun (was Re: clarification needed: shell 'exec' + function (builtin, ???))

[Thorsten Glaser]

Steffen Nurpmeso via austin-group-l at The Open Group dixit:

> |This is because m4.opengroup.org runs qmail, the arsehole under the MTAs,
> |which auto-converted the mail from quoted-printable to 8bit, sending it
> |as 8bit even to MTAs that don't offer 8BITMIME (I configured my sendmail
> |not to do that as well, so I got the same truncated mail back :( other
> |than qmail, exim is known to break the MIME and SMTP standards like that).
>
>Naaah, not true Thorsten.  At least this time.

This one *is* correct, as I got the broken message back as well.
It contains an embedded NUL.

But apparently, this was not the cause of J�rg’s problem ☻

>Related to my MUA.
[…]
>I have been able to save the mail as file and to run iconv(1) on the content.

oic

>Maybe a problem is that the first missing line is a line with a character that
>is not part of ISO-8859-1

Yes, of course, I have been writing in UTF-8 for a while.

bye,
//mirabilos
-- 
[00:02]  gecko: benutzt du emacs ?
[00:03]  nö  [00:03]  nur n normalen mac
[00:04]  argl   [00:04]  ne den editor
-- Vutral und gecko2 in #deutsch (NB: Editor? Betriebssystem.)

Re: clarification needed: shell 'exec' + function (builtin, ???)

[Thorsten Glaser]

Joerg Schilling via austin-group-l at The Open Group dixit:

>here is where the original mail ended for me. Interesting that you did get

This is because m4.opengroup.org runs qmail, the arsehole under the MTAs,
which auto-converted the mail from quoted-printable to 8bit, sending it
as 8bit even to MTAs that don't offer 8BITMIME (I configured my sendmail
not to do that as well, so I got the same truncated mail back :( other
than qmail, exim is known to break the MIME and SMTP standards like that).

Here's it from my sent-mail folder, reduced to ASCII:

>>From: Thorsten Glaser 
>>Message-ID: 
>>To: austin-grou...@opengroup.org
>>Cc: miros-mksh@mirbsd.org
>>Followup-To: austin-grou...@opengroup.org, miros-mksh@mirbsd.org
>>Date: Wed, 9 Dec 2020 21:15:37 + (UTC)
>>Subject: clarification needed: shell 'exec' + function (builtin, ...)
>>Reply-To: austin-grou...@opengroup.org, miros-mksh@mirbsd.org
>>
>>Hi *,
>>
>>I've got a report in IRC by a user who spotted a cross-shell difference.
>>
>>In my opinion, the invocation...
>>
>>  sh -c 'ls() { echo meow; }; exec ls'
>>
>>... is supposed to output "meow\n and return to the caller with a zero
>>errorlevel.
>>
>>Some shells execve() the ls(1) binary instead.
>>In particular, this was ksh88 behaviour, according to the comments
>>found in the pdksh-originating mksh source code.
>>
>>My reading of this is:
>>
>>https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#exec
>>
>>=> exec is specified with 'command'
>>=> it will replace the shell with 'command' and never return to the shell
>>
>>(note this does NOT mandate an actual execve(2) syscall or something)
>>
>>https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09
>>
>>   A command is one of the following:
>> * Simple command (see [134]Simple Commands)
>> * Pipeline (see [135]Pipelines)
>> * List compound-list (see [136]Lists)
>> * Compound command (see [137]Compound Commands)
>> * Function definition (see [138]Function Definition Command)
>>
>>In the subsequent section 2.9.1 Simple Commands, Command Search and Execution,
>>step 1.c. finds the function.
>>
>>Therefore, I believe that exec shall invoke the function, then terminate
>>the shell with the function's $? as exit status.
>>
>>(For builtins, 1.a. and 1.d. and 1.e.i.a. will find them.)
>>
>>Thanks in advance,
>>//mirabilos
>>-- 
>>(gnutls can also be used, but if you are compiling lynx for your own use,
>>there is no reason to consider using that package)
>>  -- Thomas E. Dickey on the Lynx mailing list, about OpenSSL


>Thorsten,
>
>do you know any shell besides mksh and zsh that call the function with this
>command?

>From IRC:

15:57 < orbea> yash matches the bash behavior fwiw
16:26 < orbea> pdksh, oksh, loksh, zsh and posh match mksh's behavior with 
'exec', everything else including
   ksh2020 and hsh match bash/yash
16:26 < orbea> as reproduced with: ls () { echo foo; }; exec ls
16:27 < miskatonic> and the difference is what?
16:28 < orbea> mksh prints 'foo', yash executes ls(1)

For what it's worth, the reporter agrees with my reading, that is,
that exec must run the function, then exit the shell.

>>From my understanding, calling the function is a bug.
>
>Important for me is that the Bourne Shell, ksh88 and ksh93 call ls(1), so this
>is historically correct and it was not seen as a problem by David Korn.

This is correct (that this is historically correct), but it is both
not what I'd expect as a user; prepending exec should not change
what's run; that historic behaviour also does not match my reading
of the standard.

I can live with it being open to implementations as well, but it's
best to clarify.

bye,
//mirabilos
-- 
 den AGP stecker anfeilen, damit er in den slot aufm 440BX board passt...
oder netzteile, an die man auch den monitor angeschlossen hat und die dann fuer
ein elektrisch aufgeladenes gehaeuse gesorgt haben [...] fuer lacher gut auf 
jeder
LAN party |  damals, als der pizzateig noch auf dem monior "gegangen" ist

clarification needed: shell 'exec' + function (builtin, …)

[Thorsten Glaser]

Hi *,

I’ve got a report in IRC by a user who spotted a cross-shell difference.

In my opinion, the invocation…

sh -c 'ls() { echo meow; }; exec ls'

… is supposed to output "meow\n and return to the caller with a zero
errorlevel.

Some shells execve() the ls(1) binary instead.
In particular, this was ksh88 behaviour, according to the comments
found in the pdksh-originating mksh source code.

My reading of this is:

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#exec

⇒ exec is specified with 'command'
⇒ it will replace the shell with 'command' and never return to the shell

(note this does NOT mandate an actual execve(2) syscall or something)

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09

   A command is one of the following:
 * Simple command (see [134]Simple Commands)
 * Pipeline (see [135]Pipelines)
 * List compound-list (see [136]Lists)
 * Compound command (see [137]Compound Commands)
 * Function definition (see [138]Function Definition Command)

In the subsequent section 2.9.1 Simple Commands, Command Search and Execution,
step 1.c. finds the function.

Therefore, I believe that exec shall invoke the function, then terminate
the shell with the function’s $? as exit status.

(For builtins, 1.a. and 1.d. and 1.e.i.a. will find them.)

Thanks in advance,
//mirabilos
-- 
(gnutls can also be used, but if you are compiling lynx for your own use,
there is no reason to consider using that package)
-- Thomas E. Dickey on the Lynx mailing list, about OpenSSL

[Bug 1783355] Re: here document + pipe inside COMSUB broken

[Thorsten Glaser]

fixed in R59c, hopefully without breaking anything more

** Changed in: mksh
   Status: Triaged => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1783355

Title:
  here document + pipe inside COMSUB broken

Status in mksh:
  Fix Released

Bug description:
  tglase@tglase:~ $ mksh x
  Foo
  BlA9907BAr
  try2
  x: syntax error: unexpected '|'
  1|tglase@tglase:~ $ bash x
  Foo
  BlA9910BAr
  try2
  
  tglase@tglase:~ $ ksh93 x
  Foo
  BlA9917BAr
  try2
  

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1783355/+subscriptions

Re: Assoc. arrays in mksh

[Thorsten Glaser]

KPECT dixit:

>Are you still planning to to implement associative arrays in mksh? This
>feature has been long awaited since 2011, but so far there are no

Yes, long-term. I’d also wish to add multi-dimensional arrays and JSON
{,de}serialisation in the same go, to avoid touching that code twice,
and the language implications are a bit tricky, spare time is also
always rare while working full-time, and other bugs are always more
important… :/

Use assockit.ksh from shellsnippets if you need it now. It scales
decently and is in productive use, even though it’s only prototyped.

bye,
//mirabilos
-- 
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)

Re: [PATCH] make 'set +o' useful and POSIX compatible

[Thorsten Glaser]

Hi Martijn,

>>> We're at R56 now but nothing seems to have changed here. Status?
>>
>> It’s on the TODO.
>
> Here's another attempt at a patch. Note that this version checks for OFF(i) !=
> OF_CMDLINE to exclude immutable command line-only options (interactive, login,
> restricted, stdin).

I’ve experimentally (reads: may change, but as it isn’t a regression,
it can be included already anyway) solved it thusly:

The current state of the options, with some normalisation¹, is saved
away at shell startup. “set +o” then outputs a command² beginning with
“-o .reset” which restores it, followed by ±o as needed.

① FPOSIX, FSH, and those they clear as side effect (FBRACEEXPAND and
  UTFMODE) are zero’d, so these will be always output if set
② FPOSIX and FSH first, all others later, due to these side effects

Note that the “-o .reset” behaviour is not consistent across shell
invocations (interactive vs. mksh -c differ e.g. in interactive,
monitor, stdin, trackall), but, as the command following it will
only show differences from default, I think it’s good enough.

Please find this in R59b (which also fixes your regression) for
your pleasure (and finding new bugs…).

bye,
//mirabilos
-- 
Support mksh as /bin/sh and RoQA dash NOW!
‣ src:bash (389 (415) bugs: 1 RC, 264 (283) I, 124 (131) M, 0 F)
‣ src:dash (89 (104) bugs: 0 RC, 47 (51) I, 42 (53) M, 0 F)
‣ src:mksh (0 bugs: 0 RC, 0 I, 0 M, 0 F)
dash has two RC bugs they just closed because they don’t care about quality…

Re: '[[' regression in R59

[Thorsten Glaser]

Martijn Dekker dixit:

> As of mksh-R59 there is a regression in glob pattern matching with '[['.

Thanks, fixed.

I’m too tired to release R59b tonight, so if you have got any other
bugs/regressions to go into it before I do so tomorrow or so… do tell.
You seem to have more weird-corner-case test coverage than I do.

Stay safe,
//mirabilos
-- 
21:49⎜ I have a question guys,
 ⎜Can I use my PC as SMTP server, I use Windows 7 .
 ⎜Already googled and Installed IIS
 ⎜but Still I can't send E-mail

Re: sleep builtin

[Thorsten Glaser]

Salve Noctambule,

>I have a question about sleep builtin. I'm writing a script and at the
>end, I want it to simply wait for SIGINT (from user) or signals with
>kill builtin.

The normal way to terminate sleep is SIGALRM though, but SIGINT
will most likely also work (try it out).

>What is the maximum number of seconds that can be used with sleep ?

This depends on your operating environment, specifically
the fifth argument to the select(2) syscall.

bye,
//mirabilos
-- 
“It is inappropriate to require that a time represented as
 seconds since the Epoch precisely represent the number of
 seconds between the referenced time and the Epoch.”
-- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2

Re: mksh trap in functions

[Thorsten Glaser]

n8dandy dixit:

>Yesterday, I was extensively using traps with mksh and I noticed something 
>unexpected.
>Let's consider the following example, from the book "Learning the Korn shell, 
>2nd

Well, mksh isn’t AT ksh ☺

In general, the *first* edition of that book (orelly/unix/ksh/)
works better with mksh, as the second edition, which I do own in
print, mostly deals with stuff new in ksh93 that doesn’t exist
in mksh anyway.

>>From the mksh manpage, we can read this paragraph :
>
>A separate trap/signal environment will be used during the execution
>of functions. This will mean that traps set inside a function will
>not affect the shell's traps
>
>Could you tell me if I missed something ?

The paragraph you cited is a list element under a list headed thus:

 In the future, the following differences may also be added:

Basically: “this is the plan, but we’re not there yet”


Also, under “trap”:

The original Korn shell's DEBUG trap and the handling of ERR and
EXIT traps in functions are not yet implemented.

bye,
//mirabilos
-- 
“It is inappropriate to require that a time represented as
 seconds since the Epoch precisely represent the number of
 seconds between the referenced time and the Epoch.”
-- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2

[Bug 1869833] [NEW] Please sync mksh 58-1 (universe) from Debian testing (main)

[Thorsten Glaser]

Public bug reported:

As indicated in
https://bugs.launchpad.net/ubuntu/+source/mksh/+bug/1867459 (the
previous sync request during DebianImportFreeze), I’ve just released a
new upstream version of mksh, which is virtually identical to what we
already shipped in 57-7 in Debian, just rolling up the changes into a
release, plus these new changes:

mksh (58-1) unstable; urgency=medium

  * New upstream release
- [tg] Improve code […] correctness
- [multiplexd] Add vi mode PgDn, similar to PgUp doing history search
- [tg] Build.sh fixup unreliable test -n / -z
- [multiplexd] Correct documentation of vi mode @c
- [tg] Update to UCD 13.0.0
- [tg] Use nanoseconds in test -nt / -ot (LP#1855325)
  * Work around debhelper issue #908845 (Niels Thykier)
  * Update lintian overrides

 -- Thorsten Glaser   Fri, 27 Mar 2020 12:59:25 +0100

With my Debian Developer hat on, the benefit is having all these changes
from 57-1 to 57-7 in the origtgz and a very minimal Debian diff. The
package is leaf and has autopkgtests and an extensive testsuite, so the
risk is minimal.

With my upstream hat on, I’d prefer to have a formal release shipped for
LTS.

** Affects: mksh (Ubuntu)
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh in Ubuntu.
Matching subscriptions: mkshlist-to-ubuntu-bugmail
https://bugs.launchpad.net/bugs/1869833

Title:
  Please sync mksh 58-1 (universe) from Debian testing (main)

Status in mksh package in Ubuntu:
  New

Bug description:
  As indicated in
  https://bugs.launchpad.net/ubuntu/+source/mksh/+bug/1867459 (the
  previous sync request during DebianImportFreeze), I’ve just released a
  new upstream version of mksh, which is virtually identical to what we
  already shipped in 57-7 in Debian, just rolling up the changes into a
  release, plus these new changes:

  mksh (58-1) unstable; urgency=medium

* New upstream release
  - [tg] Improve code […] correctness
  - [multiplexd] Add vi mode PgDn, similar to PgUp doing history search
  - [tg] Build.sh fixup unreliable test -n / -z
  - [multiplexd] Correct documentation of vi mode @c
  - [tg] Update to UCD 13.0.0
  - [tg] Use nanoseconds in test -nt / -ot (LP#1855325)
* Work around debhelper issue #908845 (Niels Thykier)
* Update lintian overrides

   -- Thorsten Glaser   Fri, 27 Mar 2020 12:59:25 +0100

  With my Debian Developer hat on, the benefit is having all these
  changes from 57-1 to 57-7 in the origtgz and a very minimal Debian
  diff. The package is leaf and has autopkgtests and an extensive
  testsuite, so the risk is minimal.

  With my upstream hat on, I’d prefer to have a formal release shipped
  for LTS.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mksh/+bug/1869833/+subscriptions

[Bug 1855167] Re: Comparatively poor += performance

[Thorsten Glaser]

** Changed in: mksh
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1855167

Title:
  Comparatively poor += performance

Status in mksh:
  Fix Released

Bug description:
  Heavy use of += notably impacts script performance.  Consider the
  following micro-benchmark (pattered after real script content):

  i=0 s=
  while ((i < 3)); do
((++i))
s+=$i
  done

  which creates 138,894 character long string.  On my system (macOS
  10.14.6, 3.5 GHz i7), this takes ~8 seconds in mksh, compared with
  ksh's ~0.1s and bash 5's ~0.3s.  Here're `real' timing figures from
  my most recent run (these figures are quite stable):

  - mksh r57: 0m8.17s
  - ksh 93u+ 2012-08-01: 0m0.10s
  - bash 5.0.11(1)-release: 0m0.30s

  It's no surprise that ksh93 is much faster, given its heavy
  optimisation.  Striking, though, is the poor performance of mksh
  relative to the latest bash.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1855167/+subscriptions

[Bug 1855325] Re: test -nt and -ot ignore the subsecond part of file timestamps

[Thorsten Glaser]

** Changed in: mksh
   Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1855325

Title:
  test -nt and -ot ignore the subsecond part of file timestamps

Status in mksh:
  Fix Released

Bug description:
  Example on Linux (RHEL 8):

  $ touch a; sleep 0.1; touch b
  $ ls --full-time a b
  -rw-r--r-- 1 kendall staff 0 2019-12-05 13:41:43.483652556 -0500 a
  -rw-r--r-- 1 kendall staff 0 2019-12-05 13:41:43.585652744 -0500 b
  $ if [ a -ot b ]; then print older; fi
  $ if [[ a -ot b ]]; then print older; fi  

   
  $ 

  The last two commands should have printed "older".

  $ print $KSH_VERSION
  @(#)MIRBSD KSH R56 2018/01/14
  $ uname -sr
  Linux 4.18.0-80.el8.x86_64

  R49 on Cygwin and R46 on RHEL 7 have the same behavior.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1855325/+subscriptions

[Bug 1857826] Re: mksh isglobal ASAN heap-buffer-overflow

[Thorsten Glaser]

** Changed in: mksh
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857826

Title:
  mksh isglobal ASAN heap-buffer-overflow

Status in mksh:
  Fix Released

Bug description:
  When compiling mksh with ASAN and running [[ -v $XX ]] ($XX being an
  undefined environment variable) mksh will crash.

  $ echo $KSH_VERSION
  @(#)MIRBSD KSH R57 2019/03/01
  $ set | grep XX=  
  
  $ [[ -v $XX ]]
  =
  ==362==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf4d024d5 at 
pc 0x56763b99 bp 0xff8cc988 sp 0xff8cc978
  READ of size 1 at 0xf4d024d5 thread T0
  #0 0x56763b98  (/usr/bin/mksh+0x193b98)

  0xf4d024d5 is located 0 bytes to the right of 5-byte region 
[0xf4d024d0,0xf4d024d5)
  allocated by thread T0 here:
  #0 0xf7a285bd in __interceptor_realloc 
(/lib/i386-linux-gnu/libasan.so.5+0x1125bd)
  #1 0x565e115d  (/usr/bin/mksh+0x1115d)

  SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/bin/mksh+0x193b98) 
  Shadow bytes around the buggy address:
0x3e9a0440: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0450: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0460: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0470: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  =>0x3e9a0490: fa fa fa fa fa fa fa fa fa fa[05]fa fa fa fd fa
0x3e9a04a0: fa fa fd fa fa fa 00 fa fa fa 00 00 fa fa 00 01
0x3e9a04b0: fa fa 00 04 fa fa 00 01 fa fa fd fd fa fa fd fa
0x3e9a04c0: fa fa 07 fa fa fa fd fa fa fa fd fa fa fa fd fd
0x3e9a04d0: fa fa 00 fa fa fa fd fa fa fa fa fa fa fa fa fa
0x3e9a04e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable:   00
Partially addressable: 01 02 03 04 05 06 07 
Heap left redzone:   fa
Freed heap region:   fd
Stack left redzone:  f1
Stack mid redzone:   f2
Stack right redzone: f3
Stack after return:  f5
Stack use after scope:   f8
Global redzone:  f9
Global init order:   f6
Poisoned by user:f7
Container overflow:  fc
Array cookie:ac
Intra object redzone:bb
ASan internal:   fe
Left alloca redzone: ca
Right alloca redzone:cb
Shadow gap:  cc
  ==362==ABORTING

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857826/+subscriptions

[Bug 1857195] Re: here string behaviour different in mksh and ksh93

[Thorsten Glaser]

** Changed in: mksh
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857195

Title:
  here string behaviour different in mksh and ksh93

Status in mksh:
  Fix Released

Bug description:
  consider

  IFS=$'\n'
  x=(a "b c")
  cat <<< ${x[*]}
  cat <<< "${x[*]}"
  cat <<< ${x[@]}
  cat <<< "${x[@]}"

  executing this in mksh (or zsh, incidentally) yields the output

  a
  b c
  a
  b c
  a
  b c
  a
  b c

  (i.e. identical output, always inserting first IFS char between
  elements, for all variants of accessing all elements of the array)
  while ksh93 (or bash, for that matter) yields

  a
  b c
  a
  b c
  a b c
  a b c

  (i.e. `*' behaves different from `@' but double quoting is
  ineffectual).

  I am not sure whether this is a bug (either in ksh93 or mksh) but wanted to 
report this inconsistency and to ask for clarification. what I _would_ have 
expected to start with is, that
  the above "here string" commands would yield the same output as

  print ${x[*]}
  print "${x[*]}"
  print ${x[@]}
  print "${x[@]}"

  which is neither true for ksh93 nor for mksh. is this all good and
  well and I am only overlooking something obvious?

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857195/+subscriptions

[Bug 1857828] Re: mksh expand ASAN heap-buffer-overflow

[Thorsten Glaser]

** Changed in: mksh
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857828

Title:
  mksh expand ASAN heap-buffer-overflow

Status in mksh:
  Fix Released

Bug description:
  ubuntu@bashfz:~/newmksh/mksh$ mksh -c 'echo ${0@#$0}'
  =
  ==4807==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf4d01559 
at pc 0x56649efd bp 0xffe0e668 sp 0xffe0e658
  READ of size 1 at 0xf4d01559 thread T0
  #0 0x56649efc  (/usr/bin/mksh+0x7befc)

  0xf4d01559 is located 0 bytes to the right of 9-byte region 
[0xf4d01550,0xf4d01559)
  allocated by thread T0 here:
  #0 0xf7aae5bd in __interceptor_realloc 
(/lib/i386-linux-gnu/libasan.so.5+0x1125bd)
  #1 0x565df15d  (/usr/bin/mksh+0x1115d)

  SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/bin/mksh+0x7befc) 
  Shadow bytes around the buggy address:
0x3e9a0250: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0260: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0270: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0290: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  =>0x3e9a02a0: fa fa fa fa fa fa fa fa fa fa 00[01]fa fa 00 01
0x3e9a02b0: fa fa 00 01 fa fa 00 01 fa fa 00 fa fa fa 00 00
0x3e9a02c0: fa fa 00 05 fa fa 00 04 fa fa fd fd fa fa fd fd
0x3e9a02d0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
0x3e9a02e0: fa fa fd fd fa fa fd fd fa fa fa fa fa fa fa fa
0x3e9a02f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable:   00
Partially addressable: 01 02 03 04 05 06 07 
Heap left redzone:   fa
Freed heap region:   fd
Stack left redzone:  f1
Stack mid redzone:   f2
Stack right redzone: f3
Stack after return:  f5
Stack use after scope:   f8
Global redzone:  f9
Global init order:   f6
Poisoned by user:f7
Container overflow:  fc
Array cookie:ac
Intra object redzone:bb
ASan internal:   fe
Left alloca redzone: ca
Right alloca redzone:cb
Shadow gap:  cc
  ==4807==ABORTING

  ubuntu@bashfz:~/newmksh/mksh$ valgrind ./mksh -c 'echo ${0@#$0}'
  ==4808== Memcheck, a memory error detector
  ==4808== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
  ==4808== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
  ==4808== Command: ./mksh -c echo\ ${0@#$0}
  ==4808== 
  ==4808== Invalid read of size 1
  ==4808==at 0x118527: expand (eval.c:821)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808==  Address 0x4a36873 is 0 bytes after a block of size 11 alloc'd
  ==4808==at 0x483453B: malloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x4836C88: realloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x10B68C: aresize (lalloc.c:154)
  ==4808==by 0x1420F0: setstr (var.c:491)
  ==4808==by 0x14300F: isglobal (var.c:272)
  ==4808==by 0x14305D: global (var.c:238)
  ==4808==by 0x11A9E5: varsub (eval.c:1378)
  ==4808==by 0x11A9E5: expand (eval.c:390)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808== 
  ==4808== Invalid read of size 1
  ==4808==at 0x1173CF: expand (eval.c:869)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808==  Address 0x4a36873 is 0 bytes after a block of size 11 alloc'd
  ==4808==at 0x483453B: malloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x4836C88: realloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x10B68C: aresize (lalloc.c:154)
  ==4808==by 0x1420F0: setstr (var.c:491)
  ==4808==by 0x14300F: isglobal (var.c:272)
  ==4808==by 0x14305D: global (var.c:238)
  ==4808==by 0x11A9E5: varsub (eval.c:1378)
  ==4808==by 0x11A9E5: expand (eval.c:390)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808== 

  ==4808== 
  ==4808== HEAP SUMMARY:
  ==4808== in use at exit: 0 bytes in 0 blocks
  ==4808==   total heap usage: 

[Bug 1857702] Re: " +=" operator does string concatenation for integer variables

[Thorsten Glaser]

** Changed in: mksh
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857702

Title:
  " +=" operator does string concatenation for integer variables

Status in mksh:
  Fix Released

Bug description:
  consider

  typeset -i x=0; x+=1; echo $x # → 1 (as in ksh/bash/zsh)

  but

  typeset -i x=1; x+=1; echo $x # → 11 (rather than 2 as in the other
  shells)

  I believe mksh should honour the integer declaration and interpret
  `+=' accordingly. currently, it does not even consistently use string
  concatentation (since the first example does not yield `01' ...).

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857702/+subscriptions

[Bug 1779179] Re: nōn-ASCII heredoc separators

[Thorsten Glaser]

** Changed in: mksh
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1779179

Title:
  nōn-ASCII heredoc separators

Status in mksh:
  Fix Released

Bug description:
  tglase@tglase-nb:~ $ mksh -c $':

[Bug 1867459] [NEW] Please sync mksh 57-7 (universe) from Debian sid (main)

[Thorsten Glaser]

Public bug reported:

A roll-up of a couple of bugfixes (this is almost identical to the
upcoming new upstream release), but, more importantly, introduction of
autopkgtests. I’d prefer these fixes to be in focal due to LTS, and
especially as mksh 58 will be very close to this code, have the chance
at an update if it gets released before hard freeze.

As DebianImportFreeze is active (as far as I can tell, anyway), here’s
an explicit sync request. It’s, as far as I can tell, a leaf package in
Ubuntu, so has low risk.

** Affects: mksh (Ubuntu)
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh in Ubuntu.
Matching subscriptions: mkshlist-to-ubuntu-bugmail
https://bugs.launchpad.net/bugs/1867459

Title:
  Please sync mksh 57-7 (universe) from Debian sid (main)

Status in mksh package in Ubuntu:
  New

Bug description:
  A roll-up of a couple of bugfixes (this is almost identical to the
  upcoming new upstream release), but, more importantly, introduction of
  autopkgtests. I’d prefer these fixes to be in focal due to LTS, and
  especially as mksh 58 will be very close to this code, have the chance
  at an update if it gets released before hard freeze.

  As DebianImportFreeze is active (as far as I can tell, anyway), here’s
  an explicit sync request. It’s, as far as I can tell, a leaf package
  in Ubuntu, so has low risk.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mksh/+bug/1867459/+subscriptions

[Bug 1857195] Re: here string behaviour different in mksh and ksh93

[Thorsten Glaser]

OK, turns out that this was indeed a bug (but the presence of DOHEREDOC
probably predates the existence of DOSCALAR so understandable). I hope
this doesn’t introduce any regressions. Testcase also committed.

** Changed in: mksh
   Status: Triaged => Fix Committed

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857195

Title:
  here string behaviour different in mksh and ksh93

Status in mksh:
  Fix Committed

Bug description:
  consider

  IFS=$'\n'
  x=(a "b c")
  cat <<< ${x[*]}
  cat <<< "${x[*]}"
  cat <<< ${x[@]}
  cat <<< "${x[@]}"

  executing this in mksh (or zsh, incidentally) yields the output

  a
  b c
  a
  b c
  a
  b c
  a
  b c

  (i.e. identical output, always inserting first IFS char between
  elements, for all variants of accessing all elements of the array)
  while ksh93 (or bash, for that matter) yields

  a
  b c
  a
  b c
  a b c
  a b c

  (i.e. `*' behaves different from `@' but double quoting is
  ineffectual).

  I am not sure whether this is a bug (either in ksh93 or mksh) but wanted to 
report this inconsistency and to ask for clarification. what I _would_ have 
expected to start with is, that
  the above "here string" commands would yield the same output as

  print ${x[*]}
  print "${x[*]}"
  print ${x[@]}
  print "${x[@]}"

  which is neither true for ksh93 nor for mksh. is this all good and
  well and I am only overlooking something obvious?

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857195/+subscriptions

Re: Bug#952970: mksh cannot handle non-BMP characters in \Uxxxxxxxx

[Thorsten Glaser]

severity 952970 wishlist
tags 952970 upstream
forwarded 952970 miros-mkshmirbsd.org
close 952970
thanks

>This isn’t a packaging bug, so I’ll probably close this, but we can
>discuss this upstream.

Closing as announced, but I’ve added it to the plans:
http://www.mirbsd.org/mksh.htm#plans

PS: Mails from Googlemail may not reach *@mirbsd.* addresses
(both me and the mailing list), or after a large delay only.
Use tg  debian.org if you need direct contact, or Launchpad,
or a different eMail provider.

Re: [MirBSD/mksh] Enable "+" sign in alias name as it was available in original KornShell (#2)

[Thorsten Glaser]

Aleksey Cheusov dixit:

>Well, to be honest I haven't check ksh88. But '+' sign is allowed for
>aliases in ksh (on NetBSD, Solaris), /bin/sh on FreeBSD, ksh93, dash,
^ pdksh   ^ ksh93^ ash   ^ ash
>pdksh, bash, zsh etc.

Yes, but, as I said, we had to restrict the allowed range due to
security implications. (Many of these you listed are seriously old,
for example pdksh’s last release was in 1999… mksh is its successor.)

Also, it’s not guaranteed by POSuX.

>Typical use for it is 'c++', 'g++' or 'clang++'.

OK, I can add that to the allowed aliases, we just cannot let an
alias begin with ‘+’ for the same reason ’-’ is not allowed as
first character. I have already put that on my TODO.

bye,
//mirabilos
-- 
“It is inappropriate to require that a time represented as
 seconds since the Epoch precisely represent the number of
 seconds between the referenced time and the Epoch.”
-- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2

Re: Bug#952970: mksh cannot handle non-BMP characters in \Uxxxxxxxx

[Thorsten Glaser]

Mingye Wang dixit:

>Mksh is unable to handle Unicode characters that are not in the BMP

(It’s “mksh”, lowercase ‘m’.)

Only short because I’m in a meeting: yes, and this is deliberate at
the moment, because mksh is the shell of MirBSD which uses 16-bit
Unicode.

I noticed the interactive problem, and will consider various changes,
but nothing in the short term.

This isn’t a packaging bug, so I’ll probably close this, but we can
discuss this upstream.

\U isn’t implemented at all anyway, just \u.

bye,
//mirabilos (mksh upstream AND Debian maintainer)
-- 
22:20⎜ The crazy that persists in his craziness becomes a master
22:21⎜ And the distance between the craziness and geniality is
only measured by the success 18:35⎜ "Psychotics are consistently
inconsistent. The essence of sanity is to be inconsistently inconsistent

[Bug 1857702] Re: " +=" operator does string concatenation for integer variables

[Thorsten Glaser]

This totally makes sense in mksh only.

$ i=4+4; echo $i

This is an assignment of the string "4+4" to the variable i. The string
is then, between assignment and storage, parsed as an arithmetic
expression, because i is of integer type. The expression is calculated
and the result stored.

$ i=i+1; echo $i

This is an assignment of the string "i+1" to the variable i. The string
is then, between assignment and storage, parsed as an arithmetic
expression, because i is of integer type. The expression is calculated
and the result stored.

This works because i=anything, if i is integer, is precisely the same as
i=$((anything)).

$ i+=1; echo $i

This is an assignment of the string “${i}1” to the variable i. The
string is first expanded as "11", then evaluated as an arithmetic
expression, and the result (the number 10#11) stored. This is the same
as: i=$((${i}1)).

“var+=text” is the same as “var=${var}text”. This is the basic tenet,
and this is the thing that will not change. (I am now decided.) The
other mentioned shells do this wrong. (This is one thing I will not
budge for cross-shell consistency.)

It’s much easier to accidentally introduce an error with += forgetting
that the target variable is integer (the integer flag may even have been
set by a caller!) than not using proper integer arithmetics.

Just write one of these:

$ let i+=1; echo $i
$ (( i += 1 )); echo $i
$ (( ++i )); echo $i
$ echo $((i+=1))

These should work the same in all shells mentioned.

At that: if a *caller* passes you an integer variable, in all other
shells you cannot use “+=”, which means you cannot use “+=” in the other
shells reliably at all.

function strconcat {
nameref _strconcat_tgt=$1
_strconcat_tgt+=$2
}
typeset -i foo=1
strconcat foo 2

Bam!

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857702

Title:
  " +=" operator does string concatenation for integer variables

Status in mksh:
  Fix Committed

Bug description:
  consider

  typeset -i x=0; x+=1; echo $x # → 1 (as in ksh/bash/zsh)

  but

  typeset -i x=1; x+=1; echo $x # → 11 (rather than 2 as in the other
  shells)

  I believe mksh should honour the integer declaration and interpret
  `+=' accordingly. currently, it does not even consistently use string
  concatentation (since the first example does not yield `01' ...).

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857702/+subscriptions

[Bug 1857702] Re: " +=" operator does string concatenation for integer variables

[Thorsten Glaser]

Warning reporting during running of a script is supremely hard, you
cannot use any file descriptors; basically, you have to reserve one with
a high number for syslog and do it that way and hope someone reads
syslog… so I never did. We really need a linting shell runner or
something ☹

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857702

Title:
  " +=" operator does string concatenation for integer variables

Status in mksh:
  Fix Committed

Bug description:
  consider

  typeset -i x=0; x+=1; echo $x # → 1 (as in ksh/bash/zsh)

  but

  typeset -i x=1; x+=1; echo $x # → 11 (rather than 2 as in the other
  shells)

  I believe mksh should honour the integer declaration and interpret
  `+=' accordingly. currently, it does not even consistently use string
  concatentation (since the first example does not yield `01' ...).

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857702/+subscriptions

[Bug 1857826] Re: mksh isglobal ASAN heap-buffer-overflow

[Thorsten Glaser]

** Changed in: mksh
   Status: New => Fix Committed

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857826

Title:
  mksh isglobal ASAN heap-buffer-overflow

Status in mksh:
  Fix Committed

Bug description:
  When compiling mksh with ASAN and running [[ -v $XX ]] ($XX being an
  undefined environment variable) mksh will crash.

  $ echo $KSH_VERSION
  @(#)MIRBSD KSH R57 2019/03/01
  $ set | grep XX=  
  
  $ [[ -v $XX ]]
  =
  ==362==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf4d024d5 at 
pc 0x56763b99 bp 0xff8cc988 sp 0xff8cc978
  READ of size 1 at 0xf4d024d5 thread T0
  #0 0x56763b98  (/usr/bin/mksh+0x193b98)

  0xf4d024d5 is located 0 bytes to the right of 5-byte region 
[0xf4d024d0,0xf4d024d5)
  allocated by thread T0 here:
  #0 0xf7a285bd in __interceptor_realloc 
(/lib/i386-linux-gnu/libasan.so.5+0x1125bd)
  #1 0x565e115d  (/usr/bin/mksh+0x1115d)

  SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/bin/mksh+0x193b98) 
  Shadow bytes around the buggy address:
0x3e9a0440: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0450: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0460: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0470: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  =>0x3e9a0490: fa fa fa fa fa fa fa fa fa fa[05]fa fa fa fd fa
0x3e9a04a0: fa fa fd fa fa fa 00 fa fa fa 00 00 fa fa 00 01
0x3e9a04b0: fa fa 00 04 fa fa 00 01 fa fa fd fd fa fa fd fa
0x3e9a04c0: fa fa 07 fa fa fa fd fa fa fa fd fa fa fa fd fd
0x3e9a04d0: fa fa 00 fa fa fa fd fa fa fa fa fa fa fa fa fa
0x3e9a04e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable:   00
Partially addressable: 01 02 03 04 05 06 07 
Heap left redzone:   fa
Freed heap region:   fd
Stack left redzone:  f1
Stack mid redzone:   f2
Stack right redzone: f3
Stack after return:  f5
Stack use after scope:   f8
Global redzone:  f9
Global init order:   f6
Poisoned by user:f7
Container overflow:  fc
Array cookie:ac
Intra object redzone:bb
ASan internal:   fe
Left alloca redzone: ca
Right alloca redzone:cb
Shadow gap:  cc
  ==362==ABORTING

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857826/+subscriptions

[Bug 1857828] Re: mksh expand ASAN heap-buffer-overflow

[Thorsten Glaser]

fix is making it to the anoncvs and github mirrors within the hour

** Changed in: mksh
   Status: Triaged => Fix Committed

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857828

Title:
  mksh expand ASAN heap-buffer-overflow

Status in mksh:
  Fix Committed

Bug description:
  ubuntu@bashfz:~/newmksh/mksh$ mksh -c 'echo ${0@#$0}'
  =
  ==4807==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf4d01559 
at pc 0x56649efd bp 0xffe0e668 sp 0xffe0e658
  READ of size 1 at 0xf4d01559 thread T0
  #0 0x56649efc  (/usr/bin/mksh+0x7befc)

  0xf4d01559 is located 0 bytes to the right of 9-byte region 
[0xf4d01550,0xf4d01559)
  allocated by thread T0 here:
  #0 0xf7aae5bd in __interceptor_realloc 
(/lib/i386-linux-gnu/libasan.so.5+0x1125bd)
  #1 0x565df15d  (/usr/bin/mksh+0x1115d)

  SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/bin/mksh+0x7befc) 
  Shadow bytes around the buggy address:
0x3e9a0250: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0260: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0270: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0290: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  =>0x3e9a02a0: fa fa fa fa fa fa fa fa fa fa 00[01]fa fa 00 01
0x3e9a02b0: fa fa 00 01 fa fa 00 01 fa fa 00 fa fa fa 00 00
0x3e9a02c0: fa fa 00 05 fa fa 00 04 fa fa fd fd fa fa fd fd
0x3e9a02d0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
0x3e9a02e0: fa fa fd fd fa fa fd fd fa fa fa fa fa fa fa fa
0x3e9a02f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable:   00
Partially addressable: 01 02 03 04 05 06 07 
Heap left redzone:   fa
Freed heap region:   fd
Stack left redzone:  f1
Stack mid redzone:   f2
Stack right redzone: f3
Stack after return:  f5
Stack use after scope:   f8
Global redzone:  f9
Global init order:   f6
Poisoned by user:f7
Container overflow:  fc
Array cookie:ac
Intra object redzone:bb
ASan internal:   fe
Left alloca redzone: ca
Right alloca redzone:cb
Shadow gap:  cc
  ==4807==ABORTING

  ubuntu@bashfz:~/newmksh/mksh$ valgrind ./mksh -c 'echo ${0@#$0}'
  ==4808== Memcheck, a memory error detector
  ==4808== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
  ==4808== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
  ==4808== Command: ./mksh -c echo\ ${0@#$0}
  ==4808== 
  ==4808== Invalid read of size 1
  ==4808==at 0x118527: expand (eval.c:821)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808==  Address 0x4a36873 is 0 bytes after a block of size 11 alloc'd
  ==4808==at 0x483453B: malloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x4836C88: realloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x10B68C: aresize (lalloc.c:154)
  ==4808==by 0x1420F0: setstr (var.c:491)
  ==4808==by 0x14300F: isglobal (var.c:272)
  ==4808==by 0x14305D: global (var.c:238)
  ==4808==by 0x11A9E5: varsub (eval.c:1378)
  ==4808==by 0x11A9E5: expand (eval.c:390)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808== 
  ==4808== Invalid read of size 1
  ==4808==at 0x1173CF: expand (eval.c:869)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808==  Address 0x4a36873 is 0 bytes after a block of size 11 alloc'd
  ==4808==at 0x483453B: malloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x4836C88: realloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x10B68C: aresize (lalloc.c:154)
  ==4808==by 0x1420F0: setstr (var.c:491)
  ==4808==by 0x14300F: isglobal (var.c:272)
  ==4808==by 0x14305D: global (var.c:238)
  ==4808==by 0x11A9E5: varsub (eval.c:1378)
  ==4808==by 0x11A9E5: expand (eval.c:390)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808== 

  ==4808== 
  ==4808== HEAP SUMMARY:
  ==4808== in 

Re: [bug] within 'eval', -e/-o errexit appears active, but is inactive

[Thorsten Glaser]

Martijn Dekker dixit:

> I noticed something strange while executing some 'eval'-ed commands: the -e/-o
> errexit appears to become active out of nowhere.
>
> $ mksh -c 'echo $-; eval '\''echo $-'\''; echo $-'
> hc
> ehc
> hc

Found to be caused by reusing bit7 of the ERREXIT flag for
nefarious purposes; fixed, but we could not get completely
rid of the flag.

> However, both pdksh and mksh render 'set -e' ineffective for eval'ed commands,
> which is a bug in itself; no other shell including ksh93 shares this 
> behaviour.
> Thus the expected output for the last command above is 'hc' and nothing else.

When run under -e of course.

Fixed, this was way harder.

Thanks,
//mirabilos
-- 
This space for rent.

https://paypal.me/mirabilos to support my work.

[Bug 1857702] Re: " +=" operator does string concatenation for integer variables

[Thorsten Glaser]

As discussed heavily on IRC, other shells can use ((…)) or let to work
like mksh, and the mksh behaviour is semantically correct. I’ve
documented this in more detail in the manual page and the mksh FAQ now
but kept the behaviour as to not break older scripts written in mksh.

You might wish to open bugs with the other shells to make behaviour
match mksh, or warn when the underlying variable is an integer.

** Changed in: mksh
   Importance: Undecided => Wishlist

** Changed in: mksh
   Status: New => Fix Committed

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857702

Title:
  " +=" operator does string concatenation for integer variables

Status in mksh:
  Fix Committed

Bug description:
  consider

  typeset -i x=0; x+=1; echo $x # → 1 (as in ksh/bash/zsh)

  but

  typeset -i x=1; x+=1; echo $x # → 11 (rather than 2 as in the other
  shells)

  I believe mksh should honour the integer declaration and interpret
  `+=' accordingly. currently, it does not even consistently use string
  concatentation (since the first example does not yield `01' ...).

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857702/+subscriptions

[Bug 1857828] Re: mksh expand ASAN heap-buffer-overflow

[Thorsten Glaser]

** Changed in: mksh
   Importance: Undecided => High

** Changed in: mksh
   Status: New => Triaged

** Changed in: mksh
 Assignee: (unassigned) => Thorsten Glaser (mirabilos)

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857828

Title:
  mksh expand ASAN heap-buffer-overflow

Status in mksh:
  Triaged

Bug description:
  ubuntu@bashfz:~/newmksh/mksh$ mksh -c 'echo ${0@#$0}'
  =
  ==4807==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf4d01559 
at pc 0x56649efd bp 0xffe0e668 sp 0xffe0e658
  READ of size 1 at 0xf4d01559 thread T0
  #0 0x56649efc  (/usr/bin/mksh+0x7befc)

  0xf4d01559 is located 0 bytes to the right of 9-byte region 
[0xf4d01550,0xf4d01559)
  allocated by thread T0 here:
  #0 0xf7aae5bd in __interceptor_realloc 
(/lib/i386-linux-gnu/libasan.so.5+0x1125bd)
  #1 0x565df15d  (/usr/bin/mksh+0x1115d)

  SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/bin/mksh+0x7befc) 
  Shadow bytes around the buggy address:
0x3e9a0250: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0260: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0270: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0290: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  =>0x3e9a02a0: fa fa fa fa fa fa fa fa fa fa 00[01]fa fa 00 01
0x3e9a02b0: fa fa 00 01 fa fa 00 01 fa fa 00 fa fa fa 00 00
0x3e9a02c0: fa fa 00 05 fa fa 00 04 fa fa fd fd fa fa fd fd
0x3e9a02d0: fa fa fd fd fa fa fd fd fa fa fd fd fa fa fd fd
0x3e9a02e0: fa fa fd fd fa fa fd fd fa fa fa fa fa fa fa fa
0x3e9a02f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable:   00
Partially addressable: 01 02 03 04 05 06 07 
Heap left redzone:   fa
Freed heap region:   fd
Stack left redzone:  f1
Stack mid redzone:   f2
Stack right redzone: f3
Stack after return:  f5
Stack use after scope:   f8
Global redzone:  f9
Global init order:   f6
Poisoned by user:f7
Container overflow:  fc
Array cookie:ac
Intra object redzone:bb
ASan internal:   fe
Left alloca redzone: ca
Right alloca redzone:cb
Shadow gap:  cc
  ==4807==ABORTING

  ubuntu@bashfz:~/newmksh/mksh$ valgrind ./mksh -c 'echo ${0@#$0}'
  ==4808== Memcheck, a memory error detector
  ==4808== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
  ==4808== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
  ==4808== Command: ./mksh -c echo\ ${0@#$0}
  ==4808== 
  ==4808== Invalid read of size 1
  ==4808==at 0x118527: expand (eval.c:821)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808==  Address 0x4a36873 is 0 bytes after a block of size 11 alloc'd
  ==4808==at 0x483453B: malloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x4836C88: realloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x10B68C: aresize (lalloc.c:154)
  ==4808==by 0x1420F0: setstr (var.c:491)
  ==4808==by 0x14300F: isglobal (var.c:272)
  ==4808==by 0x14305D: global (var.c:238)
  ==4808==by 0x11A9E5: varsub (eval.c:1378)
  ==4808==by 0x11A9E5: expand (eval.c:390)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808== 
  ==4808== Invalid read of size 1
  ==4808==at 0x1173CF: expand (eval.c:869)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:704)
  ==4808==  Address 0x4a36873 is 0 bytes after a block of size 11 alloc'd
  ==4808==at 0x483453B: malloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x4836C88: realloc (in 
/usr/lib/i386-linux-gnu/valgrind/vgpreload_memcheck-x86-linux.so)
  ==4808==by 0x10B68C: aresize (lalloc.c:154)
  ==4808==by 0x1420F0: setstr (var.c:491)
  ==4808==by 0x14300F: isglobal (var.c:272)
  ==4808==by 0x14305D: global (var.c:238)
  ==4808==by 0x11A9E5: varsub (eval.c:1378)
  ==4808==by 0x11A9E5: expand (eval.c:390)
  ==4808==by 0x11AABD: eval (eval.c:154)
  ==4808==by 0x11C630: execute (exec.c:124)
  ==4808==by 0x1335E1: shell (main.c:908)
  ==4808==by 0x10B118: main (main.c:7

[Bug 1857826] Re: mksh ASAN heap-buffer-overflow

[Thorsten Glaser]

** Changed in: mksh
   Importance: Undecided => Medium

** Changed in: mksh
 Assignee: (unassigned) => Thorsten Glaser (mirabilos)

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857826

Title:
  mksh isglobal ASAN heap-buffer-overflow

Status in mksh:
  New

Bug description:
  When compiling mksh with ASAN and running [[ -v $XX ]] ($XX being an
  undefined environment variable) mksh will crash.

  $ echo $KSH_VERSION
  @(#)MIRBSD KSH R57 2019/03/01
  $ set | grep XX=  
  
  $ [[ -v $XX ]]
  =
  ==362==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf4d024d5 at 
pc 0x56763b99 bp 0xff8cc988 sp 0xff8cc978
  READ of size 1 at 0xf4d024d5 thread T0
  #0 0x56763b98  (/usr/bin/mksh+0x193b98)

  0xf4d024d5 is located 0 bytes to the right of 5-byte region 
[0xf4d024d0,0xf4d024d5)
  allocated by thread T0 here:
  #0 0xf7a285bd in __interceptor_realloc 
(/lib/i386-linux-gnu/libasan.so.5+0x1125bd)
  #1 0x565e115d  (/usr/bin/mksh+0x1115d)

  SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/bin/mksh+0x193b98) 
  Shadow bytes around the buggy address:
0x3e9a0440: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0450: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0460: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0470: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x3e9a0480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  =>0x3e9a0490: fa fa fa fa fa fa fa fa fa fa[05]fa fa fa fd fa
0x3e9a04a0: fa fa fd fa fa fa 00 fa fa fa 00 00 fa fa 00 01
0x3e9a04b0: fa fa 00 04 fa fa 00 01 fa fa fd fd fa fa fd fa
0x3e9a04c0: fa fa 07 fa fa fa fd fa fa fa fd fa fa fa fd fd
0x3e9a04d0: fa fa 00 fa fa fa fd fa fa fa fa fa fa fa fa fa
0x3e9a04e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable:   00
Partially addressable: 01 02 03 04 05 06 07 
Heap left redzone:   fa
Freed heap region:   fd
Stack left redzone:  f1
Stack mid redzone:   f2
Stack right redzone: f3
Stack after return:  f5
Stack use after scope:   f8
Global redzone:  f9
Global init order:   f6
Poisoned by user:f7
Container overflow:  fc
Array cookie:ac
Intra object redzone:bb
ASan internal:   fe
Left alloca redzone: ca
Right alloca redzone:cb
Shadow gap:  cc
  ==362==ABORTING

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857826/+subscriptions

[Bug 1857702] Re: " +=" operator does string concatenation for integer variables

[Thorsten Glaser]

and it’s most definitely not emulation of ksh93

some ksh88 and little parts of ksh93, but e.g. no float and other crap
or complicated things

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857702

Title:
  " +=" operator does string concatenation for integer variables

Status in mksh:
  New

Bug description:
  consider

  typeset -i x=0; x+=1; echo $x # → 1 (as in ksh/bash/zsh)

  but

  typeset -i x=1; x+=1; echo $x # → 11 (rather than 2 as in the other
  shells)

  I believe mksh should honour the integer declaration and interpret
  `+=' accordingly. currently, it does not even consistently use string
  concatentation (since the first example does not yield `01' ...).

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857702/+subscriptions

[Bug 1857702] Re: " +=" operator does string concatenation for integer variables

[Thorsten Glaser]

01 in an integer variable is still 1 on output (010 may be 10 or 8
depending on the posix flag)

more later

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857702

Title:
  " +=" operator does string concatenation for integer variables

Status in mksh:
  New

Bug description:
  consider

  typeset -i x=0; x+=1; echo $x # → 1 (as in ksh/bash/zsh)

  but

  typeset -i x=1; x+=1; echo $x # → 11 (rather than 2 as in the other
  shells)

  I believe mksh should honour the integer declaration and interpret
  `+=' accordingly. currently, it does not even consistently use string
  concatentation (since the first example does not yield `01' ...).

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857702/+subscriptions

[Bug 1857195] Re: here string behaviour different in mksh and ksh93

[Thorsten Glaser]

Thanks for the extensive debugging/comparing work.

I think I’ll change mksh to match AT ksh93 for consistency and will
try to hunt down the cause for the difference.

** Changed in: mksh
   Importance: Undecided => Low

** Changed in: mksh
   Status: New => Triaged

** Changed in: mksh
 Assignee: (unassigned) => Thorsten Glaser (mirabilos)

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857195

Title:
  here string behaviour different in mksh and ksh93

Status in mksh:
  Triaged

Bug description:
  consider

  IFS=$'\n'
  x=(a "b c")
  cat <<< ${x[*]}
  cat <<< "${x[*]}"
  cat <<< ${x[@]}
  cat <<< "${x[@]}"

  executing this in mksh (or zsh, incidentally) yields the output

  a
  b c
  a
  b c
  a
  b c
  a
  b c

  (i.e. identical output, always inserting first IFS char between
  elements, for all variants of accessing all elements of the array)
  while ksh93 (or bash, for that matter) yields

  a
  b c
  a
  b c
  a b c
  a b c

  (i.e. `*' behaves different from `@' but double quoting is
  ineffectual).

  I am not sure whether this is a bug (either in ksh93 or mksh) but wanted to 
report this inconsistency and to ask for clarification. what I _would_ have 
expected to start with is, that
  the above "here string" commands would yield the same output as

  print ${x[*]}
  print "${x[*]}"
  print ${x[@]}
  print "${x[@]}"

  which is neither true for ksh93 nor for mksh. is this all good and
  well and I am only overlooking something obvious?

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857195/+subscriptions

[Bug 1857702] Re: " +=" operator does string concatenation for integer variables

[Thorsten Glaser]

Erm… that’s right, += is string concatenation.

Write “let” before the line to make it integer addition.

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857702

Title:
  " +=" operator does string concatenation for integer variables

Status in mksh:
  New

Bug description:
  consider

  typeset -i x=0; x+=1; echo $x # → 1 (as in ksh/bash/zsh)

  but

  typeset -i x=1; x+=1; echo $x # → 11 (rather than 2 as in the other
  shells)

  I believe mksh should honour the integer declaration and interpret
  `+=' accordingly. currently, it does not even consistently use string
  concatentation (since the first example does not yield `01' ...).

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857702/+subscriptions

[Bug 1857195] Re: here string behaviour different in mksh and ksh93

[Thorsten Glaser]

“it should be deterministic/always the same” this is a very good point,
and at this, well, point, I’ve not yet got any opinion.

I modelled the here strings after here documents: first, the first two
“<<” are parsed indicating it’s a here document, then the here document
delimiter is parsed; if that begins with another “<” it’s stripped off,
and a flag is set that it’s a here string (that the delimiter is the
document), otherwise, a document is read until the end delimiter.
Nowhere did I deliberately do anything wrt. field splitting or not-
splitting or joining.

I’m somewhat afraid even that, if this is wrong, it’s systematically
wrong somewhere…

At this point… curious:

What do the zsh developers say regarding this? I assume you also
contacted them?

What does the GNU bash developer say? Did you contact Chet?
For that matter, situ (ksh2020), although I could talk to him directly (#ksh on 
IRC).

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857195

Title:
  here string behaviour different in mksh and ksh93

Status in mksh:
  New

Bug description:
  consider

  IFS=$'\n'
  x=(a "b c")
  cat <<< ${x[*]}
  cat <<< "${x[*]}"
  cat <<< ${x[@]}
  cat <<< "${x[@]}"

  executing this in mksh (or zsh, incidentally) yields the output

  a
  b c
  a
  b c
  a
  b c
  a
  b c

  (i.e. identical output, always inserting first IFS char between
  elements, for all variants of accessing all elements of the array)
  while ksh93 (or bash, for that matter) yields

  a
  b c
  a
  b c
  a b c
  a b c

  (i.e. `*' behaves different from `@' but double quoting is
  ineffectual).

  I am not sure whether this is a bug (either in ksh93 or mksh) but wanted to 
report this inconsistency and to ask for clarification. what I _would_ have 
expected to start with is, that
  the above "here string" commands would yield the same output as

  print ${x[*]}
  print "${x[*]}"
  print ${x[@]}
  print "${x[@]}"

  which is neither true for ksh93 nor for mksh. is this all good and
  well and I am only overlooking something obvious?

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857195/+subscriptions

[Bug 1857195] Re: here string behaviour different in mksh and ksh93

[Thorsten Glaser]

Almost a good point, considering…

$ x=(a 'b c')
$ IFS=,
$ a="${x[*]}"; print -r -- "<$a>"

$ a="${x[@]}"; print -r -- "<$a>"

$ a=${x[*]}; print -r -- "<$a>"

$ a=${x[@]}; print -r -- "<$a>"

$ a="${u:-"${x[*]}"}"; print -r -- "<$a>"

$ a="${u:-"${x[@]}"}"; print -r -- "<$a>"


Oh the other hand, according to POSIX, using “$@” (contrary to “$*”) in
double-quotes in scalar context is unspecified unless part of “word” in
“${foo:-word}”… I guess mksh and zsh just equal it to $* there, whereas
AT ksh93 and GNU bash don’t.

As for the mailing list… writing to postmaster@, but if your mail
provider is either on the blacklist (Yahoo, OVH, …) or too stupid for
SMTP (Hotmail, Googlemail, …) I guess you’re somewhat out of luck. (I
didn’t see any connection to postmaster@ in my logs (from Dec 20, 16:00,
onwards, I don’t retain older logs), nor in my greylisting, if that’s a
consolation.) You can try another eMail service or read at https://www
.mail-archive.com/miros-mksh@mirbsd.org/ or so…

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857195

Title:
  here string behaviour different in mksh and ksh93

Status in mksh:
  New

Bug description:
  consider

  IFS=$'\n'
  x=(a "b c")
  cat <<< ${x[*]}
  cat <<< "${x[*]}"
  cat <<< ${x[@]}
  cat <<< "${x[@]}"

  executing this in mksh (or zsh, incidentally) yields the output

  a
  b c
  a
  b c
  a
  b c
  a
  b c

  (i.e. identical output, always inserting first IFS char between
  elements, for all variants of accessing all elements of the array)
  while ksh93 (or bash, for that matter) yields

  a
  b c
  a
  b c
  a b c
  a b c

  (i.e. `*' behaves different from `@' but double quoting is
  ineffectual).

  I am not sure whether this is a bug (either in ksh93 or mksh) but wanted to 
report this inconsistency and to ask for clarification. what I _would_ have 
expected to start with is, that
  the above "here string" commands would yield the same output as

  print ${x[*]}
  print "${x[*]}"
  print ${x[@]}
  print "${x[@]}"

  which is neither true for ksh93 nor for mksh. is this all good and
  well and I am only overlooking something obvious?

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857195/+subscriptions

[Bug 1857195] Re: here string behaviour different in mksh and ksh93

[Thorsten Glaser]

Args, hit Return too early.

- contact me in IRC or so if you’re up for SMTP debugging
- 
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_02
 for the POSuX reference

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857195

Title:
  here string behaviour different in mksh and ksh93

Status in mksh:
  New

Bug description:
  consider

  IFS=$'\n'
  x=(a "b c")
  cat <<< ${x[*]}
  cat <<< "${x[*]}"
  cat <<< ${x[@]}
  cat <<< "${x[@]}"

  executing this in mksh (or zsh, incidentally) yields the output

  a
  b c
  a
  b c
  a
  b c
  a
  b c

  (i.e. identical output, always inserting first IFS char between
  elements, for all variants of accessing all elements of the array)
  while ksh93 (or bash, for that matter) yields

  a
  b c
  a
  b c
  a b c
  a b c

  (i.e. `*' behaves different from `@' but double quoting is
  ineffectual).

  I am not sure whether this is a bug (either in ksh93 or mksh) but wanted to 
report this inconsistency and to ask for clarification. what I _would_ have 
expected to start with is, that
  the above "here string" commands would yield the same output as

  print ${x[*]}
  print "${x[*]}"
  print ${x[@]}
  print "${x[@]}"

  which is neither true for ksh93 nor for mksh. is this all good and
  well and I am only overlooking something obvious?

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857195/+subscriptions

[Bug 1857195] Re: here string behaviour different in mksh and ksh93

[Thorsten Glaser]

Not a definite decision, but consider this:

print -r -- "${x[@]}"

- vs. -

a="${x[@]}"

In the first code, the elements of array x are expanded in list context,
that is, each element as separate word, unset elements generating no
words.

In the second code, the elements are expanded in scalar context (because
they are then assigned to a scalar), that is, the same as in:

a="${x[*]}"

First, one string is built from *all* array elements, IFS[0]-separated,
then it is passed to the assignee.

I think it’s not unusual to consider the argument of a here string
scalar context.

I built here stings in mksh to parallel here documents, where the
separator is scalar, so this is by coïncidence, but I don’t see anything
wrong with the mksh behaviour, upon a short two-minute reflection.

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1857195

Title:
  here string behaviour different in mksh and ksh93

Status in mksh:
  New

Bug description:
  consider

  IFS=$'\n'
  x=(a "b c")
  cat <<< ${x[*]}
  cat <<< "${x[*]}"
  cat <<< ${x[@]}
  cat <<< "${x[@]}"

  executing this in mksh (or zsh, incidentally) yields the output

  a
  b c
  a
  b c
  a
  b c
  a
  b c

  (i.e. identical output, always inserting first IFS char between
  elements, for all variants of accessing all elements of the array)
  while ksh93 (or bash, for that matter) yields

  a
  b c
  a
  b c
  a b c
  a b c

  (i.e. `*' behaves different from `@' but double quoting is
  ineffectual).

  I am not sure whether this is a bug (either in ksh93 or mksh) but wanted to 
report this inconsistency and to ask for clarification. what I _would_ have 
expected to start with is, that
  the above "here string" commands would yield the same output as

  print ${x[*]}
  print "${x[*]}"
  print ${x[@]}
  print "${x[@]}"

  which is neither true for ksh93 nor for mksh. is this all good and
  well and I am only overlooking something obvious?

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1857195/+subscriptions

[Bug 1855167] Re: Comparatively poor += performance

[Thorsten Glaser]

np you’re welcome

Note that this is special-cased (but easy enough to do and justify in
that place).

The variable must be set and have its content dynamically allocated
(normally so for strings previously set), not a special variable (like
IFS, PATH, etc.), not an integer variable, and not have the uppercase,
lowercase, or any of the left‑ or right-padding flags set.

For mere assignment (=) instead of appending (+=) the normal path is
fast enough, AFAICT. (0.29s to 0.43s appending on my $dayjob desktop,
though the difference is largely due to the string length.)

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1855167

Title:
  Comparatively poor += performance

Status in mksh:
  Fix Committed

Bug description:
  Heavy use of += notably impacts script performance.  Consider the
  following micro-benchmark (pattered after real script content):

  i=0 s=
  while ((i < 3)); do
((++i))
s+=$i
  done

  which creates 138,894 character long string.  On my system (macOS
  10.14.6, 3.5 GHz i7), this takes ~8 seconds in mksh, compared with
  ksh's ~0.1s and bash 5's ~0.3s.  Here're `real' timing figures from
  my most recent run (these figures are quite stable):

  - mksh r57: 0m8.17s
  - ksh 93u+ 2012-08-01: 0m0.10s
  - bash 5.0.11(1)-release: 0m0.30s

  It's no surprise that ksh93 is much faster, given its heavy
  optimisation.  Striking, though, is the poor performance of mksh
  relative to the latest bash.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1855167/+subscriptions

[Bug 1855167] Re: Comparatively poor += performance

[Thorsten Glaser]

Please wait a bit until the anoncvs mirror has caught up. Thanks!

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1855167

Title:
  Comparatively poor += performance

Status in mksh:
  Fix Committed

Bug description:
  Heavy use of += notably impacts script performance.  Consider the
  following micro-benchmark (pattered after real script content):

  i=0 s=
  while ((i < 3)); do
((++i))
s+=$i
  done

  which creates 138,894 character long string.  On my system (macOS
  10.14.6, 3.5 GHz i7), this takes ~8 seconds in mksh, compared with
  ksh's ~0.1s and bash 5's ~0.3s.  Here're `real' timing figures from
  my most recent run (these figures are quite stable):

  - mksh r57: 0m8.17s
  - ksh 93u+ 2012-08-01: 0m0.10s
  - bash 5.0.11(1)-release: 0m0.30s

  It's no surprise that ksh93 is much faster, given its heavy
  optimisation.  Striking, though, is the poor performance of mksh
  relative to the latest bash.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1855167/+subscriptions

[Bug 1855606] Re: Can't parse command substitution containing here-document and pipeline

[Thorsten Glaser]

Duplicate of https://bugs.launchpad.net/mksh/+bug/1783355

** Changed in: mksh
   Status: New => Invalid

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1855606

Title:
  Can't parse command substitution containing here-document and pipeline

Status in mksh:
  Invalid

Bug description:
  This scriptlet, boiled down from production code, fails with "syntax
  error: '|' unexpected":

  a=$(cat 

[Bug 1855167] Re: Comparatively poor += performance

[Thorsten Glaser]

Ugh… do you already have any idea why this is so? I’m not aiming for
optimised-to-death, but this should be better, agreed.

Thanks for tracking this down and reporting, will fix this when I have
the chance.

** Changed in: mksh
   Importance: Undecided => Medium

** Changed in: mksh
 Assignee: (unassigned) => Thorsten Glaser (mirabilos)

** Changed in: mksh
   Status: New => Confirmed

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1855167

Title:
  Comparatively poor += performance

Status in mksh:
  Confirmed

Bug description:
  Heavy use of += notably impacts script performance.  Consider the
  following micro-benchmark (pattered after real script content):

  i=0 s=
  while ((i < 3)); do
((++i))
s+=$i
  done

  which creates 138,894 character long string.  On my system (macOS
  10.14.6, 3.5 GHz i7), this takes ~8 seconds in mksh, compared with
  ksh's ~0.1s and bash 5's ~0.3s.  Here're `real' timing figures from
  my most recent run (these figures are quite stable):

  - mksh r57: 0m8.17s
  - ksh 93u+ 2012-08-01: 0m0.10s
  - bash 5.0.11(1)-release: 0m0.30s

  It's no surprise that ksh93 is much faster, given its heavy
  optimisation.  Striking, though, is the poor performance of mksh
  relative to the latest bash.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1855167/+subscriptions

passing variable assignments to functions (was Re: Bug#935115: bash: [regression] passing variable assignments to functions broken in POSIX mode, violating POSIX)

[Thorsten Glaser]

Dixi quod…

>┌──┬┬──┬┬──┐
>│ shell \ what │ inside visible │ exported │ afterwards visible │ exported │
>├──┼┼──┼┼──┤
>│ ksh93│ ✔ as required  │ ✗ (ok)   │ ✗ no, permitted│ –│
>│ mksh ±posix  │ ✔ as required  │ ✓ (good) │ ✓ yes, permitted   │ ✓ (ok)   │
>│ posh (old)   │ ✔ as required  │ ✓ (good) │ ✗ no, permitted│ –│
>└──┴┴──┴┴──┘

>The “afterwards visible” property is dividing. Hiding it would be
>consistent related to how assignments are handled running non-functions,

Looking at POSIX again, same place:

 * If the command name is a standard utility implemented as a function
   (see XBD [142]Utility), the effect of variable assignments shall be
   as if the utility was not implemented as a function.

This is easier, not to mention more reliable to do, if all function
invocations are handled as if the callee is not a function, that is,
inside visible plus exported, outside untouched.

I’m strongly considering changing mksh to match; note this was not
changed deliberately before / from pdksh/oksh.

bye,
//mirabilos
-- 
This space for rent.

https://paypal.me/mirabilos to support my work.

Re: recent CVS: 'command .' permanently overrides PPs

[Thorsten Glaser]

Martijn Dekker dixit:

> One of the recent commits introduced a new bug: if a dot script is sourced 
> with
> 'command .', the positional parameters are permanently overridden by whatever
> they were at the point of sourcing, and any changes to them are ignored.

Ouch, so it’s not all that easy… thanks, fixed.

bye,
//mirabilos
-- 
[16:04:33] bkix: "veni vidi violini"
[16:04:45] bkix: "ich kam, sah und vergeigte"...

Re: [Bug] 'command readonly/export' exit on error

[Thorsten Glaser]

Martijn Dekker dixit:

> The 'readonly' and 'export' commands exit the shell on error even if
> they are prefixed with 'command', which should stop that exit from
> happening.

tricky but fixed, thanks
//mirabilos
-- 
11:56⎜«liwakura:#!/bin/mksh» also, i wanted to add mksh to my own distro │
i was disappointed that there is no makefile │ but somehow the Build.sh is
the least painful built system i've ever seen │ honours CC, {CPP,C,LD}FLAGS
properly │ looks cleary like done by someone who knows what they are doing

Re: [PATCH] return status 126 on failure to execute

[Thorsten Glaser]

Martijn Dekker dixit:

> Expected output:
>
> longargs.sh[8]: /bin/sh: Argument list too long
> status 126

merged, thanks

Re: 'command set' does not set PPs

[Thorsten Glaser]

Martijn Dekker dixit:

>$ mksh -c 'command set -- one two three; printf %s\\n "$@"'

fixed, thanks

Re: [PATCH] return status 126 on failure to execute

[Thorsten Glaser]

Hi Martijn,

> Currently, when a utility fails to execute due to something like E2BIG, mksh
> sets the exit status to 1. That is not satisfactory because there is no way to

agreed; thanks for the patch, it’s consistent, will apply.

bye,
//mirabilos
-- 
21:12⎜ sogar bei opensolaris haben die von der community so
ziemlich jeden mist eingebaut │ man sollte unices nich so machen das
desktopuser zuviel intresse kriegen │ das macht die code base kaputt
21:13⎜ linux war früher auch mal besser :D

[Bug 1817789] Re: misleading error message for SELinux denials

[Thorsten Glaser]

I changed the error message, as indicated.

** Changed in: mksh
   Status: Opinion => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1817789

Title:
  misleading error message for SELinux denials

Status in mksh:
  Fix Released

Bug description:
  Given a stat(2) failure caused by an SELinux denial (rather than a
  stat(2) success and an access(2) failure, as with a regular `chmod
  a-x` failure), mksh reports "not found" rather than the more correct
  "Permission denied".

  Expected:
  * Permission Denied error message

  Actual:

$ sh -c /system/bin/vold
sh: /system/bin/vold: not found

"not found" error message.

  
  here's the behind-the-scenes SELinux denial:

  02-25 22:37:11.023  4571  4571 W sh  : type=1400 audit(0.0:347):
  avc: denied { getattr } for path="/system/bin/vold" dev="dm-0" ino=717
  scontext=u:r:shell:s0 tcontext=u:object_r:vold_exec:s0 tclass=file
  permissive=0

  
  here's what strace says happened:

  newfstatat(AT_FDCWD, "/system/bin/vold", 0x7ffcc3ef20, 0) = -1 EACCES 
(Permission denied)
  write(2, "/system/bin/sh: /system/bin/vold"..., 44/system/bin/sh: 
/system/bin/vold: not found
  ) = 44

  versus the normal `chmod a-x` case where stat succeeds but access
  fails:

  newfstatat(AT_FDCWD, "/data/local/tmp/date2", {st_mode=S_IFREG|0644, 
st_size=482560, ...}, 0) = 0
  faccessat(AT_FDCWD, "/data/local/tmp/date2", X_OK) = -1 EACCES (Permission 
denied)
  write(2, "sh: /data/local/tmp/date2: can't"..., 60sh: /data/local/tmp/date2: 
can't execute: Permission denied
  ) = 60

  
  this patch fixes the issue:

  ```
  diff --git a/src/exec.c b/src/exec.c
  index 8330174..3f6d876 100644
  --- a/src/exec.c
  +++ b/src/exec.c
  @@ -1279,8 +1279,8 @@ search_access(const char *fn, int mode)
  struct stat sb;
   
  if (stat(fn, ) < 0)
  -   /* file does not exist */
  -   return (ENOENT);
  +   /* file may or may not exist: check errno */
  +   return errno;
  /* LINTED use of access */
  if (access(fn, mode) < 0) {
  /* file exists, but we can't access it */
  ```

  i don't know if you want to elaborate further in the comment along the
  lines of "...for example, an SELinux denial may mean that we get
  EACCES here, or if the file doesn't exist and we're allowed to know
  that, we'll get ENOENT".

  
  result with patch:

  $ sh -c /system/bin/vold
  sh: /system/bin/vold: can't execute: Permission denied

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1817789/+subscriptions

[Bug 1817959] Re: "test -e" inaccurately returns false when stat() is disallowed

[Thorsten Glaser]

I don’t consider it a bug.

If you break stat(2) in your system, expect follow-up breakage, as per
the GIGO principle.

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1817959

Title:
  "test -e" inaccurately returns false when stat() is disallowed

Status in mksh:
  Invalid

Bug description:
  From "man 1 test"

NAME
 test - check file types and compare values
DESCRIPTION
 Exit with the status determined by EXPRESSION.
 [deleted]
 -e FILE
FILE exists

  When "test -e" is called, it is intended to determine the existence or
  non-existence of a file. However, the "test" command is implemented
  using stat(), which may be disallowed by security policy. If stat() is
  disallowed, "test" will falsely claim a file doesn't exist when it
  really exists.

  Replacing "stat() == 0" with "access(F_OK) == 0" fixes this problem.
  See attached patch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1817959/+subscriptions

[Bug 1817789] Re: misleading error message for SELinux denials

[Thorsten Glaser]

I vaguely recall something about access() succeeding for file foo when
foo.exe exists on some platforms, or the other way round, and that the
modes are also not always correct. The manpage supports the latter…

 Even if a process has appropriate privileges and indicates success for
 X_OK, the file may not actually have execute permission bits set. Like-
 wise for R_OK and W_OK.

… but uses stronger language:

CAVEATS

 access() is a potential security hole and should never be used.


Ceterum censeo, a non-broken stat() is mandatory.

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1817789

Title:
  misleading error message for SELinux denials

Status in mksh:
  Opinion

Bug description:
  Given a stat(2) failure caused by an SELinux denial (rather than a
  stat(2) success and an access(2) failure, as with a regular `chmod
  a-x` failure), mksh reports "not found" rather than the more correct
  "Permission denied".

  Expected:
  * Permission Denied error message

  Actual:

$ sh -c /system/bin/vold
sh: /system/bin/vold: not found

"not found" error message.

  
  here's the behind-the-scenes SELinux denial:

  02-25 22:37:11.023  4571  4571 W sh  : type=1400 audit(0.0:347):
  avc: denied { getattr } for path="/system/bin/vold" dev="dm-0" ino=717
  scontext=u:r:shell:s0 tcontext=u:object_r:vold_exec:s0 tclass=file
  permissive=0

  
  here's what strace says happened:

  newfstatat(AT_FDCWD, "/system/bin/vold", 0x7ffcc3ef20, 0) = -1 EACCES 
(Permission denied)
  write(2, "/system/bin/sh: /system/bin/vold"..., 44/system/bin/sh: 
/system/bin/vold: not found
  ) = 44

  versus the normal `chmod a-x` case where stat succeeds but access
  fails:

  newfstatat(AT_FDCWD, "/data/local/tmp/date2", {st_mode=S_IFREG|0644, 
st_size=482560, ...}, 0) = 0
  faccessat(AT_FDCWD, "/data/local/tmp/date2", X_OK) = -1 EACCES (Permission 
denied)
  write(2, "sh: /data/local/tmp/date2: can't"..., 60sh: /data/local/tmp/date2: 
can't execute: Permission denied
  ) = 60

  
  this patch fixes the issue:

  ```
  diff --git a/src/exec.c b/src/exec.c
  index 8330174..3f6d876 100644
  --- a/src/exec.c
  +++ b/src/exec.c
  @@ -1279,8 +1279,8 @@ search_access(const char *fn, int mode)
  struct stat sb;
   
  if (stat(fn, ) < 0)
  -   /* file does not exist */
  -   return (ENOENT);
  +   /* file may or may not exist: check errno */
  +   return errno;
  /* LINTED use of access */
  if (access(fn, mode) < 0) {
  /* file exists, but we can't access it */
  ```

  i don't know if you want to elaborate further in the comment along the
  lines of "...for example, an SELinux denial may mean that we get
  EACCES here, or if the file doesn't exist and we're allowed to know
  that, we'll get ENOENT".

  
  result with patch:

  $ sh -c /system/bin/vold
  sh: /system/bin/vold: can't execute: Permission denied

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1817789/+subscriptions

[Bug 1817959] Re: "test -e" inaccurately returns false when stat() is disallowed

[Thorsten Glaser]

On unix. test whether a file exists is done using stat. access is not
portable and buggy on many platforms.

Honestly, write your security policies to not block stat. I’m not even
going to bother with this one.

** Changed in: mksh
   Importance: Undecided => Wishlist

** Changed in: mksh
   Status: New => Invalid

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1817959

Title:
  "test -e" inaccurately returns false when stat() is disallowed

Status in mksh:
  Invalid

Bug description:
  From "man 1 test"

NAME
 test - check file types and compare values
DESCRIPTION
 Exit with the status determined by EXPRESSION.
 [deleted]
 -e FILE
FILE exists

  When "test -e" is called, it is intended to determine the existence or
  non-existence of a file. However, the "test" command is implemented
  using stat(), which may be disallowed by security policy. If stat() is
  disallowed, "test" will falsely claim a file doesn't exist when it
  really exists.

  Replacing "stat() == 0" with "access(F_OK) == 0" fixes this problem.
  See attached patch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1817959/+subscriptions

[Bug 1817789] Re: misleading error message for SELinux denials

[Thorsten Glaser]

“5) A failure of the stat() system call is not an accurate indicator of
the exec()-ability of the file. The only way to determine if a file is
executable is to execute it.”

This is not generally true. We need to check for +x, and there was
something with strange operating systems, *and* the shell does not only
use the exec*() family syscalls but also “executes” a script by doing
stuff itself.

I’m willing to compromise on the error message right now. Everything
else needs more thought, and (probably) tomorrow will see a new mksh
version.

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1817789

Title:
  misleading error message for SELinux denials

Status in mksh:
  Opinion

Bug description:
  Given a stat(2) failure caused by an SELinux denial (rather than a
  stat(2) success and an access(2) failure, as with a regular `chmod
  a-x` failure), mksh reports "not found" rather than the more correct
  "Permission denied".

  Expected:
  * Permission Denied error message

  Actual:

$ sh -c /system/bin/vold
sh: /system/bin/vold: not found

"not found" error message.

  
  here's the behind-the-scenes SELinux denial:

  02-25 22:37:11.023  4571  4571 W sh  : type=1400 audit(0.0:347):
  avc: denied { getattr } for path="/system/bin/vold" dev="dm-0" ino=717
  scontext=u:r:shell:s0 tcontext=u:object_r:vold_exec:s0 tclass=file
  permissive=0

  
  here's what strace says happened:

  newfstatat(AT_FDCWD, "/system/bin/vold", 0x7ffcc3ef20, 0) = -1 EACCES 
(Permission denied)
  write(2, "/system/bin/sh: /system/bin/vold"..., 44/system/bin/sh: 
/system/bin/vold: not found
  ) = 44

  versus the normal `chmod a-x` case where stat succeeds but access
  fails:

  newfstatat(AT_FDCWD, "/data/local/tmp/date2", {st_mode=S_IFREG|0644, 
st_size=482560, ...}, 0) = 0
  faccessat(AT_FDCWD, "/data/local/tmp/date2", X_OK) = -1 EACCES (Permission 
denied)
  write(2, "sh: /data/local/tmp/date2: can't"..., 60sh: /data/local/tmp/date2: 
can't execute: Permission denied
  ) = 60

  
  this patch fixes the issue:

  ```
  diff --git a/src/exec.c b/src/exec.c
  index 8330174..3f6d876 100644
  --- a/src/exec.c
  +++ b/src/exec.c
  @@ -1279,8 +1279,8 @@ search_access(const char *fn, int mode)
  struct stat sb;
   
  if (stat(fn, ) < 0)
  -   /* file does not exist */
  -   return (ENOENT);
  +   /* file may or may not exist: check errno */
  +   return errno;
  /* LINTED use of access */
  if (access(fn, mode) < 0) {
  /* file exists, but we can't access it */
  ```

  i don't know if you want to elaborate further in the comment along the
  lines of "...for example, an SELinux denial may mean that we get
  EACCES here, or if the file doesn't exist and we're allowed to know
  that, we'll get ENOENT".

  
  result with patch:

  $ sh -c /system/bin/vold
  sh: /system/bin/vold: can't execute: Permission denied

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1817789/+subscriptions

[Bug 1817789] Re: misleading error message for SELinux denials

[Thorsten Glaser]

stat() must always succeed, because we must check that the executable
bits are set before accepting the file as command (if not for anything
else, then because POSIX was changed to demand distinguishing $? between
126 and 127 for these cases).

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1817789

Title:
  misleading error message for SELinux denials

Status in mksh:
  Opinion

Bug description:
  Given a stat(2) failure caused by an SELinux denial (rather than a
  stat(2) success and an access(2) failure, as with a regular `chmod
  a-x` failure), mksh reports "not found" rather than the more correct
  "Permission denied".

  Expected:
  * Permission Denied error message

  Actual:

$ sh -c /system/bin/vold
sh: /system/bin/vold: not found

"not found" error message.

  
  here's the behind-the-scenes SELinux denial:

  02-25 22:37:11.023  4571  4571 W sh  : type=1400 audit(0.0:347):
  avc: denied { getattr } for path="/system/bin/vold" dev="dm-0" ino=717
  scontext=u:r:shell:s0 tcontext=u:object_r:vold_exec:s0 tclass=file
  permissive=0

  
  here's what strace says happened:

  newfstatat(AT_FDCWD, "/system/bin/vold", 0x7ffcc3ef20, 0) = -1 EACCES 
(Permission denied)
  write(2, "/system/bin/sh: /system/bin/vold"..., 44/system/bin/sh: 
/system/bin/vold: not found
  ) = 44

  versus the normal `chmod a-x` case where stat succeeds but access
  fails:

  newfstatat(AT_FDCWD, "/data/local/tmp/date2", {st_mode=S_IFREG|0644, 
st_size=482560, ...}, 0) = 0
  faccessat(AT_FDCWD, "/data/local/tmp/date2", X_OK) = -1 EACCES (Permission 
denied)
  write(2, "sh: /data/local/tmp/date2: can't"..., 60sh: /data/local/tmp/date2: 
can't execute: Permission denied
  ) = 60

  
  this patch fixes the issue:

  ```
  diff --git a/src/exec.c b/src/exec.c
  index 8330174..3f6d876 100644
  --- a/src/exec.c
  +++ b/src/exec.c
  @@ -1279,8 +1279,8 @@ search_access(const char *fn, int mode)
  struct stat sb;
   
  if (stat(fn, ) < 0)
  -   /* file does not exist */
  -   return (ENOENT);
  +   /* file may or may not exist: check errno */
  +   return errno;
  /* LINTED use of access */
  if (access(fn, mode) < 0) {
  /* file exists, but we can't access it */
  ```

  i don't know if you want to elaborate further in the comment along the
  lines of "...for example, an SELinux denial may mean that we get
  EACCES here, or if the file doesn't exist and we're allowed to know
  that, we'll get ENOENT".

  
  result with patch:

  $ sh -c /system/bin/vold
  sh: /system/bin/vold: can't execute: Permission denied

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1817789/+subscriptions

[Bug 1817789] Re: misleading error message for SELinux denials

[Thorsten Glaser]

What do you think of “not found or inaccessible”? (I hope I spelt that
right.)

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1817789

Title:
  misleading error message for SELinux denials

Status in mksh:
  Opinion

Bug description:
  Given a stat(2) failure caused by an SELinux denial (rather than a
  stat(2) success and an access(2) failure, as with a regular `chmod
  a-x` failure), mksh reports "not found" rather than the more correct
  "Permission denied".

  Expected:
  * Permission Denied error message

  Actual:

$ sh -c /system/bin/vold
sh: /system/bin/vold: not found

"not found" error message.

  
  here's the behind-the-scenes SELinux denial:

  02-25 22:37:11.023  4571  4571 W sh  : type=1400 audit(0.0:347):
  avc: denied { getattr } for path="/system/bin/vold" dev="dm-0" ino=717
  scontext=u:r:shell:s0 tcontext=u:object_r:vold_exec:s0 tclass=file
  permissive=0

  
  here's what strace says happened:

  newfstatat(AT_FDCWD, "/system/bin/vold", 0x7ffcc3ef20, 0) = -1 EACCES 
(Permission denied)
  write(2, "/system/bin/sh: /system/bin/vold"..., 44/system/bin/sh: 
/system/bin/vold: not found
  ) = 44

  versus the normal `chmod a-x` case where stat succeeds but access
  fails:

  newfstatat(AT_FDCWD, "/data/local/tmp/date2", {st_mode=S_IFREG|0644, 
st_size=482560, ...}, 0) = 0
  faccessat(AT_FDCWD, "/data/local/tmp/date2", X_OK) = -1 EACCES (Permission 
denied)
  write(2, "sh: /data/local/tmp/date2: can't"..., 60sh: /data/local/tmp/date2: 
can't execute: Permission denied
  ) = 60

  
  this patch fixes the issue:

  ```
  diff --git a/src/exec.c b/src/exec.c
  index 8330174..3f6d876 100644
  --- a/src/exec.c
  +++ b/src/exec.c
  @@ -1279,8 +1279,8 @@ search_access(const char *fn, int mode)
  struct stat sb;
   
  if (stat(fn, ) < 0)
  -   /* file does not exist */
  -   return (ENOENT);
  +   /* file may or may not exist: check errno */
  +   return errno;
  /* LINTED use of access */
  if (access(fn, mode) < 0) {
  /* file exists, but we can't access it */
  ```

  i don't know if you want to elaborate further in the comment along the
  lines of "...for example, an SELinux denial may mean that we get
  EACCES here, or if the file doesn't exist and we're allowed to know
  that, we'll get ENOENT".

  
  result with patch:

  $ sh -c /system/bin/vold
  sh: /system/bin/vold: can't execute: Permission denied

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1817789/+subscriptions

[Bug 1817789] Re: misleading error message for SELinux denials

[Thorsten Glaser]

No.

When you do
mksh -c /tmp/d/f
you ask mksh to find it as a command. The “not found” error message there is 
more general; for example, it also happens if the interpreter is not found:

print '#!/nonexistent-int' >/tmp/ff
chmod 755 /tmp/ff
mksh -c /tmp/ff

(This even results in “No such file or directory”.)

When you ask mksh to read it as a script file, all is well:

mksh /tmp/d/f

(This results in “Permission denied”.)

** Changed in: mksh
   Importance: Undecided => Wishlist

** Changed in: mksh
   Status: New => Opinion

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1817789

Title:
  misleading error message for SELinux denials

Status in mksh:
  Opinion

Bug description:
  Given a stat(2) failure caused by an SELinux denial (rather than a
  stat(2) success and an access(2) failure, as with a regular `chmod
  a-x` failure), mksh reports "not found" rather than the more correct
  "Permission denied".

  Expected:
  * Permission Denied error message

  Actual:

$ sh -c /system/bin/vold
sh: /system/bin/vold: not found

"not found" error message.

  
  here's the behind-the-scenes SELinux denial:

  02-25 22:37:11.023  4571  4571 W sh  : type=1400 audit(0.0:347):
  avc: denied { getattr } for path="/system/bin/vold" dev="dm-0" ino=717
  scontext=u:r:shell:s0 tcontext=u:object_r:vold_exec:s0 tclass=file
  permissive=0

  
  here's what strace says happened:

  newfstatat(AT_FDCWD, "/system/bin/vold", 0x7ffcc3ef20, 0) = -1 EACCES 
(Permission denied)
  write(2, "/system/bin/sh: /system/bin/vold"..., 44/system/bin/sh: 
/system/bin/vold: not found
  ) = 44

  versus the normal `chmod a-x` case where stat succeeds but access
  fails:

  newfstatat(AT_FDCWD, "/data/local/tmp/date2", {st_mode=S_IFREG|0644, 
st_size=482560, ...}, 0) = 0
  faccessat(AT_FDCWD, "/data/local/tmp/date2", X_OK) = -1 EACCES (Permission 
denied)
  write(2, "sh: /data/local/tmp/date2: can't"..., 60sh: /data/local/tmp/date2: 
can't execute: Permission denied
  ) = 60

  
  this patch fixes the issue:

  ```
  diff --git a/src/exec.c b/src/exec.c
  index 8330174..3f6d876 100644
  --- a/src/exec.c
  +++ b/src/exec.c
  @@ -1279,8 +1279,8 @@ search_access(const char *fn, int mode)
  struct stat sb;
   
  if (stat(fn, ) < 0)
  -   /* file does not exist */
  -   return (ENOENT);
  +   /* file may or may not exist: check errno */
  +   return errno;
  /* LINTED use of access */
  if (access(fn, mode) < 0) {
  /* file exists, but we can't access it */
  ```

  i don't know if you want to elaborate further in the comment along the
  lines of "...for example, an SELinux denial may mean that we get
  EACCES here, or if the file doesn't exist and we're allowed to know
  that, we'll get ENOENT".

  
  result with patch:

  $ sh -c /system/bin/vold
  sh: /system/bin/vold: can't execute: Permission denied

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1817789/+subscriptions

Re: Bug in kill-region? (not actually killing region)

[Thorsten Glaser]

Hi John,

> It seems I might have come across an actual bug this time :-)

perhaps ;) But thanks for writing anyway, it’s always nice to
see what users run into.

> From the man page:
>
> kill-region: ^W
> Deletes the input between the cursor and the mark.

But also from the manpage:

 delete-word-backward: [n] Pfx1+ERASE (^[^H), WERASE (^W), ^[^?, ^[^H, ^[h

I fear that, when WERASE is defined by stty(1), it overwrites the
prior definition of kill-region.

If you want to rebind it to kill-region independent on stty(1),
add “bind ^W=kill-region” to your ~/.mkshrc file.

If your WERASE is set to something else, it’d likewise overwrite
that. HP/UX is famous for having # bound to VINTR (IIRC), meaning
every time you write # there it acts as if you had pressed ^C.

> Is this a bug?

This is how I was taught Unix terminals behave. Note that those
settings documented in the manual page are only the default; run
“bind” to find out the currently active ones. We even define some
default bindings for OS/2…

I guess this is “a missing FAQ entry”. (Feel free to write in
with suggestions for that as well.)

> Other commands, like evaluate-region, work fine.

There’s a bug in evaluate-region that it seems to work only once,
sometimes, which I have yet to find, unfortunately.

bye,
//mirabilos
-- 
 den AGP stecker anfeilen, damit er in den slot aufm 440BX board passt…
oder netzteile, an die man auch den monitor angeschlossen hat und die dann für
ein elektrisch aufgeladenes gehäuse gesorgt haben […] für lacher gut auf jeder
LAN party │  damals, als der pizzateig noch auf dem monior "gegangen" ist

[Bug 1804504] Re: -o pipefail makes while read loop ugly

[Thorsten Glaser]

This is, unfortunately, by design.

I’ve committed a manpage change (in CAVEATS) with a rationale, though.

** Changed in: mksh
   Status: Triaged => Invalid

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1804504

Title:
  -o pipefail makes while read loop ugly

Status in mksh:
  Invalid

Bug description:
  tc.sh:

  set -x
  set -e
  cat >tc.in <<\EOF
  one eins
  two zwei
  three drei
  EOF
  
  while IFS=' ' read a b; do
  #  $a = t* and it works
  [[ $a = o* ]] && echo $b
  done tc.out
  
  echo after the loop
  cat tc.out
  echo and out

  This works with t* both with set ±o pipefail, and with t* in ksh93,
  but t* in mksh with set -o pipefail makes it error out *after* the
  “while” because the last comparison error’d out.

  Workarounds include negating the comparison…

  [[ $a != o* ]] || echo $b

  … so that the loop always has $?=0, or:

  while :; do
  IFS=' ' read a b || break

  Interestingly enough, removing the “| sort -u” also makes it work.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1804504/+subscriptions

Re: Bug#910276: mksh does never execute an "EXIT trap", if it is created with the "trap" command in a sub shell

[Thorsten Glaser]

>I still cannot confirm either way, but some preliminary research
>with an extended test script:

I’ve found something interesting:

tg@blau:~ $ mksh -c 'echo a; (trap -- "echo subshell_exit >&2" EXIT); echo b'
a
b
tg@blau:~ $ mksh -c 'echo a; (trap -- "echo subshell_exit >&2" EXIT;); echo b'
a
b
tg@blau:~ $ mksh -c 'echo a; (trap -- "echo subshell_exit >&2" EXIT;:); echo b'
a
subshell_exit
b

There’s also already a regression test checking that EXIT traps
defined in subshells are in fact run. So it’s probably a Tree
composition or execution or optimisation issue… I’ll have a look.

bye,
//mirabilos
-- 
11:56⎜«liwakura:#!/bin/mksh» also, i wanted to add mksh to my own distro │
i was disappointed that there is no makefile │ but somehow the Build.sh is
the least painful built system i've ever seen │ honours CC, {CPP,C,LD}FLAGS
properly │ looks cleary like done by someone who knows what they are doing

Re: Bug in mksh when counting line length

[Thorsten Glaser]

Hi John,

>> no, you’ve discovered you did not read the manpage ☺
>
> I did read it, but I didn't understand it...!  Adding the \r and surrounding
> all escape sequences with \1 solves my problem, but I don't understand why 
> (?).

Ah okay.

Basically, the first \1\r says: make the \1 into a marker.

After that, \1 switches between “counted / visible” and
“not counted / invisible”.

So, in an example: $'\1\r\1\033[1m\1$PWD\1\033[0m\1 % '

\1\r - define marker
\1\033[1m\1 - invisible part
$PWD - visible part
\1\033[0m\1 - invisible part
 %  - visible part

Hope this clears it up.

By the way, use just $? instead of $(echo $?) ☺ or look how
dot.mkshrc handles that (it shows $?| if $?≠0).

bye,
//mirabilos
-- 
11:56⎜«liwakura:#!/bin/mksh» also, i wanted to add mksh to my own distro │
i was disappointed that there is no makefile │ but somehow the Build.sh is
the least painful built system i've ever seen │ honours CC, {CPP,C,LD}FLAGS
properly │ looks cleary like done by someone who knows what they are doing

Re: Bug in mksh when counting line length

[Thorsten Glaser]

Hi John,

> I think I've discovered a bug in mksh.  According to the man page:

no, you’ve discovered you did not read the manpage ☺

 PS1  The primary prompt for interactive shells. Parameter, com-
[…]
  Since backslashes and other special characters may be inter-
  preted by the shell, to set PS1 either escape the backslash
  itself or use double quotes. The latter is more practical.
  This is a more complex example, avoiding to directly enter
  special characters (for example with ^V in the emacs editing
  mode), which embeds the current working directory, in re-
  verse video (colour would work, too), in the prompt string:

x=$(print \\001) # otherwise unused char
PS1="$x$(print \\r)$x$(tput so)$x\$PWD$x$(tput se)$x> "

  Due to a strong suggestion from David G. Korn, mksh now also
  supports the following form:

PS1=$'\1\r\1\e[7m\1$PWD\1\e[0m\1> '

> It turns out that the invisible ANSI escape sequences count towards
> the length of the line! This means that mksh "breaks" the line much
> earlier than necessary, making it difficult to use colors in the
> prompt.

Yes, of course. It does not “know” that those are ANSI escapes, after
all (not all terminals use them, you know). That is the reason for the
above mechanism, from ksh88, to exist.

See also: http://www.mirbsd.org/mksh-faq.htm#ps1conv

bye,
//mirabilos
-- 
“The final straw, to be honest, was probably my amazement at the volume of
petty, peevish whingeing certain of your peers are prone to dish out on
d-devel, telling each other how to talk more like a pretty princess, as though
they were performing some kind of public service.” (someone to me, privately)

Re: [Bug] 'command readonly/export' exit on error

[Thorsten Glaser]

Hoi Martijn,

> they are prefixed with 'command', which should stop that exit from
> happening.

Almost certainly a bug (from what I remember without needing to
look it up), indeed.

> Test cases:

Thanks! Putting this on my TODO.

bye,
//mirabilos
-- 
“ah that reminds me, thanks for the stellar entertainment that you and certain
other people provide on the Debian mailing lists │ sole reason I subscribed to
them (I'm not using Debian anywhere) is the entertainment factor │ Debian does
not strike me as a place for good humour, much less German admin-style humour”

Re: mksh -l login shell bug on Fedora 29 fc29 in etc profile.d PackageKit.sh

[Thorsten Glaser]

Calvin Crowley dixit:

>mksh: /etc/profile.d/PackageKit.sh[14]: syntax error: unexpected 
>operator/operand '=~'

Please report this as a bug against the PackageKit packaging in
Fedora instead. Make clear that files in /etc/profile.d/ can be
sourced by any POSIX sh-compatible shell and thus MUST NOT use
any non-POSIX-sh constructs without consulting a whitelist of
shells supporting it.

In concrete, using [[ foo =~ bar ]] can be replaced by converting
“bar” from RE to extglob and using [[ foo = *bar* ]], and this
will work in all Korn shells, all shells supporting =~ (in GNU bash
you need “shopt -s extglob” additionally), and some compatibles.

>the broken method: command_not_found_handler ()

This has nothing to do with command_not_found_handler.

>my workaround is to not use:  mksh -l
>and instead just source my ~/.mkshrc

These are also independent of each other.

${ENV:-~/.mkshrc} is *always* read.

/etc/profile is read from a login shell or with “mksh -l”.

If “mksh -l” does not work, then using mksh as login shell
is broken on that system, which I’d consider a serious bug.
I don’t know Fedora policies, though.

>that gets me by so I can work on my ed.mksh editor kit
>for my Android tablets (FireHD 10 and Samsung Tab A 8)
>running Termius local shells.

Cool! (You also get mksh from stock methods like adb shell
and vx.connectbot, no need to install a GNU chroot for it.)

Contact me once you believe it in a workable state, and I’ll
link to it as user examples or something from the mksh webpage,
if you wish.

bye,
//mirabilos
-- 
„Cool, /usr/share/doc/mksh/examples/uhr.gz ist ja ein Grund,
mksh auf jedem System zu installieren.“
-- XTaran auf der OpenRheinRuhr, ganz begeistert
(EN: “[…]uhr.gz is a reason to install mksh on every system.”)

[Bug 1804504] [NEW] -o pipefail makes while read loop ugly

[Thorsten Glaser]

Public bug reported:

tc.sh:

set -x
set -e
cat >tc.in <<\EOF
one eins
two zwei
three drei
EOF

while IFS=' ' read a b; do
#  $a = t* and it works
[[ $a = o* ]] && echo $b
done tc.out

echo after the loop
cat tc.out
echo and out

This works with t* both with set ±o pipefail, and with t* in ksh93, but
t* in mksh with set -o pipefail makes it error out *after* the “while”
because the last comparison error’d out.

Workarounds include negating the comparison…

[[ $a != o* ]] || echo $b

… so that the loop always has $?=0, or:

while :; do
IFS=' ' read a b || break

Interestingly enough, removing the “| sort -u” also makes it work.

** Affects: mksh
 Importance: Medium
 Assignee: Thorsten Glaser (mirabilos)
 Status: Triaged

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1804504

Title:
  -o pipefail makes while read loop ugly

Status in mksh:
  Triaged

Bug description:
  tc.sh:

  set -x
  set -e
  cat >tc.in <<\EOF
  one eins
  two zwei
  three drei
  EOF
  
  while IFS=' ' read a b; do
  #  $a = t* and it works
  [[ $a = o* ]] && echo $b
  done tc.out
  
  echo after the loop
  cat tc.out
  echo and out

  This works with t* both with set ±o pipefail, and with t* in ksh93,
  but t* in mksh with set -o pipefail makes it error out *after* the
  “while” because the last comparison error’d out.

  Workarounds include negating the comparison…

  [[ $a != o* ]] || echo $b

  … so that the loop always has $?=0, or:

  while :; do
  IFS=' ' read a b || break

  Interestingly enough, removing the “| sort -u” also makes it work.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1804504/+subscriptions

[Bug 1783355] Re: here document + pipe inside COMSUB broken

[Thorsten Glaser]

The entire tree printing code is currently not suited for that, a major
rewrite is needed.

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1783355

Title:
  here document + pipe inside COMSUB broken

Status in mksh:
  Triaged

Bug description:
  tglase@tglase:~ $ mksh x
  Foo
  BlA9907BAr
  try2
  x: syntax error: unexpected '|'
  1|tglase@tglase:~ $ bash x
  Foo
  BlA9910BAr
  try2
  
  tglase@tglase:~ $ ksh93 x
  Foo
  BlA9917BAr
  try2
  

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1783355/+subscriptions

[Bug 1719035] Re: mksh.htm "Patching" section should mention KSH_VERSIONNAME_VENDOR_EXT

[Thorsten Glaser]

** Changed in: mksh
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1719035

Title:
  mksh.htm "Patching" section should mention KSH_VERSIONNAME_VENDOR_EXT

Status in mksh:
  Fix Released

Bug description:
  currently there's a dead link to debian and a suggestion of keeping a
  manual local patch that's likely to get lost. overriding
  KSH_VERSIONNAME_VENDOR_EXT seems more robust.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1719035/+subscriptions

[Bug 1779179] Re: nōn-ASCII heredoc separators

[Thorsten Glaser]

** Changed in: mksh
   Status: Triaged => Fix Committed

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1779179

Title:
  nōn-ASCII heredoc separators

Status in mksh:
  Fix Committed

Bug description:
  tglase@tglase-nb:~ $ mksh -c $':

[Bug 1798397] Re: too eager expansion in substitution

[Thorsten Glaser]

Found to be not a bug:

$ i=42; : ${var#${q[i=777]}}; echo $i; var=meow; : ${var#${q[i=777]}};
echo $i

must return 777 twice, because the pattern must always be expanded
before looking at $var.

$ i=42; : ${var+${q[i=777]}}; echo $i; var=meow; : ${var+${q[i=777]}};
echo $i

already behaves correctly (42 then 777).

** Changed in: mksh
   Status: Triaged => Invalid

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1798397

Title:
  too eager expansion in substitution

Status in mksh:
  Invalid

Bug description:
  “If word is not needed, it shall not be expanded.” (POSIX)

  00:52 < izabera> i=42; : ${var#${q[i=777]}}; echo $i; var=meow; : 
${var#${q[i=777]}}; echo $i
  00:52 < izabera> prints 42 777 in bash and ksh93
  00:52 < izabera> and 777 777 in mkmsh

  The #/##/%/%% of course need to expand word first.

  Thanks to jilles for the heads-up.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1798397/+subscriptions

[Bug 1798397] [NEW] too eager expansion in substitution

[Thorsten Glaser]

Public bug reported:

“If word is not needed, it shall not be expanded.” (POSIX)

00:52 < izabera> i=42; : ${var#${q[i=777]}}; echo $i; var=meow; : 
${var#${q[i=777]}}; echo $i
00:52 < izabera> prints 42 777 in bash and ksh93
00:52 < izabera> and 777 777 in mkmsh

The #/##/%/%% of course need to expand word first.

Thanks to jilles for the heads-up.

** Affects: mksh
 Importance: Medium
 Status: Triaged

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1798397

Title:
  too eager expansion in substitution

Status in mksh:
  Triaged

Bug description:
  “If word is not needed, it shall not be expanded.” (POSIX)

  00:52 < izabera> i=42; : ${var#${q[i=777]}}; echo $i; var=meow; : 
${var#${q[i=777]}}; echo $i
  00:52 < izabera> prints 42 777 in bash and ksh93
  00:52 < izabera> and 777 777 in mkmsh

  The #/##/%/%% of course need to expand word first.

  Thanks to jilles for the heads-up.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1798397/+subscriptions

Re: Bug#910276: mksh does never execute an "EXIT trap", if it is created with the "trap" command in a sub shell

[Thorsten Glaser]

Bernd Schumacher dixit:

>Please confirm, that this is a bug and not the expected behaviour of mksh.

I still cannot confirm either way, but some preliminary research
with an extended test script:

$ cat script
fkt()
{
  trap -- "echo $1 >&2" EXIT
}
fkt shell_exit
$(fkt fn_exit)
$(trap -- "echo comsub_exit >&2" EXIT)
(trap -- "echo subshell_exit >&2" EXIT)

$ mksh script
shell_exit

$ bash2.05b script
subshell_exit
shell_exit

$ ksh93 script
fn_exit
comsub_exit
subshell_exit
shell_exit

$ bash4 script
fn_exit
comsub_exit
subshell_exit
shell_exit

$ nbsh script
fn_exit
comsub_exit
subshell_exit
shell_exit

$ dash script
fn_exit
comsub_exit
subshell_exit
shell_exit

$ yash script
fn_exit
comsub_exit
subshell_exit
shell_exit

$ zsh script
shell_exit
fn_exit
comsub_exit
subshell_exit

$ zsh --emulate sh script
fn_exit
comsub_exit
subshell_exit
shell_exit


With the addition of subshell_exit (I renamed yours in fn_exit),
it’s consistent with no other shell I have, not even GNU bash 2.05b
(Heirloom Shell doesn’t know of the EXIT trap, and I’m ignoring
the C shell). zsh is a notable outlyer but easily fixed.

The manual page has something to say about EXIT traps defined in
functions declared using “function foo { … }” (Korn Shell syntax),
but that’s not used here.

I think that I’ll adapt it to the other shells independent of
whether it’s really a bug or not, for the sake of consistency.
If you’re still interested in semantics, I can continue the
research, though.

bye,
//mirabilos
-- 
“It is inappropriate to require that a time represented as
 seconds since the Epoch precisely represent the number of
 seconds between the referenced time and the Epoch.”
-- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2

[Bug 1790571] [NEW] Zombies left when backgrounded jobs run for longer than until the main script execs away

[Thorsten Glaser]

Public bug reported:

The following construct causes a zombie to hang around:

#!/bin/sh
cd /home/tglase/Misc/vpn.w
(sleep 20; sudo route -n add -host x.x.x.x gw 172.28.0.1) &
exec sudo openvpn --config /home/tglase/Misc/vpn.w/vpn-bn-01.ovpn "$@"

Adding an “exec” before the sudo in the subshell fixes this, so perhaps
the shell could clean this up itself? (Ideas and patches welcome…)

** Affects: mksh
 Importance: Low
 Status: Confirmed

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1790571

Title:
  Zombies left when backgrounded jobs run for longer than until the main
  script execs away

Status in mksh:
  Confirmed

Bug description:
  The following construct causes a zombie to hang around:

  #!/bin/sh
  cd /home/tglase/Misc/vpn.w
  (sleep 20; sudo route -n add -host x.x.x.x gw 172.28.0.1) &
  exec sudo openvpn --config /home/tglase/Misc/vpn.w/vpn-bn-01.ovpn "$@"

  Adding an “exec” before the sudo in the subshell fixes this, so
  perhaps the shell could clean this up itself? (Ideas and patches
  welcome…)

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1790571/+subscriptions

[Bug 1783355] [NEW] here document + pipe inside COMSUB broken

[Thorsten Glaser]

Public bug reported:

tglase@tglase:~ $ mksh x
Foo
BlA9907BAr
try2
x: syntax error: unexpected '|'
1|tglase@tglase:~ $ bash x
Foo
BlA9910BAr
try2

tglase@tglase:~ $ ksh93 x
Foo
BlA9917BAr
try2


** Affects: mksh
 Importance: Medium
 Status: Triaged

** Attachment added: "bug reproducer"
   https://bugs.launchpad.net/bugs/1783355/+attachment/5167176/+files/x

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1783355

Title:
  here document + pipe inside COMSUB broken

Status in mksh:
  Triaged

Bug description:
  tglase@tglase:~ $ mksh x
  Foo
  BlA9907BAr
  try2
  x: syntax error: unexpected '|'
  1|tglase@tglase:~ $ bash x
  Foo
  BlA9910BAr
  try2
  
  tglase@tglase:~ $ ksh93 x
  Foo
  BlA9917BAr
  try2
  

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1783355/+subscriptions

[Bug 1783355] Re: here document + pipe inside COMSUB broken

[Thorsten Glaser]

Syntax is valid POSuX as per https://stackoverflow.com/a/7046926/2171120

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1783355

Title:
  here document + pipe inside COMSUB broken

Status in mksh:
  Triaged

Bug description:
  tglase@tglase:~ $ mksh x
  Foo
  BlA9907BAr
  try2
  x: syntax error: unexpected '|'
  1|tglase@tglase:~ $ bash x
  Foo
  BlA9910BAr
  try2
  
  tglase@tglase:~ $ ksh93 x
  Foo
  BlA9917BAr
  try2
  

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1783355/+subscriptions

Re: mksh: [bug] command line option -m ineffective

[Thorsten Glaser]

Seb dixit:

>As if '-m' was just ignored.

It gets worse: if “-m” is passed, interactive is also broken.

I’ve tracked this down to between pdksh 4.0 (usenet posting)
and pdksh-5.0.6.tar.gz, even the changelog entry I believe
responsible, and I agree it’s probably a thinko instead of
intended behaviour.

Fixed in: @(#)MIRBSD KSH R56 2018/05/08

Thanks,
//mirabilos
-- 
Yay for having to rewrite other people's Bash scripts because bash
suddenly stopped supporting the bash extensions they make use of
-- Tonnerre Lombard in #nosec

Re: tty state not restored after restarted stopped job

[Thorsten Glaser]

Hi again,

>Thinking a bit more about this, I found a valid reason not to keep the
>terminal state after a restarted job exits (successfully): the tty
>state in which that job was started can be obsolete at the time it is
>restarted because the tty state may have been modified while the job was
>stopped (or running in the background).

good thinking.

>It would be a rare case where mksh is more "advanced" than ksh if the
>behaviour is intentional.

This is all from pdksh/oksh, so… ;-)

>I join a patch that adds a manage subsection "Terminal state" explaining
>how to change the terminal state and describing the exception for
>temporarily stopped jobs (and mentioning that the line editing
>code saves/restores the terminal state too).  The saving/restoring of
>the tty state for stopped/restarted jobs is already documented in the
>subsection "Job control".

From looking at the code, I don’t think it entirely correct.
There’s a tty_hasstate, and one of the comments around it says
/*
 * Only restore tty settings if job was originally
 * started in the foreground. Problems can be
 * caused by things like 'more foobar &' which will
 * typically get and save the shell's vi/emacs tty
 * settings before setting up the tty for itself;
 * when more exits, it restores the 'original'
 * settings, and things go down hill from there...
 */
although, slightly below, it also says
/*-
 * Don't use tty mode if job is stopped and
 * later restarted and exits. Consider
 * the sequence:
 *  vi foo (stopped)
 *  ...
 *  stty something
 *  ...
 *  fg (vi; ZZ)
 * mode should be that of the stty, not what
 * was before the vi started.
 */
which matches your description, so it’s probably merely incomplete
(or in the Job control subsection?).

*looks* “and the state of the terminal is saved or restored when a
 foreground job is stopped or restarted, respectively”
it says… perhaps we should reword this slightly?


I’ve now moved the new subsection *below* the one on job control and
removed the tty stuff from the job control one, to have it all in one
place. The wording is currently like this:

  Terminal state
 The state of the controlling terminal can be modified by a command exe-
 cuted in the foreground, whether or not job control is enabled, but the
 modified terminal state is only kept past the job's lifetime and used for
 later command invocations if the command exits successfully (i.e. with an
 exit status of 0). When such a job is momentarily stopped or restarted,
 the terminal state is saved and restored, respectively, but it will not
 be kept afterwards. In interactive mode, when line editing is enabled,
 the terminal state is saved before being reconfigured by the shell for
 the line editor, then restored before running a command.

I think this matches our understanding of the code; if not, please feel
free to enter further comments into this discussion. The same goes for
what the code actually does; with it being documented like this, are you
happy with it?

Thanks,
//mirabilos (still ill at home, but decided to tackle some TODO items)
-- 
Yay for having to rewrite other people's Bash scripts because bash
suddenly stopped supporting the bash extensions they make use of
-- Tonnerre Lombard in #nosec

Re: tty state not restored after restarted stopped job

[Thorsten Glaser]

G.raud Meyer dixit:

>First one should know what is the correct behaviour.  The feature of
[…]
>Before a patch fixing the strange case of a "successful but temporarily
>stopped job has failed" it would be good to document the feature (and
>the bug, if it is one).

I honestly don’t know.

It is desirable to be able to change all sorts of terminal state
with stty, so *some* not-resetting and/or saving is desirable.
On the other hand, the editor needs a certain state, too.

Basically, we need to not break the scenarios:

$ stty something; some other command
$ ← back to the line editor

(fully)

$ stty something
$ some other command ← back to the line editor

(partially? OTOH the line editor can cope with a LOT of states)

We also want the line editor to work.

It gets more complicated when backgrounded jobs come
into play. As I said, I honestly have absolutely no
idea…

Comparing this with, say, the last pdksh release,
would be a start: did it do the exact same thing,
and, if not, why the differences.

bye,
//mirabilos
-- 
Solange man keine schmutzigen Tricks macht, und ich meine *wirklich*
schmutzige Tricks, wie bei einer doppelt verketteten Liste beide
Pointer XORen und in nur einem Word speichern, funktioniert Boehm ganz
hervorragend.   -- Andreas Bogk über boehm-gc in d.a.s.r

Re: [PATCH] quote empties

[Thorsten Glaser]

Hoi Martijn,

>Here's a patch.

Indeed. Thanks, applied.

I’ll look at your other and Stéphane’s bugreport once I’m over
the current edition of the “common cold” ☹

Much appreciated,
//mirabilos
-- 
(gnutls can also be used, but if you are compiling lynx for your own use,
there is no reason to consider using that package)
-- Thomas E. Dickey on the Lynx mailing list, about OpenSSL

[Bug 1749741] Re: bash-style for (()) loops

[Thorsten Glaser]

No.

They’re ugly enough in C, and they can easily be transformed into
something portable:

for ((a; b; c)); do
blah
done

↓

(( a ))
while (( b )); do
blah
(( c ))
done

So, *really*, no. I’m translating for loops in C source into while loops
as I go, even.

** Changed in: mksh
   Importance: Undecided => Wishlist

** Changed in: mksh
   Status: New => Opinion

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1749741

Title:
  bash-style for (()) loops

Status in mksh:
  Opinion

Bug description:
  LTP (https://github.com/linux-test-project/ltp) has a lot of scripts
  using the bash for (()) loops, as described in the bash manual:
  ```
  An alternate form of the for command is also supported:

  for (( expr1 ; expr2 ; expr3 )) ; do commands ; done

  First, the arithmetic expression expr1 is evaluated according to the
  rules described below (see Shell Arithmetic). The arithmetic
  expression expr2 is then evaluated repeatedly until it evaluates to
  zero. Each time expr2 evaluates to a non-zero value, commands are
  executed and the arithmetic expression expr3 is evaluated. If any
  expression is omitted, it behaves as if it evaluates to 1. The return
  value is the exit status of the last command in commands that is
  executed, or false if any of the expressions is invalid.
  ``` (https://www.gnu.org/software/bash/manual/bash.html#Looping-Constructs)

  these aren't supported by mksh (or by dash, according to
  http://man7.org/linux/man-pages/man1/dash.1.html).

  do you feel like this bash extension is in scope for mksh? i might be
  able to contribute an implementation (though i have a long-standing
  line editing bug i should fix first), but i wanted to check first
  whether this would be considered useful enough?

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1749741/+subscriptions

Re: [Patch] Support for changing locale in running mksh

[Thorsten Glaser]

Hi,

>for better support of the UTF-8 mode of the mksh I've done a small
>patch which enables the mksh to change to UTF-8 mode even if the
>locale is changed at runtime ', e.g in ~/.profile, ~/.mkshrc or
>in the system wide /etc/profile.  The attached patch is for R54.

locale tracking is on the TODO. There's one problem with the
patch (which technically is very well done):

mksh (well, lksh -o posix) is only a POSIX-compatible shell
in the C locale. I fear complaints from users.

I do not wish to have mksh use locale tracking at this point,
before this has been thought out more. It does not use the locale
internally at all anyway. (Hm, the OS' strerror() and friends
might, which *also* requires consideration.) There's also an
additional thing in MirBSD to take care of first...

The manpage documents that...

case ${KSH_VERSION:-} in
*MIRBSD KSH*|*LEGACY KSH*)
case ${LC_ALL:-${LC_CTYPE:-${LANG:-}}} in
*[Uu][Tt][Ff]8*|*[Uu][Tt][Ff]-8*) set -U ;;
*) set +U ;;
esac ;;
esac

... can be used to make set ±U consistent with the current
user locale (UTF-8 or not-UTF-8, anyway). This snippet is
in sh; an mksh-specific example (from Debian skel/.mkshrc)
would be:

\\builtin set -U; [[ ${LC_ALL:-${LC_CTYPE:-${LANG:-}}} = *[Uu][Tt][Ff]?(-)8* ]] 
|| \\builtin set +U

There are multiple issues bearing consideration; I was, at
some point in time, even considering a full rewrite that
would use the system POSIX locale functions if available
(they're not everywhere), possibly extending them with
UTF-8 support (which is not everywhere). Of course, in the
GNU/Linux camp, this would be easy, but mksh supports a ton
of OSes...

I'd ask you to include the shell snippet from above into a
late shell rc file at the current point in time, for users'
consistencies (runtime changes to LC_* aren't effective in
mksh on any other platform currently, so this would be in-
consistent), but if you can't do that...

bye,
//mirabilos
-- 
“It is inappropriate to require that a time represented as
 seconds since the Epoch precisely represent the number of
 seconds between the referenced time and the Epoch.”
-- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2

Re: strerror/strsignal or sys_errlist/sys_signame

[Thorsten Glaser]

Dr. Werner Fink dixit:

>> Just to make sure: this is with LTO disabled?

>... AFAICS ftom the build log the link-time optimizer is not disabled

Just a short look, don't have much time atm:
call Build.sh *without* "-c lto".

>> This will indicate it uses those functions instead of the
>> arrays deprecated by SuSE.
>
>Ohmm ... that is not SUSE but glibc, from /usr/include/signal.h
[...]
>disappear.  Don't know what the advantage of `strsignal' and
>`strerror' is over BSD string arrays sys_siglist[] and sys_errlist[].

Ah, okay.

bye,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Re: Request 543328 requires review (submit openSUSE:Leap:15.0/mksh)

[Thorsten Glaser]

Dr. Werner Fink dixit:

>Morning Thorsten,

Morning ☺

>On Wed, Nov 22, 2017 at 12:42:52AM +, Thorsten Glaser wrote:

>> what’s -D_DEFAULT_SOURCE for?

> -- Macro: _DEFAULT_SOURCE
> If you define this macro, most features are included apart from
> X/Open, LFS and GNU extensions: the effect is to enable features

Oh, I needed something like that the other day when glibc did not
have _ALL_SOURCE and _GNU_SOURCE broke strerror_r(3)… but this is
really new, isn’t it?

>As long as mksh does not use hard coded lists I'll do this.

It needs a list of signal names (hence sys_signame[] is
highly recommended), but strsignal and strerror work and
can be used as fallback (sys_siglist[] and sys_errlist[]
are completely skipped then).

See the other mail, then check for the presence of
-DHAVE_STRERROR=1 -DHAVE_STRSIGNAL=1 (instead of =0)
in the generated CPPFLAGS.

>> What’s -ftree-loop-linear for? If it makes sense, I can
>> include it in the list of CFLAGS to check for.
>
>This should help to do loop nest optimizations, at least
>gcc manual page does show this
>
>   -ftree-loop-linear
>   -floop-interchange
>   -floop-strip-mine
>   -floop-block
>   -floop-unroll-and-jam
>   Perform loop nest optimizations.  Same as -floop-nest-optimize.  To
>   use this code transformation, GCC has to be configured with
>   --with-isl to enable the Graphite loop transformation
>   infrastructure.

Hmm.

>> Interesting approach in using GNU screen, not script(1),
>> to provide a tty. But if it works… sure.
>
>Yep, this I had done a few years back for building clisp and
>also for original ksh to have a pty/tty pair and as I'm familiar
>with screen I decided to use this.

OK.

Thanks,
//mirabilos
-- 
Solange man keine schmutzigen Tricks macht, und ich meine *wirklich*
schmutzige Tricks, wie bei einer doppelt verketteten Liste beide
Pointer XORen und in nur einem Word speichern, funktioniert Boehm ganz
hervorragend.   -- Andreas Bogk über boehm-gc in d.a.s.r

Re: strerror/strsignal or sys_errlist/sys_signame

[Thorsten Glaser]

Hi,

>HAVE_SYS_SIGLIST=0
>HAVE_SYS_ERRLIST=0
>HAVE_SYS__SIGLIST=0
>HAVE_SYS__ERRLIST=0
>export HAVE_SYS_SIGLIST HAVE_SYS_ERRLIST HAVE_SYS_SIGLIST HAVE_SYS__ERRLIST

you have no less than *three* mistakes in there ;)

1. You export HAVE_SYS_SIGLIST twice (second one is missing the
   extra underscore)

2./3. The extra underscore is not after the “sys”.

I originally wrote:
export HAVE_SYS_ERRLIST=0
export HAVE__SYS_ERRLIST=0
export HAVE_SYS_SIGLIST=0
export HAVE__SYS_SIGLIST=0

I’ve just tested a build with those four in the environment,
on Debian/x32, and it succeeded.

>[   61s] /home/abuild/rpmbuild/BUILD/mksh/mksh: [5]: 
>trap: bad signal 'INT'

Good to see that the testsuite works! ;-)

bye,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Re: [PATCH] make 'set +o' useful and POSIX compatible

[Thorsten Glaser]

Martijn Dekker dixit:

>> I’ll put the issue on my TODO only, for now, but thanks anyway.
>
>We're at R56 now but nothing seems to have changed here. Status?

It’s on the TODO.

bye,
//mirabilos
-- 
11:56⎜«liwakura:#!/bin/mksh» also, i wanted to add mksh to my own distro │
i was disappointed that there is no makefile │ but somehow the Build.sh is
the least painful built system i've ever seen │ honours CC, {CPP,C,LD}FLAGS
properly │ looks cleary like done by someone who knows what they are doing

Re: mksh R56 - change in '0' movement in vi command mode

[Thorsten Glaser]

Hi Larry,

>In mksh R56, with 'set -o vi' in effect, entering '0' no longer
>moves the cursor to the beginning of the line. That seems like a

thanks, indeed a regression, fixed in commitid 10059A356DB182F3791.

bye,
//mirabilos
-- 
22:20⎜ The crazy that persists in his craziness becomes a master
22:21⎜ And the distance between the craziness and geniality is
only measured by the success 18:35⎜ "Psychotics are consistently
inconsistent. The essence of sanity is to be inconsistently inconsistent

[Bug 1694943] Re: history from one of many parallel shell sessions sometimes not stored

[Thorsten Glaser]

According to multiplexd this is likely fixed in R56. If not, pleas e
open a new issue.

** Changed in: mksh
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1694943

Title:
  history from one of many parallel shell sessions sometimes not stored

Status in mksh:
  Fix Released

Bug description:
  Natureshadow reports that, when having multiple parallel mksh sessions
  that share the same HISTFILE, after exiting one of them and then
  starting a new one, the history from the shell just terminated is not
  available.

  We currently believe this is due to history truncation (i.e. number of
  history lines approaching $HISTSIZE). There are known issues with the
  code (which is another reason HISTFILE is unset by default, and
  persistent history support so -><- close to having been removed). A
  workaround is:

  ① make HISTSIZE bigger
  ② truncate manually by setting HISTSIZE small while only one shell is 
accessing the HISTFILE

  Ideally we’d have a PostgreSQL history backend, but that won’t work
  for obvious reasons. I’m not sure how much we can do with the current
  ⓐ history file format and ⓑ command line editing code, but
  improvements are planned for libmkshedit when at least the latter is
  going to be redesigned anyway and the former can change.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1694943/+subscriptions

[Bug 1709716] Re: FAIL ./check.t:arith-ternary-prec-1 with GCC >= 6.3 and LTO

[Thorsten Glaser]

It’s known that mksh’s testsuite exhibits problems in the infrastructure
(kernel, libraries, toolchain, compiler).

In this case, GCC’s LTO (and before that, -fwhole-program --combine) is
a repeat offender of breaking code.

If your builds exhibit this problem, you should *disable LTO for your
_entire_ operating system*. (I’ve seen this GCC version produce LTO bugs
on other distros as well, it’s wrong code generation, but LTO bugs are
not interesting to GCC developers.)


** Changed in: mksh
   Status: New => Invalid

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1709716

Title:
  FAIL ./check.t:arith-ternary-prec-1 with GCC >= 6.3 and LTO

Status in mksh:
  Invalid

Bug description:
  Not really sure where the issue is, whether it's mksh or gcc, but I'll
  try here first.

  Since GCC 6.3 (also present in 6.4, didn't check 7.x) I get
  reproducible failure in tests if mksh was built with LTO:

  sh ./Build.sh -c lto
  ...
  ./tests.sh
  ...
  FAIL ./check.t:arith-ternary-prec-1
Description:
Check precedence of ternary operator vs assignment
unexpected exit status 0 (exit-code 0), expected e != 0
unexpected stderr - wanted pattern:
/.*:.*1 \? 20 : x\+=2.*lvalue.*\n$/
got nothing

  If I run input from test:

  typeset -i x=2
  y=$((1 ? 20 : x+=2))

  the value of y is 22.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1709716/+subscriptions

Re: 'exec' runs shell functions and builtins

[Thorsten Glaser]

Robert Elz dixit:

>there is no right answer - and that the only rational result is that portable
>scripts cannot expect to be able to exec a function (or any builtin that is

Yes, but the others in this thread want to allow portable scripts
to rely on the exec builtin always resulting in a PATH search.
This is extra effort on my side *and* limiting.


Martijn Dekker dixit:

>Another fact is that pdksh was intended to be a clone of ksh88. Thus,

Not fully, and never exactly. With mksh that’s no part of the
mission statement any more anyway, although I believe it has
not been in pdksh for a while before its death in 1999 either.

>Every other shell except zsh acts like ksh88. And the zsh devs just
>fixed the problem for its POSIX mode the other day.

This sucks…

>Since you appear to feel strongly about this, maybe a compromise would
>be to make mksh act like ksh88 for POSIX mode only.

Yeah, let’s add more conditional code… I just did that ☹

>| The use of "program" and "overlaid the shell" here means that the
>| standard clearly does not allow the execution of built-in utilities and
>| functions.

I completely disagree, especially when combined with other wording.

But, well, POSIX was never among the more useful standards…

bye,
//mirabilos
-- 
“It is inappropriate to require that a time represented as
 seconds since the Epoch precisely represent the number of
 seconds between the referenced time and the Epoch.”
-- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2

Re: 'exec' runs shell functions and builtins

[Thorsten Glaser]

Martijn Dekker dixit:

>> And this is utter nonsense, the builtin does replace the shell process,
>
>No. On mksh, 'exec builtin' is exactly equivalent to 'builtin; exit'.
>The builtin is run in the existing shell process. No
>replacing/overlaying of the shell process takes place. It's easy to
>verify this in the source code.

This is an implementation detail. Whether it’s running the same
binary is independent of this.

Compare:

ln -s $(which mksh) print
PATH=.
exec print foo

This does the same, except the “replacing the shell by the builtin”
without an actual exec is cheaper. The *functionality* of the program
is still substituted from shell to builtin.

bye,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Re: 'exec' runs shell functions and builtins

[Thorsten Glaser]

Robert Elz dixit:

>  | This specifically means that builtins MAY be made available
>  | to exec, and that thatbs an expected modus operandi.
>
>It means nothing of the kind.   The System Interfaces volume of POSIX.1-2008
>is the part that defines the system calls.

There’s no “exec” system call, so that’s the wrong part
you’re searching in for this.

>What the text you quoted is saying (if you had quoted it all) is that
>all of the built-in utilities (except the special built-ins) must be
>able to be accessed by execle(2) (and its sibling interfaces.)
>
>Note not may, must.

Yes, that *too*, but it also says that they *may* be builtins.

>Aside from the possibility that the sh exec special-builtin might be
>intended to make an exec*() system call (the spec does not say that)

Note that the exec built-in utility does not need to perfom
an exec*() system call.

>  | Heck, you can exec a function!
>
>That is where we started.   That is, that is the question.  Can you?

Of course you do, see the definition of “command” near the
top of the shell section.

bye,
//mirabilos
-- 
“ah that reminds me, thanks for the stellar entertainment that you and certain
other people provide on the Debian mailing lists │ sole reason I subscribed to
them (I'm not using Debian anywhere) is the entertainment factor │ Debian does
not strike me as a place for good humour, much less German admin-style humour”

Re: 'exec' runs shell functions and builtins

[Thorsten Glaser]

Martijn Dekker dixit:

>And under what theory should 'exec' run a shell function?

Read up on what a “command” can be, pretty far up in
the Shell part.

Although I did not manage to get a pipeline or loop run.
Not sure if that’s a bug, but looking at the way statements
are parsed, probably not.

bye,
//mirabilos
-- 
18:47⎜ well channels… you see, I see everything in the
same window anyway  18:48⎜ i know, you have some kind of
telnet with automatic pong 18:48⎜ haha, yes :D
18:49⎜ though that's more tinyirc – sirc is more comfy

Re: 'exec' runs shell functions and builtins

[Thorsten Glaser]

Martijn Dekker dixit:

>to be a bug in POSIX terms; 'exec' is supposed to launch a program that
>overlays the current shell[2], implying the program launched by 'exec'
>is always external to the shell.

The built-in utility is “the program implementing command”,
and, with exec, it is not returning to the shell.

So, NO.

bye,
//mirabilos
-- 
11:56⎜«liwakura:#!/bin/mksh» also, i wanted to add mksh to my own distro │
i was disappointed that there is no makefile │ but somehow the Build.sh is
the least painful built system i've ever seen │ honours CC, {CPP,C,LD}FLAGS
properly │ looks cleary like done by someone who knows what they are doing

Re: 'exec' runs shell functions and builtins

[Thorsten Glaser]

Martijn Dekker dixit:

>This behaviour also appears to be contrary to the documentation in
>mksh(1) ("The command is executed without forking, replacing the shell
>process"). The test script below demonstrates that neither "exec"

And this is utter nonsense, the builtin does replace the shell process,

>replaces the shell process as the first executes a shell function and
>the second (within the function) executes a mksh builtin.

and you can even execute builtins by symlinking mksh to them,
try e.g. a symlink from mksh to print some day.

//mirabilos
-- 
(gnutls can also be used, but if you are compiling lynx for your own use,
there is no reason to consider using that package)
-- Thomas E. Dickey on the Lynx mailing list, about OpenSSL

[Bug 1694943] [NEW] history from one of many parallel shell sessions sometimes not stored

[Thorsten Glaser]

Public bug reported:

Natureshadow reports that, when having multiple parallel mksh sessions
that share the same HISTFILE, after exiting one of them and then
starting a new one, the history from the shell just terminated is not
available.

We currently believe this is due to history truncation (i.e. number of
history lines approaching $HISTSIZE). There are known issues with the
code (which is another reason HISTFILE is unset by default, and
persistent history support so -><- close to having been removed). A
workaround is:

① make HISTSIZE bigger
② truncate manually by setting HISTSIZE small while only one shell is accessing 
the HISTFILE

Ideally we’d have a PostgreSQL history backend, but that won’t work for
obvious reasons. I’m not sure how much we can do with the current ⓐ
history file format and ⓑ command line editing code, but improvements
are planned for libmkshedit when at least the latter is going to be
redesigned anyway and the former can change.

** Affects: mksh
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/1694943

Title:
  history from one of many parallel shell sessions sometimes not stored

Status in mksh:
  New

Bug description:
  Natureshadow reports that, when having multiple parallel mksh sessions
  that share the same HISTFILE, after exiting one of them and then
  starting a new one, the history from the shell just terminated is not
  available.

  We currently believe this is due to history truncation (i.e. number of
  history lines approaching $HISTSIZE). There are known issues with the
  code (which is another reason HISTFILE is unset by default, and
  persistent history support so -><- close to having been removed). A
  workaround is:

  ① make HISTSIZE bigger
  ② truncate manually by setting HISTSIZE small while only one shell is 
accessing the HISTFILE

  Ideally we’d have a PostgreSQL history backend, but that won’t work
  for obvious reasons. I’m not sure how much we can do with the current
  ⓐ history file format and ⓑ command line editing code, but
  improvements are planned for libmkshedit when at least the latter is
  going to be redesigned anyway and the former can change.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/1694943/+subscriptions

Re: An additional quote test for check.t

[Thorsten Glaser]

Hi Steffen,

>i think, the \r is expanded, the output is

a bit unsure what exactly you’re talking about, but you cannot
ever use echo to display strings portably.

Use 'print -r -- "$foo"' to display the contents of the foo parameter.

Is there anything you think mksh fails to do correctly? If so,
please indicate _exactly_ where and what and how.

I guess I’ll have to look into the Austin ML archive, what is
the Subject of the thread?

Thanks,
//mirabilos
-- 
Stéphane, I actually don’t block Googlemail, they’re just too utterly
stupid to successfully deliver to me (or anyone else using Greylisting
and not whitelisting their ranges). Same for a few other providers such
as Hotmail. Some spammers (Yahoo) I do block.

Re: \uXXXX on EBCDIC systems

[Thorsten Glaser]

Daniel Richard G. dixit:

>While UTF-8 isn't a thing in the z/OS environment, I think there could
>be value in printing something that will be converted by the existing
>EBCDIC->ASCII terminal/NFS conversion into correctly-formed UTF-8
>characters.
[…]
>in the terminal. Effectively an ASCII->EBCDIC->ASCII round trip.

While I still cringe at the way you Mainframe types (also the IBM
docs I read) write ASCII and mean 8-bit charsets, yes, that’s exactly
what I was thinking about.

>I don't know if there are use cases where this may yield unintuitive
>results... perhaps if this "nega-UTF-8" were redirected to a file and
>then processed further in z/OS, that may lead to some surprises. But in

That was *also* what I was thinking about. But then I remembered (and
this is why I replied to the old thread for this) that you said that
UTF-8 is rarely seen and UTF-EBCDIC isn’t seen at all and UCS-{2,4}
is just used on the system if needed.

>terms of doing something sensible when using a "\u" escape in an
>environment that shouldn't support it, it seems no worse than producing
>actual UTF-8 bytes.

Indeed — especially if you have an autoconvert layer between that
and the devices you interact with the mainframe with.

Of course, ASCII-mksh on z/OS would do away with all this; out of
curiosity, IIRC you said something about wanting to create an ASCII
environment for z/OS, have you come any closer to that?

(There’s that added data point of \u on EBCDIC-mksh writing
stuff to local files that ASCII-mksh could not read… unless it
were able to access the files “as if” they were autoconverted to
extended ASCII. Oh well, or just use iconv…)

So, to summarise, I believe we both agree in saying that this
(botch \u (and base-1 integers in utf8-mode) to output not
UTF-8 but UTF-8-converted-as-extended-ASCII-to-EBCDIC) makes
sense, or, at least, more than not doing it?

Thanks,
//mirabilos
-- 
>> Why don't you use JavaScript? I also don't like enabling JavaScript in
> Because I use lynx as browser.
+1
-- Octavio Alvarez, me and ⡍⠁⠗⠊⠕ (Mario Lang) on debian-devel

Re: mksh on EBCDIC, testing

[Thorsten Glaser]

Daniel Richard G. dixit:

[…]
>but that's about it.

OK.

>Note that "cc" isn't even the same type of compiler as xlc. The option
>syntax is different/incompatible, for starters.

It’s likely still an xlc frontend, as it’s detected as xlc.

>I saw that you put in a change to use xlc on this platform, however, so
>this should no longer be an issue.

True.

>> It’s used later on, but it could conceivably be optional.
>> My perl-foo is not very good either though… I’ll try something.
>
>I remember seeing somewhere that it's possible to make a "use" fail non-
>fatally. I think an eval was involved...

Yeah, an eval and some added magic. We have that now.

>Glad to help! Let me know if you want another go.

OK, please do so… I’ll mail you a tarball preview version.

>of that test, and the raw bytes therein, I would point out that the file
>did go through an EBCDIC->ASCII conversion when I copied it out of the

Yes, I expected that. I tried to fix a fair amount of the testcases
already… well, let’s see what to make of it. Unexpectedly, I also
managed to fix with not too much effort a case of where it tried to
use 0x80 as flag, applied to things like '0', which of course went
wrong (and was the likely cause for most of the ^G and ^L in the
testsuite output).

Thanks and goodnight,
//mirabilos
-- 
 you introduced a merge commit│ % g rebase -i HEAD^^
 sorry, no idea and rebasing just fscked │ Segmentation
 should have cloned into a clean repo  │  fault (core dumped)
 if I rebase that now, it's really ugh │ wuahh

mksh on EBCDIC, testing

[Thorsten Glaser]

Hi,

I’m starting a new thread so I can archive the old one.

I’ve merged most of your patch, only did some things differently.
I also added your copyright to misc.c which had the most logic
from you; feel free to send patches to change your copyright info
on files you feel you have co-authorship now.

I’ve disabled the Vi editing mode for EBCDIC for now, it’s got its
own lookup table (although we can probably handle that with the
rtt2asc() macro as well, now that I think about it).

I’ve not merged the huuuge “intro to EBCDIC in the comments” part,
as some of it is no longer entirely true after our discussion.
Please *do* update it and send me an appropriate patch when you
have time.

Can you please test mksh CVS HEAD? (CVS and github mirror should
update within the next couple of minutes.)

$ sh Build.sh -r -E && ./test.sh

The new -E option is necessary to enable EBCDIC, I chose to not
rely on autodetection as we will need this in the testsuite, too.

Thanks,
//mirabilos
-- 
[...] if maybe ext3fs wasn't a better pick, or jfs, or maybe reiserfs, oh but
what about xfs, and if only i had waited until reiser4 was ready... in the be-
ginning, there was ffs, and in the middle, there was ffs, and at the end, there
was still ffs, and the sys admins knew it was good. :)  -- Ted Unangst über *fs

[Bug 625164] Re: some extended Korn shell globs are really slow

[Thorsten Glaser]

mksh’s globbing really sucks. Way out: parse them as special kind of
regex (NFA with making $KSH_MATCH an array, possibly).

References:

- https://research.swtch.com/glob

- https://swtch.com/~rsc/regexp/regexp1.html

-- 
You received this bug notification because you are a member of mksh
Mailing List, which is subscribed to mksh.
Matching subscriptions: mkshlist-to-mksh-bugmail
https://bugs.launchpad.net/bugs/625164

Title:
  some extended Korn shell globs are really slow

Status in mksh:
  In Progress

Bug description:
  Attaching a testcase.

  The first glob is decently fast, the second one is noticeable (about a 
second) on a 3 GHz Athlon.
  The third glob needs to be killed with SIGKILL out of all things.

  Also, I have another shell script, where replacing
  [[ $foo = *@(x)* ]] with [[ $foo = *'x'* ]] and
  [[ $foo = @(1|2………),* ]] with [[ $foo = 1,* || $foo = 2………,* ]
  made it noticeable faster.

  The probable culprit is gmatchx, do_gmatch, and friends, mostly from
  misc.c.

  This bug serves as documentation for now, because I have no idea how
  to tackle it, but if someone takes it up and submits patches, be my
  guest.

To manage notifications about this bug go to:
https://bugs.launchpad.net/mksh/+bug/625164/+subscriptions