Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Alan Coopersmith]

On 05/29/17 01:56 PM, Joerg Schilling wrote:

Alan Coopersmith  wrote:


We still ship a couple closed source shells in Solaris - the SVR4 Bourne Shell
as /usr/sunos/bin/sh and a modified ksh88 as /usr/xpg4/bin/sh, but our primary
day to day shells are the common open source ones - bash, ksh93, zsh, etc.


Is there any bugfix in /usr/sunos/bin/sh that is not in the Svr4 compatile
copmpile variant of bosh? Or is it unmodified since the Sun updates for
OpenSolaris stopped?


I have no idea what is or is not in bosh.  Most of the changes on our side
since that source stopped being published were to adapt to changing build
environments and to prepare for it's EOL.


Is your ksh88 managed in a way that keeps it compatible with the evolving POSIX
standard?


We aren't chasing moving targets here - we're targeting stable releases like
UNIXv7.  We have fixed several bugs in our /usr/xpg4/bin/sh that were needed
to pass various cases in the UNIXv7 test suite in past years.


Did you upgrade ksh93 since 2010?


Yes - you can see that we're using the 2012-08-01 release in our open source 
trees at:

https://github.com/oracle/solaris-userland/tree/master/components/ksh93


If yes, have you been able to keep it as fast as it was in OpenSolaris?


I have no idea - I don't know what commonly accepted shell benchmark we'd
use to tell.

--
-Alan Coopersmith-   alan.coopersm...@oracle.com
 Oracle Solaris Engineering - https://blogs.oracle.com/alanc

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Alan Coopersmith  wrote:

> We still ship a couple closed source shells in Solaris - the SVR4 Bourne Shell
> as /usr/sunos/bin/sh and a modified ksh88 as /usr/xpg4/bin/sh, but our primary
> day to day shells are the common open source ones - bash, ksh93, zsh, etc.

Is there any bugfix in /usr/sunos/bin/sh that is not in the Svr4 compatile 
copmpile variant of bosh? Or is it unmodified since the Sun updates for 
OpenSolaris stopped?

Is your ksh88 managed in a way that keeps it compatible with the evolving POSIX 
standard?

Did you upgrade ksh93 since 2010? If yes, have you been able to keep it as fast 
as it was in OpenSolaris? When I compile a recent ksh93 source, it is 21% 
slower than the OpenSolaris ksh93 from April 2010 and 7% slower than a recent 
bosh.

Parts of the slowdown are caused by the fact that the cat builtin is not in the 
search path, and parts of the slowdown is caused by the huge binary that is not 
using shared linking (fork becomes slow) but I doubt that this could sum up to 
21%.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Alan Coopersmith]

On 05/29/17 04:37 AM, Robert Elz wrote:

There is no requirement that you, or I, or anyone else in particular,
be able to build and run any particular shell.   While that's nice if
possible, a shell is still a shell even if its source code is not openly
available at all (like any that are in closed source systems, if any of
those still really exist.)   Users get to experience those shells too.


We still ship a couple closed source shells in Solaris - the SVR4 Bourne Shell
as /usr/sunos/bin/sh and a modified ksh88 as /usr/xpg4/bin/sh, but our primary
day to day shells are the common open source ones - bash, ksh93, zsh, etc.

--
-Alan Coopersmith-   alan.coopersm...@oracle.com
 Oracle Solaris Engineering - https://blogs.oracle.com/alanc

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Stephane CHAZELAS  wrote:

> 2017-05-24 17:21:33 +0200, Joerg Schilling:

> > Do you have a pointer to the POSIX text that forbids a subshell in this 
> > case 
> > when stdin is redirected?
> [...]
>
> I don't know if you'll find some text that *explicitly* forbids
> it to run in a subshell, but you won't find text that *allows*
> it (just like you won't see text that allows { a=2; } alone to
> run in a subshell environment)
>
> On the other hand, you'll find text that explicitely allows it
> to run in a subshell environment in:
>
> { a=2; } | cmd
>
> cmd | { a=2; }
>
> (even though for the latter, many shells don't run it in a
> subshell environment).

Given that I planned a related change since ~2 years, I now implemented an 
enhancement to the parser and to this did even result in an aprox. speedup of 2%
with my "configure" performance test.

The Bourne Shell now auto-exports imported variables when it starts up un POSIX 
mode.

Multiple non-space IFS characters now result in empty arguments.


The new schilytools bundle is here:


https://sourceforge.net/projects/schilytools/files/schily-2017-05-29.tar.bz2

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Robert Elz]

Date:Mon, 29 May 2017 14:38:02 +0200
From:Joerg Schilling 
Message-ID:  <592c162a.syoz+dozzpogtqep%joerg.schill...@fokus.fraunhofer.de>

  | The problem is that I have a FreeBSD system available for testing,
  | but not a NetBSD system...

It is fairly easy to set up NetBSD - it runs (kind of badly...) under
qemu, and quite well under Xen, virtualbox and vmware.

(If you want to try any of that, for this purpose, definitely use one
of the NetBSD current daily builds, not a release version, the shell in
those is ancient...)

  | Moreover, I like to run frequent tests with various shells on my main 
  | development machine.

Yes, so do I, but I don't have them all (too much of a good thing!)

  | *BSD typically uses things like "err()" that are not portable.

Yes, a lot of utilities do, using that encourages more consistent behaviour,
but the shell does not (err() is kind of obvious - what it does is almost
never what the shell would want to do, but we don't use warn() either).

For the utilities that do use them, when they're being compiled on some other
system (NetBSD can be built almost anywhere, but we need a sane environment
to do that, so the first step of the build process is to build a set of
NetBSD tools on the host) there is a nbcompat library that can be used that
provides essentially all of the pieces missing from other libc's.

  | The fact that a real vfork() implementation changes the timing,
  | causes some constructs to fail unless there is special code inside.

Yes, there is "special code" - our shell can only use vfork() if it is
a real vfork() - if a system provides a vfork() that is just fork() wrapper
(or some abomination in between) for the benefit or programs that assume
that vfork() exists, and use it blindly, then that's no use at all, and
our shell won't (can't) use that, it will just fork() instead.

  | You may like to make your shell available if you are interested in getting 
  | other people to test it.

Yes, one day, I will.   That's definitely an objective.   But right now
I have enough bugs (and missing pieces) that I already know about that need
fixing, that I don't really need people finding more which aren't currently
bothering anyone (many of the bugs are in exotica that no real script ever
uses.)

kre

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Robert Elz  wrote:

> Date:Mon, 29 May 2017 11:47:04 +0200
> From:Joerg Schilling 
> Message-ID:  
> <592bee18.kvZiYpLd6LS/Evf/%joerg.schill...@fokus.fraunhofer.de>
>
>   | If the related shells would be available as portable code,
>   | it would be easy to test them.
>
> There is no requirement that you, or I, or anyone else in particular,
> be able to build and run any particular shell.   While that's nice if
> possible, a shell is still a shell even if its source code is not openly
> available at all (like any that are in closed source systems, if any of
> those still really exist.)   Users get to experience those shells too.

The problem is that I have a FreeBSD system available for testing, but not a 
NetBSD system...

Moreover, I like to run frequent tests with various shells on my main 
development machine.

> If no-one can test the shell, that would be different, but as long as someone
> can run tests, we have all we absolutely need.
>
>   | For now, the problem is that these shells massively rely on non-POSIX 
>   | extensions in /usr/include and in libc.
>
> I don't think the NetBSD shell has any such code in it (the debugging code
> relies on asprintf() and friends, but you don't need that to compile it.)
> Formatter output (for messages, and such) will use vasprintf() if available,
> but that's conditionally compiled, and can be done without.

*BSD typically uses things like "err()" that are not portable.

> Similarly, use of vfork() is conditionally compiled (it runs the same
> without, just slower).  We only use waitpid() currently (which is POSIX)
> though the shell is supposed to be able to build and work (with some loss
> of functionality) with just wait() for ancient systems.   (I am adding
> waitid() support as one of my back-burner tasks, but that doesn't really
> fix anything that's broken, so it is low priority.)

vfork() is not a real problem except when your vfork() support code might not 
be correct enough to correctly deal with process groups and similar. The fact 
that a real vfork() implementation changes the timing, causes some constructs 
to fail unless there is special code inside.

You may like to make your shell available if you are interested in getting 
other people to test it.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Robert Elz]

Date:Mon, 29 May 2017 11:47:04 +0200
From:Joerg Schilling 
Message-ID:  <592bee18.kvZiYpLd6LS/Evf/%joerg.schill...@fokus.fraunhofer.de>

  | If the related shells would be available as portable code,
  | it would be easy to test them.

There is no requirement that you, or I, or anyone else in particular,
be able to build and run any particular shell.   While that's nice if
possible, a shell is still a shell even if its source code is not openly
available at all (like any that are in closed source systems, if any of
those still really exist.)   Users get to experience those shells too.

If no-one can test the shell, that would be different, but as long as someone
can run tests, we have all we absolutely need.

  | For now, the problem is that these shells massively rely on non-POSIX 
  | extensions in /usr/include and in libc.

I don't think the NetBSD shell has any such code in it (the debugging code
relies on asprintf() and friends, but you don't need that to compile it.)
Formatter output (for messages, and such) will use vasprintf() if available,
but that's conditionally compiled, and can be done without.

Similarly, use of vfork() is conditionally compiled (it runs the same
without, just slower).  We only use waitpid() currently (which is POSIX)
though the shell is supposed to be able to build and work (with some loss
of functionality) with just wait() for ancient systems.   (I am adding
waitid() support as one of my back-burner tasks, but that doesn't really
fix anything that's broken, so it is low priority.)

The reason I don't make a tarball of the NetBSD shell available (anyone who
wants to look at it can get it from the NetBSD anoncvs repo of course, none
of it is secret) isn't that the code won't compile, but that there's no
build system for it ... the Makefile more or less just lists the names of
the source files, and says how to run the (build time) scripts that are
used to generate the files that come from a non-C format (like the list of
builtin commands - which gets converted into appropriate .h and .c files
using a script.)

Everything else is handled by NetBSD make system magic - including finding the
sources of the programs (like kill, printf, test ...) that are included as
builtins (normally anyway) - we use the same sources as are used for the
/bin (or /usr/bin or whatever) versions of the commands, so none of that gets
included with the sh sources.

Building a portable build system is one of my long term goals, but I think
that (for now) a better objective is to fix bugs, and (where appropriate)
posix conformance, rather than worrying about how someone else would build it.

If you want to test what it does with some particular script, just send me
the script, and I'll run it and send you back the results.

kre

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Robert Elz  wrote:

>   | It also would make sense to check with shells have maintainers
>   | that are on this list and willing to come to a common agreement.
>
> Neither were on your list, but the FreeBSD shell and NetBSD shell maintainers
> are here too...

If the related shells would be available as portable code, it would be easy to 
test them.

For now, the problem is that these shells massively rely on non-POSIX 
extensions in /usr/include and in libc.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Robert Elz  wrote:

> To get away from the dick waving "my shell is bigger than yours"
> discussions for a minute ...
>
> In note 3745 attached to http://austingroupbugs.net/view.php?id=767
> Joerg Schilling proposes making a list of shells to use to help
> guide what can be regarded as "standard" for the purposes of compliance.
>
> In that he said ...
>
>First, I believe that "yash" and "posh" should not be in that list
>because these implementations are too buggy or have unexplained
>deviations
>
> with which I don't agree.   First, because any existing deviations with
> the standard aren't really (or at least, necessarily) relevant when
> considering some new issue, and second if a shell (any shell) does not

Well I believe it matters when a shell like "posh" fails in _many_ places 
because the code of the shell is not POSIX compliant.

It may be that the authors of posh did never compile it on a POSIX system.
Last weekend, I had a look into the source and discovered that it is _expected_ 
to fail, the way it was written, because it ignores the getopt() documentation.

It looks interesting that the problem is inside one of two files that is (unlike
the rest) using a GPL header. The rest of the code is OK, so it looks like one 
of
those cases where Debian people modified code and thus caused problems.

The commands that fail (are unusable) because of that problem are:

break, cd, exit, export, eval (dumps core), pwd,  readonly, return, 
set, shift, trap, umask

To me, posh thus makes no sense to be tested at all until it is fixed.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Robert Elz]

Date:Wed, 24 May 2017 20:51:25 +0200
From:Joerg Schilling 
Message-ID:  <5925d62d.mr6gk0tc0ybr6h6k%joerg.schill...@fokus.fraunhofer.de>


  | The problem would be if there is deviating behavior only between the shells 
  | that are closer to POSIX and yash/posh, it may be a good idea not to mark
  | the specific behavior as "unspecified" in the standard.

That would depend upon what the specific behaviour is, and why it is
deviating, not upon the name of the shell that displays it.

  | It also would make sense to check with shells have maintainers
  | that are on this list and willing to come to a common agreement.

Neither were on your list, but the FreeBSD shell and NetBSD shell maintainers
are here too...

kre

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Robert Elz  wrote:

> To get away from the dick waving "my shell is bigger than yours"
> discussions for a minute ...
>
> In note 3745 attached to http://austingroupbugs.net/view.php?id=767
> Joerg Schilling proposes making a list of shells to use to help
> guide what can be regarded as "standard" for the purposes of compliance.
>
> In that he said ...
>
>First, I believe that "yash" and "posh" should not be in that list
>because these implementations are too buggy or have unexplained
>deviations
>
> with which I don't agree.   First, because any existing deviations with
> the standard aren't really (or at least, necessarily) relevant when
> considering some new issue, and second if a shell (any shell) does not
> behave as expected (or desired) and that is generally agreed to be a
> bug (or even an unexplained deviation) then for that issue, that shell
> is simply discounted (or it could even be regarded as supporting the
> proposed standard if the implementor intends to fix that shell.)

This is not the problem. 

The problem would be if there is deviating behavior only between the shells 
that are closer to POSIX and yash/posh, it may be a good idea not to mark the 
specific behavior as "unspecified" in the standard.

It also would make sense to check with shells have maintainers that are on this 
list and willing to come to a common agreement. From what I can say, the 
authors of

bash
bosh
mksh
ksh

are on this list.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Robert Elz]

To get away from the dick waving "my shell is bigger than yours"
discussions for a minute ...

In note 3745 attached to http://austingroupbugs.net/view.php?id=767
Joerg Schilling proposes making a list of shells to use to help
guide what can be regarded as "standard" for the purposes of compliance.

In that he said ...

   First, I believe that "yash" and "posh" should not be in that list
   because these implementations are too buggy or have unexplained
   deviations

with which I don't agree.   First, because any existing deviations with
the standard aren't really (or at least, necessarily) relevant when
considering some new issue, and second if a shell (any shell) does not
behave as expected (or desired) and that is generally agreed to be a
bug (or even an unexplained deviation) then for that issue, that shell
is simply discounted (or it could even be regarded as supporting the
proposed standard if the implementor intends to fix that shell.)

What matters is what users will actually see, what they can count on.
And for that, the shells that don't measure up to someone's private
view of what is important are just as relevant as the others.

kre

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Stephane CHAZELAS  wrote:

> > > > $ ./sh -o posix -c 'a=1; { a=2; } < /dev/null; echo "$a"'
> > > > 1
> > > >
> > > > Again 2 required by POSIX.
> > 
> > Do you have a pointer to the POSIX text that forbids a subshell in this 
> > case 
> > when stdin is redirected?
> [...]
>
> I don't know if you'll find some text that *explicitly* forbids
> it to run in a subshell, but you won't find text that *allows*
> it (just like you won't see text that allows { a=2; } alone to
> run in a subshell environment)

Whether this is relevant with POSIX mainly depends on whether you expect a 
specific behavior with this command:

  lastline=
  while read line
  do

  lastline=$line
  done < /etc/passwd
  echo "lastline=$lastline"


The Bourne Shell always created a subshell for a complex command with I/O 
redirection, but it is on my TODO list for useful changes in future.

Just as a note: "posh" and the Bourne Shell both print an empty line here.


> On the other hand, you'll find text that explicitely allows it
> to run in a subshell environment in:
>
> { a=2; } | cmd
>
> cmd | { a=2; }
>
> (even though for the latter, many shells don't run it in a
> subshell environment).

This is a general problem: POSIX does not mention the way pipes are set up and 
probably never will as this yould enforce many shell to be rewritten.

The historic Bourne Shell did create many sub shells with pipes because this 
could be done with less code and RAM was rare in the 1970s.

bosh has been rewritten completely with repect to pipe handling and this is one 
of the reasons why it is noticeably faster than the historic Bourne Shell.

This is because the new pipe handling in bosh makes the rightmost process the 
foreground process which allows:

echo bla | read var

to behave as people would expect.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Chet Ramey]

On 5/24/17 11:40 AM, Stephane CHAZELAS wrote:
> 2017-05-24 17:21:33 +0200, Joerg Schilling:

>> Do you have a pointer to the POSIX text that forbids a subshell in this case 
>> when stdin is redirected?
> [...]
> 
> I don't know if you'll find some text that *explicitly* forbids
> it to run in a subshell, but you won't find text that *allows*
> it (just like you won't see text that allows { a=2; } alone to
> run in a subshell environment)

"Command substitution, commands that are grouped with parentheses, and
asynchronous lists shall be executed in a subshell environment.
Additionally, each command of a multi-command pipeline is in a subshell
environment; as an extension, however, any or all commands in a pipeline
may be executed in the current environment. All other commands shall be
executed in the current shell environment."

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_12

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRUc...@case.eduhttp://cnswww.cns.cwru.edu/~chet/

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Stephane CHAZELAS]

2017-05-24 17:21:33 +0200, Joerg Schilling:
> Joerg Schilling  wrote:
> 
> > I did not change the Bourne Shell behavior as I believe that the 
> > export behavior is compatibile with POSIX and as this behavior is more 
> > useful 
> > than what other shells do.
> >
> > > $ ./sh -o posix -c 'a=1; { a=2; } < /dev/null; echo "$a"'
> > > 1
> > >
> > > Again 2 required by POSIX.
> 
> Do you have a pointer to the POSIX text that forbids a subshell in this case 
> when stdin is redirected?
[...]

I don't know if you'll find some text that *explicitly* forbids
it to run in a subshell, but you won't find text that *allows*
it (just like you won't see text that allows { a=2; } alone to
run in a subshell environment)

On the other hand, you'll find text that explicitely allows it
to run in a subshell environment in:

{ a=2; } | cmd

cmd | { a=2; }

(even though for the latter, many shells don't run it in a
subshell environment).

-- 
Stephane

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Dan Kegel]

On Wed, May 24, 2017 at 8:15 AM, Joerg Schilling
 wrote:
>> $ a=1 ./sh -o posix -c 'a=2; printenv a'
>> 1
>>
>> POSIX requires 2 to be output there (assuming printenv a command
>> to print the value of a given environment variable).
>
> Do you have a pointer to the related text?

I write scripts every day that depend on POSIX behavior here, so I
would be cross at a shell that outputs '1' in this case.

FWIW, http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
says "If a variable is initialized from the environment, it shall be
marked for export immediately".

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Stephane CHAZELAS  wrote:

> > >   ZSH_EMULATION=sh CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure
> > 
> > Ok this is what I get:
> > 
> > ZSH_EMULATION=sh CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure
> > ./configure:777: no matches found: conftest*
> > creating cache ./config.cache
> > ^C./configure:828: no matches found: conftest*
> [...]
>
> You possibly overlooked the part about adding
>
> if [ -n "$ZSH_EMULATION" ]; then
>   emulate "$ZSH_EMULATION"
> fi
>
> to your ~/.zshenv.

OK, thank you it now works...

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Joerg Schilling  wrote:

> I did not change the Bourne Shell behavior as I believe that the 
> export behavior is compatibile with POSIX and as this behavior is more useful 
> than what other shells do.
>
> > $ ./sh -o posix -c 'a=1; { a=2; } < /dev/null; echo "$a"'
> > 1
> >
> > Again 2 required by POSIX.

Do you have a pointer to the POSIX text that forbids a subshell in this case 
when stdin is redirected?


> What I had to do to come to that point was to support /bin/sh as automatic 
> POSIX enabler, add support for "local" and implement the IFS field splitting 
> as 
> done by ksh88 after I assumed to be ready.

Sorry for the wrong order, the paragrap above was related to Gentoo.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Stephane CHAZELAS]

2017-05-24 17:00:25 +0200, Joerg Schilling:
[...]
> > Let's be serious, if "posh" had such a bug it would have been
> > reported long ago and fixed. Presumably, you've compiled
> > it in a such way or on such a system that has never been tested
> > before and triggered a bug.
> 
> A shell that does not work correctly when compiled from the latest source has 
> a 
> problem. This definitely applies to posh.

Definitely. But here, that's (most probably) a compatibility
bug. It says nothing about posh usability as a POSIX shell on
systems where it builds successfully.

> > to your ~/.zshenv and run your "configure" script as:
> >
> >   ZSH_EMULATION=sh CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure
> 
> Ok this is what I get:
> 
> ZSH_EMULATION=sh CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure
> ./configure:777: no matches found: conftest*
> creating cache ./config.cache
> ^C./configure:828: no matches found: conftest*
[...]

You possibly overlooked the part about adding

if [ -n "$ZSH_EMULATION" ]; then
  emulate "$ZSH_EMULATION"
fi

to your ~/.zshenv.

Verify by running:

ZSH_EMULATION=sh zsh -xc 'echo /configtest*'

It should show something like:

+/home/chazelas/.zshenv:1> [ -n sh ']'
+/home/chazelas/.zshenv:2> emulate sh
+zsh:1> echo '/configtest*'
/configtest*

-- 
Stephane

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Stephane CHAZELAS  wrote:

> I reported 3 as that was the first 3 I tested:
>
> $ a=1 ./sh -o posix -c 'a=2; printenv a'
> 1
>
> POSIX requires 2 to be output there (assuming printenv a command
> to print the value of a given environment variable).

Do you have a pointer to the related text?

I did not change the Bourne Shell behavior as I believe that the 
export behavior is compatibile with POSIX and as this behavior is more useful 
than what other shells do.

> $ ./sh -o posix -c 'a=1; { a=2; } < /dev/null; echo "$a"'
> 1
>
> Again 2 required by POSIX.

I have to check the reason for that...

> $ a=1::2: ./sh -o posix -f -c 'IFS=:; printf "<%s>\n" $a'
> <1>
> <2>
>
> POSIX requires:
>
> <1>
> <>
> <2>

This is the only problem I remember to have received from you so far.
It seems that the field separation changes introduced a years ago have been 
buggy.

> > has been verified to correctly work as the system shell (/bin/sh) on 
> > Gentoo Linux in case that it was compiled with -DDO_POSIX_SH
>
> Do you mean that you have tested all the scripts ever written
> for gentoo and verified that they worked as expected?

Well, if the system comes up to multi user and is otherwise OK and with no 
warning or error, this seems to be a fairly good coverage. 

On OpenSolaris, things have been easier:

-   All scripts have been known to be Bourne Shell compatible
before April 2010

-   After April 2010, a number of scripts have been changed to use ksh93
syntay, so I needed to change these scripts to require no more than
what POSIX grants in order to make bosh the system shell for SchilliX.

What I had to do to come to that point was to support /bin/sh as automatic 
POSIX enabler, add support for "local" and implement the IFS field splitting as 
done by ksh88 after I assumed to be ready.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Stephane CHAZELAS]

2017-05-24 16:21:05 +0200, Joerg Schilling:
> Stephane Chazelas  wrote:
> 
> > 2017-05-24 14:48:12 +0100, Stephane Chazelas:
> > [...]
> > > > bash
> > > > bosh
> > > 
> > > As discussed, as of the 2017-05-16 release, posh was not POSIX
> > > and still had many of non-conformances of the Bourne shell.
> > [...]
> >
> > Sorry, I meant "bosh", not "posh" above.
> 
> Well, you reported one non-conformance for the Bourne Shell and the Bourne 
> Shell has been verified to correctly work as the system shell (/bin/sh) on 
> Gentoo Linux in case that it was compiled with -DDO_POSIX_SH
[...]

I reported 3 as that was the first 3 I tested:

$ a=1 ./sh -o posix -c 'a=2; printenv a'
1

POSIX requires 2 to be output there (assuming printenv a command
to print the value of a given environment variable).

$ ./sh -o posix -c 'a=1; { a=2; } < /dev/null; echo "$a"'
1

Again 2 required by POSIX.

$ a=1::2: ./sh -o posix -f -c 'IFS=:; printf "<%s>\n" $a'
<1>
<2>

POSIX requires:

<1>
<>
<2>

The 

> has been verified to correctly work as the system shell (/bin/sh) on 
> Gentoo Linux in case that it was compiled with -DDO_POSIX_SH

Do you mean that you have tested all the scripts ever written
for gentoo and verified that they worked as expected?

-- 
Stephane

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Stephane CHAZELAS  wrote:

> 2017-05-24 16:15:03 +0200, Joerg Schilling:

> > A configure script (if done correctly) is a script with _very_ 
> > pessimistic assumptions on portability. Given that "posh" completely fails 
> > here, makes it unusable for every day work.
>
> Without being more specific than a script named "configure", I
> can't really help. Even if you mean a GNU autoconf generated
> "configure" script, that will still depend on whatever extra
> "check" code snipets the developer has haded.

This is "Schily configure" that has been build starting with GNU configure-2.13.
There are no non-portable parts in the script and any other halfway OK shell 
has no problems with it. So this is definitely a "posh" problem that would need 
to be addressed by the posh people before takting posh for serious.


> > $ exit
> > posh: exit: bad number
>
> Let's be serious, if "posh" had such a bug it would have been
> reported long ago and fixed. Presumably, you've compiled
> it in a such way or on such a system that has never been tested
> before and triggered a bug.

A shell that does not work correctly when compiled from the latest source has a 
problem. This definitely applies to posh.

> to your ~/.zshenv and run your "configure" script as:
>
>   ZSH_EMULATION=sh CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure

Ok this is what I get:

ZSH_EMULATION=sh CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure
./configure:777: no matches found: conftest*
creating cache ./config.cache
^C./configure:828: no matches found: conftest*

Does not look convincing... ^C is as I needed to stop a hang.

> > If you like to use the Bourne Shell as Linux default shell, compile it with:
> [...]
>
> As discussed, even with -o posix, in my tests it's still not
> POSIX (for instance with regards to the environment processing,
> with regards to redirected compound commands being run in
> subshells, and $IFS processing at least).

I did get one single bugreport from you related to field splitting.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Vincent Lefevre  wrote:

> On 2017-05-24 16:15:03 +0200, Joerg Schilling wrote:
> > I have no idea what "posh" targets, but given that:
> > 
> > posh
> > $ exit
> > posh: exit: bad number
>
> I can't reproduce this issue:
>
> cventin:~> posh
> $ exit
> cventin:~> 
>
> on a Debian/unstable machine (posh 0.12.6, 14 Feb 2016, so with
> a version from more than 1 year ago).

I fetched the source tree posh_0.12.6.tar.xz from Debian.

md5sum posh_0.12.6.tar.xz
7f8ba439620ea64cc37c32b2255b27c9  posh_0.12.6.tar.xz

The newest file in that tar archive is:

  23566 -rwxr-xr-x0/0   Feb 14 18:57 2016 posh/depcomp

There was just a problem with creating the file "siglist.out"
as the given Makefile is broken, so I gessed how to create the file and did it 
manually.

The rest was just a vanilla make call.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Stephane CHAZELAS]

2017-05-24 16:15:03 +0200, Joerg Schilling:
> Stephane Chazelas  wrote:
> 
> > If your "configure" script works with "mksh" but not with
> > "posh", I suspect it's because that script is using non-POSIX
> > ksh extensions, as basically mksh is pdksh with a few  fixes and
> > extensions added, while posh is pdksh with a few fixes and most
> > ksh extensions (over POSIX, not over Bourne) removed.
> 
> This is not true...
> 
> A configure script (if done correctly) is a script with _very_ 
> pessimistic assumptions on portability. Given that "posh" completely fails 
> here, makes it unusable for every day work.

Without being more specific than a script named "configure", I
can't really help. Even if you mean a GNU autoconf generated
"configure" script, that will still depend on whatever extra
"check" code snipets the developer has haded.

> > Note that AFAIK, posh's target is the Debian policy, not POSIX. Though
> > the Debian policy is for the most part a superset of POSIX (and
> > a few parts from Unix/XSI)
> 
> I have no idea what "posh" targets, but given that:
> 
> posh
> $ exit
> posh: exit: bad number

Let's be serious, if "posh" had such a bug it would have been
reported long ago and fixed. Presumably, you've compiled
it in a such way or on such a system that has never been tested
before and triggered a bug.

"exit" definitely works in the Debian package of "posh" (the
primary target system as its goal is to be a tool for Debian
developers to identify non-POSIX usages).

[...]
> > > bosh
> >
> > As discussed, as of the 2017-05-16 release, posh was not POSIX
> > and still had many of non-conformances of the Bourne shell.
> 
> "posh" has a 2017-05-16 release?

Sorry again. As I clarified in a later email, I meant "bosh",
not "posh".

[...]
> Back to zsh:
> 
> If there is no way to modify this command line:
> 
>   CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure
> 
> in a way that allows to run the configure script, this is a bad
> state for the current usability of zsh. I currently need to do the
> following:
> 
>   cp /usr/bin/zsh /tmp
>   CONFIG_SHELL=/tmp/sh /tmp/sh ./configure
[...]

I don't see how that has anything to do with the usability of
zsh, but you can always add:

if [ -n "$ZSH_EMULATION" ]; then
  emulate "$ZSH_EMULATION"
fi

to your ~/.zshenv and run your "configure" script as:

  ZSH_EMULATION=sh CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure

> > mksh, bash and yash at least have a "posix" option to enforce a
> > more compliant mode (at least bash enables it when invoked as
> > "sh"). See also the xpg_echo option in bash which you also need
> > to have a compliant echo.
> 
> There are a few features where the "Bourne Shell" would cause problems
> with existing Bourne Shell scripts if it was by default 100% POSIX.

Yes of course, POSIX is not backward compatible with the Bourne
shell, nobody denies that. The POSIX spec was never based on the
Bourne shell.

You cannot make a shell both Bourne compatible and POSIX
compatible at the same time.

> For this reason it implements set -o posix that is auto-activated if it was
> installed as /usr/xpg4/bin/sh (note that a symlink will not work here).
> 
> If you like to use the Bourne Shell as Linux default shell, compile it with:
[...]

As discussed, even with -o posix, in my tests it's still not
POSIX (for instance with regards to the environment processing,
with regards to redirected compound commands being run in
subshells, and $IFS processing at least).

-- 
Stephane

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Vincent Lefevre]

On 2017-05-24 16:15:03 +0200, Joerg Schilling wrote:
> I have no idea what "posh" targets, but given that:
> 
> posh
> $ exit
> posh: exit: bad number

I can't reproduce this issue:

cventin:~> posh
$ exit
cventin:~> 

on a Debian/unstable machine (posh 0.12.6, 14 Feb 2016, so with
a version from more than 1 year ago).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Stephane Chazelas  wrote:

> 2017-05-24 14:48:12 +0100, Stephane Chazelas:
> [...]
> > > bash
> > > bosh
> > 
> > As discussed, as of the 2017-05-16 release, posh was not POSIX
> > and still had many of non-conformances of the Bourne shell.
> [...]
>
> Sorry, I meant "bosh", not "posh" above.

Well, you reported one non-conformance for the Bourne Shell and the Bourne 
Shell has been verified to correctly work as the system shell (/bin/sh) on 
Gentoo Linux in case that it was compiled with -DDO_POSIX_SH

BTW: bosh is faster than dash even though bosh implements support for 
multi-byte chars and dash does not. My benchmarks show that 30% of the total
CPU time in bosh is needed for multi-byte support.

The Bourne Shell is already used as the system shelll on SchilliX and so I
see the need to fix a reported bug, but I would not call it non-POSIX.

yash and posh however really have dozens of hard deviations from POSIX.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Stephane Chazelas  wrote:

> If your "configure" script works with "mksh" but not with
> "posh", I suspect it's because that script is using non-POSIX
> ksh extensions, as basically mksh is pdksh with a few  fixes and
> extensions added, while posh is pdksh with a few fixes and most
> ksh extensions (over POSIX, not over Bourne) removed.

This is not true...

A configure script (if done correctly) is a script with _very_ 
pessimistic assumptions on portability. Given that "posh" completely fails 
here, makes it unusable for every day work.


> Note that AFAIK, posh's target is the Debian policy, not POSIX. Though
> the Debian policy is for the most part a superset of POSIX (and
> a few parts from Unix/XSI)

I have no idea what "posh" targets, but given that:

posh
$ exit
posh: exit: bad number

and that a similar error is created with "break", I cannot take it for serious.


also:

CONFIG_SHELL=/tmp/posh /tmp/posh ./configure

./configure:728: bad signal rm -fr conftest* confdefs* core core.* *.core 
$ac_clean_files; exit 1, not setting trap
creating cache ./config.cache
./configure:872: break: break: bad number

The line in question is:

trap 'rm -fr conftest* confdefs* core core.* *.core $ac_clean_files; exit 1' 1 
2 15


> > So a useful list should have:
> > 
> > bash
> > bosh
>
> As discussed, as of the 2017-05-16 release, posh was not POSIX
> and still had many of non-conformances of the Bourne shell.

"posh" has a 2017-05-16 release?

> > I am not sure whether we should add "zsh" to this list as zsh
> > by default implements many intentional deviations to the standard.
> > 
> > Note that I know of no way of doing zsh / POSIX related tests
> > that work except for copying zsh to /tmp/sh and then test with
> > /tmp/sh, as I could not find a better way to make zsh compliant
> > enough to be able to run a "configure" script.
> [...]
>
> Aside from the fact that zsh is in my opinion the Bourne-like
> shell which has made some of the best designs decisions and
> should be at least considered as a source of inspiration for the
> future of the specification (again, my opinion only), zsh is
> relevant in that it has a "sh" mode that is typically used to
> interpret POSIX code.
>
> emulate sh # to emulate sh
> emulate -L sh # to emulate sh in the local scope only
> emulate sh -c 'some code' # to evaluate the code as "sh"
>   # (and also stick the sh emulation to
> # the functions defined in that code)
>
> That "sh" emulation used to target the SysV shell, but has
> gradually switched to POSIX over the last decades.
>
> zsh enters the "sh" emulation mode when invoked as "sh", but I'm
> not sure I would use zsh for the system's sh (note however that
> when it comes to multibyte processing, it's more compliant than
> shells like dash, posh or mksh).

Multi byte support is really a problem.

dash and posh do not implement multi byte support at all and thus
may only certified as system shell for an embedded UNIX version.

Moreover, dash is a _very_ slow shell. If it did ever implement
multi byte support it would be slower than bash.

mksh needs "-o utf8-mode -o posix" to work as expected.

Back to zsh:

If there is no way to modify this command line:

CONFIG_SHELL=/usr/bin/zsh  /usr/bin/zsh ./configure

in a way that allows to run the configure script, this is a bad
state for the current usability of zsh. I currently need to do the
following:

cp /usr/bin/zsh /tmp
CONFIG_SHELL=/tmp/sh /tmp/sh ./configure

> mksh, bash and yash at least have a "posix" option to enforce a
> more compliant mode (at least bash enables it when invoked as
> "sh"). See also the xpg_echo option in bash which you also need
> to have a compliant echo.

There are a few features where the "Bourne Shell" would cause problems
with existing Bourne Shell scripts if it was by default 100% POSIX.

For this reason it implements set -o posix that is auto-activated if it was
installed as /usr/xpg4/bin/sh (note that a symlink will not work here).

If you like to use the Bourne Shell as Linux default shell, compile it with:

-DDO_POSIX_SH   to enable -o posix whenever argv[0] ends in /sh

-DPOSIX_BOSH_PATH="other-path" to enable something different
than /usr/xpg4/bin/sh

>From the man page for -o posix:

  This  currently  disallows  to  use
  ``^''  as the pipe symbol, switches
  off support for test -t  without  a
  parameter  and sets the default for
  directory operations to -L.
  POSIX mode is enabled by default if
  the  executable  path  of the shell
  equals a compiled  in  value,  e.g.
  /usr/xpg4/bin/sh.


> So in that regard, the main difference between bash and zsh is
> the amo

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Stephane Chazelas]

2017-05-24 14:48:12 +0100, Stephane Chazelas:
[...]
> > bash
> > bosh
> 
> As discussed, as of the 2017-05-16 release, posh was not POSIX
> and still had many of non-conformances of the Bourne shell.
[...]

Sorry, I meant "bosh", not "posh" above.

The bash/posh/bosh trio sounds like it's been especially
concieved to cause lapsus :-)

-- 
Stephane