Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Alan Coopersmith]

On 05/29/17 01:56 PM, Joerg Schilling wrote:

Alan Coopersmith  wrote:


We still ship a couple closed source shells in Solaris - the SVR4 Bourne Shell
as /usr/sunos/bin/sh and a modified ksh88 as /usr/xpg4/bin/sh, but our primary
day to day shells are the common open source ones - bash, ksh93, zsh, etc.


Is there any bugfix in /usr/sunos/bin/sh that is not in the Svr4 compatile
copmpile variant of bosh? Or is it unmodified since the Sun updates for
OpenSolaris stopped?


I have no idea what is or is not in bosh.  Most of the changes on our side
since that source stopped being published were to adapt to changing build
environments and to prepare for it's EOL.


Is your ksh88 managed in a way that keeps it compatible with the evolving POSIX
standard?


We aren't chasing moving targets here - we're targeting stable releases like
UNIXv7.  We have fixed several bugs in our /usr/xpg4/bin/sh that were needed
to pass various cases in the UNIXv7 test suite in past years.


Did you upgrade ksh93 since 2010?


Yes - you can see that we're using the 2012-08-01 release in our open source 
trees at:

https://github.com/oracle/solaris-userland/tree/master/components/ksh93


If yes, have you been able to keep it as fast as it was in OpenSolaris?


I have no idea - I don't know what commonly accepted shell benchmark we'd
use to tell.

--
-Alan Coopersmith-   alan.coopersm...@oracle.com
 Oracle Solaris Engineering - https://blogs.oracle.com/alanc

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Alan Coopersmith]

On 05/29/17 04:37 AM, Robert Elz wrote:

There is no requirement that you, or I, or anyone else in particular,
be able to build and run any particular shell.   While that's nice if
possible, a shell is still a shell even if its source code is not openly
available at all (like any that are in closed source systems, if any of
those still really exist.)   Users get to experience those shells too.


We still ship a couple closed source shells in Solaris - the SVR4 Bourne Shell
as /usr/sunos/bin/sh and a modified ksh88 as /usr/xpg4/bin/sh, but our primary
day to day shells are the common open source ones - bash, ksh93, zsh, etc.

--
-Alan Coopersmith-   alan.coopersm...@oracle.com
 Oracle Solaris Engineering - https://blogs.oracle.com/alanc

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Stephane CHAZELAS  wrote:

> 2017-05-24 17:21:33 +0200, Joerg Schilling:

> > Do you have a pointer to the POSIX text that forbids a subshell in this 
> > case 
> > when stdin is redirected?
> [...]
>
> I don't know if you'll find some text that *explicitly* forbids
> it to run in a subshell, but you won't find text that *allows*
> it (just like you won't see text that allows { a=2; } alone to
> run in a subshell environment)
>
> On the other hand, you'll find text that explicitely allows it
> to run in a subshell environment in:
>
> { a=2; } | cmd
>
> cmd | { a=2; }
>
> (even though for the latter, many shells don't run it in a
> subshell environment).

Given that I planned a related change since ~2 years, I now implemented an 
enhancement to the parser and to this did even result in an aprox. speedup of 2%
with my "configure" performance test.

The Bourne Shell now auto-exports imported variables when it starts up un POSIX 
mode.

Multiple non-space IFS characters now result in empty arguments.


The new schilytools bundle is here:


https://sourceforge.net/projects/schilytools/files/schily-2017-05-29.tar.bz2

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Robert Elz]

Date:Mon, 29 May 2017 14:38:02 +0200
From:Joerg Schilling 
Message-ID:  <592c162a.syoz+dozzpogtqep%joerg.schill...@fokus.fraunhofer.de>

  | The problem is that I have a FreeBSD system available for testing,
  | but not a NetBSD system...

It is fairly easy to set up NetBSD - it runs (kind of badly...) under
qemu, and quite well under Xen, virtualbox and vmware.

(If you want to try any of that, for this purpose, definitely use one
of the NetBSD current daily builds, not a release version, the shell in
those is ancient...)

  | Moreover, I like to run frequent tests with various shells on my main 
  | development machine.

Yes, so do I, but I don't have them all (too much of a good thing!)

  | *BSD typically uses things like "err()" that are not portable.

Yes, a lot of utilities do, using that encourages more consistent behaviour,
but the shell does not (err() is kind of obvious - what it does is almost
never what the shell would want to do, but we don't use warn() either).

For the utilities that do use them, when they're being compiled on some other
system (NetBSD can be built almost anywhere, but we need a sane environment
to do that, so the first step of the build process is to build a set of
NetBSD tools on the host) there is a nbcompat library that can be used that
provides essentially all of the pieces missing from other libc's.

  | The fact that a real vfork() implementation changes the timing,
  | causes some constructs to fail unless there is special code inside.

Yes, there is "special code" - our shell can only use vfork() if it is
a real vfork() - if a system provides a vfork() that is just fork() wrapper
(or some abomination in between) for the benefit or programs that assume
that vfork() exists, and use it blindly, then that's no use at all, and
our shell won't (can't) use that, it will just fork() instead.

  | You may like to make your shell available if you are interested in getting 
  | other people to test it.

Yes, one day, I will.   That's definitely an objective.   But right now
I have enough bugs (and missing pieces) that I already know about that need
fixing, that I don't really need people finding more which aren't currently
bothering anyone (many of the bugs are in exotica that no real script ever
uses.)

kre

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Robert Elz  wrote:

> Date:Mon, 29 May 2017 11:47:04 +0200
> From:Joerg Schilling 
> Message-ID:  
> <592bee18.kvZiYpLd6LS/Evf/%joerg.schill...@fokus.fraunhofer.de>
>
>   | If the related shells would be available as portable code,
>   | it would be easy to test them.
>
> There is no requirement that you, or I, or anyone else in particular,
> be able to build and run any particular shell.   While that's nice if
> possible, a shell is still a shell even if its source code is not openly
> available at all (like any that are in closed source systems, if any of
> those still really exist.)   Users get to experience those shells too.

The problem is that I have a FreeBSD system available for testing, but not a 
NetBSD system...

Moreover, I like to run frequent tests with various shells on my main 
development machine.

> If no-one can test the shell, that would be different, but as long as someone
> can run tests, we have all we absolutely need.
>
>   | For now, the problem is that these shells massively rely on non-POSIX 
>   | extensions in /usr/include and in libc.
>
> I don't think the NetBSD shell has any such code in it (the debugging code
> relies on asprintf() and friends, but you don't need that to compile it.)
> Formatter output (for messages, and such) will use vasprintf() if available,
> but that's conditionally compiled, and can be done without.

*BSD typically uses things like "err()" that are not portable.

> Similarly, use of vfork() is conditionally compiled (it runs the same
> without, just slower).  We only use waitpid() currently (which is POSIX)
> though the shell is supposed to be able to build and work (with some loss
> of functionality) with just wait() for ancient systems.   (I am adding
> waitid() support as one of my back-burner tasks, but that doesn't really
> fix anything that's broken, so it is low priority.)

vfork() is not a real problem except when your vfork() support code might not 
be correct enough to correctly deal with process groups and similar. The fact 
that a real vfork() implementation changes the timing, causes some constructs 
to fail unless there is special code inside.

You may like to make your shell available if you are interested in getting 
other people to test it.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'

Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")

[Joerg Schilling]

Robert Elz  wrote:

> To get away from the dick waving "my shell is bigger than yours"
> discussions for a minute ...
>
> In note 3745 attached to http://austingroupbugs.net/view.php?id=767
> Joerg Schilling proposes making a list of shells to use to help
> guide what can be regarded as "standard" for the purposes of compliance.
>
> In that he said ...
>
>First, I believe that "yash" and "posh" should not be in that list
>because these implementations are too buggy or have unexplained
>deviations
>
> with which I don't agree.   First, because any existing deviations with
> the standard aren't really (or at least, necessarily) relevant when
> considering some new issue, and second if a shell (any shell) does not

Well I believe it matters when a shell like "posh" fails in _many_ places 
because the code of the shell is not POSIX compliant.

It may be that the authors of posh did never compile it on a POSIX system.
Last weekend, I had a look into the source and discovered that it is _expected_ 
to fail, the way it was written, because it ignores the getopt() documentation.

It looks interesting that the problem is inside one of two files that is (unlike
the rest) using a GPL header. The rest of the code is OK, so it looks like one 
of
those cases where Debian people modified code and thus caused problems.

The commands that fail (are unusable) because of that problem are:

break, cd, exit, export, eval (dumps core), pwd,  readonly, return, 
set, shift, trap, umask

To me, posh thus makes no sense to be tested at all until it is fixed.

Jörg

-- 
 EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin
joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/
 URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'