Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
On 05/29/17 01:56 PM, Joerg Schilling wrote: Alan Coopersmith wrote: We still ship a couple closed source shells in Solaris - the SVR4 Bourne Shell as /usr/sunos/bin/sh and a modified ksh88 as /usr/xpg4/bin/sh, but our primary day to day shells are the common open source ones - bash, ksh93, zsh, etc. Is there any bugfix in /usr/sunos/bin/sh that is not in the Svr4 compatile copmpile variant of bosh? Or is it unmodified since the Sun updates for OpenSolaris stopped? I have no idea what is or is not in bosh. Most of the changes on our side since that source stopped being published were to adapt to changing build environments and to prepare for it's EOL. Is your ksh88 managed in a way that keeps it compatible with the evolving POSIX standard? We aren't chasing moving targets here - we're targeting stable releases like UNIXv7. We have fixed several bugs in our /usr/xpg4/bin/sh that were needed to pass various cases in the UNIXv7 test suite in past years. Did you upgrade ksh93 since 2010? Yes - you can see that we're using the 2012-08-01 release in our open source trees at: https://github.com/oracle/solaris-userland/tree/master/components/ksh93 If yes, have you been able to keep it as fast as it was in OpenSolaris? I have no idea - I don't know what commonly accepted shell benchmark we'd use to tell. -- -Alan Coopersmith- alan.coopersm...@oracle.com Oracle Solaris Engineering - https://blogs.oracle.com/alanc
Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
Alan Coopersmith wrote: > We still ship a couple closed source shells in Solaris - the SVR4 Bourne Shell > as /usr/sunos/bin/sh and a modified ksh88 as /usr/xpg4/bin/sh, but our primary > day to day shells are the common open source ones - bash, ksh93, zsh, etc. Is there any bugfix in /usr/sunos/bin/sh that is not in the Svr4 compatile copmpile variant of bosh? Or is it unmodified since the Sun updates for OpenSolaris stopped? Is your ksh88 managed in a way that keeps it compatible with the evolving POSIX standard? Did you upgrade ksh93 since 2010? If yes, have you been able to keep it as fast as it was in OpenSolaris? When I compile a recent ksh93 source, it is 21% slower than the OpenSolaris ksh93 from April 2010 and 7% slower than a recent bosh. Parts of the slowdown are caused by the fact that the cat builtin is not in the search path, and parts of the slowdown is caused by the huge binary that is not using shared linking (fork becomes slow) but I doubt that this could sum up to 21%. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
On 05/29/17 04:37 AM, Robert Elz wrote: There is no requirement that you, or I, or anyone else in particular, be able to build and run any particular shell. While that's nice if possible, a shell is still a shell even if its source code is not openly available at all (like any that are in closed source systems, if any of those still really exist.) Users get to experience those shells too. We still ship a couple closed source shells in Solaris - the SVR4 Bourne Shell as /usr/sunos/bin/sh and a modified ksh88 as /usr/xpg4/bin/sh, but our primary day to day shells are the common open source ones - bash, ksh93, zsh, etc. -- -Alan Coopersmith- alan.coopersm...@oracle.com Oracle Solaris Engineering - https://blogs.oracle.com/alanc
Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
Stephane CHAZELAS wrote: > 2017-05-24 17:21:33 +0200, Joerg Schilling: > > Do you have a pointer to the POSIX text that forbids a subshell in this > > case > > when stdin is redirected? > [...] > > I don't know if you'll find some text that *explicitly* forbids > it to run in a subshell, but you won't find text that *allows* > it (just like you won't see text that allows { a=2; } alone to > run in a subshell environment) > > On the other hand, you'll find text that explicitely allows it > to run in a subshell environment in: > > { a=2; } | cmd > > cmd | { a=2; } > > (even though for the latter, many shells don't run it in a > subshell environment). Given that I planned a related change since ~2 years, I now implemented an enhancement to the parser and to this did even result in an aprox. speedup of 2% with my "configure" performance test. The Bourne Shell now auto-exports imported variables when it starts up un POSIX mode. Multiple non-space IFS characters now result in empty arguments. The new schilytools bundle is here: https://sourceforge.net/projects/schilytools/files/schily-2017-05-29.tar.bz2 Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
Date:Mon, 29 May 2017 14:38:02 +0200 From:Joerg Schilling Message-ID: <592c162a.syoz+dozzpogtqep%joerg.schill...@fokus.fraunhofer.de> | The problem is that I have a FreeBSD system available for testing, | but not a NetBSD system... It is fairly easy to set up NetBSD - it runs (kind of badly...) under qemu, and quite well under Xen, virtualbox and vmware. (If you want to try any of that, for this purpose, definitely use one of the NetBSD current daily builds, not a release version, the shell in those is ancient...) | Moreover, I like to run frequent tests with various shells on my main | development machine. Yes, so do I, but I don't have them all (too much of a good thing!) | *BSD typically uses things like "err()" that are not portable. Yes, a lot of utilities do, using that encourages more consistent behaviour, but the shell does not (err() is kind of obvious - what it does is almost never what the shell would want to do, but we don't use warn() either). For the utilities that do use them, when they're being compiled on some other system (NetBSD can be built almost anywhere, but we need a sane environment to do that, so the first step of the build process is to build a set of NetBSD tools on the host) there is a nbcompat library that can be used that provides essentially all of the pieces missing from other libc's. | The fact that a real vfork() implementation changes the timing, | causes some constructs to fail unless there is special code inside. Yes, there is "special code" - our shell can only use vfork() if it is a real vfork() - if a system provides a vfork() that is just fork() wrapper (or some abomination in between) for the benefit or programs that assume that vfork() exists, and use it blindly, then that's no use at all, and our shell won't (can't) use that, it will just fork() instead. | You may like to make your shell available if you are interested in getting | other people to test it. Yes, one day, I will. That's definitely an objective. But right now I have enough bugs (and missing pieces) that I already know about that need fixing, that I don't really need people finding more which aren't currently bothering anyone (many of the bugs are in exotica that no real script ever uses.) kre
Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
Robert Elz wrote: > Date:Mon, 29 May 2017 11:47:04 +0200 > From:Joerg Schilling > Message-ID: > <592bee18.kvZiYpLd6LS/Evf/%joerg.schill...@fokus.fraunhofer.de> > > | If the related shells would be available as portable code, > | it would be easy to test them. > > There is no requirement that you, or I, or anyone else in particular, > be able to build and run any particular shell. While that's nice if > possible, a shell is still a shell even if its source code is not openly > available at all (like any that are in closed source systems, if any of > those still really exist.) Users get to experience those shells too. The problem is that I have a FreeBSD system available for testing, but not a NetBSD system... Moreover, I like to run frequent tests with various shells on my main development machine. > If no-one can test the shell, that would be different, but as long as someone > can run tests, we have all we absolutely need. > > | For now, the problem is that these shells massively rely on non-POSIX > | extensions in /usr/include and in libc. > > I don't think the NetBSD shell has any such code in it (the debugging code > relies on asprintf() and friends, but you don't need that to compile it.) > Formatter output (for messages, and such) will use vasprintf() if available, > but that's conditionally compiled, and can be done without. *BSD typically uses things like "err()" that are not portable. > Similarly, use of vfork() is conditionally compiled (it runs the same > without, just slower). We only use waitpid() currently (which is POSIX) > though the shell is supposed to be able to build and work (with some loss > of functionality) with just wait() for ancient systems. (I am adding > waitid() support as one of my back-burner tasks, but that doesn't really > fix anything that's broken, so it is low priority.) vfork() is not a real problem except when your vfork() support code might not be correct enough to correctly deal with process groups and similar. The fact that a real vfork() implementation changes the timing, causes some constructs to fail unless there is special code inside. You may like to make your shell available if you are interested in getting other people to test it. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
Date:Mon, 29 May 2017 11:47:04 +0200 From:Joerg Schilling Message-ID: <592bee18.kvZiYpLd6LS/Evf/%joerg.schill...@fokus.fraunhofer.de> | If the related shells would be available as portable code, | it would be easy to test them. There is no requirement that you, or I, or anyone else in particular, be able to build and run any particular shell. While that's nice if possible, a shell is still a shell even if its source code is not openly available at all (like any that are in closed source systems, if any of those still really exist.) Users get to experience those shells too. If no-one can test the shell, that would be different, but as long as someone can run tests, we have all we absolutely need. | For now, the problem is that these shells massively rely on non-POSIX | extensions in /usr/include and in libc. I don't think the NetBSD shell has any such code in it (the debugging code relies on asprintf() and friends, but you don't need that to compile it.) Formatter output (for messages, and such) will use vasprintf() if available, but that's conditionally compiled, and can be done without. Similarly, use of vfork() is conditionally compiled (it runs the same without, just slower). We only use waitpid() currently (which is POSIX) though the shell is supposed to be able to build and work (with some loss of functionality) with just wait() for ancient systems. (I am adding waitid() support as one of my back-burner tasks, but that doesn't really fix anything that's broken, so it is low priority.) The reason I don't make a tarball of the NetBSD shell available (anyone who wants to look at it can get it from the NetBSD anoncvs repo of course, none of it is secret) isn't that the code won't compile, but that there's no build system for it ... the Makefile more or less just lists the names of the source files, and says how to run the (build time) scripts that are used to generate the files that come from a non-C format (like the list of builtin commands - which gets converted into appropriate .h and .c files using a script.) Everything else is handled by NetBSD make system magic - including finding the sources of the programs (like kill, printf, test ...) that are included as builtins (normally anyway) - we use the same sources as are used for the /bin (or /usr/bin or whatever) versions of the commands, so none of that gets included with the sh sources. Building a portable build system is one of my long term goals, but I think that (for now) a better objective is to fix bugs, and (where appropriate) posix conformance, rather than worrying about how someone else would build it. If you want to test what it does with some particular script, just send me the script, and I'll run it and send you back the results. kre
Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
Robert Elz wrote: > | It also would make sense to check with shells have maintainers > | that are on this list and willing to come to a common agreement. > > Neither were on your list, but the FreeBSD shell and NetBSD shell maintainers > are here too... If the related shells would be available as portable code, it would be easy to test them. For now, the problem is that these shells massively rely on non-POSIX extensions in /usr/include and in libc. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'
Re: What shell implementations to consider (Was: [1003.1(2008)/Issue 7 0000767]: Add built-in "local")
Robert Elz wrote: > To get away from the dick waving "my shell is bigger than yours" > discussions for a minute ... > > In note 3745 attached to http://austingroupbugs.net/view.php?id=767 > Joerg Schilling proposes making a list of shells to use to help > guide what can be regarded as "standard" for the purposes of compliance. > > In that he said ... > >First, I believe that "yash" and "posh" should not be in that list >because these implementations are too buggy or have unexplained >deviations > > with which I don't agree. First, because any existing deviations with > the standard aren't really (or at least, necessarily) relevant when > considering some new issue, and second if a shell (any shell) does not Well I believe it matters when a shell like "posh" fails in _many_ places because the code of the shell is not POSIX compliant. It may be that the authors of posh did never compile it on a POSIX system. Last weekend, I had a look into the source and discovered that it is _expected_ to fail, the way it was written, because it ignores the getopt() documentation. It looks interesting that the problem is inside one of two files that is (unlike the rest) using a GPL header. The rest of the code is OK, so it looks like one of those cases where Debian people modified code and thus caused problems. The commands that fail (are unusable) because of that problem are: break, cd, exit, export, eval (dumps core), pwd, readonly, return, set, shift, trap, umask To me, posh thus makes no sense to be tested at all until it is fixed. Jörg -- EMail:jo...@schily.net(home) Jörg Schilling D-13353 Berlin joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.org/private/ http://sf.net/projects/schilytools/files/'