More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI This information is Copyright 2010 Sun Microsystems 1. Introduction 1.1. Project/Component Working Name: More ksh93 builtins 1.2. Name of Document Author/Supplier: Author: Olga Kryzhanovska 1.3 Date of This Document: 18 March, 2010 4. Technical Description I'm sponsoring this fast-track request on behalf of the ksh93-integration project. Please note that this is an *open* case. The release binding is the same as with the ksh93 project: a patch/micro release of Solaris delivering through ON Stability levels are as described below. This project is an amendment to the Korn Shell 93 Integration project (PSARC/2006/550 and PSARC/2007/035, PSARC/2008/094, PSARC/2008/344 and PSARC/2008/589) specifying the following additional interfaces: Addition of /usr/gnu/bin, /usr/xpg4/bin and /usr/bin built in mappings in ksh93 Bug/RFE Number(s): 6935110 ksh93 needs (more) XPG, GNU built ins This case proposes to deliver the following features as a set of independent putbacks as they become available. Each feature is self contained and independent of the others, so out of order and partial putbacks at this granularity should have no adverse impact on the functionality and behavior of the system as a whole. Part 1: ksh93 built in mappings for /usr/gnu/bin utilities -- The case proposes to enable the following mappings to ksh93 built in commands for utilities in /usr/gnu/bin. When the utility is called using it's command name, and not the full path, the ksh93 builtin will be used instead of executing the /usr/gnu/bin binary. Interface StabilityDescription - ---- ksh93 '/usr/gnu/bin/basename' built in Uncommitted basename utility with GNU extensions ksh93 '/usr/gnu/bin/cksum' built in Uncommitted cksum utility with GNU extensions ksh93 '/usr/gnu/bin/comm' built in Uncommitted comm utility with GNU extensions ksh93 '/usr/gnu/bin/cut' built in Uncommitted cut utility with GNU extensions ksh93 '/usr/gnu/bin/dirname' built in Uncommitted dirname utility with GNU extensions ksh93 '/usr/gnu/bin/expr' built in Uncommitted expr utility with GNU extensions ksh93 '/usr/gnu/bin/fold' built in Uncommitted fold utility with GNU extensions ksh93 '/usr/gnu/bin/join' built in Uncommitted join utility with GNU extensions ksh93 '/usr/gnu/bin/logname' built in Uncommitted logname utility with GNU extensions ksh93 '/usr/gnu/bin/mkdir' built in Uncommitted mkdir utility with GNU extensions ksh93 '/usr/gnu/bin/mkfifo' built inUncommitted mkfifo utility with GNU extensions ksh93 '/usr/gnu/bin/mktemp' built inUncommitted mktemp utility with GNU extensions ksh93 '/usr/gnu/bin/pathchk' built in Uncommitted pathchk utility with GNU extensions ksh93 '/usr/gnu/bin/paste' built in Uncommitted paste utility with GNU extensions ksh93 '/usr/gnu/bin/sleep' built in Uncommitted sleep utility with GNU extensions ksh93 '/usr/gnu/bin/sync' built in Uncommitted sync utility with GNU extensions ksh93 '/usr/gnu/bin/tee' built in Uncommitted tee utility with GNU extensions ksh93 '/usr/gnu/bin/tty' built in Uncommitted tty utility with GNU extensions ksh93 '/usr/gnu/bin/uniq' built in Uncommitted uniq utility with GNU extensions ksh93 '/usr/gnu/bin/rmdir' built in Uncommitted rmdir utility with GNU extensions ksh93 '/usr/gnu/bin/wc' built inUncommitted wc utility with GNU extensions Notes - Compatibility of AT&T AST ksh93 built in utilities and GNU coreutils implementations: Modulo correction bugs in the implementation, we believe that the ksh93 built in utilities act as a 100% compatible drop in for the GNU programs sans wording of --help, --version and error message output (error messages are Not-An-Interface). Compatibility between ksh93 built in utility implementation and GNU coreutils implementation: Should a future ARC case will add new features to the GNU coreutils utilities the project team will update the corresponding ksh93 built in utility. Should this not be possible the ksh93 project team will remove the mapping. Part 2: ksh93 built in mappings for /usr/xpg4/bin utilities -- The case proposes to enable the following mappings to ksh93 built in commands for utilities in /usr/xpg4/bin. When the utility is called using it's command name, and not the full path, the ksh93 builtin will be used instead of executing the /usr/xpg4/bin binary. InterfaceStabilityDescription -----
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Maybe I don't understand enough about ksh93 (since I'm a zsh user for interactive shell work) but I don't understand what this case is about. What benefit does this case bring ? How does this interact with PSARC/2009/377 in kernel pfexec, maybe it doesn't need to and that is an okay answer, when ksh93 is the profile shell ? Why would I want to use ksh93 builtins if I have /usr/gnu/bin explicitly in my path ? Are the ksh93 builtin versions 100% compatible in all respects with the GNU ones ? If so then I wonder why we are even shipping the GNU ones. -- Darren J Moffat
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
This project raises a concern in my mind with respect to a very old and generally accepted UNIX architectural principle: "Tools, Not Policy". If i understand it correctly, this case effectively vacates the principle stated above, and replaces it with its exact opposite: "Policy, Not Tools". Because the only possible rationale for having /usr/gnu/bin/grep transparently and silently replaced by a shell builtin grep is as a result of some mandatory policy in effect, which would trump explicit user selection. It would be very helpful if the project team would kindly explain how it intends to address the architectural concern above, and also how it intends to mitigate the proliferation of userland commands with identical names, but providing, in many cases, different semantics. Thank you. --Stefan -- Garrett D'Amore - sun microsystems wrote: > This project is an amendment to the Korn Shell 93 Integration project > (PSARC/2006/550 and PSARC/2007/035, PSARC/2008/094, PSARC/2008/344 > and PSARC/2008/589) specifying the following additional > interfaces: > Addition of /usr/gnu/bin, /usr/xpg4/bin and /usr/bin built in > mappings in ksh93 -- Stefan Teleman Oracle Corporation stefan.teleman at Sun.COM
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Garrett D'Amore - sun microsystems wrote: > Compatibility between ksh93 built in utility implementation and GNU > coreutils implementation: > Should a future ARC case will add new features to the GNU coreutils > utilities the project team will update the corresponding ksh93 built > in utility. Should this not be possible the ksh93 project team will > remove the mapping. So this constrains all future GNU coreutils update cases to coordinate with the ksh93 builtins, right? Have those responsible for the GNU coreutils agreed to this constraint? Should this be expressed as an ARC contract for cross-consolidation agreement of Volatile interfaces? -- -Alan Coopersmith- alan.coopersmith at sun.com Oracle Solaris Platform Engineering: X Window System
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 4:41 PM, Darren J Moffat wrote: > Maybe I don't understand enough about ksh93 (since I'm a zsh user for > interactive shell work) but I don't understand what this case is about. > > What benefit does this case bring ? One advantage is MUCH HIGHER performance. A simple loop iterating over my source tree with basename takes 26 seconds without a builtin and 0.1 seconds when basename is a builtin: $ timex ksh93 -c 'find work/src | while read i ; do x=$(basename $i) ; done' real 26.35 user 9.34 sys1.34 $ timex ksh93 -c 'builtin basename ; find work/src | while read i ; do x=$(basename $i) ; done' real 0.11 user 0.07 sys0.03 This ROCKS incredibly :) > How does this interact with PSARC/2009/377 in kernel pfexec, maybe it > doesn't need to and that is an okay answer, when ksh93 is the profile shell > ? ksh93-integration-discuss@ currently has an ongoing discussion about this and the broken profile shell concept. There are two concurrent proposals to integrate the concepts of shell builtins and profile shells. > Why would I want to use ksh93 builtins if I have /usr/gnu/bin explicitly in > my path ? The default end user path in Opensolaris contains /usr/gnu/bin at the beginning. Scripts which run from that environment will have lower performance in ksh93 except when they set their own PATH. Enabling the builtins in /usr/gnu/bin solves the problem and restores full performance again. > Are the ksh93 builtin versions 100% compatible in all respects > with the GNU ones ? I think yes. Otherwise Olga wouldn't propose them. > If so then I wonder why we are even shipping the GNU > ones. I don't see the point either since the ksh93 commands have both features from GNU AND BSD Jenny -- Jennifer Pioch, Uni Frankfurt
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 08:41 AM, Darren J Moffat wrote: > Maybe I don't understand enough about ksh93 (since I'm a zsh user for > interactive shell work) but I don't understand what this case is about. > > What benefit does this case bring ? (Note: I'm not the project team, just the ARC sponsor. They may have different ideas here, so they should respond if appropriate.) ksh93 already has the code. As I understand it, this change benefits ksh93 users and scripts, by reducing the total number of fork/exec calls. It makes use of the common code base that we already have available. I am not aware of any other benefits. > > How does this interact with PSARC/2009/377 in kernel pfexec, maybe it > doesn't need to and that is an okay answer, when ksh93 is the profile > shell ? I don't think this gets involved with that, because the builtins are only used when a default path search is applied, not when a fully specified path is used. > > Why would I want to use ksh93 builtins if I have /usr/gnu/bin > explicitly in my path ? Are the ksh93 builtin versions 100% > compatible in all respects with the GNU ones ? If so then I wonder > why we are even shipping the GNU ones. > My understanding is that yes, the ksh93 ones are fully compatible modulo some bugs which are fixed in the ksh93 versions. I've always questioned the idea that we have to ship all GNU tools, instead of modernizing our own set. I think the idea is to make people who prefer Linux happy. That said, its possible that the GNU tools will evolve in the future, at a rate differently than the ksh93 versions do. (At that point, the case says that the ksh93 version will either be adapted, or they'll stop supplying the built-in.) -- Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Which policy, which tools? Could you please a bit clearer? Jenny On Thu, Mar 18, 2010 at 4:53 PM, Stefan Teleman wrote: > This project raises a concern in my mind with respect to a very old and > generally accepted UNIX architectural principle: > > "Tools, Not Policy". > > If i understand it correctly, this case effectively vacates the principle > stated above, and replaces it with its exact opposite: > > "Policy, Not Tools". > > Because the only possible rationale for having /usr/gnu/bin/grep > transparently and silently replaced by a shell builtin grep is as a result > of some mandatory policy in effect, which would trump explicit user > selection. > > It would be very helpful if the project team would kindly explain how it > intends to address the architectural concern above, and also how it intends > to mitigate the proliferation of userland commands with identical names, but > providing, in many cases, different semantics. > > Thank you. > > --Stefan > > -- > > Garrett D'Amore - sun microsystems wrote: > >> This project is an amendment to the Korn Shell 93 Integration project >> (PSARC/2006/550 and PSARC/2007/035, PSARC/2008/094, PSARC/2008/344 >> and PSARC/2008/589) specifying the following additional >> interfaces: >> Addition of /usr/gnu/bin, /usr/xpg4/bin and /usr/bin built in >> mappings in ksh93 > > > > -- > Stefan Teleman > Oracle Corporation > stefan.teleman at Sun.COM > > ___ > opensolaris-arc mailing list > opensolaris-arc at opensolaris.org > -- Jennifer Pioch, Uni Frankfurt
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 08:58 AM, Alan Coopersmith wrote: > Garrett D'Amore - sun microsystems wrote: > >> Compatibility between ksh93 built in utility implementation and GNU >> coreutils implementation: >> Should a future ARC case will add new features to the GNU coreutils >> utilities the project team will update the corresponding ksh93 built >> in utility. Should this not be possible the ksh93 project team will >> remove the mapping. >> > So this constrains all future GNU coreutils update cases to coordinate > with the ksh93 builtins, right? Have those responsible for the GNU > coreutils agreed to this constraint? Should this be expressed as an > ARC contract for cross-consolidation agreement of Volatile interfaces? > > The ksh93 versions adhere to the public interfaces from the GNU versions. If they change the semantics, then it will require some extra work, but I'm not sure that a contract properly captures this. That said, this case explicitly states that it will revert any builtin if the GNU version evolves incompatibly (or even offers a new feature that the ksh93 doesn't) in the future. Personally, I despise the fact that we have to have GNU at the front of the user path. My $PATH doesn't list /usr/gnu at all. I'd far rather break this dependency upon GNU, and supply our own versions based on the ksh93 code base (which seems to be actively maintained), and just scrap /usr/gnu. But that's just my opinion, and not this case in any event. - Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
I believe I was very clear. --Stefan Jennifer Pioch wrote: > Which policy, which tools? Could you please a bit clearer? > > Jenny > > On Thu, Mar 18, 2010 at 4:53 PM, Stefan Teleman > wrote: >> This project raises a concern in my mind with respect to a very old and >> generally accepted UNIX architectural principle: >> >> "Tools, Not Policy". >> >> If i understand it correctly, this case effectively vacates the principle >> stated above, and replaces it with its exact opposite: >> >> "Policy, Not Tools". >> >> Because the only possible rationale for having /usr/gnu/bin/grep >> transparently and silently replaced by a shell builtin grep is as a result >> of some mandatory policy in effect, which would trump explicit user >> selection. >> >> It would be very helpful if the project team would kindly explain how it >> intends to address the architectural concern above, and also how it intends >> to mitigate the proliferation of userland commands with identical names, but >> providing, in many cases, different semantics. >> >> Thank you. >> >> --Stefan >> >> -- >> >> Garrett D'Amore - sun microsystems wrote: >> >>> This project is an amendment to the Korn Shell 93 Integration project >>> (PSARC/2006/550 and PSARC/2007/035, PSARC/2008/094, PSARC/2008/344 >>> and PSARC/2008/589) specifying the following additional >>> interfaces: >>> Addition of /usr/gnu/bin, /usr/xpg4/bin and /usr/bin built in >>> mappings in ksh93 >> >> >> -- >> Stefan Teleman >> Oracle Corporation >> stefan.teleman at Sun.COM >> >> ___ >> opensolaris-arc mailing list >> opensolaris-arc at opensolaris.org >> > > > -- Stefan Teleman Oracle Corporation stefan.teleman at Sun.COM
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 4:24 PM, Garrett D'Amore - sun microsystems wrote: > > Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI > This information is Copyright 2010 Sun Microsystems > 1. Introduction >1.1. Project/Component Working Name: > More ksh93 builtins >1.2. Name of Document Author/Supplier: > Author: Olga Kryzhanovska >1.3 Date of This Document: >18 March, 2010 > 4. Technical Description > > I'm sponsoring this fast-track request on behalf of the > ksh93-integration project. > Please note that this is an *open* case. > > The release binding is the same as with the ksh93 project: a > patch/micro release of Solaris delivering through ON > Stability levels are as described below. > > > This project is an amendment to the Korn Shell 93 Integration project > (PSARC/2006/550 and PSARC/2007/035, PSARC/2008/094, PSARC/2008/344 > and PSARC/2008/589) specifying the following additional > interfaces: > Addition of /usr/gnu/bin, /usr/xpg4/bin and /usr/bin built in > mappings in ksh93 > > Bug/RFE Number(s): > > 6935110 ksh93 needs (more) XPG, GNU built ins > > This case proposes to deliver the following features as a set of > independent putbacks as they become available. Each feature is > self contained and independent of the others, so out of order > and partial putbacks at this granularity should have no adverse > impact on the functionality and behavior of the system as a whole. > > > Part 1: ksh93 built in mappings for /usr/gnu/bin utilities > -- > The case proposes to enable the following mappings to ksh93 built in > commands for utilities in /usr/gnu/bin. When the utility is called > using it's command name, and not the full path, the ksh93 builtin > will be used instead of executing the /usr/gnu/bin binary. > > > Interface StabilityDescription > - ---- > ksh93 '/usr/gnu/bin/basename' built in Uncommitted basename utility > with GNU extensions > ksh93 '/usr/gnu/bin/cksum' built in Uncommitted cksum utility with > GNU extensions > ksh93 '/usr/gnu/bin/comm' built in Uncommitted comm utility with > GNU extensions > ksh93 '/usr/gnu/bin/cut' built in Uncommitted cut utility with GNU > extensions > ksh93 '/usr/gnu/bin/dirname' built in Uncommitted dirname utility with > GNU extensions > ksh93 '/usr/gnu/bin/expr' built in Uncommitted expr utility with > GNU extensions > ksh93 '/usr/gnu/bin/fold' built in Uncommitted fold utility with > GNU extensions > ksh93 '/usr/gnu/bin/join' built in Uncommitted join utility with > GNU extensions > ksh93 '/usr/gnu/bin/logname' built in Uncommitted logname utility with > GNU extensions > ksh93 '/usr/gnu/bin/mkdir' built in Uncommitted mkdir utility with > GNU extensions > ksh93 '/usr/gnu/bin/mkfifo' built inUncommitted mkfifo utility with > GNU extensions > ksh93 '/usr/gnu/bin/mktemp' built inUncommitted mktemp utility with > GNU extensions > ksh93 '/usr/gnu/bin/pathchk' built in Uncommitted pathchk utility with > GNU extensions > ksh93 '/usr/gnu/bin/paste' built in Uncommitted paste utility with > GNU extensions > ksh93 '/usr/gnu/bin/sleep' built in Uncommitted sleep utility with > GNU extensions > ksh93 '/usr/gnu/bin/sync' built in Uncommitted sync utility with > GNU extensions > ksh93 '/usr/gnu/bin/tee' built in Uncommitted tee utility with GNU > extensions > ksh93 '/usr/gnu/bin/tty' built in Uncommitted tty utility with GNU > extensions > ksh93 '/usr/gnu/bin/uniq' built in Uncommitted uniq utility with > GNU extensions > ksh93 '/usr/gnu/bin/rmdir' built in Uncommitted rmdir utility with > GNU extensions > ksh93 '/usr/gnu/bin/wc' built inUncommitted wc utility with GNU > extensions > > Notes > - > Compatibility of AT&T AST ksh93 built in utilities and GNU > coreutils implementations: > Modulo correction bugs in the implementation, we believe that the > ksh93 built in utilities act as a 100% compatible drop in for the > GNU programs sans wording of --help, --version and error message > output (error messages are Not-An-Interface). > > Compatibility between ksh93 built in utility implementation and GNU > coreutils implementation: > Should a future ARC case will add new features to the GNU coreutils > utilities the project team will update the corresponding ksh93 built > in utility. Should this not be possible the ksh93 project team will > remove the mapping. > > > > Part 2: ksh93 built in mappings for /usr/xpg4/bin utilities > -- > The case proposes to enable the following mappings to ksh93 built in > commands for utilities in /usr/xpg4/bin. When the utility is called > using it's command name, and not the
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 04:58:39PM +0100, Jennifer Pioch wrote:
> On Thu, Mar 18, 2010 at 4:41 PM, Darren J Moffat
> wrote:
> > Maybe I don't understand enough about ksh93 (since I'm a zsh user for
> > interactive shell work) but I don't understand what this case is about.
> >
> > What benefit does this case bring ?
>
> One advantage is MUCH HIGHER performance. A simple loop iterating over
> my source tree with basename takes 26 seconds without a builtin and
> 0.1 seconds when basename is a builtin:
BTW, I've been using ${var##*/} and ${var%/*} for years as a built-in
replacement for basename(1) and dirname(1). Making those two commands
built-ins won't help me! :)
> This ROCKS incredibly :)
I agree that more builtins -> better.
> > Are the ksh93 builtin versions 100% compatible in all respects
> > with the GNU ones ?
>
> I think yes. Otherwise Olga wouldn't propose them.
>
> > If so then I wonder why we are even shipping the GNU
> > ones.
>
> I don't see the point either since the ksh93 commands have both
> features from GNU AND BSD
Bash users still need the commands in the bin directories. This case
has a note indicating that eventually even bash4 weill be able to use
ksh93 builtins, but that would still leave zsh, tcsh, ...
Nico
--
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 18/03/2010 15:58, Jennifer Pioch wrote: > On Thu, Mar 18, 2010 at 4:41 PM, Darren J Moffat > wrote: >> Maybe I don't understand enough about ksh93 (since I'm a zsh user for >> interactive shell work) but I don't understand what this case is about. >> >> What benefit does this case bring ? > > One advantage is MUCH HIGHER performance. A simple loop iterating over > my source tree with basename takes 26 seconds without a builtin and > 0.1 seconds when basename is a builtin: > $ timex ksh93 -c 'find work/src | while read i ; do x=$(basename $i) ; done' > > real 26.35 > user 9.34 > sys1.34 > > $ timex ksh93 -c 'builtin basename ; find work/src | while read i ; do > x=$(basename $i) ; done' > > real 0.11 > user 0.07 > sys0.03 > > This ROCKS incredibly :) > >> How does this interact with PSARC/2009/377 in kernel pfexec, maybe it >> doesn't need to and that is an okay answer, when ksh93 is the profile shell >> ? > > ksh93-integration-discuss@ currently has an ongoing discussion about > this and the broken profile shell concept. There are two concurrent > proposals to integrate the concepts of shell builtins and profile > shells. Then this case should be put in waiting need spec until that discussion reaches some consensus. The profile shell concept is not broken, some people may not like what it is or how it works but it is approved a delivered architecture. It has been deliver as part of Solaris since Solaris 8 and is based on functionality going back more than 15 years in various Trusted Solaris (and even before that name) releases of SunOS. >> If so then I wonder why we are even shipping the GNU >> ones. > > I don't see the point either since the ksh93 commands have both > features from GNU AND BSD Do the have 100% of the features with exactly the same behaviour though ? -- Darren J Moffat
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 08:53 AM, Stefan Teleman wrote: > This project raises a concern in my mind with respect to a very old > and generally accepted UNIX architectural principle: > > "Tools, Not Policy". > > If i understand it correctly, this case effectively vacates the > principle stated above, and replaces it with its exact opposite: > > "Policy, Not Tools". > > Because the only possible rationale for having /usr/gnu/bin/grep > transparently and silently replaced by a shell builtin grep is as a > result of some mandatory policy in effect, which would trump explicit > user selection. > > It would be very helpful if the project team would kindly explain how > it intends to address the architectural concern above, and also how it > intends to mitigate the proliferation of userland commands with > identical names, but providing, in many cases, different semantics. Users can override the policy and disable the builtins if they so choose. They could also choose another shell instead of ksh93 -- these changes don't apply to bash or tcsh or zsh. I guess I don't really understand the objection -- the builtin behaves the same way, but performs better. What's the issue? (We've do this with other subsystems in the past, such as in-kernel accelerations for SSL, HTTP, etc. These all provide a behind the scenes "improvement" using an alternate implementation.) - Garrett > > Thank you. > > --Stefan > > -- > > Garrett D'Amore - sun microsystems wrote: > >> This project is an amendment to the Korn Shell 93 Integration project >> (PSARC/2006/550 and PSARC/2007/035, PSARC/2008/094, PSARC/2008/344 >> and PSARC/2008/589) specifying the following additional >> interfaces: >> Addition of /usr/gnu/bin, /usr/xpg4/bin and /usr/bin built in >> mappings in ksh93 > > >
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 09:09 AM, Darren J Moffat wrote: > > >> ksh93-integration-discuss@ currently has an ongoing discussion about >> this and the broken profile shell concept. There are two concurrent >> proposals to integrate the concepts of shell builtins and profile >> shells. > > Then this case should be put in waiting need spec until that > discussion reaches some consensus. > > The profile shell concept is not broken, some people may not like what > it is or how it works but it is approved a delivered architecture. It > has been deliver as part of Solaris since Solaris 8 and is based on > functionality going back more than 15 years in various Trusted Solaris > (and even before that name) releases of SunOS. Architecturally, I have to agree with Darren here. I don't know what the concerns are here where this would fail to operate with the current pfexec... I thought that it was just the case that pfexec would bypass the builtin and use the filesystem supplied binary. If the above isn't true, and there will be breakage for any of these, then yes, this case needs to be put in waiting need spec. If however the only problem is that pfexec'd versions of the builtins will not use the builtin but suffer a performance penalty (a real fork/exec), then I think we can proceed. Can folks more familiar with the problem(s) involved here please clarify? > >>> If so then I wonder why we are even shipping the GNU >>> ones. >> >> I don't see the point either since the ksh93 commands have both >> features from GNU AND BSD > > Do the have 100% of the features with exactly the same behaviour though ? > Olga has assured me that yes, this is the case. (Modulo fixing some edge case bugs, such as handling of multi-byte locales.) - Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 5:09 PM, Darren J Moffat wrote: > > > On 18/03/2010 15:58, Jennifer Pioch wrote: >> >> On Thu, Mar 18, 2010 at 4:41 PM, Darren J Moffat >> wrote: >>> >>> Maybe I don't understand enough about ksh93 (since I'm a zsh user for >>> interactive shell work) but I don't understand what this case is about. >>> >>> What benefit does this case bring ? >> >> One advantage is MUCH HIGHER performance. A simple loop iterating over >> my source tree with basename takes 26 seconds without a builtin and >> 0.1 seconds when basename is a builtin: >> $ timex ksh93 -c 'find work/src | while read i ; do x=$(basename $i) ; >> done' >> >> real 26.35 >> user 9.34 >> sys1.34 >> >> $ timex ksh93 -c 'builtin basename ; find work/src | while read i ; do >> x=$(basename $i) ; done' >> >> real 0.11 >> user 0.07 >> sys0.03 >> >> This ROCKS incredibly :) >> >>> How does this interact with PSARC/2009/377 in kernel pfexec, maybe it >>> doesn't need to and that is an okay answer, when ksh93 is the profile >>> shell >>> ? >> >> ksh93-integration-discuss@ currently has an ongoing discussion about >> this and the broken profile shell concept. There are two concurrent >> proposals to integrate the concepts of shell builtins and profile >> shells. > > Then this case should be put in waiting need spec until that discussion > reaches some consensus. Why? The majority in the discussion already said that this is not a bug in ksh93. Why should this case wait for something which is not a bug? Why should this case wait for something where not even a consensus has been reached whether the issue is a bug or not? Or is this a delay tactic again to stall? > > The profile shell concept is not broken, Go to the bash authors and ask them. Ask them why they never accepted the patches for profile shell support. Why did BSD never adopt the profile shell system? Did Linux adopt this concept? The answer is all a big, sticky NO. > some people may not like what it is > or how it works but it is approved a delivered architecture. It has been > deliver as part of Solaris since Solaris 8 and is based on functionality > going back more than 15 years in various Trusted Solaris (and even before > that name) releases of SunOS. > >>> If so then I wonder why we are even shipping the GNU >>> ones. >> >> I don't see the point either since the ksh93 commands have both >> features from GNU AND BSD > > Do the have 100% of the features with exactly the same behaviour though ? Yes, I think so. Jenny -- Jennifer Pioch, Uni Frankfurt
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Garrett D'Amore wrote: > On 03/18/10 08:53 AM, Stefan Teleman wrote: >> This project raises a concern in my mind with respect to a very old >> and generally accepted UNIX architectural principle: >> >> "Tools, Not Policy". >> >> If i understand it correctly, this case effectively vacates the >> principle stated above, and replaces it with its exact opposite: >> >> "Policy, Not Tools". >> >> Because the only possible rationale for having /usr/gnu/bin/grep >> transparently and silently replaced by a shell builtin grep is as a >> result of some mandatory policy in effect, which would trump explicit >> user selection. >> >> It would be very helpful if the project team would kindly explain how >> it intends to address the architectural concern above, and also how it >> intends to mitigate the proliferation of userland commands with >> identical names, but providing, in many cases, different semantics. > > Users can override the policy and disable the builtins if they so > choose. They could also choose another shell instead of ksh93 -- these > changes don't apply to bash or tcsh or zsh. Why do I need to explicitly disable something I don't want in the first place ? I want to use the GNU grep in /usr/gnu/bin/grep. I don't want to have to tell the shell NOT to silently override my explicit choices. Having the shell override explicit choices by default is in my opinion inherently broken. And the rationale behind this breakage appears to be: "We broke the expected behavior. If you don't like this breakage, change your shell." > I guess I don't really understand the objection -- the builtin behaves > the same way, but performs better. What's the issue? The issue is that there are now 4 different grep's in Solaris, and that the shell can pick a different one from the one i explicitly chose by setting my PATH. I find that having 4 different greps is excessive, and very difficult to justify architecturally. And it is not acceptable to say "get rid of /usr/gnu and make room for the ksh93 builtins", because that's not how it is supposed to work. The GNU coreutils/binutils were integrated first. --Stefan -- Stefan Teleman Oracle Corporation stefan.teleman at Sun.COM
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
>Architecturally, I have to agree with Darren here. I don't know what >the concerns are here where this would fail to operate with the current >pfexec... I thought that it was just the case that pfexec would bypass >the builtin and use the filesystem supplied binary. That is not currently the case and "pfsh" in OpenSolaris doesn't work as OpenSolaris did for the builtins. I'm hoping to fix that by making sure we ignore the builtin commands in profile shells. (By default and as only option inside of a profile shell) Casper
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
userland command name conflict resolution is handled by crafting PATH around the conflicts and user preferences ksh93 builtins can also be pluguins located in shared-libs/dlls and plugin lookup can be interspersed with PATH lookup if /usr/gnu/bin is searched before a plugin containing a grep builtin then /usr/gnu/bin/grep will take precedence over the grep builtin On Thu, 18 Mar 2010 11:53:57 -0400 Stefan Teleman wrote: > This project raises a concern in my mind with respect to a very old > and generally accepted UNIX architectural principle: > "Tools, Not Policy". > If i understand it correctly, this case effectively vacates the > principle stated above, and replaces it with its exact opposite: > "Policy, Not Tools". > Because the only possible rationale for having /usr/gnu/bin/grep > transparently and silently replaced by a shell builtin grep is as a > result of some mandatory policy in effect, which would trump explicit > user selection. > It would be very helpful if the project team would kindly explain how > it intends to address the architectural concern above, and also how it > intends to mitigate the proliferation of userland commands with > identical names, but providing, in many cases, different semantics.
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 3:24 PM, Garrett D'Amore - sun microsystems wrote: > > Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI > This information is Copyright 2010 Sun Microsystems > 1. Introduction > ? ?1.1. Project/Component Working Name: > ? ? ? ? More ksh93 builtins > ? ?1.2. Name of Document Author/Supplier: > ? ? ? ? Author: ?Olga Kryzhanovska > ? ?1.3 ?Date of This Document: > ? ? ? ?18 March, 2010 > 4. Technical Description > > I'm sponsoring this fast-track request on behalf of the > ksh93-integration project. > Please note that this is an *open* case. > > The release binding is the same as with the ksh93 project: a > patch/micro release of Solaris delivering through ON > Stability levels are as described below. > > > This project is an amendment to the Korn Shell 93 Integration project > (PSARC/2006/550 and PSARC/2007/035, PSARC/2008/094, PSARC/2008/344 > and PSARC/2008/589) specifying the following additional > interfaces: > Addition of /usr/gnu/bin, /usr/xpg4/bin and /usr/bin built in > mappings in ksh93 > > Bug/RFE Number(s): > > 6935110 ?ksh93 needs (more) XPG, GNU built ins > > This case proposes to deliver the following features as a set of > independent putbacks as they become available. Each feature is > self contained and independent of the others, so out of order > and partial putbacks at this granularity should have no adverse > impact on the functionality and behavior of the system as a whole. > > > Part 1: ksh93 built in mappings for /usr/gnu/bin utilities > -- > The case proposes to enable the following mappings to ksh93 built in > commands for utilities in /usr/gnu/bin. When the utility is called > using it's command name, and not the full path, the ksh93 builtin > will be used instead of executing the /usr/gnu/bin binary. > > > Interface ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Stability ? ? ? ?Description > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? - ? ? ? ?--- > ksh93 '/usr/gnu/bin/basename' built in ?Uncommitted ? ? ?basename utility > with GNU extensions So, if I call a plain basename, what gets executed? Does it look at the PATH setting and call the gnu/solaris/xpg4 (or for the more perverse of us, ucb) variant of the builtin? Is there just one builtin that always behaves the same way? Or, another way, if I don't have /usr/gnu/bin in my PATH, will the builtin basename have the GNU extensions? (Differences for something like basename are minor; for other utilities the differences may matter.) -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 18/03/2010 16:28, Casper.Dik at sun.com wrote: > > >> Architecturally, I have to agree with Darren here. I don't know what >> the concerns are here where this would fail to operate with the current >> pfexec... I thought that it was just the case that pfexec would bypass >> the builtin and use the filesystem supplied binary. > > That is not currently the case and "pfsh" in OpenSolaris doesn't work as > OpenSolaris did for the builtins. I'm hoping to fix that by making sure > we ignore the builtin commands in profile shells. (By default and as only > option inside of a profile shell) Thank you Casper, that is all I needed to know. I'm happy to see that you and Glenn are working on it. -- Darren J Moffat
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 09:21 AM, Stefan Teleman wrote: > > The issue is that there are now 4 different grep's in Solaris, and > that the shell can pick a different one from the one i explicitly > chose by setting my PATH. I find that having 4 different greps is > excessive, and very difficult to justify architecturally. We already have them even without this case. I hate this too. I would love to just get rid of *all* the others, provide a single implementation based on ksh93 that offers all the most often requested features. But that's not this case. > > And it is not acceptable to say "get rid of /usr/gnu and make room for > the ksh93 builtins", because that's not how it is supposed to work. > The GNU coreutils/binutils were integrated first. If you feel that way, then either don't use ksh93, or use a .profile that disables the builtins. The point of this case is that the builtins are drop-in compatible. You should not care about the implementation. If you *do*, then you're an edge case user and its not unreasonable that you have to suffer some extra pain to get exactly the implementation bits you want. - Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 09:28 AM, Casper.Dik at Sun.COM wrote: > > >> Architecturally, I have to agree with Darren here. I don't know what >> the concerns are here where this would fail to operate with the current >> pfexec... I thought that it was just the case that pfexec would bypass >> the builtin and use the filesystem supplied binary. >> > That is not currently the case and "pfsh" in OpenSolaris doesn't work as > OpenSolaris did for the builtins. I'm hoping to fix that by making sure > we ignore the builtin commands in profile shells. (By default and as only > option inside of a profile shell) > > > Casper > > This sounds like a showstopper for this case then. - Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 3:41 PM, Darren J Moffat wrote: > > Why would I want to use ksh93 builtins if I have /usr/gnu/bin explicitly in > my path ? ?Are the ksh93 builtin versions 100% compatible in all respects > with the GNU ones ? No. It's explicitly stated that --version in particular is worded differently. I've seen configure scripts and applications take apart --version output in order to divine version-specific behaviour >?If so then I wonder why we are even shipping the GNU > ones. Because that's what many users and applications expect. And there's a world outside ksh93. -- -Peter Tribble http://www.petertribble.co.uk/ - http://ptribble.blogspot.com/
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 18/03/2010 16:38, Garrett D'Amore wrote: > On 03/18/10 09:28 AM, Casper.Dik at Sun.COM wrote: >> >>> Architecturally, I have to agree with Darren here. I don't know what >>> the concerns are here where this would fail to operate with the current >>> pfexec... I thought that it was just the case that pfexec would bypass >>> the builtin and use the filesystem supplied binary. >> That is not currently the case and "pfsh" in OpenSolaris doesn't work as >> OpenSolaris did for the builtins. I'm hoping to fix that by making sure >> we ignore the builtin commands in profile shells. (By default and as only >> option inside of a profile shell) >> >> >> Casper >> > > This sounds like a showstopper for this case then. This case doesn't make the current situation worse. I also now understand that isn't really that relevant to this case anyway because this is all about when /usr/gnu/bin is first in your path and about providing builtins for that. Since we have no /usr/gnu/bin entries in exec_attr anyway there is no issue with pfexec and this case. -- Darren J Moffat
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
>On 18/03/2010 16:38, Garrett D'Amore wrote: >> On 03/18/10 09:28 AM, Casper.Dik at Sun.COM wrote: >>> Architecturally, I have to agree with Darren here. I don't know what the concerns are here where this would fail to operate with the current pfexec... I thought that it was just the case that pfexec would bypass the builtin and use the filesystem supplied binary. >>> That is not currently the case and "pfsh" in OpenSolaris doesn't work as >>> OpenSolaris did for the builtins. I'm hoping to fix that by making sure >>> we ignore the builtin commands in profile shells. (By default and as only >>> option inside of a profile shell) >>> >>> >>> Casper >>> >> >> This sounds like a showstopper for this case then. > >This case doesn't make the current situation worse. I also now >understand that isn't really that relevant to this case anyway because >this is all about when /usr/gnu/bin is first in your path and about >providing builtins for that. Since we have no /usr/gnu/bin entries in >exec_attr anyway there is no issue with pfexec and this case. I agree with Darren (assuming there are no profiles with now builtin GNU commands) Casper
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 09:37 AM, Peter Tribble wrote: > On Thu, Mar 18, 2010 at 3:41 PM, Darren J Moffat > wrote: > >> Why would I want to use ksh93 builtins if I have /usr/gnu/bin explicitly in >> my path ? Are the ksh93 builtin versions 100% compatible in all respects >> with the GNU ones ? >> > No. It's explicitly stated that --version in particular is worded differently. > I've seen configure scripts and applications take apart --version output > in order to divine version-specific behaviour > If those versions specify a full path, then they'll not see a change. It does sound like this is kind of busted. I can understand why this happens though. > >> If so then I wonder why we are even shipping the GNU >> ones. >> > Because that's what many users and applications expect. And there's > a world outside ksh93. > > I have a couple of opinions about all this, which I'll restate here: 1) In an ideal world, we'd supply (by "default") a single implementation of these commands. It seems like ksh93 is the most well-maintained POSIX conforming implementation, and offers all the features people seem to want, so that seems like the best choice. 2) /usr/gnu versions would still ship for people who *insist*, but we should not be advocating GNU versions in preference to the POSIX conforming versions. /usr/gnu has no place (IMO) on the default user's path. The fact that we put it there at all now is just a crutch to workaround the lack of investment in the aging (rotting!) Sun userland (ksh93 not included) 3) IMO, the configuration of builtins should be enabled the same way that other PATH elements are enabled -- via a new PATH element. I would not mind seeing /usr/ksh93/bin. This could be prepended to the default PATH for new users. It would provide POSIX conforming (plus any of the popular desired features from GNU, BSD, etc.) implementations of commands, but would also allow for simple builtins to be used by libcmd and ksh93. Since ksh93 uses libcmd, there would be *zero* functional difference apart from the savings of a fork/exec, and everyone would be happy. This whole nonsense IMO, comes about because the ksh93 folks want to enable higher performance, but someone decided that /usr/gnu should be at the front of default users PATHs. I think *that* decision was busted. -- Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 09:41 AM, Darren J Moffat wrote: > > > On 18/03/2010 16:38, Garrett D'Amore wrote: >> On 03/18/10 09:28 AM, Casper.Dik at Sun.COM wrote: >>> Architecturally, I have to agree with Darren here. I don't know what the concerns are here where this would fail to operate with the current pfexec... I thought that it was just the case that pfexec would bypass the builtin and use the filesystem supplied binary. >>> That is not currently the case and "pfsh" in OpenSolaris doesn't >>> work as >>> OpenSolaris did for the builtins. I'm hoping to fix that by making sure >>> we ignore the builtin commands in profile shells. (By default and as >>> only >>> option inside of a profile shell) >>> >>> >>> Casper >>> >> >> This sounds like a showstopper for this case then. > > This case doesn't make the current situation worse. I also now > understand that isn't really that relevant to this case anyway because > this is all about when /usr/gnu/bin is first in your path and about > providing builtins for that. Since we have no /usr/gnu/bin entries in > exec_attr anyway there is no issue with pfexec and this case. > Thanks for the clarification. So, this particular objection is removed. - Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Jennifer Pioch wrote: > Why? The majority in the discussion already said that this is not a > bug in ksh93. Why should this case wait for something which is not a > bug? A majority of people who don't know what profile shells are or how they are used is not relevant - even if they did, bugs are not decided by democratic vote of the masses, but by review of the experts in that area. -- -Alan Coopersmith-alan.coopersmith at oracle.com Oracle Solaris Platform Engineering: X Window System
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Garrett D'Amore wrote: > The point of this case is that the builtins are drop-in compatible. You > should not care about the implementation. If you *do*, then you're an > edge case user and its not unreasonable that you have to suffer some > extra pain to get exactly the implementation bits you want. I think the relevant architectural question should be: "if a project team wants to update an existing GNU utility, what must that team do in order to avoid regression for ksh93 users, and how does that project team know that they need to do this?" That's the part that needs to be described adequately, so that any team doing that work doesn't have to stumble in the dark. If the answer is, "they must coordinate first with the ksh93 team," that seems reasonable, but there needs to be _some_ clear answer. I see nothing wrong with silently replacing existing utilities with better-performing ones having exactly the same interface, but I do think it's a problem if some users may accidentally see "old" versions of something that has been updated. (In particular, seeing a newly updated man or GNU info page, and having a new feature listed there not work because the command invocation is redirected by the shell would be pretty confusing.) -- James Carlson 42.703N 71.076W
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Garrett D'Amore wrote: > On 03/18/10 08:58 AM, Alan Coopersmith wrote: >> Garrett D'Amore - sun microsystems wrote: >> >>> Compatibility between ksh93 built in utility implementation and GNU >>> coreutils implementation: >>> Should a future ARC case will add new features to the GNU coreutils >>> utilities the project team will update the corresponding ksh93 built >>> in utility. Should this not be possible the ksh93 project team will >>> remove the mapping. >>> >> So this constrains all future GNU coreutils update cases to coordinate >> with the ksh93 builtins, right? Have those responsible for the GNU >> coreutils agreed to this constraint? Should this be expressed as an >> ARC contract for cross-consolidation agreement of Volatile interfaces? >> >> > > The ksh93 versions adhere to the public interfaces from the GNU > versions. If they change the semantics, then it will require some extra > work, but I'm not sure that a contract properly captures this. Even just having a note added to the SFW metadata file would be better than doing nothing, which goes back to the original question of whether those responsible for the GNU coreutils in SFW have agreed to this or even been made aware of it? -- -Alan Coopersmith-alan.coopersmith at oracle.com Oracle Solaris Platform Engineering: X Window System
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 10:45 AM, Alan Coopersmith wrote: > Garrett D'Amore wrote: > >> On 03/18/10 08:58 AM, Alan Coopersmith wrote: >> >>> Garrett D'Amore - sun microsystems wrote: >>> >>> Compatibility between ksh93 built in utility implementation and GNU coreutils implementation: Should a future ARC case will add new features to the GNU coreutils utilities the project team will update the corresponding ksh93 built in utility. Should this not be possible the ksh93 project team will remove the mapping. >>> So this constrains all future GNU coreutils update cases to coordinate >>> with the ksh93 builtins, right? Have those responsible for the GNU >>> coreutils agreed to this constraint? Should this be expressed as an >>> ARC contract for cross-consolidation agreement of Volatile interfaces? >>> >>> >>> >> The ksh93 versions adhere to the public interfaces from the GNU >> versions. If they change the semantics, then it will require some extra >> work, but I'm not sure that a contract properly captures this. >> > Even just having a note added to the SFW metadata file would be better > than doing nothing, which goes back to the original question of whether > those responsible for the GNU coreutils in SFW have agreed to this or > even been made aware of it? > I doubt they have been formally notified. If they have, I'm unaware of it. I'm actually of the opinion that this entire case, along with the /usr/gnu situation, is a mess, and we should go back to the drawing board (so to speak) and address the more significant concerns that are the root cause for this case (in particular the /usr/gnu replacements) to come up. The problem is, I'm not sure there is any desire on anyone doing the actual work to revisit the overall architecture surrounding user environments. Even if it is desperately needed. -- Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
I have two arguments against this. 1) GNU tools are sometimes more flexible (you can do a lot with one command, something which is quite tricky with Solaris tools.) E.g.: - grep -r something somewhere - find -print0 (without it its' difficult to deal with files with complex paths) - xargs -0 2) A lot of software depend on GNU tools, and you can't build it without gmake, gcc and even GNU patch, because Linux is the most widespread Unix-like system. Even on FreeBSD you have to install gmake to compile a lot of stuff. The first problem you may partly solve with evolving Solaris native tools (thaks OpenSolaris developers for BSD ps syntax, without it to see full program path and arguments was a problem :)) , but the second part, IMHO, is not possible to solve. (At least you could use find + grep to emulate this behavior), > 2) /usr/gnu versions would still ship for people who > *insist*, but we > should not be advocating GNU versions in preference > to the POSIX > conforming versions. /usr/gnu has no place (IMO) on > the default user's > path. The fact that we put it there at all now is > just a crutch to > workaround the lack of investment in the aging > (rotting!) Sun userland > (ksh93 not included) -- This message posted from opensolaris.org
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 10:53, Garrett D'Amore wrote: > I'm actually of the opinion that this entire case, along with the > /usr/gnu situation, is a mess, and we should go back to the drawing > board (so to speak) and address the more significant concerns that are > the root cause for this case (in particular the /usr/gnu replacements) > to come up. Regardless of your feelings about the existence of /usr/gnu, the fact is that there are multiple versions of various commands out there, and users who expect those commands to work in their manifold versions. It seems broken to suggest that the incorporation of ksh93 built-ins in any way should in any way constrain the addition of new features into the emulated commands. ksh93 is merely one shell among many, and despite a somewhat vociferous and adulatory fan base is not the only programming environment out there. If you need to add built-ins to the shell to solve your script performance issues I think you've chosen the wrong programming environment, and should consider something other than a shell. - Bart -- Bart Smaalders Solaris Kernel Performance bart.smaalders at oracle.comhttp://blogs.sun.com/barts "You will contribute more with mercurial than with thunderbird."
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
I'm not a member of PSARC and can't actually vote here, but I would have to give this case a -1. It's not that I don't think that the ksh93 built-ins have a place and that they couldn't be a perfectly reasonable default for most people. In some cases, they seem to provide something that is sorely needed in the Solaris userland, a blend of POSIX conforming and GNUish commands. Several things concern me about this case. * When I (read any user) explicitly set my PATH to include /usr/bin, /usr/xpg4/bin, /usr/xpg6/bin, ..., I expect that my selection will be honored. I am asking for a specific toolset and expect a bug for bug compatible version. * "default" behaviour should apply across the board here. For several paths on the system, customers expect certain behaviour /usr/xpg4 XPG4 compatible behaviour /usr/xpg6XPG6 compatible behaviour /usr/gnuGNU/Linux compatible behaviour /usr/ucbSunOS/BSD 4.X behaviour /usr/5binSVR3 compatible behaviour ... On 03/18/10 10:24 AM, Garrett D'Amore - sun microsystems wrote: > Template Version: @(#)sac_nextcase 1.69 02/15/10 SMI > This information is Copyright 2010 Sun Microsystems > 1. Introduction > 1.1. Project/Component Working Name: >More ksh93 builtins > 1.2. Name of Document Author/Supplier: >Author: Olga Kryzhanovska > 1.3 Date of This Document: > 18 March, 2010 > 4. Technical Description > > I'm sponsoring this fast-track request on behalf of the > ksh93-integration project. > Please note that this is an *open* case. > > The release binding is the same as with the ksh93 project: a > patch/micro release of Solaris delivering through ON > Stability levels are as described below. > > > This project is an amendment to the Korn Shell 93 Integration project > (PSARC/2006/550 and PSARC/2007/035, PSARC/2008/094, PSARC/2008/344 > and PSARC/2008/589) specifying the following additional > interfaces: > Addition of /usr/gnu/bin, /usr/xpg4/bin and /usr/bin built in > mappings in ksh93 > > Bug/RFE Number(s): > > 6935110 ksh93 needs (more) XPG, GNU built ins > > This case proposes to deliver the following features as a set of > independent putbacks as they become available. Each feature is > self contained and independent of the others, so out of order > and partial putbacks at this granularity should have no adverse > impact on the functionality and behavior of the system as a whole. > > > Part 1: ksh93 built in mappings for /usr/gnu/bin utilities > -- > The case proposes to enable the following mappings to ksh93 built in > commands for utilities in /usr/gnu/bin. When the utility is called > using it's command name, and not the full path, the ksh93 builtin > will be used instead of executing the /usr/gnu/bin binary. > > > Interface StabilityDescription > - ---- > ksh93 '/usr/gnu/bin/basename' built in Uncommitted basename utility > with GNU extensions > ksh93 '/usr/gnu/bin/cksum' built in Uncommitted cksum utility with > GNU extensions > ksh93 '/usr/gnu/bin/comm' built in Uncommitted comm utility with > GNU extensions > ksh93 '/usr/gnu/bin/cut' built in Uncommitted cut utility with GNU > extensions > ksh93 '/usr/gnu/bin/dirname' built in Uncommitted dirname utility with > GNU extensions > ksh93 '/usr/gnu/bin/expr' built in Uncommitted expr utility with > GNU extensions > ksh93 '/usr/gnu/bin/fold' built in Uncommitted fold utility with > GNU extensions > ksh93 '/usr/gnu/bin/join' built in Uncommitted join utility with > GNU extensions > ksh93 '/usr/gnu/bin/logname' built in Uncommitted logname utility with > GNU extensions > ksh93 '/usr/gnu/bin/mkdir' built in Uncommitted mkdir utility with > GNU extensions > ksh93 '/usr/gnu/bin/mkfifo' built inUncommitted mkfifo utility with > GNU extensions > ksh93 '/usr/gnu/bin/mktemp' built inUncommitted mktemp utility with > GNU extensions > ksh93 '/usr/gnu/bin/pathchk' built in Uncommitted pathchk utility with > GNU extensions > ksh93 '/usr/gnu/bin/paste' built in Uncommitted paste utility with > GNU extensions > ksh93 '/usr/gnu/bin/sleep' built in Uncommitted sleep utility with > GNU extensions > ksh93 '/usr/gnu/bin/sync' built in Uncommitted sync utility with > GNU extensions > ksh93 '/usr/gnu/bin/tee' built in Uncommitted tee utility with GNU > extensions > ksh93 '/usr/gnu/bin/tty' built in Uncommitted tty utility with GNU > extensions > ksh93 '/usr/gnu/bin/uniq' built in Uncommitted uniq utility with > GNU extensions > ksh93 '/usr/gnu/bin/rmdir' built in Uncommitted rmdir utility with > GNU extensions > ksh93 '/usr/gnu/bin/wc' built inUncommitted wc utility with GNU > extensions >
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 18/03/2010 23:57, Norm Jacobs wrote: > For several paths on the system, customers expect certain behaviour > /usr/xpg4 XPG4 compatible behaviour > /usr/xpg6 XPG6 compatible behaviour > /usr/gnu GNU/Linux compatible behaviour > /usr/ucb SunOS/BSD 4.X behaviour > /usr/5bin SVR3 compatible behaviour standards(5) mandates specific directory paths for different levels of POSIX etc compliance. I'm not clear how this 'hidden override' mechanism impacts that, an explanation would be helpful. And I don't see any mention of /usr/xpg6/bin in there either. -- Alan Burlison --
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Norm, On 03/18/10 07:57 PM, Norm Jacobs wrote: > It's not that I don't think that the ksh93 built-ins have a place and > that they couldn't be a perfectly reasonable default for most people. In > some cases, they seem to provide something that is sorely needed in the > Solaris userland, a blend of POSIX conforming and GNUish commands. I'm afraid I don't understand your initial point. That's not what the ksh93 shell built-ins provide. $PATH is still honored, and the built-ins are compatible with the commands that are resolved from $PATH. You do not suddenly get a blend of POSIX and GNU commands. I believe that architecture introduced by PSARC 2006/550 maintains the semantics of $PATH by binding a built-in to a pathname, and only executing a built-in when a $PATH search matches the binding. > > Several things concern me about this case. > > * When I (read any user) explicitly set my PATH to include /usr/bin, > /usr/xpg4/bin, /usr/xpg6/bin, ..., I expect that my selection will > be honored. I am asking for a specific toolset and expect a bug > for bug compatible version. > * "default" behaviour should apply across the board here. > > For several paths on the system, customers expect certain behaviour > /usr/xpg4 XPG4 compatible behaviour > /usr/xpg6 XPG6 compatible behaviour > /usr/gnu GNU/Linux compatible behaviour > /usr/ucb SunOS/BSD 4.X behaviour > /usr/5bin SVR3 compatible behaviour > ... Given that the shell honors compatibility within each of the applicable paths for which it provides built-in bindings, there doesn't seem to be any issue regarding "compatible behavior". -Seb
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/18/10 09:54 PM, Sebastien Roy wrote:
> Norm,
>
> On 03/18/10 07:57 PM, Norm Jacobs wrote:
>> It's not that I don't think that the ksh93 built-ins have a place and
>> that they couldn't be a perfectly reasonable default for most people. In
>> some cases, they seem to provide something that is sorely needed in the
>> Solaris userland, a blend of POSIX conforming and GNUish commands.
>
> I'm afraid I don't understand your initial point. That's not what the
> ksh93 shell built-ins provide. $PATH is still honored, and the
> built-ins are compatible with the commands that are resolved from
> $PATH. You do not suddenly get a blend of POSIX and GNU commands.
I guess I just assumed that there would be one ksh93 built-in here for
something like basename and that it would take the same arguments and
function the same regardless of whether ksh93 were executing it in place
of the /usr/gnu, /usr/xpg4, or /usr/bin version. Perhaps that's not the
case.
> I believe that architecture introduced by PSARC 2006/550 maintains
> the semantics of $PATH by binding a built-in to a pathname, and only
> executing a built-in when a $PATH search matches the binding.
Perhaps I am misunderstanding this part of the current case.
> Part 1: ksh93 built in mappings for /usr/gnu/bin utilities
> --
> The case proposes to enable the following mappings to ksh93 built in
> commands for utilities in /usr/gnu/bin. When the utility is called
> using it's command name, and not the full path, the ksh93 builtin
> will be used instead of executing the /usr/gnu/bin binary.
>
As I understand that statement
$ PATH=/usr/gnu/bin:/usr/bin /usr/bin/ksh93 -c "basename --version"
and
$ PATH=/usr/gnu/bin:/usr/bin
/usr/bin/{your-favorite-non-ksh93-shell} -c "basename --version"
will return different results. Given that I explicitly put /usr/gnu/bin
at the front of the PATH in the examples, I would expect
/usr/gnu/bin/basename warts and all. Though perhaps it's only slightly
different and perhaps slightly better than the shell built-in bits that
are things like test, echo, ...
I think that in part, I misread this:
> Interface StabilityDescription
> - ----
> ksh93 '/usr/gnu/bin/basename' built in Uncommitted basename utility
> with GNU extensions
> ksh93 '/usr/gnu/bin/cksum' built in Uncommitted cksum utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/comm' built in Uncommitted comm utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/cut' built in Uncommitted cut utility with GNU
> extensions
> ksh93 '/usr/gnu/bin/dirname' built in Uncommitted dirname utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/expr' built in Uncommitted expr utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/fold' built in Uncommitted fold utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/join' built in Uncommitted join utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/logname' built in Uncommitted logname utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/mkdir' built in Uncommitted mkdir utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/mkfifo' built inUncommitted mkfifo utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/mktemp' built inUncommitted mktemp utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/pathchk' built in Uncommitted pathchk utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/paste' built in Uncommitted paste utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/sleep' built in Uncommitted sleep utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/sync' built in Uncommitted sync utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/tee' built in Uncommitted tee utility with GNU
> extensions
> ksh93 '/usr/gnu/bin/tty' built in Uncommitted tty utility with GNU
> extensions
> ksh93 '/usr/gnu/bin/uniq' built in Uncommitted uniq utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/rmdir' built in Uncommitted rmdir utility with
> GNU extensions
> ksh93 '/usr/gnu/bin/wc' built inUncommitted wc utility with GNU
> extensions
to mean that
$ ksh93 -c "/usr/gnu/bin/basename"
would use the ksh93 basename built-in, which supports GNU extensions.
Though that does seem in conflict with the paragraph that precedes the
table. I gather that it means that
$ PATH=/usr/gnu/bin:/usr/bin /usr/bin/ksh93 -c "basename"
will use the ksh93 basename built-in, which happens to support GNU
extensions.
>>
>> For several paths on the system, customers expect certain behaviour
>> /usr/xpg4 XPG4 compatible behaviour
>> /usr/xpg6 XPG6 compatible behaviour
>> /usr/gnu GNU/Linux compatible behaviour
>> /usr/ucb SunOS/BSD 4.X behaviour
>> /usr/5bin SVR3 compatible behaviou
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
Norm, On 03/19/10 04:34 AM, Norm Jacobs wrote: > I think that in part, I misread this: >> Interface Stability Description >> - - --- >> ksh93 '/usr/gnu/bin/basename' built in Uncommitted basename utility >> with GNU extensions ... > to mean that > > $ ksh93 -c "/usr/gnu/bin/basename" > > would use the ksh93 basename built-in, which supports GNU extensions. > Though that does seem in conflict with the paragraph that precedes the > table. I gather that it means that > > $ PATH=/usr/gnu/bin:/usr/bin /usr/bin/ksh93 -c "basename" > > will use the ksh93 basename built-in, which happens to support GNU > extensions. Correct: Fully qualified command names do not use built-ins. This is specified by 2006/550. >> Given that the shell honors compatibility within each of the >> applicable paths for which it provides built-in bindings, there >> doesn't seem to be any issue regarding "compatible behavior". > > If ksh93 built-ins are really drop-in replacements for the interfaces > they intend on interposing on under ksh93 (/usr/bin, /usr/xpg4, > /usr/gnu, ...), then we should be delivering a wrapper/link to the ksh93 > version so that we have consistency regardless of shell/execution > environment. It could be the intention of this case and that I just > missed it in my read through. If they are direct replacements, then they > should use delivered as such. If they are not direct replacements, then > they should probably not be used in place of their intended target under > ksh93 only. > > This may also have some added benefit in helping us be able to better > deal with conflict/divergence down the road. Yes, that's a good point, and perhaps that can be looked into in the future. However, I think this discussion is veering off-topic for this case, as you're now debating the implementation and architecture of the shell built-ins in general, which is not being introduced by this case. That was introduced by PSARC 2006/550. This case is building upon previously approved architecture. -Seb
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/19/10 07:19 AM, Sebastien Roy wrote: > Norm, > > On 03/19/10 04:34 AM, Norm Jacobs wrote: >> I think that in part, I misread this: >>> Interface Stability Description >>> - - --- >>> ksh93 '/usr/gnu/bin/basename' built in Uncommitted basename utility >>> with GNU extensions > ... >> to mean that >> >> $ ksh93 -c "/usr/gnu/bin/basename" >> >> would use the ksh93 basename built-in, which supports GNU extensions. >> Though that does seem in conflict with the paragraph that precedes the >> table. I gather that it means that >> >> $ PATH=/usr/gnu/bin:/usr/bin /usr/bin/ksh93 -c "basename" >> >> will use the ksh93 basename built-in, which happens to support GNU >> extensions. > > Correct: Fully qualified command names do not use built-ins. This is > specified by 2006/550. > >>> Given that the shell honors compatibility within each of the >>> applicable paths for which it provides built-in bindings, there >>> doesn't seem to be any issue regarding "compatible behavior". >> >> If ksh93 built-ins are really drop-in replacements for the interfaces >> they intend on interposing on under ksh93 (/usr/bin, /usr/xpg4, >> /usr/gnu, ...), then we should be delivering a wrapper/link to the ksh93 >> version so that we have consistency regardless of shell/execution >> environment. It could be the intention of this case and that I just >> missed it in my read through. If they are direct replacements, then they >> should use delivered as such. If they are not direct replacements, then >> they should probably not be used in place of their intended target under >> ksh93 only. >> >> This may also have some added benefit in helping us be able to better >> deal with conflict/divergence down the road. > > Yes, that's a good point, and perhaps that can be looked into in the > future. However, I think this discussion is veering off-topic for > this case, as you're now debating the implementation and architecture > of the shell built-ins in general, which is not being introduced by > this case. That was introduced by PSARC 2006/550. This case is > building upon previously approved architecture. > Perhaps I delve slightly into implementation detail here, but the issue of consistency is something that needs to be addressed here and now, not wait for the future. -Norm
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/19/10 07:58 AM, Norm Jacobs wrote: > >> Yes, that's a good point, and perhaps that can be looked into in the >> future. However, I think this discussion is veering off-topic for >> this case, as you're now debating the implementation and architecture >> of the shell built-ins in general, which is not being introduced by >> this case. That was introduced by PSARC 2006/550. This case is >> building upon previously approved architecture. >> > > Perhaps I delve slightly into implementation detail here, but the > issue of consistency is something that needs to be addressed here and > now, not wait for the future. I am coming to agree. While I'm the sponsor on this case, I'm on the verge of derailing this case and asking that a new case to examine userland shell architecture be created. The fact that we have to put /usr/gnu at the head of $PATH of new users is a bit of a travesty, and I'm of the opinion that we should reexamine *that* particular decision, in which case much of the motivation behind *this* case comes into question. (If /usr/gnu isn't the default for most users, then there is little motivation to provide builtin wrappers for them.) I'd rather see ksh93 based utilities (or rather libcmd based) with all the bells and whistles delivered into /usr/bin or perhaps /usr/ksh93/bin (and put at the head of $PATH) and leave /usr/gnu as a dumping ground for people who insist that they want GNU warts. -- Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/19/10 08:13, Garrett D'Amore wrote: > I'd rather see ksh93 based utilities (or rather libcmd based) with all > the bells and whistles delivered into /usr/bin or perhaps /usr/ksh93/bin > (and put at the head of $PATH) and leave /usr/gnu as a dumping ground > for people who insist that they want GNU warts. Perhaps you could leave the pejoratives out next time; it really doesn't add anything to the discussion. Like it or not, a significant number of our customers (and a larger number of those we'd like to have as customers) are familiar and accustomed to the GNU commands; referring to their environment of choice as "warts" and "dumping ground" does nothing useful. - Bart -- Bart Smaalders Solaris Kernel Performance bart.smaalders at oracle.comhttp://blogs.sun.com/barts "You will contribute more with mercurial than with thunderbird."
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Fri, Mar 19, 2010 at 08:13:48AM -0700, Garrett D'Amore wrote: > The fact that we have to put /usr/gnu at the head of $PATH of new > users is a bit of a travesty, and I'm of the opinion that we should > reexamine *that* particular decision... This is merely one opinion. There are compelling business and architecture cases for having the default userland be approachable by the majority of users of other popular unix-like operating systems. The /usr/gnu isn't the default in my path either, but it makes a lot of sense to present a userland that's familiar to users of Linux, and similar environments. Anyone is free to create a distro with a different default shell, or default path. Anyone is free to change their path as well as their shell. Your fixation on /usr/gnu's presence in the default path isn't productive. Why make it harder to get users from Linux and elsewhere to adopt Solaris? > ... in which case much of the motivation behind *this* case comes into > question. (If /usr/gnu isn't the default for most users, then there > is little motivation to provide builtin wrappers for them.) I disagree. Modulo the issues about the profile shell, I see no reason why it matters that the ARC delve into the minutia of shell builtins. In general, that's an implementation or configuration detail of the shell. I would recommend against derailing this case in favor of one about a grand shell [re-]architecture. We should be making it easier add different shells for Solaris. Using this case as an opportunity to rail about Gnu is just divisive. -j
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/19/10 03:52 PM, johansen at sun.com wrote: > On Fri, Mar 19, 2010 at 08:13:48AM -0700, Garrett D'Amore wrote: > >> The fact that we have to put /usr/gnu at the head of $PATH of new >> users is a bit of a travesty, and I'm of the opinion that we should >> reexamine *that* particular decision... >> > This is merely one opinion. There are compelling business and > architecture cases for having the default userland be approachable by > the majority of users of other popular unix-like operating systems. The > /usr/gnu isn't the default in my path either, but it makes a lot of > sense to present a userland that's familiar to users of Linux, and > similar environments. > Approachable (and even familiar) does not necessarily == /usr/gnu. It does mean having the main command flags work the same way that people coming from foreign environments (and not just Linux, but also *BSD and maybe other platforms like AIX or HPUX) are used to. > Anyone is free to create a distro with a different default shell, or > default path. Anyone is free to change their path as well as their > shell. Your fixation on /usr/gnu's presence in the default path isn't > productive. Why make it harder to get users from Linux and elsewhere to > adopt Solaris? > I'm not proposing that it should be. I'm proposing that putting /usr/gnu first is not necessarily the only way to achieve that goal. I'd rather see us modernize our own tools. I resent abdication of our own engineering, and the necessity of abandoning all good innovations (like shell builtins) because some people feel its critical that the only way to achieve these goals is to provide these 3rd party tools. Its more offensive to me specifically because there is no good reason why we can't use tools from the ksh93 community (who seems to be a lot more willing to work with us on key engineering issues than the GNU folks who are mostly fixated on Linux) to achieve this. I'm also of the opinion that it is a mistake to sacrifice familiarity for our paying Solaris 10 customers in favor of familiarity for people coming from Linux. Which group do you think contributes more towards the $$ that pay our salaries? > >> ... in which case much of the motivation behind *this* case comes into >> question. (If /usr/gnu isn't the default for most users, then there >> is little motivation to provide builtin wrappers for them.) >> > I disagree. Modulo the issues about the profile shell, I see no reason > why it matters that the ARC delve into the minutia of shell builtins. > In general, that's an implementation or configuration detail of the shell. > I guess reasonable people can disagree here. The points have been made that these drop ins are not 100% compatible drop ins... they have ever so slightly differing interfaces. While normal folks should not care about the differences (they are so tiny, like the format of --version's output), there are situations where it can matter. That makes it a matter of concern for ARC. > I would recommend against derailing this case in favor of one about a > grand shell [re-]architecture. We should be making it easier add > different shells for Solaris. Using this case as an opportunity to rail > about Gnu is just divisive. > The case isn't derailed. Its waiting need spec. At the (more or less) request of the upstream technology supplier. - Garrett
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Fri, Mar 19, 2010 at 04:08:09PM -0700, Garrett D'Amore wrote: > I'd rather see us modernize our own tools. I resent abdication of > our own engineering, and the necessity of abandoning all good > innovations (like shell builtins) because some people feel its > critical that the only way to achieve these goals is to provide > these 3rd party tools. Its more offensive to me specifically > because there is no good reason why we can't use tools from the > ksh93 community (who seems to be a lot more willing to work with us > on key engineering issues than the GNU folks who are mostly fixated > on Linux) to achieve this. Instead of re-inventing the wheel at every opportunity, it makes more sense to take the open source projects that have wide acceptance and incorporate them into our product. I think that both ksh93 and gnu fall into this category. It's much better for us to focus our engineering efforts on areas where we can actually differentiate our product from our competitors. I don't have a problem with ksh93 or the builtins, nor am I advocating an entirely GNU userland. What I am suggesting, however, is that the folks who decided to put /usr/gnu in the default path did talk to our customers, and also took note of the fact that Linux is widely adoped across the industry. > I'm also of the opinion that it is a mistake to sacrifice > familiarity for our paying Solaris 10 customers in favor of > familiarity for people coming from Linux. This is a false dilemma. It should be entirely possible for customers to configure whatever default path they desire and deploy that in their enterprise via AI. > Which group do you think contributes more towards the $$ that pay our > salaries? I would love to argue this point with you, but it's not appropriate to discuss it on a public mailing list. -j
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Sat, Mar 20, 2010 at 12:26 AM, wrote: > On Fri, Mar 19, 2010 at 04:08:09PM -0700, Garrett D'Amore wrote: >> I'd rather see us modernize our own tools. I resent abdication of >> our own engineering, and the necessity of abandoning all good >> innovations (like shell builtins) because some people feel its >> critical that the only way to achieve these goals is to provide >> these 3rd party tools. Its more offensive to me specifically >> because there is no good reason why we can't use tools from the >> ksh93 community (who seems to be a lot more willing to work with us >> on key engineering issues than the GNU folks who are mostly fixated >> on Linux) to achieve this. > > Instead of re-inventing the wheel at every opportunity, it makes more > sense to take the open source projects that have wide acceptance and > incorporate them into our product. I think that both ksh93 and gnu fall > into this category. It's much better for us to focus our engineering > efforts on areas where we can actually differentiate our product from > our competitors. > > I don't have a problem with ksh93 or the builtins, nor am I advocating > an entirely GNU userland. What I am suggesting, however, is that the > folks who decided to put /usr/gnu in the default path did talk to our > customers, Yes, they did. But they ignored our counsel. Adopting GNU coreutils as default or replacements for /usr/bin is not the answer (and never was). Sun needs a community centric progressive development and adopt POSIX, BSD and GNU features and semantics and not drop in GNU coreutils in the current swallow&die manner. > and also took note of the fact that Linux is widely adoped > across the industry. BSD is widely adopted in the industry, too. Many servers run FreeBSD and many embedded machines run NetBSD. Why was /usr/bsd/bin not adopted instead of /usr/gnu/bin? Sun's management seems to have a obsession with Linux, doesn't bother to look at alternatives and abandoned its own development of the userland utilities (there won't be a UNIX certification for Opensolaris) and now even starts to beat the community which has developed an alternative solution. The real strength of Linux is its community, the *OPEN* development process (not the behind-the-closed-curtain ARC process, i.e. 2010/067) and cooperation. Sun's management still has to learn that. Or Opensolaris will FAIL. >> I'm also of the opinion that it is a mistake to sacrifice >> familiarity for our paying Solaris 10 customers in favor of >> familiarity for people coming from Linux. > > This is a false dilemma. It should be entirely possible for customers > to configure whatever default path they desire and deploy that in their > enterprise via AI. Possible. Yes. Yet it's the *default* configuration we're discussing here. A *default* which broke too many things on my companies side. Guess why we did not port our applications from Solaris 10 to Opensolaris yet? The costs are too high. >> Which group do you think contributes more towards the $$ that pay our >> salaries? > > I would love to argue this point with you, but it's not appropriate to > discuss it on a public mailing list. What is cheaper for Sun? Put /usr/gnu/bin in front of PATH and break almost all user scripts or progressively enhance the utilities in /usr/bin to adopt BSD and GNU features with POSIX as foundation. The latter sounds like the better (and cheaper!) alternative as Opensolaris has already the POSIX community for that purpose. Irek
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 4:41 PM, Darren J Moffat wrote: > Maybe I don't understand enough about ksh93 (since I'm a zsh user for > interactive shell work) but I don't understand what this case is about. > > What benefit does this case bring ? First at all you do not go through fork() and be a lot faster. Second you do not have ARG_MAX and other process-based limitations, e.g. the list and size of arguments passed to builtin commands and shell functions is only limited by memory (thank again Roland for giving us a 64bit ksh93 :) ). With builtin our scripts can't trip over the ancient Solaris limits for ARG_MAX (if anyone has time, please file a bug to get ARG_MAX removed completely.). Chris -- ^---^ (@)v(@) Chris Pickett |/ IT consultant ===m==m=== pkchris at users.sourceforge.net
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/27/2010 03:39 PM, Chris Pickett wrote: > On Thu, Mar 18, 2010 at 4:41 PM, Darren J Moffat > wrote: >> What benefit does this case bring ? > > First at all you do not go through fork() and be a lot faster. The intent appears to be better performance; generally a good thing. I'm concerned about the implementation and its implications. It seems to me that this is an example of all problems in programming being an exercise in caching. The item being cached is a utility executable, and the cache load time is during compilation of the proposed new ksh93 implementation. Unfortunately, the cache invalidation and/or reload is also the latter time. I think this is a mistake. If I, with suitable permissions, cannot replace the binary of a utility in the filesystem of my system and get the expected changed behaviour, it is no longer Unix. This is an architectural issue. Gnu-ness or otherwise is not relevant to this argument. As a possible alternate architecture, I suggest 1) kernel support for pre-linking arbitrary executables with a running process, such that a call into such an executable image does not require a system-call 2) kernel support for de-linking the above, from the running process 3) means for a running process to be notified of changes to the filesystem object backing an executable, without the use of a system-call when no such change has been made We've been into this area before. Let's not make such a mistake as vfork was again. -- Jeremy Harris
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Sat, Mar 27, 2010 at 05:02:45PM +, Jeremy Harris wrote: > Unfortunately, the cache invalidation and/or reload is also the latter > time. I think this is a mistake. If I, with suitable permissions, cannot > replace the binary of a utility in the filesystem of my system and > get the expected changed behaviour, it is no longer Unix. This > is an architectural issue. If you replace programs delivered by Solaris itself they you've rendered your system unsupportable and, indeed, we will not support it. Nico --
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Mon, Mar 29, 2010 at 8:42 AM, Nicolas Williams wrote: > If you replace programs delivered by Solaris itself they you've rendered > your system unsupportable and, indeed, we will not support it. That may be true of Oracle's commercial Solaris Product, but we are talking about OpenSolaris here. The architectural point is that the user/admin needs control of things like this; with ksh93 builtins, they have that ability (i.e., they can turn builtins off...) and update binutils packages and the like. There is no magic here. -John
More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/29/2010 07:18 PM, John Plocher wrote: > The architectural point is that the user/admin needs control of things > like this; with ksh93 builtins, they have that ability (i.e., they can > turn builtins off...) and update binutils packages and the like. I'm suggesting that with better architecture there would be no need for such manual intervention, nor the surprise element of the system apparently doing the wrong thing (with the builtins enabled by default). > There is no magic here. There is insufficient magic here. Cheers, Jeremy
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
>That said, its possible that the GNU tools will evolve in the future, at >a rate differently than the ksh93 versions do. (At that point, the case >says that the ksh93 version will either be adapted, or they'll stop >supplying the built-in.) And, unfortunately, anyone who wants to deploy a newer version of a GNU tool in OpenSolaris needs to make sure that ksh93 is updated at the same time. Casper
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, 18 Mar 2010 09:38:17 -0700 Garrett D'Amore wrote:
> On 03/18/10 09:28 AM, Casper.Dik at Sun.COM wrote:
> >
> >
> >> Architecturally, I have to agree with Darren here. I don't know what
> >> the concerns are here where this would fail to operate with the current
> >> pfexec... I thought that it was just the case that pfexec would bypass
> >> the builtin and use the filesystem supplied binary.
> >>
> > That is not currently the case and "pfsh" in OpenSolaris doesn't work as
> > OpenSolaris did for the builtins. I'm hoping to fix that by making sure
> > we ignore the builtin commands in profile shells. (By default and as only
> > option inside of a profile shell)
> >
> >
> > Casper
> >
> >
> This sounds like a showstopper for this case then.
> - Garrett
the fix to disable builtins for pfksh is only a few lines
dgk and I are checking out the code now
there is another alternative if we can pfexec bracket sections of code inline
I beleieve a message yesterday, about 1000 posts ago:) mentioned this is
possible
e.g., for the builtin b_mkdir() (with made-up function names)
pfexec_pretend("mkdir")
b_mkdir()
pfexec_stop_pretending()
if this is possible I would like to see example code
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
>the fix to disable builtins for pfksh is only a few lines
>dgk and I are checking out the code now
>
>there is another alternative if we can pfexec bracket sections of code inline
>I beleieve a message yesterday, about 1000 posts ago:) mentioned this is
>possible
>e.g., for the builtin b_mkdir() (with made-up function names)
> pfexec_pretend("mkdir")
> b_mkdir()
> pfexec_stop_pretending()
>if this is possible I would like to see example code
No, not quite: the kernel-code will only work on exec and not inside
a shell.
Casper
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, 18 Mar 2010 17:51:26 +0100 Casper.Dik at sun.com wrote:
> >the fix to disable builtins for pfksh is only a few lines
> >dgk and I are checking out the code now
> >
> >there is another alternative if we can pfexec bracket sections of code inline
> >I beleieve a message yesterday, about 1000 posts ago:) mentioned this is
> >possible
> >e.g., for the builtin b_mkdir() (with made-up function names)
> > pfexec_pretend("mkdir")
> > b_mkdir()
> > pfexec_stop_pretending()
> >if this is possible I would like to see example code
> No, not quite: the kernel-code will only work on exec and not inside
> a shell.
ok, thanks
anyway, we should have an alpha package for you to check tomorrow
that disables the -lcmd builtins in pfksh
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Fri, 19 Mar 2010 08:13:48 -0700 Garrett D'Amore wrote: > I am coming to agree. While I'm the sponsor on this case, I'm on the > verge of derailing this case and asking that a new case to examine > userland shell architecture be created. The fact that we have to put > /usr/gnu at the head of $PATH of new users is a bit of a travesty, and > I'm of the opinion that we should reexamine *that* particular decision, > in which case much of the motivation behind *this* case comes into > question. (If /usr/gnu isn't the default for most users, then there is > little motivation to provide builtin wrappers for them.) > I'd rather see ksh93 based utilities (or rather libcmd based) with all > the bells and whistles delivered into /usr/bin or perhaps /usr/ksh93/bin > (and put at the head of $PATH) and leave /usr/gnu as a dumping ground > for people who insist that they want GNU warts. dgk are discussing this right now we had somehow missed the detail that the proposed ksh builtin binding dir is "/usr/gnu/bin" just because a libcmd builtin handles some gnu options does not make it gnu there are most likely gnu features that libcmd builtins will never implement e.g., the gnu getopt(3) "feature" that allows options to appear after operands once we solidify the ideas where should we post, and under what subject? -- Glenn Fowler -- at&t Research, Florham Park NJ --
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On 03/19/10 08:27 AM, Glenn Fowler wrote: > On Fri, 19 Mar 2010 08:13:48 -0700 Garrett D'Amore wrote: > >> I am coming to agree. While I'm the sponsor on this case, I'm on the >> verge of derailing this case and asking that a new case to examine >> userland shell architecture be created. The fact that we have to put >> /usr/gnu at the head of $PATH of new users is a bit of a travesty, and >> I'm of the opinion that we should reexamine *that* particular decision, >> in which case much of the motivation behind *this* case comes into >> question. (If /usr/gnu isn't the default for most users, then there is >> little motivation to provide builtin wrappers for them.) >> > >> I'd rather see ksh93 based utilities (or rather libcmd based) with all >> the bells and whistles delivered into /usr/bin or perhaps /usr/ksh93/bin >> (and put at the head of $PATH) and leave /usr/gnu as a dumping ground >> for people who insist that they want GNU warts. >> > dgk are discussing this right now > we had somehow missed the detail that the proposed ksh builtin binding dir > is "/usr/gnu/bin" > > just because a libcmd builtin handles some gnu options does not make it gnu > there are most likely gnu features that libcmd builtins will never implement > e.g., the gnu getopt(3) "feature" that allows options to appear after operands > > once we solidify the ideas where should we post, and under what subject? > Reply to this message. I'm going to put this case on waiting-needs-spec. My guess is that ultimately the thing to do will be to "not" override the GNU builtins, because you're not providing truly 100% compatible drop-ins. (And it sounds like unless you're willing to introduce any bugs that are present in the GNU versions, you never will be.) - Garrett
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Sat, Mar 20, 2010 at 12:56 AM, Alan Coopersmith wrote: > Garrett D'Amore wrote: >> I really do think that the way this was handled in OpenSolaris -- which >> occurred without any significant ARC discussion of the concerns >> surrounding this -- is unfortunate. I am half tempted to bring forward >> a case to change this if only to cause the conversation (that I think >> didn't happen) to occur. > > PSARC already has that case under review - PSARC 2010/067. Does a closed ARC case even matter? I don't think a non-public ARC case should be allowed to affect an open source project at all. Not at least if ARC wants to be taken serious in the future. Irek
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Sat, Mar 20, 2010 at 2:58 AM, Jason King wrote: > On Fri, Mar 19, 2010 at 5:26 PM, wrote: >> On Fri, Mar 19, 2010 at 04:08:09PM -0700, Garrett D'Amore wrote: >>> I'd rather see us modernize our own tools. I resent abdication of >>> our own engineering, and the necessity of abandoning all good >>> innovations (like shell builtins) because some people feel its >>> critical that the only way to achieve these goals is to provide >>> these 3rd party tools. Its more offensive to me specifically >>> because there is no good reason why we can't use tools from the >>> ksh93 community (who seems to be a lot more willing to work with us >>> on key engineering issues than the GNU folks who are mostly fixated >>> on Linux) to achieve this. >> >> Instead of re-inventing the wheel at every opportunity, it makes more >> sense to take the open source projects that have wide acceptance and >> incorporate them into our product. I think that both ksh93 and gnu fall >> into this category. It's much better for us to focus our engineering >> efforts on areas where we can actually differentiate our product from >> our competitors. >> >> I don't have a problem with ksh93 or the builtins, nor am I advocating >> an entirely GNU userland. What I am suggesting, however, is that the >> folks who decided to put /usr/gnu in the default path did talk to our >> customers, and also took note of the fact that Linux is widely adoped >> across the industry. > > Yet now, I see (at least online) complaints because the 'default' > (GNU) tools can't actually make use of the features that differentiate > Opensolaris. Why are we encouraging defaults that hide the features > that set us apart from the others? The long term goal [1] for the POSIX community is to add the missing features from BSD and GNU to the utilities in /usr/bin, using the AST tools as template. I do not think that /usr/gnu/bin in $PATH matters then. [1] Depending how fast Oracle will sponsor the putbacks from Olga. Irek
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Fri, Mar 19, 2010 at 4:30 PM, Garrett D'Amore wrote: > On 03/19/10 08:27 AM, Glenn Fowler wrote: >> >> On Fri, 19 Mar 2010 08:13:48 -0700 Garrett D'Amore wrote: >> >>> >>> I am coming to agree. While I'm the sponsor on this case, I'm on the >>> verge of derailing this case and asking that a new case to examine >>> userland shell architecture be created. The fact that we have to put >>> /usr/gnu at the head of $PATH of new users is a bit of a travesty, and >>> I'm of the opinion that we should reexamine *that* particular decision, >>> in which case much of the motivation behind *this* case comes into >>> question. (If /usr/gnu isn't the default for most users, then there is >>> little motivation to provide builtin wrappers for them.) >>> >> >> >>> >>> I'd rather see ksh93 based utilities (or rather libcmd based) with all >>> the bells and whistles delivered into /usr/bin or perhaps /usr/ksh93/bin >>> (and put at the head of $PATH) and leave /usr/gnu as a dumping ground >>> for people who insist that they want GNU warts. >>> >> >> dgk are discussing this right now >> we had somehow missed the detail that the proposed ksh builtin binding dir >> is "/usr/gnu/bin" >> >> just because a libcmd builtin handles some gnu options does not make it >> gnu >> there are most likely gnu features that libcmd builtins will never >> implement >> e.g., the gnu getopt(3) "feature" that allows options to appear after >> operands >> >> once we solidify the ideas where should we post, and under what subject? >> > > Reply to this message. I'm going to put this case on waiting-needs-spec. > My guess is that ultimately the thing to do will be to "not" override the > GNU builtins, because you're not providing truly 100% compatible drop-ins. > (And it sounds like unless you're willing to introduce any bugs that are > present in the GNU versions, you never will be.) Glenn may be confusing glibc getopt(3) with coreutils getopt. I've been experimenting with ksh93 and its shell builtins and I am not able to find a difference except that cut, paste and friends accept multibyte characters (GNU coreutils spew errors like /usr/gnu/bin/cut: the delimiter must be a single character) while GNU coreutils do not. Irek
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 5:37 PM, Peter Tribble wrote: > On Thu, Mar 18, 2010 at 3:41 PM, Darren J Moffat > wrote: >> >> Why would I want to use ksh93 builtins if I have /usr/gnu/bin explicitly in >> my path ? Are the ksh93 builtin versions 100% compatible in all respects >> with the GNU ones ? > > No. It's explicitly stated that --version in particular is worded differently. > I've seen configure scripts and applications take apart --version output > in order to divine version-specific behaviour This will break anyway in the future. GNU coreutils changed the 'wording' of the --version output a couple of times in the past. ARC...ARC... the ARC case for GNU coreutils nor the manpages specify a specific wording 'syntax'. Sounds like from an ARC perspective this does not matter. >> If so then I wonder why we are even shipping the GNU >> ones. > > Because that's what many users and applications expect. And there's > a world outside ksh93. Right. But within ksh93 scripts this optimization makes sense. And the tools in /usr/bin still require modernisation. Irek
[shell-discuss] More ksh93 builtins [PSARC/2010/095 FastTrack timeout 03/25/2010]
On Thu, Mar 18, 2010 at 5:52 PM, Garrett D'Amore wrote: > On 03/18/10 09:37 AM, Peter Tribble wrote: > I have a couple of opinions about all this, which I'll restate here: > > 1) In an ideal world, we'd supply (by "default") a single implementation of > these commands. It seems like ksh93 is the most well-maintained POSIX > conforming implementation, and offers all the features people seem to want, > so that seems like the best choice. +1 > 2) /usr/gnu versions would still ship for people who *insist*, but we should > not be advocating GNU versions in preference to the POSIX conforming > versions. /usr/gnu has no place (IMO) on the default user's path. The fact > that we put it there at all now is just a crutch to workaround the lack of > investment in the aging (rotting!) Sun userland (ksh93 not included) Oracle could hire either Olga Kryzhanovska or Roland Mainz to work on the Solaris userland. Both are experts in the area and the driving force behind the ksh93 and POSIX community efforts. > 3) IMO, the configuration of builtins should be enabled the same way that > other PATH elements are enabled -- via a new PATH element. I would not mind > seeing /usr/ksh93/bin. This could be prepended to the default PATH for new > users. It would provide POSIX conforming (plus any of the popular desired > features from GNU, BSD, etc.) implementations of commands, but would also > allow for simple builtins to be used by libcmd and ksh93. Since ksh93 uses > libcmd, there would be *zero* functional difference apart from the savings > of a fork/exec, and everyone would be happy. > > This whole nonsense IMO, comes about because the ksh93 folks want to enable > higher performance, but someone decided that /usr/gnu should be at the front > of default users PATHs. I think *that* decision was busted. Right. +1 Irek
