Copy Reduction Interfaces [PSARC/2009/478 FastTrack timeout09/16/2009]

[Roland Mainz]

Mahesh Siddheshwar wrote:
> Roland Mainz wrote:
> [snip]
> > How do you handle "sparse files", e.g. files with one or more "holes" ?
>
> Sparse files are not handled any differently in VOP_READ/VOP_WRITE calls
> when using the zero-copy interface. Modules that want to seek/skip holes
> can use the _FIO_SEEK_DATA/_FIO_SEEK_HOLE  commands
> of VOP_IOCTL, to do so.

Ok...

[snip]
> >>   The definition of xuio_t is:
> >>
> >>   typedef struct xuio {
> >> uio_t xu_uio;   /* Embedded UIO structure */
> >>
> >> /* Extended uio fields */
> >> enum xuio_type xu_type; /* What kind of uio structure? */
> >>
> >> union {
> >>
> >> /* Async I/O Support */
> >> struct {
> >> uint32_t xu_a_state;/* state of async i/o */
> >> uint32_t xu_a_state;/* state of async i/o */
> >> ssize_t xu_a_mbytes;/* bytes that have been 
> >> uioamove()ed */
> >> uioa_page_t *xu_a_lcur; /* pointer into uioa_locked[] */
> >> void **xu_a_lppp;   /* pointer into lcur->uioa_ppp[] */
> >> void *xu_a_hwst[4]; /* opaque hardware state */
> >> uioa_page_t xu_a_locked[UIOA_IOV_MAX];   /* Per iov locked 
> >> pages */
> >> } xu_aio;
> >>
> >> /* Zero Copy Support */
> >> struct {
> >> enum uio_rw xu_zc_rw;   /* the use of the buffer */
> >> void *xu_zc_priv;   /* fs specific */
> >>
> >
> > Does it make sense to have a |xu_flags| field here for future
> > enhancments ?
> >
> If future enhancements are needed to extend xuio_t,  a new xuio_type
> can be defined and extended that way.  For extensions not specific
> to xuio, there also exists uio_extflg in the uio_t.  Without a particular
> purpose an additional flag seems unnecessary for zero-copy right now.

Right now... yes. But Unix has a little (IMO) ugly tradition of not
adding such flag fields and instead swamping the headers with many many
variations of one interface over time which could be avoided by use
having a flags field as argument (that's a generic issue).

> Please note that there is a typo in the spec, in the definition
> for struct xu_aio. The below line is printed twice:
> 
> uint32_t xu_a_state;/* state of async i/o */
> 
> A corrected final spec will be posted in the case directory.

Ok...

> >> } xu_zc;
> >>
> >> } xu_ext;
> >>   } xuio_t;
> >>
> >>   where xu_type is currently defined as:
> >>
> >>   typedef enum xuio_type {
> >> UIOTYPE_ASYNCIO,
> >> UIOTYPE_ZEROCOPY
> >>   } xuio_type_t;
> >>
> >>   New uio extensions can be added by defining a new xuio_type_t, and 
> >> adding a
> >>   new member to the xu_ext union.
> >>
> >>  b. Requesting zero-copy buffers
> >>
> >> #define VOP_REQZCBUF(vp, rwflag, uiozcp, cr, ct) \
> >> fop_reqzcbuf(vp, rwflag, uiozcp, cr, ct)
> >>
> >> int fop_reqzcbuf(vnode_t *, enum uio_rw, xuio_t *, cred_t *,
> >> caller_context_t *);
> >>
> >
> > AFAIK the prototype should have a flags field to allow future
> > changes/extenstions without adding another VOP_*-hook ...
>
> Roland, if the extensions/changes are for the purpose of
> copy reduction/buffer sharing,  we don't need to add
> additional VOP_* routines. The current xuio_t extension is
> defined just for that.

Erm... the idea of having a flags field in |fop_reqzcbuf()| was to allow
slight modifications in behaviour - for example in the future there
could be flags which describe where (in a NUMA system) the buffer memory
resides (e.g. near the calling thread, near a point which is optimal for
all consumers, or near the hardware which fills the buffer etc.),
whether it should be in the L2 cache or not etc. etc.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Copy Reduction Interfaces [PSARC/2009/478 FastTrack timeout09/16/2009]

[Roland Mainz]

Rich.Brown at sun.com wrote:
[snip] 
>  == Introduction/Background ==
> 
>  Zero-copy (copy avoidance) is essentially buffer sharing
>  among multiple modules that pass data between the modules.
>  This proposal avoids the data copy in the READ/WRITE path
>  of filesystems, by providing a mechanism to share data buffers
>  between the modules. It is intended to be used by network file
>  sharing services like NFS, CIFS or others.
> 
>  Although the buffer sharing can be achieved through a few different
>  solutions, any such solution must work with File Event Monitors
>  (FEM monitors)[1] installed on the files. The solution must
>  allow the underlying filesystem to maintain any existing file
>  range locking in the filesystem.
> 
>  The proposed solution provides extensions to the existing VOP
>  interface to request and return buffers from a filesystem. The
>  buffers are then used with existing VOP_READ/VOP_WRITE calls with
>  minimal changes.

How do you handle "sparse files", e.g. files with one or more "holes" ?

>  == Proposed Changes ==
> 
>  VOP Extensions for Zero-Copy Support
>  
> 
>  a. Extended struct uio, xuio_t
[snip]
>   The project team has commitment from the networking team to remove
>   the current use of uioa_t and use the proposed extensions (CR 6880095).
> 
>   The definition of xuio_t is:
> 
>   typedef struct xuio {
> uio_t xu_uio;   /* Embedded UIO structure */
> 
> /* Extended uio fields */
> enum xuio_type xu_type; /* What kind of uio structure? */
> 
> union {
> 
> /* Async I/O Support */
> struct {
> uint32_t xu_a_state;/* state of async i/o */
> uint32_t xu_a_state;/* state of async i/o */
> ssize_t xu_a_mbytes;/* bytes that have been uioamove()ed 
> */
> uioa_page_t *xu_a_lcur; /* pointer into uioa_locked[] */
> void **xu_a_lppp;   /* pointer into lcur->uioa_ppp[] */
> void *xu_a_hwst[4]; /* opaque hardware state */
> uioa_page_t xu_a_locked[UIOA_IOV_MAX];   /* Per iov locked pages 
> */
> } xu_aio;
> 
> /* Zero Copy Support */
> struct {
> enum uio_rw xu_zc_rw;   /* the use of the buffer */
> void *xu_zc_priv;   /* fs specific */

Does it make sense to have a |xu_flags| field here for future
enhancments ?

> } xu_zc;
> 
> } xu_ext;
>   } xuio_t;
> 
>   where xu_type is currently defined as:
> 
>   typedef enum xuio_type {
> UIOTYPE_ASYNCIO,
> UIOTYPE_ZEROCOPY
>   } xuio_type_t;
> 
>   New uio extensions can be added by defining a new xuio_type_t, and adding a
>   new member to the xu_ext union.
> 
>  b. Requesting zero-copy buffers
> 
> #define VOP_REQZCBUF(vp, rwflag, uiozcp, cr, ct) \
> fop_reqzcbuf(vp, rwflag, uiozcp, cr, ct)
> 
> int fop_reqzcbuf(vnode_t *, enum uio_rw, xuio_t *, cred_t *,
> caller_context_t *);

AFAIK the prototype should have a flags field to allow future
changes/extenstions without adding another VOP_*-hook ...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2009/455 - Environment Modules

[Roland Mainz]

Garrett D'Amore wrote:
> Bruce Rothermal wrote:
> > This package is needed for a $Supported code project we are working on
> > in HPC. Since we are already porting this for our project we are
> > directed to make the package available to all and also so it is
> > available via the IPS package server on opensolaris.org.
> 
> Does this mean that we should be scrutinizing this project much more
> closely?  If the project is being put into use as a building block by
> our own developers, then it is no longer a "familiarity" project, and
> probably deserves much closer consideration for architectural
> correctness and completeness.

Some related questions:
1. Is there _any_ of the major Linux distributions which ships this
software ? I couldn't find it in SuSE and RedHat
2. Is it possible that this software is completely outdated, e.g. it
seems this stuff completely predates the "invention" of machine-global
config files like /etc/ksh.kshrc, /etc/bash.bashrc etc. which are
available on Linux since many many years ?
3. What happens if /etc/ksh.kshrc, /etc/bash.bashrc etc. are being used
by a shell - does the "modules" stuff just try to override such things
(e.g. clear all variables and shell settings and then start to define
it's own settings) or can it be modified to just pass-through such
"machine-global" settings ?
4. The "modules" stuff touches files of the user which are usually
_private_ - did anyone look at this issue from the _legal_ side ? It's
Ok if the user manually invokes the "modules" command but you'll be
walking in _very_ thin ice if some kind of automated script touches such
files without permission of the user. AFAIK Sun's lawyers need to check
at least the EU privacy laws and regulations in such a case.
5. IMO the shipping of per-user ~/.profile and ~/.bashrc+~/.kshrc should
be _discontinued_ in the future because updating them is impossible in
an usefull way and the legal troubles (see [4]) may make this even
impossible. That's why SuSE for example no longer installs such per-user
files (beyond empty "dummies") and has switched to machine-wide files
(e.g. /etc/profile, /etc/ksh.kshrc, /etc/bash.bashrc etc.) to define
their "vendor defaults" (and IMO Solaris should do the same).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2009/455 - Environment Modules

[Roland Mainz]

John Fischer wrote:
> I am sponsoring this project for Bruce Rothermal who will be integrating
> this project via the SFW consolidation.  I have set the timer for Friday,
> August 28th.  The case directory contains this proposal, FOSS check list,
> and appropriate man pages.
> 
> This project proposes to integrate the Environment Modules within a
> Minor release of Solaris (i.e., Open Solaris).  The environment modules
> provides an easy modification to a user's environment via TCL scripts.
> These scripts set various environmental variables such as PATH, MANPATH,
> etc.
[snip]
> 4. Technical Description:
> 
> The environment modules package provides for an easy dynamic
> modification of a user's environment via modulefiles. which
> typically instruct the module command to alter or set shell
> environment variables such as PATH, MANPATH, etc.
> 
>Command name   Notes
>===
>module   Link or alias to modulecmd
>modulecmdCommand interface to the Modules package.
[snip]
>Imported interfaceClassification   Comments
>===   ==   
> ===
>/lib/ld.so.1  Uncommitted  Library provided by SUNWcslr
>/usr/lib/libtcl8.4.so Uncommitted  Library provided by SUNWTcl
>/lib/libdl.so.1   Uncommitted  Library provided by SUNWcslr
>/lib/libm.so.2Uncommitted  Library provided by SUNWlibmsr
>/usr/lib/libX11.so.4  Uncommitted  Library provided by SUNWxwrtl
>/lib/libnsl.so.1  Uncommitted  Library provided by SUNWcslr
>/lib/libsocket.so.1   Uncommitted  Library provided by SUNWcslr
>/lib/libc.so.1Uncommitted  Library provided by SUNWcslr
>/lib/libpthread.so.1  Uncommitted  Library provided by SUNWcslr
>/usr/lib/libXext.so.0 Uncommitted  Library provided by SUNWxwrtl

I took a dive in the actual code at
http://modules.cvs.sourceforge.net/viewvc/modules/modules/ to see how it
works...
1. ... aren't the shells and their environment files imported interfaces
too, e.g. shouldn't the list include /usrbin/bash (and ~/.bashrc +
~/.bash_profile), /usr/bin/ksh (and ~/.profile, ~/.kshrc + ~/.kshrc),
/usr/bin/ksh93 (and ~/.profile,~/.kshrc + ~/.kshrc), /usr/bin/zsh
(erm... I don't know exactly which files zsh uses for startup) etc. ?
2. Please correct me if I am wrong... but does the "modules" stuff
differ between "login shell" and "interactive shell" ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2009/455 - Environment Modules

[Roland Mainz]

John Fischer wrote:
> 
> PSARC,
> 
> I am sponsoring this project for Bruce Rothermal who will be integrating
> this project via the SFW consolidation.  I have set the timer for Friday,
> August 28th.  The case directory contains this proposal, FOSS check list,
> and appropriate man pages.
> 
> This project proposes to integrate the Environment Modules within a
> Minor release of Solaris (i.e., Open Solaris).  The environment modules
> provides an easy modification to a user's environment via TCL scripts.
> These scripts set various environmental variables such as PATH, MANPATH,
> etc.
[snip]
> 4. Technical Description:
> 
> The environment modules package provides for an easy dynamic
> modification of a user's environment via modulefiles. which
> typically instruct the module command to alter or set shell
> environment variables such as PATH, MANPATH, etc.

How does this work ? AFAIK the shells have no API which allow another
process to just write or override the values of shell-internal values ?
Or is it expected that existing commands in /usr/bin/ are replaced with
the "module" wrapper and then select a matching binariy from {
/usr/gnu/bin/, /usr/oldsystemvcmd/bin/, /usr/bsd/bin/ etc. } ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2009/453 - futimens, utimensat

[Roland Mainz]

Garrett D'Amore wrote:
> Don Cragun wrote:
> > On Fri, 21 Aug 2009 00:00:53 -0700, Hugh McIntyre wrote:
> >> Joerg Schilling wrote:
> int futimens(int fd, const struct timespec times[2]);
> 
> int utimensat(int fd, const char *path,
>  const struct timespec times[2], int flag);
> 
>  In order to allow programs like star to be able to work correctly, we 
>  would
>  also need to have a pathconf()/fpathconf() call to retrieve the _actual_
>  resolution of a filesystem.
> 
> >> It looks like there's already a standard for this (perhaps just
> >> proposed?): _POSIX_TIMESTAMP_RESOLUTION and _PC_TIMESTAMP_RESOLUTION.
> >
> > These are not just proposed; tHey were added to the standards at the
> > same time futimens() and utimensat() were added.
> >
> > I believe this project is incomplete without also adding support for
> > these to the fpathconf() and pathconf() functions and the getconf
> > utility.
> 
> Hm  it seems that these should definitely be added.  I'm happy for
> them to be added to this project, but I'm not sure it is *required*.

I think it is required since otherwise there is no good way to know why
the timestamp changed between "set" and "get" ...

> Indeed, the pathconf/fpathconf changes should probably have been made at
> the time ZFS introduced nanosecond resolution.

Doesn't tmpfs have nanosecond resolution since a long time ?

> So while there is a relationship here, I'm not sure that the projects
> are necessarily locked together.

I think these are indirectly locked together and this issue needs to be
fixed - otherwise there's no way to use the proposed API correctly for
some applications.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2009/453 - futimens, utimensat

[Roland Mainz]

Don Cragun wrote:
> On Fri, 21 Aug 2009 00:00:53 -0700, Hugh McIntyre wrote:
> > Joerg Schilling wrote:
> > >>> >>>int futimens(int fd, const struct timespec times[2]);
> > >>> >>>
> > >>> >>>int utimensat(int fd, const char *path,
> > >>> >>> const struct timespec times[2], int flag);
> > > >
> > > > In order to allow programs like star to be able to work correctly, we 
> > > > would
> > > > also need to have a pathconf()/fpathconf() call to retrieve the _actual_
> > > > resolution of a filesystem.
> >
> > It looks like there's already a standard for this (perhaps just
> > proposed?): _POSIX_TIMESTAMP_RESOLUTION and _PC_TIMESTAMP_RESOLUTION.
> 
> These are not just proposed; tHey were added to the standards at the
> same time futimens() and utimensat() were added.
> 
> I believe this project is incomplete without also adding support for
> these to the fpathconf() and pathconf() functions and the getconf
> utility.

I agree with Don - we need a properly to detect such support on a
filesystem level, otherwise there will be some difficulties to use these
APIs in normal applications.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

changing stability levels (was: Fast Crash Dump)

[Roland Mainz]

Alan Coopersmith wrote:
> Roland Mainz wrote:
> > ... what's the general procedure to make a "project private" API
> > "consolidation private" ?
> 
> File an ARC fasttrack (or maybe even a self-review case if simple
> enough) declaring the list of interfaces and their new stability
> level.
> 
> Examples:
> PSARC/2007/154 DOF stability to Committed
> PSARC/2007/355 ddi_modopen Stability Promotion
> PSARC/2006/407 [TCP|UDP]_EXCLBIND Stability Change

Erm... does this even apply to "Private" interfaces (the question was
for a switch from "Project Private" to "Consolidation Private") ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Fast Crash Dump [PSARC/2009/330 FastTrack timeout 08/18/2009]

[Roland Mainz]

Garrett D'Amore wrote:
> Roland Mainz wrote:
> > Garrett D'Amore wrote:
> >> Dave Plauger wrote:
> >>> Ivek Szczesniak wrote:
> >>>> The stdio implementation in libc is among the slowest stdio
> >>>> versions out there. If you want to archive better performance you
> >>>> should use the stdio implementation in libast or use mmap(2).
> >>>>
> >>> This is an interesting implementation suggestion, but is outside the
> >>> scope of PSARC because it does not affect the interfaces
> >>> being proposed. We did achieve quite a speedup over the old method.
> >>> We'll take another look.
> >>>
> >> Using libast might well incur extra PSARC oversight -- are the libast
> >> interfaces public?  Consolidation private?  If they are *project
> >> private* then you'll need to get a contract for them.  Using mmap()
> >> would be free of those issues, and is likely to be the fastest without
> >> imposing any new interdependencies.
> >
> > Erm... I think it shouldn't be a problem to make the |malloc()| and
> >  parts of libast "consolidation private" for use within OS/Net
> > (e.g. this are the most stable APIs of libast and very unlikely to
> > change) ...
> >
> > ... question is: How do I do that ?
> 
> Lets find out if this is even necessary. 

Erm... the question above is more generic (since we had the idea to make
libast::stdio "consolidation private" several times in the past and
never had the time to do it) ...
... what's the general procedure to make a "project private" API
"consolidation private" ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Fast Crash Dump [PSARC/2009/330 FastTrack timeout 08/18/2009]

[Roland Mainz]

Garrett D'Amore wrote:
> Dave Plauger wrote:
> > Ivek Szczesniak wrote:
> >> The stdio implementation in libc is among the slowest stdio
> >> versions out there. If you want to archive better performance you
> >> should use the stdio implementation in libast or use mmap(2).
> > This is an interesting implementation suggestion, but is outside the
> > scope of PSARC because it does not affect the interfaces
> > being proposed. We did achieve quite a speedup over the old method.
> > We'll take another look.
> 
> Using libast might well incur extra PSARC oversight -- are the libast
> interfaces public?  Consolidation private?  If they are *project
> private* then you'll need to get a contract for them.  Using mmap()
> would be free of those issues, and is likely to be the fastest without
> imposing any new interdependencies.

Erm... I think it shouldn't be a problem to make the |malloc()| and
 parts of libast "consolidation private" for use within OS/Net
(e.g. this are the most stable APIs of libast and very unlikely to
change) ...

... question is: How do I do that ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

sys/stdbool.h [PSARC/2009/429 FastTrack timeout 2009/09/17]

[Roland Mainz]

Garrett D'Amore wrote:
> Peter Cudhea wrote:
> > It would be useful to take a definitive stand on whether or not
> > operations on two adjacent bool values are atomic with respect to one
> > another.  E.g. back in the days of the DEC Alpha, there existed no
> > instructions that could modify the value of one bool in a word without
> > potentially interfering in other uses of a differnt bool in the same
> > word.   That consideration does not apply to either of the current
> > Solaris platforms.   But are we introducing a constraint that Solaris
> > could never be ported to such a platform?   Or are we introducing a
> > constraint that any values that should be updatable atomically should
> > be stored in a boolean_t and not in a bool?   Or does the compiler
> > guarantee to allocate bools on word boundaries in platforms such as
> > this?   Either way, we should spell it out.
> 
> I would suggest that in situations where this matters, there are a
> couple of options:
> 
> 1) use locking primitives to guard an associated group of booleans.
> 2) find out what C99 says about this (if anything)
> 3) if you're really that concerned, such as in kernel space, use boolean_t.
> 
> I raised some of these concerns in private review with Roland before
> submitting the case.   Personally, I have reservations about using C99
> bool in places where atomicity is a requirement,

What about adding an |atomic_swap_bool()| (e.g. an alias for
|atomic_swap_uchar()|) to  ?

> or in public headers
> where it might force C99, or in structures where strict packing is required.

What do you mean with "strict packaging" in this case ? |bool| will
behave like the platform's smallest datatype (e.g. |char| (one byte on
SPARC+x86)) within a struct and therefore consume a lot less than
|boolean_t| (which may result in less data cache usage since |boolean_t|
is AFAIK an |enum| which consumes the space of an |int| (four bytes on
SPARC+x86)) ...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

sys/stdbool.h [PSARC/2009/429 FastTrack timeout 2009/09/17]

[Roland Mainz]

Peter Cudhea wrote:
> On 08/10/09 19:35, Garrett D'Amore - sun microsystems wrote:
> > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
> > This information is Copyright 2009 Sun Microsystems
> > 1. Introduction
> > 1.1. Project/Component Working Name:
> >sys/stdbool.h
> > 1.2. Name of Document Author/Supplier:
> >Author:  Garrett D'Amore
> > 1.3  Date of This Document:
> >   10 August, 2009
> > 4. Technical Description
> > I am sponsoring this fast track for Roland Mainz.  The case times out in one
> > week (8/19/2009), and minor binding is requested.
[snip]
> > 3.  Interface table
> >
> >Interface  | committment level
> >---+--
> >| Commited
> >C99 datatype "bool"| Commited
> >C99 value "true"   | Commited
> >C99 value "false"  | Commited
> >C99 CPP symbol "__bool_true_false_are_defined" | Commited
> >
> >Notes:
> >1.  precisely implements the functionality for
> >defined in IEEE Std 1003.1-2001.
> >2. The ability to undefine and redefine the macros "bool",
> >  "true", and "false" is an _implementation_detail_ in the ISO C
> >  standard which may be withdrawn in a future version of the
> >  standards specifications, kernel code shall _not_ rely on this
> >  _implementation_detail_ (e.g. "#undef true", "#undef false",
> >  "#undef bool" are not allowed in kernel code as they may cause
> >  problems with future compiler versions).
> >
> > 4.  References
> >- Bugster CR #6870093 RFE: C99  should be visible for
> >  kernel modules
> >- IEEE Std 1003.1-2001
> >- http://en.wikipedia.org/wiki/Stdbool.h
> >- ISO C99 draft (final document is not available for free)
> >  http://www.open-std.org/JTC1/SC22/WG14/www/docs/n1256.pdf
[snip]
> 
> It would be useful to take a definitive stand on whether or not
> operations on two adjacent bool values are atomic with respect to one
> another.  E.g. back in the days of the DEC Alpha, there existed no
> instructions that could modify the value of one bool in a word without
> potentially interfering in other uses of a differnt bool in the same
> word.

AFAIK Solaris supports |atomic_swap_uchar()| for userland applications -
since |bool| is a byte ([1]) on Solaris this may be used for atomic
operations...

[1]=Assuming the compiler cannot "reduce" it somehow further, e.g. if
the variable is "standalone" (e.g. not part of a struct, not accessed
via pointer etc.) the compiler may tweak it as it likes it (the only
other factor is the amount of states the variable may hold - but that's
exactly defined to be two ({ |true, |false| }) and therefore eliminates
that issue for the compiler).

> That consideration does not apply to either of the current
> Solaris platforms.   But are we introducing a constraint that Solaris
> could never be ported to such a platform?   Or are we introducing a
> constraint that any values that should be updatable atomically should be
> stored in a boolean_t and not in a bool?   Or does the compiler
> guarantee to allocate bools on word boundaries in platforms such as
> this?   Either way, we should spell it out.

Erm... this case is only to make the C99 header |stdbool.h> (and the
associated C99 datatype |bool|) available for kernel code (which IMO
should've been done at the time when C99 support was added to Solaris
(e.g. this case is basically "janitor work")) and _not_ to interpret
more on top of it, e.g. we only gurantee what the ISO C standard says
about |bool| and nothing beyond that.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2009/414 _withdrawn_ / was: Re: Some clarifications about"--help" and "--man" / Re: [busybox-dev]AST versions of fold, mktemp,pathchk,& tty [PSARC/2009/414 FastTrack timeout07/31/2009]

[Roland Mainz]

Roland Mainz wrote:
> Garrett D'Amore wrote:
> > Roland Mainz wrote:
> > > Alan Coopersmith wrote:
> [snip]
> > > What I still try to offer is the "truce" of letting us work in _peace_
> > > on the idea of having one master DocBook/XML manual page file from which
> > > both the normal manpages and the getopts strings are generated from
> > > (manpages have the full information, the getopts stuff just the terse
> > > information to keep it small).
> > >
> > > Garret, would you accept the "truce" and remove the "de-rail", please ?
> >
> > No, its too late for that.
> 
> Ok... I really tried to mediate a way though this but it seems we're
> stuck and right now it looks we're getting too much damage...
> 
> ... here hereby _withdraw_ this ARC case.

s/here/I/

I think the best option is now to drop this subject and discuss this on
the next OpenSolaris summit...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2009/414 _withdrawn_ / was: Re: Some clarifications about "--help" and "--man" / Re: [busybox-dev]AST versions of fold, mktemp, pathchk,& tty [PSARC/2009/414 FastTrack timeout07/31/2009]

[Roland Mainz]

Garrett D'Amore wrote:
> Roland Mainz wrote:
> > Alan Coopersmith wrote:
[snip]
> > What I still try to offer is the "truce" of letting us work in _peace_
> > on the idea of having one master DocBook/XML manual page file from which
> > both the normal manpages and the getopts strings are generated from
> > (manpages have the full information, the getopts stuff just the terse
> > information to keep it small).
> >
> > Garret, would you accept the "truce" and remove the "de-rail", please ?
> 
> No, its too late for that. 

Ok... I really tried to mediate a way though this but it seems we're
stuck and right now it looks we're getting too much damage...

... here hereby _withdraw_ this ARC case.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Some clarifications about "--help" and "--man" / Re: [busybox-dev] AST versions of fold, mktemp, pathchk, & tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

Alan Coopersmith wrote:
> I'm sponsoring this fast-track request on behalf of the
> ksh93-integration and busybox projects.  The timeout is
> set for Friday, July 31, 2009.
[snip]

Just to clarify (since both points have IMHO been either ignored or
misinterpreted several times):
1. We do _not_ intent to remove or discontinue normal documentation,
including manual pages (in fact I've been a long-term advocate of
getting Solaris moved to DocBook/XML-based manual pages (including
_shipping_ them as part of the installation instead of the nroff
versions (that's why I even worked on a /usr/bin/man replacement
codebase))).

2. The whole idea of "--man" was to re-use the existing information used
by the |libast::getopts()| call to provide quick access to _never_
information. It was _never_ intended (by this project (but not upstream
(for which the --nroff/--man output is _sufficient_))) to be a
_complete_ manual page with all manpage sections required by Solaris.
>From an ARC point of view I only see this (and intent to treat it) as an
extended form of the "--help" option which AFAIK has significant amount
of precendet. I really wish ARC would simply see "--man" as such, too.

What I still try to offer is the "truce" of letting us work in _peace_
on the idea of having one master DocBook/XML manual page file from which
both the normal manpages and the getopts strings are generated from
(manpages have the full information, the getopts stuff just the terse
information to keep it small).

Garret, would you accept the "truce" and remove the "de-rail", please ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[busybox-dev] AST versions of fold, mktemp,pathchk,& tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

John Sonnenschein wrote:
> On 25-Jul-09, at 4:59 PM, James Carlson wrote:
> > John Sonnenschein wrote:
> >> I've got a question about this...
> >> Whose responsibility is it to update the man pages and --man
> >> command then? The people whose jobs it is to update man pages, or
> >> the people whose jobs it is to update the command line utility?
> >> Basically if a new flag is added in the future for some reason, how
> >> will one synchronize the man pages?
> >
> > Usually, that's done by filing a bug against the man pages.
> >
> > The advantage of keeping the documentation separate is that it's in
> > the hands of professional documentation writers, who are able to
> > keep a consistent style across all of the system man pages.
> >
> > I'm with Garrett about the inadvisability of baking man page
> > documentation into executables, but for ksh93-related things, I
> > think that ship has unfortunately sailed.
> 
> Sure but for the sake of argument if we have some tools that have --
> man and also man pages, does that mean that the docs people will be
> putting back to ON,

Erm... why should the documentation people do putback into OS/Net ? The
strings used by getopts are used for argument parsing and are - as "nice
side-effect" - reused to generate the output for --help, --version,
--man etc.

> or will there be a desynchronization between the
> man pages and the --man pages ?

They _may_ be out-of-sync shortly after code putback if we add new
options to the |getopts()| string until the documentation folks caught
up with the code changes. But as I am trying to say over and over again
(and I am starting to feel _ignored_) that the output for --help, --man
etc. is generated from the getopts string template used for argument
parsing. This string is there to "drive" the argument parsing and is
absolutely the wrong place for Solaris-specific edits. We have a real,
seperate and maintained manual page for that purpose ([1]).

[1]=(And as said _several_ times that we could use a DocBook/XML manual
page as master source file shared between documentation and code folks
in the future from which both the Solaris manpage and the getopts string
can be generated from (this would eliminate all the "manpages
out-of-sync" concerns described in this thread))



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[busybox-dev] AST versions of fold, mktemp,pathchk,& tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

Garrett D'Amore wrote:
> Alan Coopersmith wrote:
> > Garrett D'Amore wrote:
> >> 1) The commands increase the size of the text segment.   Not only does
> >> it add new parsing requirements (you have to at least have enough code
> >> to handle --man, for example), but you also have the text of the man
> >> pages themselves.   While you might like to maintain the fiction that
> >> this comes for free, it *is* a fiction.  Run sum --man or some of the
> >> other commands and you'll see content that was not automatically generated.
> >
> > The minimum memory requirement for the release this is integrating into
> > is 512Mb.   A few k of memory, in a shared library that can be shared 
> > amongst
> > all processes using it, is far down in the noise.
> 
> This attitude, spread across dozens, or hundreds of projects, is part of
> the reason why software still struggles even in the face of Moore's
> law.   Nobody cares about a few kB, but those kB add up in aggregate.

Actually I _do_ care. That's why we came up with the AST-based busybox
design - to save memory and disk space _and_ make the commands faster at
the same time. The whole stuff should fully scale from 8MB ARM-based
machines upto SF25k class machines

> >> 2) We also have traditional manual pages.  On a normal system, the
> >> default installation will include now *two* copies of the manual page.
> >> This is wasted space.
> >
> > We also install commands users never use, drivers for hardware they don't
> > have (and could possibly never have), sample audio files they'll never
> > listen to, etc.   Text that helps them use a command is far more useful
> > than any of those.
> 
> Yes, but *two* copies of the text?  And in many cases those drivers
> *can* be easily removed, as can the audio files.  They are
> uninstallable.  (In fact, I recently pushed a change to delete a 300k
> audio file that nobody listens to anymore.  The remaining audio files
> are much smaller, and *may* be used by various applications or user
> configs.)

Did you actually notice my comment that it in the mid-term (not
long-term but not for the next putback) we may simple generate both the
normal manual pages and the getopts strings put into
libcmd/libshell/libbusybox from a DocBook/XML master file ?
That was actually my "truce offer" (e.g. let this case pass as-is and
then let me work in _peace_ on the DocBook/XML--->getopts XSLT
stylesheet stuff (as already said this puts a huge strain on my
resources but I would be willing to push this as a priority project))
... and I'm very very unhappy that this was completely ignored and moved
forward and de-railed the case (which will now likely lead to a
wishdrawl of the case) ...

> >> 3) Worse, the pages can be out of synch with each other.  (The sum man
> >> pages are again a good example of this.)  Which is correct?  (*Probably*
> >> the  --man command output.)
> >
> > This documentation is less likely to be out of sync with the implementation
> > than the traditional man page.   That's an improvement to the architecture,
> > making sure the user always has at least a correct, if less detailed, source
> > of documentation of the options.   It will also always document the version
> > the user is using - I've seen far too many questions on OpenSolaris e-mail
> > and IRC from users confused because their PATH & MANPATH don't match and
> > they're seeing /usr/gnu man pages but running /usr/bin commands, or vice 
> > versa.
> 
> I agree that this is likely to be an improvement (although it turns out
> that the documentation in the command still has to be maintained.)  Does
> that mean we abandon traditional man pages?

No, that was _never_ intended. The idea was simply to re-use some of the
information which is already present in the "getopts" string and format
it a bit like a ASCII rendering of a man page (I'm still stunned that we
have so many emails about this).

> >> 4) Furthermore, the --man output doesn't reflect standards required for
> >> Sun man pages.  For example, there is no
> >> ATTRIBUTES table.
> >
> > That's a good reason why there should also be a traditional man page, and
> > --man shouldn't replace it, just supplement it.
> 
> Your response here conflicts with your response to point #3 above.
> Either we have one canonical copy of the documentation that should
> always be correct and maintained, or we risk having the two copies get
> out of sync or become inaccurate.  Which should it be?

The out-of-sync can never be avoided since this stuff is done in two
seperate departments within Sun. The the man-like ASCII rendering and
the output for "--help" is obviously _always_ in sync because it uses
exactly the same information as the |getopts()| function uses for the
actual parsing but we have two "safeguards" in place to make sure that
the real manual page is updated (the current large lag was caused by the
lay-off of April Chin and the resulting havoc).

> >> 5) Some users elect to *remove* manual pages (or not install them

[ksh93-integration-discuss] [busybox-dev] AST versions of fold, mktemp, pathchk, & tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

"I. Szczesniak" wrote:
> On 7/25/09, Garrett D'Amore  wrote:
> > Roland Mainz wrote:
> > > Garrett D'Amore wrote:
> > > > Alan Coopersmith wrote:
> > > > > Garrett D'Amore wrote:
[snip]
> > > > Strongly enough that
> > > > I'm contemplating derailing the case.
> > >
> > > And what should we do then ? The only thing we can do is to remove it
> > > from the case materials - removing it from the code can only be done
> > > globally (e.g. libast) and that really will break existing&&ARC'ed
> > > parts.
> >
> >  #ifdef SOLARIS ?
> 
> Are you SERIOUSLY suggesting the project team has to add 2196 #ifdef
> SOLARIS statements (45 commands, 4 per option, 20 to strip further
> text strings in the getopt template) in the code of libcmd?

Erm... technically it may then be easier to |#ifdef SOLARIS ... #else
... #endif| the whole codeblock containg the getopts string... but as
said this is something which this project tried to avoid at all cost in
the past to prevent that we end-up in the same situation as Solaris
ksh88 (which was at the end considered unmaintainable and unsalvageable
by the people at Sun who did the maintaince on it).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[busybox-dev] AST versions of fold, mktemp,pathchk,& tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

Garrett D'Amore wrote:
> Roland Mainz wrote:
> > Garrett D'Amore wrote:
> >> Alan Coopersmith wrote:
> >>> Garrett D'Amore wrote:
[snip]
> >> There's yet another concern, which is that I've found that man 
> >> and command --man do not generate the same document.  So we know
> >> introduce a problem were documentation delivered on the system can be
> >> inconsistent.
> >
> > Erm... no. As said in my other email the "--man" output is basically a
> > short/terse format and more or less exactly what the "getopts" parser
> > sees (it may even be usefull if main documentation and actual code are
> > out-of-sync (which is currently the case for many commands)).
> 
> No, it isn't.  (Well, you might have "extra" text in the getopts parser,
> but for an example look at the output from sum --help.  Its quite a rich
> manual page, far beyond the normal getopts kind of messaging.)

Right... but it cannot be easily stripped either because there are
callbacks tied into the getopts code. Most of this specific getopts
stuff (which is an exception (AFAIK the only one beyond "ulimit")) is
actually _dynamically_ generated based on the capabilities of the
underlying crypto libraries. IMO you hit the worst example of all... ;-(

[snip]
> > And what should we do then ? The only thing we can do is to remove it
> > from the case materials - removing it from the code can only be done
> > globally (e.g. libast) and that really will break existing&&ARC'ed
> > parts.
> >
> 
> #ifdef SOLARIS ?  Seriously, if you want Solaris to adopt these commands
> in favor of our current native implementations, then there has to be
> some willingness to address architectural issues found on, or even
> specific to, Solaris.

Right... but I also remind you that we try to _avoid_ the history of
Solaris's ksh88 which was gradually fork'ed over time with the "best
intentions" which later rendered the code incompatible to other ksh88
versions on other Unices and pretty much unmaintainable, too. The
ksh93-integration project was then created with the _strong_ intention
to _prevent_ this kind of problems to happen _ever_ again, which
resulted in several major and unbreakable rules for this project which
includes:
- WE DO NOT FORK THE CODE
- WE DO NOT BREAK THE KSH93 TEST SUITE
- THE KSH93 TEST SUITE IS COMPLETELY OFF-LIMITS FOR CHANGES
(I am using uppercase here to make it very clear that this project was
founded to prevent the history from repeating. Sun unfortunately has a
long tradition of shooting itself into it's many feet and when this
project was founded we really hoped to prevent these mistakes at least
for this project).

That's why I am very unhappy about the suggestion to "litter" the code
with lots of (IMHO) unneccesary |#ifdef SOLARIS|. It would mean we fork
the code, create a huge additional maintaince and testing burden (and
testing is already a _pain_, we're now working on the ksh93-integration
update2 for more than six month and more than four of them are spend in
testing) and make the code (slightly) incompatible. It's only "slightly"
for _now_ but I know poeple love precedents and once this is established
there is no way to stop the flood anymore.

[snip]
> > I don't understand the connection here:
> > 1. "i18n" is "internationalisation", e.g. the support for non-ASCII
> > characters&co. and this is fully covered by the new commands (and I am
> > _very_ picky about this detail).
> 
> The point is that it must be possible for the commands to be localized.
> While there is no *technical* difference imposed by the length of the
> string, the string itself must be localizable.  That means you can't
> elide handling of this when you localize the rest of the command, I think.

Erm... why ? l10n catalogs are always allowed to be sparse.

> > 2. "l10n" means "localistion" and mainly rotates around error
> > strings/messages/etc. being provided in non-english languages.
> 
> Yes.
> 
> Now let me break down the architectural problems I have with --man (and
> also with --nroff and --troff), as they pertain to Solaris:
> 
> 1) The commands increase the size of the text segment.   Not only does
> it add new parsing requirements (you have to at least have enough code
> to handle --man, for example), but you also have the text of the man
> pages themselves.   While you might like to maintain the fiction that
> this comes for free, it *is* a fiction.  Run sum --man or some of the
> other commands and you'll see content that was not automatically generated.
> 
> 2) We also have traditional manual pages.  On a

[busybox-dev] AST versions of fold, mktemp, pathchk,& tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

John Sonnenschein wrote:
> On 24-Jul-09, at 11:01 PM, Garrett D'Amore wrote:
> > Roland Mainz wrote:
> >> Garrett D'Amore wrote:
> >>> Alan Coopersmith wrote:
[snip]
> > I will say just one more thing.   Where it not for the --man, --
> > nroff, and --html options, I think I would unhesitatingly give this
> > case a +1.   I think the rest of the case has a great deal of
> > technical merit, and I actually would like to see the changes
> > integrated -- just without manual pages integrated into the binaries.
> 
> I've got a question about this...
> 
> Whose responsibility is it to update the man pages and --man command
> then? The people whose jobs it is to update man pages, or the people
> whose jobs it is to update the command line utility?
> 
> Basically if a new flag is added in the future for some reason, how
> will one synchronize the man pages?

There are two independent methods ([1] is mandatory, [2] acts as
"safeguard"):
1. If we putback into the OS/Net gate and had an ARC case with manual
page changes associated with it the putback rules require that we notify
the documentation folks at Sun to get the matching manpages updated
(e.g. we need to document a milestone/build id when the mapage changes
will be completed (which is usually required not to further away than
two builds from the code putback build id))
2. If we add new options we edit the internal string passed to
|libast::getopts()| from which the output for
--man/--help/--html/--nroff/--version/etc. is generated from. Once the
new code runs we have script in place which compares the output against
the last output stored in a Subversion tree and warns us if there is a
difference in the options part.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[busybox-dev] AST versions of fold, mktemp, pathchk,& tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

Garrett D'Amore wrote:
> Alan Coopersmith wrote:
> > Garrett D'Amore wrote:
> >> Personally, I think --man, --html and --nroff and such is a dangerous
> >> precedent to set.  I'd rather not have them, and instead rely on the
> >> "man" command to provide this functionality.
> >
> > Isn't it a bit late to raise such a concern, since the precedent was set
> > in the long list of previous cases that used AST/ksh93 implementations?
> 
> It might be.  I certainly should have raised the issue back then.  I'm
> still not happy about this.

Why ?

> There's yet another concern, which is that I've found that man 
> and command --man do not generate the same document.  So we know
> introduce a problem were documentation delivered on the system can be
> inconsistent.

Erm... no. As said in my other email the "--man" output is basically a
short/terse format and more or less exactly what the "getopts" parser
sees (it may even be usefull if main documentation and actual code are
out-of-sync (which is currently the case for many commands)).

> I feel really strongly that this was a bad idea.

IMO it was a nice idea - see my other email where this feature
originated from.

> Strongly enough that
> I'm contemplating derailing the case.

And what should we do then ? The only thing we can do is to remove it
from the case materials - removing it from the code can only be done
globally (e.g. libast) and that really will break existing&&ARC'ed
parts.

[snip]
> > No matter what you multiply $0 by, it's still $0.   (We don't localize man
> > pages in Solaris.   A subset of man pages used to be translated to Japanese,
> > but I believe even that is no longer done.)
> 
> Really?  That comes as a surprise.  But we *do* localize commands.

Actually the situation is AFAIK currently that there is not really much
funding for this left and the basic system commands are very low
priority. That's why I am currently working on getting a rag-tag team
set-up to get l10n catalogs for the AST commands (e.g. covering ksh93
itself and all commands which go through the busybox-like "alias"
wrapper (including those commands covered by this ARC case)) integrated
(first covering Japanese, Chinese, French and later German, Spanish,
Russian, Urkainian locales).

> So
> does putting --man content in the command suddenly mean that in order to
> be I18N compliant they have to be localized?  That would certainly add
> to the cost.

I don't understand the connection here:
1. "i18n" is "internationalisation", e.g. the support for non-ASCII
characters&co. and this is fully covered by the new commands (and I am
_very_ picky about this detail).
2. "l10n" means "localistion" and mainly rotates around error
strings/messages/etc. being provided in non-english languages. 



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[busybox-dev] AST versions of fold, mktemp,pathchk,& tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

Roland Mainz wrote:
> Garrett D'Amore wrote:
> > Alan Coopersmith wrote:
[snip]
> Yes, at least we cover the following goals:
> - Familarity: GNU+BSD command line options (which increases
> interoperabilty, not only across GNU but BSD and MacOSX, too)
> - Performance:
>   1. The AST implementions are usually a lot faster than the current
> commands (we seen with the replacements for /usr/bin/cut, /usr/bin/paste
> etc. which are sometimes eight, ten or twelve times faster (partially
> because the |libc::stdio| implementation is _extremely_ slow))
>   2. Performance boost for OpenSolaris/Indiana since the tool is a
> builtin shell command for /usr/bin/sh, /sbin/sh, /usr/bin/ksh and
> /usr/bin/ksh93
> - 64bit clean codebase: Right now OS/Net is _not_ being 64bit clean (the
> tools we are touching in ksh93-integration update2 and this case are in
> particular the worst offenders, followed by the CTF tools and some minor
> other areas). This situation causes serious problems (e.g. accounting
> for 1/5 of the engineering time required to port Solaris) for ports to
> other hardware - for example the Solaris/SyetemZ port was forced to
> implement a 32bit emulation layer (!!) on pure 64bit hardware because
> there was no other easy way to get Solaris ported (and IMO this
> situation _sucks_ (<-- sorry... but I really don't like it that the code
> was never cleaned-up)).
[snip]

I forgot two items:
- Make all tools 100% largefile-aware. Right now some tools are still
not largefile-aware, related bugs include:
1. CR #4808051 ("pathchk of file larger than 2GB will fail") [1]
2. CR #5082249 ('*fold* is not large file aware "value too large for
defined data type"')
- Implement  IEEE Std 1003.1-2008 features (e.g. pathchk's "-P" option)

[1]=I'm wondering how Solaris 10 passed the
Unix&co. certifications with this bug... grrr...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[busybox-dev] AST versions of fold, mktemp, pathchk,& tty [PSARC/2009/414 FastTrack timeout 07/31/2009]

[Roland Mainz]

Garrett D'Amore wrote:
> Alan Coopersmith wrote:
> > I'm sponsoring this fast-track request on behalf of the
> > ksh93-integration and busybox projects.  The timeout is
> > set for Friday, July 31, 2009.
> >
> >   -Alan Coopersmith-   alan.coopersmith at sun.com
> >Sun Microsystems, Inc. - X Window System Engineering
> >
> > Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
> > This information is Copyright 2009 Sun Microsystems
> > 1. Introduction
> > 1.1. Project/Component Working Name:
> >AST versions of fold, mktemp, pathchk, & tty
> > 1.2. Name of Document Author/Supplier:
> >Author:  Roland Mainz
> > 1.3  Date of This Document:
> >   24 July, 2009
> > 4. Technical Description
[snip]
> My main concern here is the integration of manual page functionality
> into the commands themselves.  I see both benefits and costs.  The
> benefit is that the documentation is more likely to match the actual
> command.  But part of the cost is a much higher cost to perform
> localization for these,

Erm... see below... normal manpages won't be discontinued - see below...
... and there is (in theory) no restriction to localise builtin manpages
- the matching string would simply appear in the l10n catalog file for
libcmd (this is however not planned (yet)).

> and (depending on implementation) a potentially
> larger minimum size of the binaries.  (I'm assuming for the moment that
> the documentation is stored in the binary, and the command is doing more
> than just executing some pipeline to access the manual content from
> /usr/share/man or whatever.)
> 
> Personally, I think --man, --html and --nroff and such is a dangerous
> precedent to set.

... which already exists since the ksh93-integration project was
started...

> I'd rather not have them, and instead rely on the
> "man" command to provide this functionality.

Erm... we never proposed to discontinue to use manpages. The builtin
support for "--man"&co. is actually only a nice side-effect of the AST
getopts function - see below...

> (Also with --html and
> --nroff an --man, how is the content stored -- does the command do
> format conversions on demand?...  It seems like this functionality also
> might add to the total code size, although I guess this functionality is
> already stored in the AST libraries.  Hopefully this is *not* storing 3
> separate copies of the documentation in the binaries, at least!

Erm... some clarifications:
1. Support for "--man"/"-nroff"/"--html" is only a side-effect of the
extensions of the |libast::getopts()| function. This extension
(available via the ksh93 "getopts" interface for scripts and
|libast::getopts()| for binaries) is used to describe the short&&long
command/utilty options and allows to tag them with some messages, too.
The libast code then converts this into a mannual page on demand.
2. Please don't worry... we don're store 666 copies of the manual page
somewhere in the code - instead the _compact_ getopts string is
dynamically converted at runtime to the requested output format.
3. The actual "extra" space being used is _tiny_ - just looking at the
(completely different) implementations of /usr/bin/fold vs. the "fold.o"
object from one of my development trees:
-- snip --
$ ls -l ./build_sparc_64bit/arch/sol11.sun4/src/lib/libcmd/fold.o
/usr/bin/fold
-rw-r--r--   1 gisburn  gisburn14136 Jun 28 04:27
./build_sparc_64bit/src/lib/libcmd/fold.o
-r-xr-xr-x   1 root bin12752 Sep 13  2006 /usr/bin/fold
-- snip --
This is on SPARC ("fold.o" is SPARCv9, "/usr/bin/fold" is SPARCv8) ...
and the size difference is AFAIK not a problem since UFS uses 8k pages
and both binaries therefore fit happily in two 8k pages and on x86 both
need three 4k pages.
4. The bulitin manpages are intentionally only a short/terse version of
the normal manpages (for example $ ksh93 --man # only gives normal shell
usage, not the full syntax of the ksh93 scripting language and all the
details. It's intended as quick reference and not to replace the full
manual page (for smaller projects like "bldenv" or "webrev" it can
replace the full manual page but that's more the exception from the
standard)).
5. For the long-term (not yet, not now, not this case (please)) it may
be interesting to generate the builtin string for AST "getopts" and the
normal manual pages from a DocBook/XML master file (with some XML
ifdef/else/endif to cut-out the non-mandatory parts). That would unify
both systems but requires that I'll finish the "shxml" work (e.g.
xmlreader/xmlwriter support for the shell) or write a dedicated
docbook2astgetopt

Update GNU coreutils to version 7.4 [PSARC/2009/355 FastTracktimeout 06/22/2009]

[Roland Mainz]

Glenn Fowler wrote:
> On Tue, 16 Jun 2009 20:24:39 +0200 Roland Mainz wrote:
> > Casper.Dik at sun.com wrote:
> > > >On Tue, Jun 16, 2009 at 09:55:30AM +0200, Casper.Dik at Sun.COM wrote:
> > > >>
> > > >> >/usr/bin already has an mktemp.  How does it differ from the GNU
> > > >> >version?  Ah, there's an option conflict, sadly (-u is "--dry-run" in
> > > >> >GNU, but "unsafe operation" in Solaris).
> > > >>
> > > >> Solaris' mktemp takes after the OpenBSD mktemp.
> > > >
> > > >What is the point of the Solaris mktemp(1) -u option though?  Why would
> > > >anyone choose "unsafe operation"?
> > >
> > > It came from OpenBSD.
> > >
> > > I can understand the reason, though, you can't always want to create a
> > > file, you may want an unique value:
> > >
> > > unique=$(mktemp -u XX)
> 
> > Ok... that makes sense... :-)
> 
> > Erm... what is the stabilty state of "-u" - wasn't it "Commited
> > Obsolete" ? If that's the case it may be nice to get rid of the
> > "Obsolete" and poke the GNU coreutils maintainers to rename the "dryrun"
> > option to something else (I'll poke AST+BSD upstream to do the same for
> > their "mktemp" command (e.g. clarify documentation)).
> 
> how would you clarify this:
> 
>   -u, --undo|unsafe|unwise
> Remove the file/directory before exiting. Who would want 
> to do that.
> 
>   version mktemp (AT&T Research) 2009-05-01

See the other email with the patch I send a few seconds to you
(ahhgllrr... I forgot to update the version number (new patch follows)).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Update GNU coreutils to version 7.4 [PSARC/2009/355 FastTracktimeout 06/22/2009]

[Roland Mainz]

Casper.Dik at sun.com wrote:
> >On Tue, Jun 16, 2009 at 09:55:30AM +0200, Casper.Dik at Sun.COM wrote:
> >>
> >> >/usr/bin already has an mktemp.  How does it differ from the GNU
> >> >version?  Ah, there's an option conflict, sadly (-u is "--dry-run" in
> >> >GNU, but "unsafe operation" in Solaris).
> >>
> >> Solaris' mktemp takes after the OpenBSD mktemp.
> >
> >What is the point of the Solaris mktemp(1) -u option though?  Why would
> >anyone choose "unsafe operation"?
> 
> It came from OpenBSD.
> 
> I can understand the reason, though, you can't always want to create a
> file, you may want an unique value:
> 
> unique=$(mktemp -u XX)

Ok... that makes sense... :-)

Erm... what is the stabilty state of "-u" - wasn't it "Commited
Obsolete" ? If that's the case it may be nice to get rid of the
"Obsolete" and poke the GNU coreutils maintainers to rename the "dryrun"
option to something else (I'll poke AST+BSD upstream to do the same for
their "mktemp" command (e.g. clarify documentation)).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Update GNU coreutils to version 7.4 [PSARC/2009/355 FastTracktimeout 06/22/2009]

[Roland Mainz]

Nicolas Williams wrote:
> On Mon, Jun 15, 2009 at 01:40:21PM -0700, Rich Burridge wrote:
> > Between 6.7 and 7.4, the following new commands were introduced:

Erm... where can I find the full ARC case ? Somehow I never got the
email for this... ;-(

> > /usr/gnu/bin/
> > chcon
> > mktemp
> > runcon
> > timeout
> > truncated
> >Location Uncommitted
> >Invocation   Uncommitted
> >Human-readable outputNot an Interface
> 
> chcon and runcon are specific to SELinux.  What would they do on
> Solaris?
> 
> /usr/bin already has an mktemp.  How does it differ from the GNU
> version?  Ah, there's an option conflict, sadly (-u is "--dry-run" in
> GNU, but "unsafe operation" in Solaris).

BTW: The next ksh93-integration update will cover /usr/bin/mktemp and
add the non-conflicting BSD+GNU options.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[i18n-discuss] EOF of @euro locales [PSARC/2009/342 FastTracktimeout 06/11/2009]

[Roland Mainz]

Nicolas Williams wrote:
> On Thu, Jun 04, 2009 at 01:36:14PM -0700, Apostolos Syropoulos wrote:
> > Indeed, I agree. But in my opinion there is no need for a UK at euro locale.
> > For example, if country X has trade relations with India then should we
> > introduce a locale for this country with support for the Indian currency?
> >
> > In other words, this is a real problem that has to be resolved accordingly.
> 
> It's easiest to resolve by EOFing all non-Unicode locales.  But that's
> not necessarily a practical solution.

IMHO it would be very "unwise". Solaris would simply "disappear" from
markets where non-UTF-8 multibyte characters are either required by law
or still have widespread coporate use (e.g. "zh_CN.GB18030", "ja_JP.PCK"
and some others fall into this category (and Sun has invested a lot to
get these locales implemented&&certified)).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

2009/327 [system_noshell]

[Roland Mainz]

Casper.Dik at sun.com wrote:
> >I think it would be helpful to see a few worked examples that show how
> >system_noshell() and its variants make things simpler than using
> >posix_spawn().
> >
> >Date: Fri, 29 May 2009 10:41:30 -0700
> >From: Sumanth Naropanth 
> >Subject: Re: system_noshell [PSARC/2009/327 FastTrack timeout
> >   06/05/2009]
> >
> >...
> >
> >system_noshell("/bin/rm /tmp/tmpfile") is simpler than
> >posix_spawn(pid, "rm", NULL, NULL, argv, NULL) which also includes
> >populating an argv vector.
> >
> >The example given above is a start, but I'd like to see something more
> >realistic.
> 
> If we're going to *parse* commands using spaces or what not, I vote
> no, right now!

I agree with Casper... we had that kind of proposal with |exec_system()|
a while ago (AFAIK in security-discuss at opensolaris.org) and that
proposal was "eaten&&trampled alive" (there should be always an option
to pass _any_ content (except '\0') via arguments and environment
variables and using a whitespace character for argument splitting
violates that).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

system_noshell [PSARC/2009/327 FastTrack timeout 06/05/2009]

[Roland Mainz]

John.Zolnowsky at sun.com wrote:
> 
> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
> This information is Copyright 2009 Sun Microsystems
> 1. Introduction
> 1.1. Project/Component Working Name:
>  system_noshell
> 1.2. Name of Document Author/Supplier:
>  Author:  Sumanth Naropanth
> 1.3  Date of This Document:
> 29 May, 2009
> 4. Technical Description
> I'm sponsoring this fasttrack on behalf of Sumanth Naropanth.
> This case will time out June 5, 2009.
> 
> SUMMARY:
> Implementation of new C library functions -- system_noshell(3C)
> (and variants) to work as secure alternatives to the system(3C)
> function, to proactively prevent security vulnerabilities resulting
> from unintended invocation of user shell.
[snip]
> PROPOSED SOLUTION:
> The system_noshell(3C) function call will be implemented to provide 
> the
> same ease of use as the system(3C) function, via a single
> (const char *) argument. Variants of this function will be
> 
> system_noshell_x(3C)
> and
> system_noshell_xv(3C)
> 
> which will allow for special arguments to be passed while executing a
> file.
> 
> Prototypes:
> ---
> 
> system_noshell(const char *abs_path);

1. Erm... just to be sure: "abs_path" may not contain any arguments,
right (remeber the proposal for |exec_system(3C)| which was torn into
pieces by wild animals^H^H^H^H^H^H^Hengineers because it picked a random
character and used it for argument splitting (which is bad for cases
when an argument should actually contain this character)) ?

2. It may be nice to add a |flags| argument to this call, too (to allow
that it can be extended on demand without requiring another symbol to be
added to libc).

> system_noshell_x(const char *abs_path, uint_t flags, const char *arg0,
> ... /* const char *argn, (char *)0 */);
> 
> system_noshell_xv(const char *abs_path, uint_t flags,
> char *const argv[]);

In general: What happens with open file descriptors when they do not
have the |FD_CLOEXEC| flag set ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Delivering ksh93 through SFWNV instead of OS/Net ? / was: Re: Update Perl to version 5.10.x [PSARC/2009/315 FastTrack timeout06/02/2009]

[Roland Mainz]

James Carlson wrote:
> I. Szczesniak writes:
> > On 5/26/09, James Carlson  wrote:
[snip]
> With either consolidation, the Perl code that's present is essentially
> just what comes from the open source, as it is with _many_ projects
> that happen to deliver through ON -- including, incidentally, ksh93,
> where we're not doing extensive code or design reviews up to the
> normal ON standards, because they'd just be infeasible.
> 
> The big difference between the two is in makefiles: you use your own
> in SFW, but you generally must use ON's in ON.  That difference is why
> I argued (more than once) for delivering ksh93 through SFW.

Erm... why is the ksh93-integration project dragged into this discussion
and why is this particular topic dragged out of it's grave (again
(anyone remeber the "Cadaver Synod" [1] ? =:-) )) ?

ksh93 was _intentionally_ integrated into OS/Net because...
- ... ksh88 is in OS/Net and ksh93 ws "grandfather'ed" into OS/Net as
replacement for ksh88 (with the explicit comment that this is a special
decision and should not be used as precendent by future projects)
- ... ksh88 and ksh93 have an "intimate" relationship with libc (e.g. to
implement |libc::wordexp()|)
- ... it was planned to gradually "upgrade" some system commands with
those in libcmd (which was implemented with ksh93-integration update1
and is now being continued with update2, update3 and update4)
- ... Bourne shell is in OS/Net and IMO the "default system shell"
(which is likely going to be ksh93 (see OpenSolaris/Indiana)) should be
part of OS/Net and not in a different consolidation
- ... we intended to use SFIO for perl (which was in OS/Net during that
time)
- ... we wanted libast::stdio for OS/Net applications (as
consolidation-private interface) for I/O-heavy applications (since
libast's stdio is significantly faster than libc's current stdio
implementation (for Solaris 3.x we may get a "stdio replacement" project
to replace the old stdio implementation with the libast one))
- ... we wanted to replace the existing (closed-source) pattern matching
functions (e.g. |regex()|, |fnmatch()|) with the code from libast (which
is opensource, faster and has very usefull extensions (which we could
propose as POSIX standard if we get two implementations))
- <... AFAIK there are more reasons but I am too tired to dig them all
out today (again) ...>

> It'd be
> simpler and lower overhead, and there's no necessary entanglement with
> the rest of the core OS.

I disagree that it would be "simpler and lower overhead". Yes, I know we
could've "dumped" the ast-ksh package into SFWNV and build ksh93 as an
monolothic all-in-one binary. However we intended to make "maximum use"
of the technology provided by ksh93 and AST and therefore we decided to
deliver ksh93 into OS/Net (after having three conference calls about his
topic (that's why I am not happy that this decision is challanged
again)), including shipping ksh93 as set of shared libraries that other
applications can use it.
Without doing that we would've _never_ done all the fine-tuning, testing
(including integrating ksh93's own test suite into OS/Net),
benchmarking, adoption for OS/Net's needs+rules (which helped us finding
lots of bugs in ksh93+libast) etc. and we would've never been able to do
all the work with updating the POSIX commands in OS/Net (which will now
grow into it's own "POSIX commands community" (which should act as
umbrella for lots of smaller projects to improve Solaris's (POSIX)
command line utilities)).

And "yes", it's possible to move ksh93 to SFWNV, but it would require
_lots_ of work to do the move (which I do not have) and we would need
lots of paperwork for interface contracts between OS/Net and SFWNV which
create so much overhead that the project would quickly drown+die from
that (and we could throw the idea for the POSIX commands community into
the bin).

Beyond that SFWNV currently has big build issues which make even a basic
contribution a _PAIN_ for external people with limited resources - for
example incrmental builds do not work (which means you have to clobber
and rebuild everything from scratch over and over again and that just
takes ~28-30h on my laptop). That's why I tried to avoid having any
involvement with SFWNV unless neccesary (well, I now signed-up to do the
"bash4" work to avoid that we end-up with the mess for "bash3" (which
doesn't even pass it's own test suite the way it was integrated into
SFWNV)) or if someone pays me for that pain.

[1]=http://en.wikipedia.org/wiki/Cadaver_Synod



Bye,
Roland

P.S.: Setting Reply-To: to on-discuss at opensolaris.org

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Update Perl to version 5.10.x [PSARC/2009/315 FastTrack timeout06/02/2009]

[Roland Mainz]

Rich Burridge wrote:
> 
>I'm sponsoring the following fast track for John Sonnenschein.
>The requested release binding is Minor. The fast track times out
>on 06/02/2009.
> 
> 
> 
> 1. Introduction
> 
>1.1. Project/Component Working Name:
> 
> PSARC/2009/315 Update Perl to version 5.10.x
[snip]
> Script interface (Uncommitted)
> ---
> 
> There have been no major changes between 5.8.x and 5.10.x
> except
> backwards compatible extensions.
> 
> Incompatible changes are as follows:
> 
> * Packing and UTF-8 strings
> * Byte/character count feature in unpack()
> * The $* and $# variables have been removed
> * substr() lvalues are no longer fixed-length
> * Parsing of -f _
> * :unique
> * Effect of pragmas in eval
> * chdir FOO
> * Handling of .pmc files
> * $^V is now a version object instead of a v-string
> * @- and @+ in patterns
> * $AUTOLOAD can now be tainted
> * Tainting and printf
> * undef and signal handlers
> * strictures and dereferencing in defined()
> * (?p{}) has been removed
> * Pseudo-hashes have been removed
>* Removal of the bytecode compiler and of perlcc
> * Removal of the JPL
> * Recursive inheritance detected earlier
> 
> Full explanation of changes are included in
> perl5100delta.pod.txt
> (in the case materials directory)

One thing I'd like to see is to swtch the "perl" build to use SFIO
(available via libast) for better I/O performance (we need to setup an
interface contract between OS/Net and SFWNV for that). Originally this
was planned for the "next perl update" but that somehow drowned with the
RIF of Tim Sparlin's team... ;-(



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

"tail +r" not equivalent to "tail + | rev -l" ?

[Roland Mainz]

Hi!



[The following is either a PSARC or OS/Net gatekeeper question - I don't
know it exactly and therefore both end-up in the CC: field]
During PIT (=PreIntegration Testing) for ksh93-integration update2
(which includes an opensource replacement for /usr/bin/tail and
/usr/xpg4/bin/tail (note that both are built from the same source) - see
http://arc.opensolaris.org/caselog/PSARC/2009/249/20090420_roland.mainz.2)
we hit a bug with "tail -r/+r" and during fixing it we've hit the
following issue:

SXCE B110 tail(1) manual page defines "tail"'s "-r" option like this:
-- snip --
 -r Reverse. Copies lines  from  the  specified  starting
point in the file in reverse order. The default for r
is to print the entire file in reverse order.
-- snip --

PSARC/2009/249 defines it the same way but with a shorter description:
-- snip --
  -r, --reverse   Output lines in reverse order.
-- snip --

Both AFAIK are thought that option "-r" works like "tail - | rev
-l" ("rev -l" outputs the incoming lines in reverse order). The
/usr/bin/tail+/usr/xpg4/bin/tail commands in SXCE B110 does this
correctly when I use something like "-r", for example:
-- snip --
$ print "1\n2\n3\n4\n5\n6" | /usr/xpg4/bin/tail
-4r 
6
5
4
3
$ print "1\n2\n3\n4\n5\n6" | /usr/xpg4/bin/tail -4 | rev
-l 
6
5
4
3
-- snip --

All other combinations for "-" are identical between "tail
-r" and "tail - | rev -l", too.


However this is _not_ true when I use "tail +r" compared to
"tail + | rev -l". In that case the output differs:
-- snip --
$ print "1\n2\n3\n4\n5\n6" | /usr/xpg4/bin/tail
+4r 
6
5
4
3
2
$ print "1\n2\n3\n4\n5\n6" | /usr/xpg4/bin/tail +4 | rev
-l 
6
5
4
-- snip --

Both outputs differ and I belive this is simply a bug in the current
SXCE implementation.

Question to gatekeepers: How should I handle this issue ? Should we...
a) ... treat this as bug and fix it with the upcoming ksh93-integration
putback
  or
b) ... make the AST/ksh93 builtin behave _exactly_ as the current SXCE
implementation of /usr/bin/tail+/usr/xpg4/bin/tail and fix this bug with
a later putback
  or
c) ... declare that this is not a bug and keep the inconsistent
behaviour of "tail +r" vs. "tail + | rev -l".

AFAIK I would prefer [a] but I have a patch for [b], too. [c] is IMO not
really an option since makes option "r" behave in an inconsistent way.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

EOF of Xprint [PSARC/2009/294 FastTrack timeout 05/14/2009]

[Roland Mainz]

Alan Coopersmith wrote:
> 
> I am sponsoring this fasttrack for the X engineering team and have set
> the timeout to one week from today, Thursday, May 14.
> 
> Since the interfaces being removed have been reviewed and imported in
> cases reviewed both at PSARC & LSARC, both ARC's are cc'ed, though the
> case is submitted to PSARC for review.
> 
> -Alan Coopersmith-   alan.coopersmith at sun.com
>  Sun Microsystems, Inc. - X Window System Engineering
> 
> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
> This information is Copyright 2009 Sun Microsystems
> 1. Introduction
> 1.1. Project/Component Working Name:
>  EOF of Xprint
> 1.2. Name of Document Author/Supplier:
>  Author:  Alan Coopersmith
> 1.3  Date of This Document:
> 07 May, 2009
> 4. Technical Description
> 
> The X.Org community recently released Xorg server version 1.6.   This
> version no longer includes the Xprint variant of the X server to generate
> output in page description formats like PostScript and PCL and to either
> save that to a file or send to the printer via the lp system.   X.Org has
> also deprecated all of the Xprint client libraries and utilities, and will
> not be including them in X11R7.5 or any future releases.
> 
> Xprint was originally created by the X Consortium under contract to the
> Open Group to serve as the printing backend for the CDE 2.1 release.
> Sun integrated Xprint into Solaris with the upgrade to X11R6.4 in
> Solaris 7 11/99, but never integrated most of the CDE or Motif support
> to use Xprint.

Sometimes I still wish to know why this was never done - it created a
lot more trouble than simply doing the update to CDE2.x ... ;-(

> Xprint was never widely adopted outside of CDE and Motif, and since
> the last maintainer of Xprint moved on to other projects (such as
> integrating ksh93 into OpenSolaris) a few years ago, little activity
> has happened on Xprint.

... well, after the STSF project went down there was little hope to get
Xprint back on it's feet... ;-(



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

faccessat(2) - determine accessibility of a file usingfiledescriptors [PSARC/2009/284 Self Review]

[Roland Mainz]

Joerg Schilling wrote:
> Roland Mainz  wrote:
> > Is it possible to make this interface "Uncommited" until the IEEE
> > P1003.1 Draft has been made an official standard (e.g. what would you do
> > if somehow the Draft changes the prototype or functionality in a subtle
> > way - add another symbol or add some more hacks which depend on the
> > |libc::__xpg6| variable) ?
> 
> The function is in the Opengroup draft since a few years and the opengroup
> standard was accepted last year.
> 
> Why do you believe this will be changed?

I just want to be carefull. Creating "commited" interfaces based on a
draft standard already resulted in "interesting" problems for Solaris
(e.g. when the draft changed again before being approved by the standard
bodies) since it wasn't possible to change that commited interface
anymore.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

faccessat(2) - determine accessibility of a file using file descriptors [PSARC/2009/284 Self Review]

[Roland Mainz]

Don Cragun wrote:
> > Casper Dik wrote:
> >>
>  ... ... ...
> >> Recent publications of the Open Group Technical Standards, 
> >> including
> >> [IEEE P1003.1 Draft 5.1, 15 May 2008] has confirmed that this
> >> interface
> >> shall be named 'faccessat' and shall have the following prototype:
> >>
> >> int faccessat(int fd, const char *path, int amode, int
> >> flag);
> > [snip]
> >>
> >> ---
> >> | Interface |  Commitment Level   |
> >> ---
> >> |faccessat()| Committed   |
> >> ---
> >
> >
> > Is it possible to make this interface "Uncommited" until the IEEE
> > P1003.1 Draft has been made an official standard (e.g. what would you do
> > if somehow the Draft changes the prototype or functionality in a subtle
> > way - add another symbol or add some more hacks which depend on the
> > |libc::__xpg6| variable) ?
> >
>
> It is no longer a draft standard.  It was formally approved by all three
> bodies (The Open Group as SUSv4, by IEEE as IEEE Std 1003.1-2008,
> and by ISO as ISO/IEC IS 9945:2008) last year.  The specification for
> this function did not change during balloting.

Ok... can we then remove the word "draft" from the text above, please
(which was the reason for my concerns - it looked like we're going to
get a "Commited" interface based on a standard _draft_ (which already
has backfired several times in the history of Solaris)) ?

> The branding program for the next UNIX brand is not yet in place.

Ok...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Requesting ARC cases 1993/685, 1994/188, 1994/196, 1994/197, 1994/302, 1995/077, 1995/082, 1995/153

[Roland Mainz]

hi!



I'd like to request to open the following ARC cases for publish access:
PSARC 1993/685 PowerPC_ABI
PSARC 1994/188 Solaris Kernel for PowerPC
PSARC 1994/196 PowerPC Header Files
PSARC 1994/197 PowerPC Commands
PSARC 1994/302 PowerPC_Booting
PSARC 1995/077 processor_info(2) binding for PowerPC and future ISAs
PSARC 1995/082 PowerPC sysconfig parameters
PSARC 1995/153 PPC installboot

We need this materials as template for a set of ARC casess to specify
support for a new architecture/hardware platform...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Long options and |libast::optget()| ... / was: Re: 2009/228 ls enhancements

[Roland Mainz]

James Carlson wrote:
> Glenn Fowler writes:
[snip]
> > a bonus is that ksh93 uses optget() to implement getopts
> > so shell scripts are not second class citizens w.r.t. long options
> > and since the options desc ription is one 0-terminated string
> > instead of a C-data-struct the description can be moved
> > freely between script<=>C-source implementations
> 
> Since this is going in ON, I think the implementor is free to choose
> among the getopt_* family and the ksh93 optget() functions as he sees
> fit. 

Erm... note that this is not ksh93 Glenn was thinking about - it's
|libast::optget()| which has this additional functionality...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

faccessat(2) - determine accessibility of a file using filedescriptors [PSARC/2009/284 Self Review]

[Roland Mainz]

Casper Dik wrote:
> 
> Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
> This information is Copyright 2009 Sun Microsystems
> 1. Introduction
> 1.1. Project/Component Working Name:
>  faccessat(2) - determine accessibility of a file using file 
> descriptors
> 1.2. Name of Document Author/Supplier:
>  Author:  Sumanth Naropanth
> 1.3  Date of This Document:
> 06 May, 2009
> 4. Technical Description
> faccessat(2) - determine accessibility of a file using file descriptors
> 
> AUTHOR:
> Sumanth Naropanth
> 
> SUMMARY:
> 
> Promotion of the private __accessat() function in OpenSolaris to 
> public
> visibility as faccessat(2)
> 
> PROBLEM:
> 
> RFE 6478917 introduced a new extended attribute system interface:
> 
> int __accessat(int fd, const char *path, int amode);
> 
> This function is equivalent to the access(2) system call, except in
> the case where the "path" variable specifies a relative path. In this
> case the file whose accessibility is to be determined shall be located
> relative to the directory associated with the file descriptor "fd"
> instead of the current working directory.
> 
> If "__accessat()" is passed the special value AT_FDCWD in the "fd"
> parameter, the current working directory is used and the behavior
> shall be identical to a call to access().
> 
> This function was kept private at the time of writing since there
> had been discussions among the standards body on the name for this
> function.
> 
> Recent publications of the Open Group Technical Standards, including
> [IEEE P1003.1 Draft 5.1, 15 May 2008] has confirmed that this 
> interface
> shall be named 'faccessat' and shall have the following prototype:
> 
> int faccessat(int fd, const char *path, int amode, int flag);
[snip]
> 
> ---
> | Interface |  Commitment Level   |
> ---
> |faccessat()| Committed   |
> ---


Is it possible to make this interface "Uncommited" until the IEEE
P1003.1 Draft has been made an official standard (e.g. what would you do
if somehow the Draft changes the prototype or functionality in a subtle
way - add another symbol or add some more hacks which depend on the
|libc::__xpg6| variable) ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC 2009/228 ls enhancements (draft opinion)

[Roland Mainz]

Garrett D'Amore wrote:
> Roland Mainz wrote:
> > Garrett D'Amore wrote:
> >> I've posted a draft opinion "filename opinion-draft.txt" (and also .ms)
> >> in the case directory for review.  Its attached to this message as
> >> well.  Thanks.
> >>
> > [snip]
> >
> >> 3.  Interfaces
> >>
> >> The project exports the following interfaces.
> >>
> >> ___
> >> | Interfaces Exported |
> >> |_||__|
> >> |Interface|  Classification|  Comments|
> >> |_||__|
> >> |/usr/bin/ls  |  Committed |  |
> >> |/usr/xpg4/bin/ls |  Committed |  |
> >> |/usr/xpg6/bin/ls |  Committed |  |
> >> |_||__|
> >
> >
> > I have a small request:
> > For the commands replaced by the AT&T AST commands (currently done via
> > the ksh93-integration project) we _carefully_ investigated which
> > options+features (including long options) are available by which
> > implementation (e.g. we investigated Solaris/SystemV, GNU, BSD and AST
> > implementations).
> > Technically we only made options "Commited" (or added them to the AST
> > codebase) which are implemented identically by two or more
> > implementations and must exist for >= two years and set-up the rule that
> > options+features which are only available in one implementation or exist
> > since less than two years must be "Uncommited" (to avoid that the
> > implementation somehow changes and therefore breaks a "Commited"
> > interface).
> >
> > IMO it may be nice to 1) apply the same set of rules to /usr/bin/ls,
> > /usr/xpg4/bin/ls and /usr/bin/xpg6/bin/ls and 2) make options which are
> > somehow "disputed" in this case either "Uncommited" or "Volatile" (I can
> > help with that on demand since we already know the drill from
> > ksh93-integration update1+update2+ammendents+etc.).
> 
> IMO, there is little value in a familiarity case where the options are
> not themselves relatively stable.

Erm... my concern is that "Commited" means "_very_ stable" and should be
used very carefully. Changing "Commited" interfaces is AFAIK non-trivial
and that's why I was suggesting to use "Uncommitted" for now until we
have at least the "proof" that more than one implementation provides the
same options with the same behaviour for at least two years.

> Anyway, the case was approved with Committed.  Unless the project team
> wants to ask ARC to demote them, they stand as is at Committed.

Ok... ;-/



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC 2009/228 ls enhancements (draft opinion)

[Roland Mainz]

Garrett D'Amore wrote:
> I've posted a draft opinion "filename opinion-draft.txt" (and also .ms)
> in the case directory for review.  Its attached to this message as
> well.  Thanks.
[snip]
> 3.  Interfaces
> 
> The project exports the following interfaces.
> 
> ___
> | Interfaces Exported |
> |_||__|
> |Interface|  Classification|  Comments|
> |_||__|
> |/usr/bin/ls  |  Committed |  |
> |/usr/xpg4/bin/ls |  Committed |  |
> |/usr/xpg6/bin/ls |  Committed |  |
> |_||__|


I have a small request:
For the commands replaced by the AT&T AST commands (currently done via
the ksh93-integration project) we _carefully_ investigated which
options+features (including long options) are available by which
implementation (e.g. we investigated Solaris/SystemV, GNU, BSD and AST
implementations).
Technically we only made options "Commited" (or added them to the AST
codebase) which are implemented identically by two or more
implementations and must exist for >= two years and set-up the rule that
options+features which are only available in one implementation or exist
since less than two years must be "Uncommited" (to avoid that the
implementation somehow changes and therefore breaks a "Commited"
interface).

IMO it may be nice to 1) apply the same set of rules to /usr/bin/ls,
/usr/xpg4/bin/ls and /usr/bin/xpg6/bin/ls and 2) make options which are
somehow "disputed" in this case either "Uncommited" or "Volatile" (I can
help with that on demand since we already know the drill from
ksh93-integration update1+update2+ammendents+etc.).



Bye,
Roland


-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] more ksh93 command conversions[PSARC/2009/249 FastTrack timeout04/27/2009]

[Roland Mainz]

James Carlson wrote:
> Garrett D'Amore - sun microsystems writes:
> > I'm sponsoring this fast-track request on behalf of Roland Mainz and the
> > ksh93-integration project.  It consists of more commands being converted
> > to use internal ksh93 builtins.
> 
> Not an ARC issue, but please do make sure that careful tests are run
> with nightly build, BFU,

Actually we tested BFU for the last big putback lots of times... the
part which we missed is to BFU and then BFU the resulting system _again_
(something which sounds locical after you know it... ;-/ ... BTW: Can
someone add this to the OS/Net documenation, please ?).

> upgrade,

Which upgrade ? Try to upgrade a BFU'ed SXCE system ?

> and PIT,

How can I do that (e.g. I need the name of a manager to contact) ? On my
todo list is the item to contact each person who reported isues after
the original ksh93-integration update1 putback (this includes one person
from PIT) and let them test the binary tarballs at
http://www.opensolaris.org/os/project/ksh93-integration/downloads/2009-04-20/

> so that we can avoid a
> repeat of the last /usr/bin conversion project, which resulted in
> several distinct failures in the formerly stable "/usr/bin/sleep."

I know... I don't want to repeat that experience... but the issue was
half "unforseen usage of commands" which can never be completely
ruled-out and stupidity (from my side). However this time we get much
greater testing coverage and I hope it'll work more smoothly than the
last one (on the other side I can't rule-out that we dig new holes to
jump-in or even discover a competely new species of hole... that's why
both ARC cases have the option to defer parts to later putbacks if we
run into trouble late during the putback process).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] more ksh93 command conversions[PSARC/2009/249 FastTrack, timeout 04/27/2009]

[Roland Mainz]

Nicolas Williams wrote:
> On Wed, Apr 22, 2009 at 11:16:09AM -0700, Don Cragun wrote:
> > The standard does not currently specify a way to count (multi-byte)
> > characters even though this means tail output may start or end in the
> > middle of a multi-byte character when using the -c option.
> 
> How... painful.  Of course, for fixed width encodings
> and UTF-8/16 it

AFAIK Solaris doesn't have an UTF-16 based locale and AFAIK UTF-16 can't
be supported by the POSIX multibyte API (at least I never saw and and
can't imagine how it should work) ...

> should be possible to automatically adjust the -c argument value so it
> starts at the start of a character, but that would require another
> argument.

Following the precedent of "wc" we could use "-C" (uppercase 'C') for
this purpose...
... but I am not sure whether it is possible for all encodings (e.g.
Shift-JIS, GBK, EUC etc.) to properly detect the start of a multibyte
character (anyone remeber ISO-2022 ? =:-) ).

> Fortunately I bet tail -c ... is fairly uncommon.

There are several consumes in Solaris which use it.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

more ksh93 command conversions [PSARC/2009/249 FastTrack, timeout 04/27/2009]

[Roland Mainz]

Don Cragun wrote:
> From: "Garrett D'Amore" 
> > I. Szczesniak wrote:
>   ... ... ...
> >> Could you elaborate the difference between Committed and Committed 
> >> Obsolete?
> >>
> >
> > Committed Obsolete means we can't remove it, but you're still
> > discouraged from using it in new code.  That's the case with -b -- it is
> > used to indicate a count in blocks (512 bytes), but the option was
> > removed from the POSIX standard over confusion about what was meant by a
> > "block" (is it a block of bytes, or a block of characters?)
> >
>   ... ... ...
> >
> > -- Garrett
> 
> Garrett,
> I think you misread the tail rationale in the POSIX standard.
> SVID3 had a -b option that counted 512-byte blocks; BSD had a -b option
> that counted 1024-byte blocks.  The -b option wasn't removed from POSIX;
> it was never added in the first place due to this difference in
> behavior.

Erm... that was my fault... I thought that "-b" was defined by POSIX
long long ago. Sorry about that.

> The SVID3 and XPG3 -c options counted in bytes, but the documentation
> said it counted characters (I don't remember what the BSD tail man page
> said).  The POSIX committee briefly considered changing the -c option to
> be -b when the description of -c changed from "characters" to "bytes",
> but left it as -c because historical implementations had the -b option
> counting blocks even though it wasn't going to be specified by the
> standard.
> 
> The standard does not currently specify a way to count (multi-byte)
> characters even though this means tail output may start or end in the
> middle of a multi-byte character when using the -c option.

Is this even possible to specify characters in this case ? AFAIK the
multibyte API doesn't have a way to seek into a random position in a
file and then find the start of the next multibyte character (it works
for UTF-8 but I am not sure for older encoding systems).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] more ksh93 command conversions[PSARC/2009/249 FastTrack timeout 04/27/2009]

[Roland Mainz]

"I. Szczesniak" wrote:
> As the case is specified it gets my +1 but I still have some minor questions:
> 
> On 4/21/09, Garrett D'Amore - sun microsystems
>  wrote:
> >
> >  I'm sponsoring this fast-track request on behalf of Roland Mainz and the
> >  ksh93-integration project.  It consists of more commands being converted
> >  to use internal ksh93 builtins.
> >
> >  Template Version: @(#)sac_nextcase 1.68 02/23/09 SMI
> >  This information is Copyright 2009 Sun Microsystems
> >  1. Introduction
> > 1.1. Project/Component Working Name:
> >  more ksh93 command conversions
> > 1.2. Name of Document Author/Supplier:
> >  Author:  Roland Mainz
> > 1.3  Date of This Document:
> > 20 April, 2009
> >  4. Technical Description
> >
> >  Please note that this is an *open* case.
> >
> >  The release binding is the same as with the previous ksh93 project: a
> >  patch/micro release of Solaris delivering through OS/Net
> >  Stability levels are as described below.
> >
> >  Additional materials (man pages and diffs) can be found in the
> >  'materials/' subdirectory.
> >
> >  This project is an amendment to the Korn Shell 93 Integration project
> >  (PSARC/2006/550, PSARC/2006/587, PSARC/2007/035, PSARC/2008/094,
> >  PSARC/2008/344, PSARC/2009/063 and PSARC/2009/248) and depends on
> >  PSARC/2009/248, specifying the following additional interfaces:
> >
> >  1) An enhanced version of the "cksum" utility and an identical ksh93
> >   built-in command
> >  2) An enhanced version of the "head" utility and an identical ksh93
> >   built-in command
> >  3) An enhanced version of the "join" utility and an identical ksh93
> >   built-in command
> >  4) An enhanced version of the "logname" utility and an identical ksh93
> >   built-in command
> >  5) An enhanced version of the "mkfifo" utility and an identical ksh93
> >   built-in command
> >  6) An enhanced version of the "tail" utility and an identical ksh93
> >   built-in command
> >  7) An enhanced version of the "tee" utility and an identical ksh93
> >   built-in command
> >
> >
> >  Bug/RFE Number(s):
> >
> >  6793763 RFE: Update /usr/bin/ksh93 to ast-ksh.2009-03-10
> >  6828692 RFE: Update /usr/bin/cksum to AT&T AST "cksum"
> >  6805812 RFE: Update /usr/bin/head to AT&T AST "head"
> >  6805813 RFE: Update /usr/bin/join to AT&T AST "join"
> >  6828644 RFE: Update /usr/bin/logname to AT&T AST "logname"
> >  6805814 RFE: Update /usr/bin/mkfifo to AT&T AST "mkfifo"
> >  6790507 RFE: Update /usr/bin/tail and /usr/xpg4/bin/tail to
> > AT&T AST "tail"
> >  6805819 RFE: Update /usr/bin/tee to AT&T AST "tee"
> >
> >
> >
> >  Interface StabilityDescription
> >  - ----
> >  /usr/bin/cksumCommited cksum command
> >  /usr/bin/head Commited head command
> >  /usr/bin/join Commited join command
> >  /usr/bin/logname  Commited logname command
> >  /usr/bin/mkfifo   Commited mkfifo command
> >  /usr/bin/tail [1]  tail command
> >  /usr/xpg4/bin/tail[1]  XPG4 tail command
> >  /usr/bin/tee  Commited tee command
> >
> >  [1]=Stabiltiy for /usr/bin/tail and /usr/xpg4/bin/tail is "Commited"
> >   except for "Uncommited" options "-q"/"--quiet" which are
> >   GNU-only and option "-b" which is now "Commited Obsolete".
> 
> Could you elaborate the difference between Committed and Committed Obsolete?

See the other emails - both mean it's a "Commited" interface but
"Commited Obsolete" means "try to avoid using it". The original idea was
based on the comment in
http://www.opengroup.org/onlinepubs/9699919799/utilities/tail.html which
says:
-- snip --
This version of tail was created to allow conformance to the Utility
Syntax Guidelines. The historical -b option was omitted because of the
general non-portability of block-sized units of text. The -c option
historically meant "characters", but this volume of POSIX.1-2008
indicates that it means "bytes". This was selected to allow reasonable
im

"Simlinks" rename (to "pseudonetlink") ? / was: Re: 2009/200 Solaris Simlinks

[Roland Mainz]

Artem Kachitchkine wrote:
> 
> > We seem to be doing a good job coming up with names that we definitely
> > shouldn't use :-)  I still hope Rishi can put forth a new name for these
> > handy objects.  (Naming is hard -- but important.)
> 
> We just need to break out of the confines of the English language. Let
> me throw in a few, just to get your creative juices flowing:
> 
> alalink
> linkola
> liaison-dangereuse
> grandenpoopen
> hastalalinko

Erm... technically (at least looking at Unix SystemV (David, Glenn or
Schilly may correct me)) the precendent is to use the prefix "pseudo",
e.g. "pty" for "psuedo terminal" etc. ...
... my preference would be to follow this (e.g. use "pseudonetlink" (=
"pseudo network link")).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2009/116 posix_spawn_file_actions_addclosefrom_np

[Roland Mainz]

"Roger A. Faulkner" wrote:
> 
> I am sponsoring this automatic case for myself.
> 
> 1.  Introduction
> 
> This case adds a new posix_spawn() support function to the C library:
> 
>int posix_spawn_file_actions_addclosefrom_np(
>posix_spawn_file_actions_t *file_actions,
>int filedes);
> 
> The committment level of this interface is Committed.
> The "_np" suffix indicates that this is a non-portable
> Solaris extension.

Two nits:
- IMO it may be nice to use a different suffix than "_np", e.g. IMO
|posix_spawn_file_actions_addclosefrom_solaris()| may be better.
- What about adding a |flags| argument to allow that future extensions
can be made, e.g. change the prototype to...
-- snip --
int posix_spawn_file_actions_addclosefrom_solaris(
posix_spawn_file_actions_t *file_actions,
int filedes,
uint32_t flags
)
-- snip --

> The release binding is "minor release" so it can go
> into SunOS 5.11 (OpenSolaris) (there is no need or
> intention to back-port the interface to SunOS 5.10).

Why ? It may be _nice_ to have it in Solaris 10 on demand, too (e.g. you
don't have to do the backport now but it keeps the option open to do a
backport to Solaris 10 without another ARC case).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 update 2[PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

"Mark J. Nelson" wrote:
> > ...I've asked around
> > whether it is possible to deliver the contents of one ARC case with
> > multiple putbacks but the answers were a bit "fuzzy"
> 
> Speaking not as an ARC member, but rather as a CRT Advocate and former
> Tech Lead:
> 
> Standard expectation is that a single ARC case will be integrated as a
> single unit.

Ok...

> If you intend anything else, either because a coordinated delivery is
> necessary, or you plan to stage multiple deliveries for different phases
> of the work, then it should be called out explicitly in the ARC case
> materials.

Looking at
http://svn.genunix.org/repos/on/branches/ksh93/gisburn/arc/ksh93_update2/onepager.txt
part 1 need to be done first and all further steps can be done
independently in parallel (since there aren't any dependicies between
the parts 2-10).

> If I evaluate a project or RTI, and find that it's only delivering a
> subset of the work specified in the ARC case materials, and that this
> was not made explicit when it was reviewed by the ARC, then I typically
> ask the submitter to send a note to the case log, describing what
> they're doing and why, and giving the members a chance to comment.

Ok... but in this case we already cut-down the ARC case to avoid that we
have to deliver the putback in multiple steps. On the other side it may
help to have something like that as "safety pad" to avoid that we run
(again) into unexpected problems shortly before the RTI, causing either
deplays or ripping-out parts of the ARC cases (for example "printf" was
ripped-out from the original ksh93-integration update1 work since Don
Cragun found a issue (see
http://bugs.opensolaris.org/view_bug.do?bug_id=6800113) very late when
the code was in the early code review stages (we droped this part in an
attempt to make it for the OpenSolaris/Indiana deadline)).
Do know any ARC case which we could use as precedent ?

> If I have to dig to find this information, I get a little bit grumpy.
> Ie if you're only delivering a subset of the case materials, please
> spell it out clearly in the RTI.  Otherwise it feels like you've either
> been careless, or that you're trying to be sneaky.  (Neither of which is
> usually true, but your advocate doesn't always have all of the
> information that you have...)

Ok...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

stpcpy, stpncpy, wcpcpy, wcpncpy in lib C [PSARC/2009/084 Self Review]

[Roland Mainz]

James Carlson wrote:
> Casper Dik writes:
> >char *stpcpy(char  *restrict s1, const char *restrict s2);
> >char *stpncpy(char *restrict s1, const char *restrict s2, size_t n);
> >wchar_t *wcpcpy(wchar_t  restrict *ws1, const wchar_t *restrict ws2);
> >wchar_t *wcpncpy(wchar_t restrict *ws1, const wchar_t *restrict ws2, 
> > size_t
> > n);
> 
> The lint maintainers be notified so that they can add stpcpy and
> wcpcpy to their list of security-questionable functions warned about
> via -errsecurity.  (At least file an RFE.)

Erm... are both |strcpy()| and |strcat()| in this list, too ?

> This looks to me like a standards-related blunder.  "stpncpy" and
> "wcpncpy" functionality (strncpy-like zero pad to end of the buffer,
> *and* leaving full destination buffers unterminated) doesn't seem to
> make much sense in this context.

My primary interest was |stpcpy()| to get performance of the locale
codepaths fixed (we assume that we check at the _beginning_ that all
allocated buffers have sufficient size (since the use of something like
|stplcpy()| would completely kill the performance advantage since more
registers and more operations are required for string copies)). Right
now these codepaths are seveal _factors_ (more than 2x) _slower_ than on
Linux and one of the primary "root cause" is the over-use of inefficient
buffer handling (whic hwe try to partially address with the use of
|stpcpy()|).

> I think it's a shame they didn't do stplcpy and wcplcpy instead.  Do
> we have anyone involved in the standards process who could add
> comments?

Don Cragun was RIF'ed and right now I don't know anyone at Sun who can
be queried for standards issues.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 update 2[PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

"Mark J. Nelson" wrote:
> > ...I've asked around
> > whether it is possible to deliver the contents of one ARC case with
> > multiple putbacks but the answers were a bit "fuzzy"
> 
> Speaking not as an ARC member, but rather as a CRT Advocate and former
> Tech Lead:
> 
> Standard expectation is that a single ARC case will be integrated as a
> single unit.

Ok...

> If you intend anything else, either because a coordinated delivery is
> necessary, or you plan to stage multiple deliveries for different phases
> of the work, then it should be called out explicitly in the ARC case
> materials.

Looking at
http://svn.genunix.org/repos/on/branches/ksh93/gisburn/arc/ksh93_update2/onepager.txt
part 1 need to be done first and all further steps can be done
independently in parallel (since there aren't any dependicies between
the parts 2-10).

> If I evaluate a project or RTI, and find that it's only delivering a
> subset of the work specified in the ARC case materials, and that this
> was not made explicit when it was reviewed by the ARC, then I typically
> ask the submitter to send a note to the case log, describing what
> they're doing and why, and giving the members a chance to comment.

Ok... but in this case we already cut-down the ARC case to avoid that we
have to deliver the putback in multiple steps. On the other side it may
help to have something like that as "safety pad" to avoid that we run
(again) into unexpected problems shortly before the RTI, causing either
deplays or ripping-out parts of the ARC cases (for example "printf" was
ripped-out from the original ksh93-integration update1 work since Don
Cragun found a issue (see
http://bugs.opensolaris.org/view_bug.do?bug_id=6800113) very late when
the code was in the early code review stages (we droped this part in an
attempt to make it for the OpenSolaris/Indiana deadline)).
Do know any ARC case which we could use as precedent ?

> If I have to dig to find this information, I get a little bit grumpy.
> Ie if you're only delivering a subset of the case materials, please
> spell it out clearly in the RTI.  Otherwise it feels like you've either
> been careless, or that you're trying to be sneaky.  (Neither of which is
> usually true, but your advocate doesn't always have all of the
> information that you have...)

Ok...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 update 2 [PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

"I. Szczesniak" wrote:
> On 2/4/09, Roland Mainz  wrote:
> >  2. The next ARC case may contain a few more non-filesystem utilties
> >  (e.g. "join", "head", "tail", "tee", "mkfifo") from libcmd and (as a
> >  seperare ARC case) cover the remaining closed-source commands defined by
> >  POSIX (e.g. "tr" in it's various incarnations, "printf", "od", "sed"
> >  ("tail" is already handled via the previous list) etc.)
> >  3. The long-term plan is to create some kind of POSIX commands community
> >  to actively develop (this includes that we try to push new developments
> >  and technologies into the POSIX standard) and maintain these utilities -
> >  see
> >  http://mail.opensolaris.org/pipermail/ogb-discuss/2009-January/006302.html
> >  for the original plan from last November (this has become even more
> >  important now that Sun RIF'ed the complete team which was doing this
> >  work for nearly a decade... ;-( )
> 
> Why aren't you going to replace head, join, mkfifo, tail and tee in
> this ARC case?

1. To simplify the putback and avoid that we have to wait lots of months
if one of the commands isn't ready and still requires adjustments
(assuming that one ARC case must result in exactly one putback (like it
was the case for ksh93-integration update1 (the primary reason for
multi-month delays was the roundtrips for testing)). I've asked around
whether it is possible to deliver the contents of one ARC case with
multiple putbacks but the answers were a bit "fuzzy" (erm... or better:
There seems to be lots of ideas/opinions, no consens and noone came up
with a precedent we could "clone" and at some point I decided to move
adhead and simply get the ksh93-integration update2 code ready for
putback)).

2. Not everything is ready right now - all commands except one covered
by this ARC case are already "ready" (e.g. passes VSC test suite, Sun's
testsuites, we made sure that the last ten bugfixed applied to the old
codebase are fixed in the new codebase, passes { OS/Net, SFWNV, FOX, KDE
} builds etc.) for the change since "ksh93-integration update1" but not
all commands in your list are ready.
The current status looks like this:
- "head" - easy to handle with current codebase
- "mkfifo" - easy to handle with updated codebase (there is a small bug
with the umask handling which the new version has fixed)
- "tee" - easy to handle with current codebase
- "tail" - will be a _pain_ since Solaris has two versions
(/usr/bin/tail and /usr/xpg4/bin/tail) with slightly different behvaiour
and there are _lots_ of bugs filed against Solaris's "tail" codebase and
we'd like to make sure we don't reintroduce any old issues.
Unfortunately the "tail" codebase is closed-source and we lack a sponsor
who can check the bug reports and extract the testcases for us.
- "join" - should be possible to handle it with the current codebase but
I think we don't have sufficient testing coverage right now (which means
either the RTI advocate or gatekeepers will "bite" us). The AST "join"
version passes the AT&T and GNU testsuites but we do not have the VSC
results and since April Chin was laid-off we don't have access to this
test suite right now. And AFAIK there isn't a consumer within OS/Net
which makes me feel a bit uncomfortable unless we find a real-world
consumer.

Beyond that "tail" and "join" should be ready in four-six weeks and then
we can run the next ARC case.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Why "print" is not bound to a PATH element... / was: Re:[ksh93-integration-discuss] ksh93 update 2 [PSARC/2009/063FastTrack timeout02/09/2009]

[Roland Mainz]

James Carlson wrote:
> Roland Mainz writes:
> > 2. Restricted shell scripts (e.g. "rsh", "rksh", "pfrksh" [2] etc.) need
> > a way to output data (e.g. counterpart to "read") and therefore "print"
> > was never bound to a PATH element since the korn shell exists. Otherwise
> > you have shell scripts which can't output anything except syntax errors
> > when running in "restricted" shell mode.
> 
> OK; that's the key bit I was looking for.  "print" is a built-in even
> when $PATH doesn't have a "print" on it, and binding would break that.

Right (BTW: the same applies to "printf" and "sleep") ...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

ksh93 update 2 [PSARC/2009/063 FastTrack timeout 02/09/2009]

[Roland Mainz]

Scott Rotondo wrote:
> John Plocher wrote:
> > Peter Tribble wrote:
> >>> I know that I'm certainly not happy about ripping out Solaris commands and
> >>> replacing them with external commands.
> >
> > Since Sun's management seems to have RIF'd the entire team that used
> > to maintain those old Solaris commands, it seems clear that *they* no
> > longer have the same commitment to them as they used to.  In that
> > case, anything that moves towards a codebase that /is/ actively
> > supported by people who are experts in the field is rather to be
> > desired, don't you think?
> 
> Point taken, but to be fair Sun does still support these commands, and
> there are Sun engineers responsible for maintaining them. Just not as
> many as before.

... which leads to a question from my side: Who is now responsible for
these commands (technically I need a name of someone who can answer some
questions (both technical and organisatorical)) ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 update 2 [PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

Casper.Dik at Sun.COM wrote:
> > > >>> It's fine to make use of the ksh builtin support for various
> > > >>> commands, but
> > > >>> can we please learn from the problems that occurred when we
> > > >>> changed sleep
> > > >>> to be a builtin recently (e.g. 6793120) and instead create trivial
> > > >>> wrapper
> > > >>> *programs* that access the builtin functionality through libshell?
> > > >>
> > > >> I already have a fix (tested and queued for my sponsor) for CR
> > > >> #6793120
> > > >> which does something similar as you've proposed...
> > > >
> > > > So there is a unique pid for each program and thus it can still be
> > > > pkill'd?
> > >
> > > If so, and if this fix involves wrappers, Wouldn't we have lost the
> > > "no fork/exec" advantage of having shell builtins in the first place,
> > > right?
> >
> >My understanding is that the driving force is code sharing, not
> >performance.  If we are really concerned about the performance of the
> >`sum' or `sleep' commands, something more fundamental is amiss.
> 
> And if you start them as individual commands, there's no fork/exec
> you can save.  If you run ksh93, then you save the fork/exec.

Right... but in the case of OpenSolaris/Indiana /usr/bin/sh, /bin/sh,
/sbin/sh, /usr/bin/ksh, /bin/ksh etc. (right now I am unsure about
/usr/xpg4/bin/sh) are all links to /usr/bin/ksh93 and therefore a large
chunk of default shells and system scripts benefit from the change.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Fix for CR #6793120 ... / was: Re: [ksh93-integration-discuss] ksh93update 2[PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

Peter Memishian wrote:
>  > > So there is a unique pid for each program and thus it can still be 
> pkill'd?
>  >
>  > If you start the command as seperate child job (e.g. $ sleep 12345 & #)
>  > it will always have a seperate pid. But that was not the problem which
>  > caused CR #6793120 - the process name changed from "sleep 12345" to
>  > something like "ksh93 sleep 12345" which caused the PIT test scripts to
>  > fail because they matched exactly for the process name "sleep 12345".
>  > The upcoming patch makes sure the processes get their expected name.
> 
> Please tell me that this does not involve horrible hacks involving faking
> up or munging process names in the kernel.

Nah... we just use a buffer overrun in the doors subsystem to hack
into nscd, from there we crawl into the kernel and patch the process
table ...

Seriously... we change the alias.sh wrapper into a binary and therefore
get the expected process name (this is a temporary solution until we
switch (as originally designed) to compiled shell scripts (which should
have the same effect)).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Fix for CR #6793120 ... / was: Re: [ksh93-integration-discuss] ksh93 update 2[PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

Peter Memishian wrote:
>  > > It's fine to make use of the ksh builtin support for various commands, 
> but
>  > > can we please learn from the problems that occurred when we changed sleep
>  > > to be a builtin recently (e.g. 6793120) and instead create trivial 
> wrapper
>  > > *programs* that access the builtin functionality through libshell?
>  >
>  > I already have a fix (tested and queued for my sponsor) for CR #6793120
>  > which does something similar as you've proposed...
> 
> So there is a unique pid for each program and thus it can still be pkill'd?

If you start the command as seperate child job (e.g. $ sleep 12345 & #)
it will always have a seperate pid. But that was not the problem which
caused CR #6793120 - the process name changed from "sleep 12345" to
something like "ksh93 sleep 12345" which caused the PIT test scripts to
fail because they matched exactly for the process name "sleep 12345".
The upcoming patch makes sure the processes get their expected name.



Bye,
Roland

P.S.: If you need a seperate process id or do not want to use builtins
you can either specify the full path of the command (e.g. use
"/usr/bin/sleep" instead of just "sleep" - this forces the shell to run
the external command since POSIX requires that calling a command with
the full path runs _exactly_ this executable and not a builtin) or use
(in the case of ksh93) the "bulitin" command to remove this builtin
command from the list of builtins.

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 update 2[PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

Peter Memishian wrote:
>  > >>> It's fine to make use of the ksh builtin support for various
>  > >>> commands, but
>  > >>> can we please learn from the problems that occurred when we
>  > >>> changed sleep
>  > >>> to be a builtin recently (e.g. 6793120) and instead create trivial
>  > >>> wrapper
>  > >>> *programs* that access the builtin functionality through libshell?
>  > >>
>  > >> I already have a fix (tested and queued for my sponsor) for CR
>  > >> #6793120
>  > >> which does something similar as you've proposed...
>  > >
>  > > So there is a unique pid for each program and thus it can still be
>  > > pkill'd?
>  >
>  > If so, and if this fix involves wrappers, Wouldn't we have lost the
>  > "no fork/exec" advantage of having shell builtins in the first place,
>  > right?
> 
> My understanding is that the driving force is code sharing, not
> performance.

"performace" is important, too. That's why we added the whole
infratructure for compiled shell script (including "shcomp" and
"shbinexec"). We just can't use it until a sufficient amount of time has
passed to make sure that all build machines have the matching
"shbinexec" kernel module (that would require a "flag day" ... in theory
it could be done ASAP but that would IMO just cause unneccesary pain for
the developers and admins (at least I would prefer to wait until at
least B116)).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 update 2 [PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

Dale Ghent wrote:
> On Feb 4, 2009, at 12:32 AM, Peter Memishian wrote:
> >>> It's fine to make use of the ksh builtin support for various
> >>> commands, but
> >>> can we please learn from the problems that occurred when we
> >>> changed sleep
> >>> to be a builtin recently (e.g. 6793120) and instead create trivial
> >>> wrapper
> >>> *programs* that access the builtin functionality through libshell?
> >>
> >> I already have a fix (tested and queued for my sponsor) for CR
> >> #6793120
> >> which does something similar as you've proposed...
> >
> > So there is a unique pid for each program and thus it can still be
> > pkill'd?
> 
> If so, and if this fix involves wrappers, Wouldn't we have lost the
> "no fork/exec" advantage of having shell builtins in the first place,
> right?

Erm... no. The wrapper is there to fill the matching filesystem entry
(for example /usr/bin/chicken). If the shell is already running and the
"chicken" command should be executed the shell will walk the PATH
elements and search for the first filesystem entry which matches this
name. If a builtin "chicken" is bound to /usr/bin/ it will be executed
instead of the /usr/bin/chicken command in the filesystem if the PATH
search reaches /usr/bin/.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 update 2 [PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

Peter Memishian wrote:
> 
> It's fine to make use of the ksh builtin support for various commands, but
> can we please learn from the problems that occurred when we changed sleep
> to be a builtin recently (e.g. 6793120) and instead create trivial wrapper
> *programs* that access the builtin functionality through libshell?

I already have a fix (tested and queued for my sponsor) for CR #6793120
which does something similar as you've proposed... and since the
commands all go through the same wrapper we won't see this problem a 2nd
time (and "yes", I'll be carefull the next time and explicitly test for
this condition - CR #6793120 was one of the mistakes in the
ksh93-integration update1 putback which was really avoidable if I
would've used my brain... ;-/ ).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

ksh93 update 2 [PSARC/2009/063 FastTrack timeout 02/09/2009]

[Roland Mainz]

Peter Tribble wrote:
> On Mon, Feb 2, 2009 at 1:28 AM, Alan Hargreaves
>  wrote:
> > I'm sponsoring this fast-track request on behalf of the
> > ksh93-integration project.
> >
> > Please note that this is an *open* case.
> >
> > Template Version: @(#)sac_nextcase %I% %G% SMI
> > This information is Copyright 2009 Sun Microsystems
> > 1. Introduction
> >1.1. Project/Component Working Name:
> > ksh93 update 2
> >1.2. Name of Document Author/Supplier:
> > Author:  Alan Hargreaves
> >1.3  Date of This Document:
> >01 February, 2009
> > 4. Technical Description
> >
> > The release binding is the same as with the previous ksh93 project: a
> > patch/micro release of Solaris delivering through OS/Net.  Stability
> > levels are as described below.
> >
> > Additional materials (man pages and diffs) can be found in the
> > 'materials' subdirectory.
> >
> > This project is an amendment to the Korn Shell 93 Integration project
> > (PSARC/2006/550, PSARC/2006/587, PSARC/2007/035 and PSARC/2008/344)
> > specifying the following additional interfaces:
> >
> > 1) Update of ksh93 interfaces
> > 1.1) New "typeset" variable type qualifier ("-C") to declare compound
> > variables
> > 1.2) New "typeset" option "-m" to rename/move a variable
> > 1.3) New "read" option "-C" to read a variable as compound variable
> > 1.4) New "print" option "-C" to print compound variables as a
> > single line
> > 1.5) New ksh93 math functions "log10", "j0", "j1", "jn", "y0",
> > "y1", "yn"
> > 1.6) Interface stability change of the POSIX shell command language
> > parts of the ksh93 language
> >
> > 2) An enhanced version of the "cmp" utility and an identical ksh93
> >   built-in command
> >
> > 3) An enhanced version of the "cut" utility and an identical ksh93
> >   built-in command
> >
> > 4) An enhanced version of the "comm" utility and an identical ksh93
> >   built-in command
> >
> > 5) An enhanced version of the "paste" utility and an identical ksh93
> >   built-in command
> >
> > 6) The addition of /usr/bin/print
> >
> > 7) An enhanced version of the "uniq" utility and an identical ksh93
> >   built-in command
> >
> > 8) An enhanced version of the "wc" utility and an identical ksh93
> >   built-in command
> 
> Replacement of important components of the userland seems rather too important
> for a fast-track.
>
> I know that I'm certainly not happy about ripping out Solaris commands and
> replacing them with external commands.

Erm... these commands aren't "external" nor are we going to "rip them
out". The new commands will still be part of OS/Net (actually they _are_
already within OS/Net as part of usr/src/lib/libcmd/common/) and shipped
like the old commands (most of them remain in the same package but some
move from "SUNWesu" to "SUNWcsu" (which isn't a problem since "SUNWesu"
depends on "SUNWcsu")). The change is that:
- we get a living and cooperative upstream with which we do active
development and enhancements instead of just maintaining the "status
quo" (which was more or less the case for the commands we change in this
case)
- we get a 64bit clean codebase (which is _urgendly_ needed for Solaris
ports like SystemZ)
- we get a significant performance[1] boost on OpenSolaris/Indiana since
all these commands then run "inline"
- we reduce the system's resource usage (both disk space footprint and
memory usage (you'll find a similar appriach on embedded Linux versions
which use the "busybox" toolkit))

[1]=Short explanation: Using builtin commands saves at least one
|fork()|+|exec()| sequence and the whole work behind |setlocale()|. The
|fork()| itself is obviously a heavywheight operation but the |exec()|
can affect the performance of the _whole_ machine since the kernel must
make crosscalls to all CPUs to tear down the matching address space. Now
imagine a T5440 with 256 threads and lots of scripts running in
parallel... in that case each saved |exec()| is a significant
performance win.

> What happens when we need or want
> to diverge from upstream behaviour?

1. Nicolas already answered that question in
http://mail.opensolaris.org/pipermail/opensolaris-arc/2009-February/013756.html
... we can diverge from the upstream codebase but AFAIK there should've
be a need to do this because we _cooperate_ with upstream.
2. The tools we change in this ARC case are all defined by POSIX+SUS and
both the old and new commands conform to these standards. There is no
practical way to diverge from the POSIX+SUS behaviour since otherwise
we'll break lots of stuff. For the extensions beyond POSIX+SUS defined
in this case the behaviour is defined by more than one party (e.g.
usually at least two of { GNU *, FreeBSD, NetBSD, AST, MacOSX } ) which
gives a high probabilty of keeping these interfaces stable (that's the
reason why we jump directly to the ARC interface status "commited").

> And is this just the tip of the iceberg?

This is AFAIK offtopic for an ARC discussion... but the answer is both
"yes

Why "print" is not bound to a PATH element... / was: Re: [ksh93-integration-discuss] ksh93 update 2 [PSARC/2009/063FastTrack timeout 02/09/2009]

[Roland Mainz]

James Carlson wrote:
> Alan Hargreaves writes:
> > Unlike other built-in commands named in PSARC/2006/550, the "print"
> > built-in in ksh93 will _not_ be bound to the /usr/bin/ pathname to
> > ensure backwards-compatiblity to existing ksh93 scripts (for example
> > scripts running in "restricted" shell mode expect that some shell
> > builtins are available independently from the value of ${PATH}).
> 
> Could someone on the ksh93 team explain this bit in a little more
> detail?  It might be just a nit, but I don't follow the logic above:
> what usage case requires that this path isn't bound?

The usage cases are:
1. "Legacy"/"Compatibilty" issue: Existing ksh88 and ksh93 scripts
assume that "print" is _always_ available, independetly how the value of
${PATH} looks like (the same applies to "sleep" and "printf" for ksh93
scripts (e.g. these are the only three non-special [1] shell builtin
commands which _must_ be there)).
2. Restricted shell scripts (e.g. "rsh", "rksh", "pfrksh" [2] etc.) need
a way to output data (e.g. counterpart to "read") and therefore "print"
was never bound to a PATH element since the korn shell exists. Otherwise
you have shell scripts which can't output anything except syntax errors
when running in "restricted" shell mode.

Or short: Binding the "print" builtin to a PATH element would blow-up
lots of scripts

[1]="special builtins" refers to shell builtin commands like "read"
which are part of the shell language and must run within the shell
process itself (e.g. to write information back into variables or change
the shell's status (e.g. "set", "unset", "shift", "alias", "exec" etc.
fall into the same category))
[2]="pfrksh" doesn't exist yet but ksh93 now has support for such a
chimera made from profile+restricted shells



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

prefetch interfaces [PSARC/2009/039 Self Review]

[Roland Mainz]

Garrett D'Amore - sun microsystems wrote:
[snip]
> PROPOSAL
> 
> Add inlines for i386/amd64 and sparcv9 for the following types of prefetch:
> 
> void prefetch_read_many(void *)
> 
> This requests data be loaded into a cache for repeated reading. (This
> equates to a 't0' prefetch on x86 CPUs).
> 
> void prefetch_read_once(void *)
> 
> This requests data be loaded into a cache for one read only, so it can
> be flushed from the cache immediately after access. (This equates to a
> 'nta' prefetch on x86 CPUs).

Why do both interfaces do not have a |length| operand ? Some
CPUs/architecturs allow to define a memory range (rounded up to the CPUs
block size) to be pre-fetched...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Issues for LSARC/2008/598: Sun Scalable Visualization

[Roland Mainz]

Steve Nash wrote:
> Alan Coopersmith wrote:
[snip]
> >> * changes the default $PATH and $LD_LIBRARY_PATH for all users
> >
> > How does it change the default $PATH?
> > How does it set $LD_LIBRARY_PATH without breaking other software?
> 
> We modify the /etc/{bashrc,csh.cshrc} to change the default environment
> (for all users).

1. Currently there is no /etc/bash.bashrc (e.g. the bash binary is
compiled without support for this file) and AFAIK no /etc/csh.cshrc
either
2. There is /etc/profile (for login shells) and /etc/ksh.kshrc (for
_interactive_ ksh93 shells) but both are "going away" in the near
future, being replaced by /etc/env.d/ (e.g. instead of modifying
/etc/profile , /etc/ksh.kshrc, /etc/bash.bashrc you put "scriptlets"
into /etc/env.d/shell_interactive_start/)



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Issues for LSARC/2008/598: Sun Scalable Visualization

[Roland Mainz]

Alan Coopersmith wrote:
[snip]
> What shell is used for these scripts?   If it's /bin/sh, have you verified 
> they
> work with both classic Solaris /bin/sh and ksh93 (which OpenSolaris uses as
> /bin/sh)?

Actually it would help a lot if projects would test their scripts
against /usr/demo/ksh/bin/shlint (that was added in B106 and may soon be
moved to /usr/bin/) - getting the warnings fixed (except the ones for
"`cmd` vs. $(cmd)")  would solve _lots_ of problems with the current
shell scripts in Solaris.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

2008/784 Sendfilev Behavior Change

[Roland Mainz]

James Carlson wrote:
> I am sponsoring this fast-track request for Vasumathi Sundaram.  The
> timer is set to 01/02/2009 in light of the coming holidays.  Please
> speak up if you need more time than that.
> 
> In an offline discussion, the project team went to lengths to find any
> sendfilev users,

Erm... ksh93 had support for |sendfile()|/|sendfilev()| support but it
was breaking UDP networking support (and there was no reliable way to
check this) and we removed this capability again (it gave a good
performance improvement for sending larger files but propper
cross-platform support is more important in this case).

> 2. Solution
> 
>   sendfilev is modified so that the output behavior is not the same as
>   writev() and when multiple threads attempt to write to the same
>   output file at the same time.  Instead, the result of the combined
>   operations is undefined.  The applications that use sendfilev() will
>   have to use their own synchronization mechanism to ensure a
>   consistent output.

... which effectively means: A shell with thread support or any other
language environment (e.g. java) shouldn't use |sendfilev()| when files
are involed, right ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2008/778 - asprintf, vasprintf

[Roland Mainz]

Nicolas Williams wrote:
> On Fri, Dec 19, 2008 at 09:14:37PM +0100, Roland Mainz wrote:
> > Will this even work if another library is used for |malloc()|&&|free()|
> > calls (IMO it should work but I am not sure about the case when an
> > application is linked via -B direct) ?
> 
> Yes, it will: malloc() and friends are marked as NODIRECT in libc, which
> means they can be interposed on freely by the application, even when
> they are called from within libc itself.

Devils advocate question:
What happens if someone intercepts |asprintf()| or |vasprintf()| with
his/her own function which doesn't use the default |malloc()| () ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

"Namespace seperation via PATH" vs. "dump all stuff into /usr/bin/" ... / was: Re: GNU Developer Collection [LSARC/2008/776 FastTrack timeout01/07/2009]

[Roland Mainz]

Nicolas Williams wrote:
> On Fri, Dec 19, 2008 at 11:39:01AM -0800, George Vasick wrote:
> > We consulted the submitter, Craig Mohrman.  The case is stalled in Sun
> > legal.  Craig agrees with the concept in our proposal of using
> > /usr/compilers/... for this and future versions of compilers allowing
> > multiple versions to coexist.  Users would then control which is the
> > default compiler version by which links package is installed.
> 
> Should we have /usr/interpreters too?  (And should Perl5, Python, ...
> move there?)  What about projects that deliver both, interpreters and
> compilers (there exist LISPs that do that)?
> 
> BTW, I like the idea of links pkgs for creating defaults in /bin.

I dislike it since it is the reason that some things can only be build
with "root" rights anymore (such as all Mozilla.org products) since you
have to remove stuff to prevent things like "autoconf" from finding it
by default. OkOk, "autoconf" is a problem anyway but the original Unix
designers used PATH for propper namespace seperation and the whole
Linux-style "dump all things into /bin" made this situation very bad.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

PSARC/2008/778 - asprintf, vasprintf

[Roland Mainz]

"Roger A. Faulkner" wrote:
> 
> I am sponsoring this automatic case for myself.
> 
> 1.  Introduction
> 
> This case adds two new functions to the C library, asprintf()
> and vasprintf() as follows:
> 
>int asprintf(char **ret, const char *format, ...);
>int vasprintf(char **ret, const char *format, va_list ap);
> 
> The committment level of these interfaces is Committed.
> 
> The release binding is "patch"
> (so it can be back-ported to Solaris 10 if required).
> 
> 2.  Discussion
> 
> Linux and the various BSD operating systems provide the *asprintf()
> family of functions.  These behave as sprintf(), except that they
> allocate a string large enough to hold the result, and return a
> pointer to that string.  The caller is later expected to free()
> this buffer.

Will this even work if another library is used for |malloc()|&&|free()|
calls (IMO it should work but I am not sure about the case when an
application is linked via -B direct) ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

GNU Developer Collection [LSARC/2008/776 FastTrack timeout01/07/2009]

[Roland Mainz]

George Vasick wrote:
> Rainer Orth wrote:
> > Raj Prakash  writes:
[snip]
> >> 2. Project Summary
> >>2.1. Project Description:
> >>  The project will provide the current releases of the GNU Compiler
> >>  Collection (GCC) and the GNU Binutils for OpenSolaris.  The primary
> >>  components are the following:
> >>  - GCC includes C, C++, FORTRAN, Objective-C, and Objective-C++.
> >
> > Why exclude GCJ (i.e. the Java compiler) and GNAT (Ada compiler)?
> 
> We based the included languages on what was done for the current
> opensolaris version of gcc, 3.4.3.  We will definitely consider
> including additional languages in the future if there is demand.

Erm... I disagree in this point. The lack of Fortan and _ADA_ was
requested often enougth (at least at universities). IMO it's easier to
do it _now_ in this ARC case than deferring it _again_ (which would
likely mean we won't see gcc Ada support in Solaris before 2010).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

GNU Developer Collection [LSARC/2008/776 FastTrack timeout01/07/2009]

[Roland Mainz]

George Vasick wrote:
> Danek Duvall wrote:
> > On Thu, Dec 18, 2008 at 10:35:36AM +1000, James C. McPherson wrote:
> >
> >> You seem to be conflicting with
> >>
> >> 6674032 Introduce GCC 4.3.x in Nevada
> >> 6674042 Introduce MPFR (Multiple Precision Floating-Point Rounding 
> >> Library) in Nevada
> >> 6674044 Introduce GNU MP 4.2.4 in Nevada
> >
> > There's also
> >
> > PSARC/2008/378 Move gcc and binutils from /usr/sfw/bin to /usr/bin
> 
> We consulted the submitter, Craig Mohrman.  The case is stalled in Sun
> legal.  Craig agrees with the concept in our proposal of using
> /usr/compilers/... for this and future versions of compilers allowing
> multiple versions to coexist.  Users would then control which is the
> default compiler version by which links package is installed.

Can we please get some feedback whether Sun Studio will use
/usr/compilers/ in the future ? From what I heared it will remain in
/opt/SUNWspro/ with links in /usr/bin/ in the future and if only gcc
resides there: Why should this directory be named
"compilers" and not /usr/lib/gcc/ ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Remove /usr/bin/printf from PSARC case 2008 094 [PSARC/2008/589Self Review]

[Roland Mainz]

Garrett D'Amore wrote:


> 
> -- Garrett
> 
> Don Cragun wrote:
> > I am sponsoring this case for Roland Mainz (an OpenSolaris contributor)
> > and April Chin (acting as sponsor for this work).
> >
> > I believe it qualifies for closed approved automatic status.   If a
> > PSARC member disagrees, let me know and I will upgrade it to a
> > fast-track case.
> >
> >   Sincerely,
> >   Don
> >
> > Template Version: @(#)sac_nextcase %I% %G% SMI
> > This information is Copyright 2008 Sun Microsystems
> > 1. Introduction
> > 1.1. Project/Component Working Name:
> >Remove /usr/bin/printf from PSARC case 2008 094
> > 1.2. Name of Document Author/Supplier:
> >Author:  Donald Cragun
> > 1.3  Date of This Document:
> >   17 September, 2008
> > 4. Technical Description
> >
> > This case updates PSARC/2008/094 (ksh93 Update 1).
> >
> > While implementing PSARC/2008/094, which (among other things) planned
> > to replace /usr/bin/printf with a link to the AT&T AST printf, the
> > project team found some irregularities in standards conformance in both
> > the default printf() function in libc and in the behavior of
> > /usr/bin/printf.
> >
> > Since some of these issues may involve official interpretation requests
> > against the POSIX Standards and the Single UNIX Specifications, they
> > may take a long time to resolve.  Therefore, this case removes the
> > changes to /usr/bin/printf that were documented in PSARC/2008/094 from
> > the deliverables provided by that case so PSARC/2008/094 changes can
> > be integrated in a timely manner.
> >
> > Since the current /usr/bin/printf will not be changed by this case,
> > there are no backwards compatibility issues.
> 
> I concur that this is an automatic approval case.  In fact, you probably
> could have just done it as an update to 2008/094 directly.

Erm... the idea is to defer this part to a later putback when the issues
have been resolved. The problem is that we don't have time anymore for
_any_ discussions around the /usr/bin/printf since the code for
PSARC/2008/094 must be ready by Friday (_last_ _date_).

> That said, out of curiosity, where do the implementations differ such
> that ksh93's

s/ksh93/AST/ , ksh93 only offers the AST version of "printf" as builtin
command

> version can't stand as a replacement?

It's the "precision" option of "%s" - does it count in "whole
charatcers" (where "character" means "multi-byte character" (like all
other ksh88/ksh93/bash2/bash3 string operators do)), "screen columns"
(since _some_ multibyte characters occupy more than one screen column)
or "bytes".

The ksh93 builtin originally used "whole characters" but the POSIX/SUS
standards indirectly define "bytes" (without having any "leeway" right
now to allow "whole characters") and the Solaris command seems to use
(accidently ?) "screen columns" (where I doubt whether this makes
sense).
In the meantime the ksh93 builtin has been changed to use "bytes" as
defined by POSIX/SUS and we're now trying to investigate two things:
1. Why Solaris's /usr/bin/printf uses "screen columns" ?
2. How can we get a |printf()| formatting option for "whole characters"
("%s"'s precision option counting in bytes isn't very healthy for
multibyte characters) ?

We're currently digging around to get an answer for [1] but we don't
have _any_ time anymore to hold-off the other parts of PSARC/2008/094
for that - people need the updated/fixed ksh93 _BADLY_, preferable
checked-in yesterday or the day before that.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)

Adding Malaysian, Indonesian,Vietnamese UTF-8 locales [PSARC/2008/420FastTrack timeout 07/11/2008]

[Roland Mainz]

Yong Sun wrote:
> This project is targeting to Indiana 2008.11 release, UTF-8 locales have
> higher priority than legacy locales,

Right (except stuff like "zh_CN.GB18030" which still should be preferred
over the UTF-8 variant since the matching goverment requires it (and is
very picky about it)) ...

> we will gather more
> information/feedback to see if we have future plans for legacy locales.

Ok... my main concern is "interoperabilty", e.g. what happens if I am
running something like the "ms_MY.UTF-8" or "vi_VN.UTF-8" locale and
then do a slogin into a Linux machine ?

> Regarding to the localization of CLI messages, yes, it would be changed
> over time, hopefully by the opensolaris l10n communities.
> 
> And for you last question, X11 terminal emulators, I believe the answer
> is Yes, since they are all unicode characters, and we will ship the
> proper fonts.

Please check whether you have _fixed_width_ fonts with all matching
point populated. Many fonts for these languages are either proportional
or don't have all points populated.

BTW: What about keyboard mappings and input methods (e.g. VIQR, VNI,
TELEX etc, for Vienamese), e.g. how does an user type characters for
these languages, either within Gnome or without having Gnome installed
(e.g. there is stuff like http://xvnkb.sourceforge.net/ for plain X11
applications) ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL 
 (;O/ \/ \O;)

Adding Malaysian, Indonesian, Vietnamese UTF-8 locales [PSARC/2008/420FastTrack timeout 07/11/2008]

[Roland Mainz]

Yong Young Sun wrote:
[snip] 
> 2. Project Summary
>2.1. Project Description:
> Add the following South East Asia locales support to current
> OpenSolaris.
> Malaysian:   ms_MY.UTF-8
> Indonesian:  id_ID.UTF-8
> Vietnamese:  vi_VN.UTF-8

Are there any plans to add a "vi_VN.TCVN" locale for interoperabilty
with Linux and other operating systems ?

> Add VISCII and TCVN code conversion feature in iconv library. So that
> the bidirectional conversion between VISCII,TCVN, and UTF-8/UCS4(2)
> will be supported in iconv(1) and iconv(3).
> 
> 3. Business Summary
>3.5. Opportunity Window/Exposure:
> Solaris Nevada
> Project Indiana
> 
> 4. Technical Description
>4.1. Details:
[snip] 
>   * CLI (Command Line Interface) messages:
> The localization for Solaris system libraries and utilities in 
> Malaysian,
> Indonesian and Vietnamese are not available.

Will this be changed over time ?

>   * L10N status for major GUI components by communities
> 
>   MalaysianIndonesianVietnamese
> --
> Gnome[3]  Yes  Yes   Yes
> Firefox [10]  N/A  Yes   N/A
> Thunderbird [11]  N/A  N/A   N/A
> Openoffice [14]   Yes [12] N/A   Yes [13]
> 
> The localization contents for these three languages of gnome had 
> been
> integrated in package SUNWgnome-l10nmessages-extra on Solaris.

One thing which is AFAIK missing here is the support status for X11
terminal emulators, e.g. can characters in these locales properly
displayed in:
1) xterm
2) dtterm
3) gnome-terminal (this may be optional since gnome-terminal has _huge_
issues with non-ASCII characters anyway)



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL 
 (;O/ \/ \O;)

Spec update and draft opinion available for 2008/351 -SwitchSPARCGNU coreutils+bash from 32 to 64bit

[Roland Mainz]

Darren J Moffat wrote:
> Roland Mainz wrote:
> > - There is one "well-known"[1] port of Solaris which runs on 64bit only
> > hardware - the problem with "under review" is that the work is _not_
> >
> > [1]=The name is not public... yet.
> 
> So well known that I don't know what it is and its name isn't public
> yet, come on get real it can't be both.

I've been told to avoid using the name or the involved companies and use
the term "well-known port" instead (welcome to "NDA"&co.). I can only
get more specific using my @sun.com address to addresses within Sun
(since I prefer to keep my job).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL 
 (;O/ \/ \O;)

Spec update and draft opinion available for 2008/351 - SwitchSPARCGNU coreutils+bash from 32 to 64bit

[Roland Mainz]

James Carlson wrote:
> John Plocher writes:
[snip] 
> Because I think the out-of-context excerpt in 4.3.2 appears to suggest
> that Sun has the final say here, and not the distributor, and is thus
> quite destructive on the face of it, I would vote against this opinion
> as drafted.
> 
> Without that section, I'd abstain.  Without a project supporting an
> actual 64-bit-only platform under review,

- There is one "well-known"[1] port of Solaris which runs on 64bit only
hardware - the problem with "under review" is that the work is _not_
done by Sun. Right now they desperately try get the problem under
control by adding "artificial" 32bit support on 64bit-only hardware to
_avoid_ having to cleanup OS/Net+SFWNV themselves. If this attempt fails
then this port is _DEAD_ (unless we find a way to make OS/Net+SFWNV
64bit clean).

- One of the early attemps (e.g. shortly after OpenSolaris.org wass
founded) of porting (Open-)Solaris Nevada to another platform simply
died when the matching company figured out that
1) OS/Net is not 64bit clean
2) the usual reply was "you have to cleanup that yourself, we won't
help"
Guess why that company lost their interest ?

At some point we need more ports to get the "diversity" from which Linux
and the *BSD communities live from - it's not only "packages packages
packages" what OpenSolaris needs - we need the ports, too. And there
we've hit the bottom ("PPC" being "dormant", one dead-born because the
company was scared away (see above) and the only currently active port
is in trouble because of the 64bit issue) and won't make _any_ progress
until the 64bit issue has been solved.

[1]=The name is not public... yet.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL 
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 Integration Update 1 Amendments1 [PSARC/2008/344 FastTrack timeout 06/03/2008]

[Roland Mainz]

James Carlson wrote:
[sorry, this one was stuck&&forgotten in my Drafts/-folder]
> Garrett D'Amore writes:
> > Have the upstream providers given thought to dealing with changes like
> > this and their impact on already-deployed scripts?  (Maybe there aren't
> > any that we care about yet, since our ksh93 is still so new.)
> 
> We've already had such problems.  See CR 6667990 for one such
> accident; you can't call your local function "start" and upgrade
> safely from Sun's old ksh88 to ksh93.

1. AFAIK the "quagga" builtin has problem with the ksh(88|93) shell
_alias_ called "stop"
2. The issue is caused by the switch from Bourne shell to ksh, not by
ksh93 itself. ksh88 already had the "stop" alias:
-- snip --
$ /usr/bin/ksh -c 'alias' | fgrep "stop"
stop='kill -STOP'
-- snip --
... and other shells have the same problem. And the trouble is that
"stop" is an shell "alias" which pretty much overrides anything else (as
a future countermeasure we'll get "shlint" at some point (once "shcomp"
is in the tree) to check for such common dragon-filled pits in the
future (in the meantime you can run your script against $ shcomp -n
 /dev/null # ...)).

> > I'm concerned, going forward, as ksh93 syntax becomes more prevalent,
> > that bringing in changes like the above may have unintended consequences
> > in scripts or even ON delivered components, which we cannot easily find
> > or test.
> 
> This probably isn't a good place to design a solution, but I share the
> sense of unease.  It puts script writers on shaky ground if they can't
> either specify a known environment or predict what's "safe."

Umpf... IMO "safe" is what's described in the POSIX shell standard and
http://www.opensolaris.org/os/project/shell/shellstyle/ (that's why I
wrote that document...).



Bye,
Roland

P.S.: Setting Reply-To: to ksh93-integration-discuss at opensolaris.org to
get this off he ARC list...

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

PSARC 2008/374 dladm parseable output

[Roland Mainz]

Garrett D'Amore wrote:
> Joseph Kowalski wrote:
[snip] 
> All of my other complaints/arguments related to the attempt to extend
> this case beyond dladm.  Creation of a new, perhaps non-obvious, parsed
> language/file format as a generic standard for everyone else to use
> starts to get difficult (and probably fails the "obvious" test for fast
> tracks.)

Erm..  it would be nice not to "threaten" other people's project (like
the libcompoundvar.so.1 one) with a "full ARC case" just because they
want to address a generic problem at the root... ;-(



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Structured information in shells... / was: Re: PSARC 2008/374 dladm parseable output

[Roland Mainz]

Garrett D'Amore wrote:
> Nicolas Williams wrote:
> > On Thu, Jun 12, 2008 at 08:34:33PM -0700, Garrett D'Amore wrote:
[snip]
> The question isn't about discouraging scripting, but rather, how far do
> we want to go to enable people to use a shell script to perform
> something that they really probably should have a better tool for?
> 
> I get the feeling that we're trying to represent what is really
> structured data, to a language that doesn't have even the slightest
> concept of data structures.

Erm... I suggest to stop designing stuff around the old Bourne shell or
treating all shells like the Bourne shell (if we go down to that level
we can easily _lower_ it by making the Thomson shell (predecessor of the
Bourne shell) the _minimum_ level for this case. IMO a reasonable
_minimum_ level should be the POSIX shell specification - which would
solve lots of trouble this case has...).

For example ksh93 has support for |struct|-like variables which can be
nested like this:
-- snip --
$ ksh93 -c 'x=( a=5 b="hello" c=( d=9 e="hello world" ) ) ; print "$x"'  
(
a=5
b=hello
c=(
d=9
e='hello world'
)
)
-- snip --

This kind of variable tree can be nested without any limits - see
http://www.nrubsig.org/people/gisburn/work/sun/xserver_license_extractor/X-src-20071107_XW_NV_open-src_tarballs_extraced_comments20080331.cpv
for a simply example of an array value.
 
The largest variable tree so far on production systems contained ~~30GB
(GigaBytes!!) of data (and it was _not_ slower than a JAVA or C++
application to build and crawl the tree (e.g. stop speading the "FUD"
that "shells are slow". It may apply to the Bourne shell but the shells
did evolve since that a bit...)).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Updated spec for [PSARC/2008/344 FastTrack timeout 06/03/2008] / was: Re: [ksh93-integration-discuss] ksh93 Integration Update 1 Amendments 1[PSARC/2008/344 FastTrack timeout 06/03/2008]

[Roland Mainz]

Alan Coopersmith wrote:
> 
> I'm sponsoring this fast-track request on behalf of the
> ksh93-integration project.
> Please note that this is an *open* case.
[snip]

As discussed in today's ARC phone call...
... attached is the updated text for "onepager.txt" and
"manpage.diff.txt" and the diff from the old to the new version
("ksh93_integration_update1_Amendments_1_PSARC_2008_344_case_update001.diff.txt")
...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)
-- next part --
I'm sponsoring this fast-track request on behalf of the
ksh93-integration project.
Please note that this is an *open* case.

The release binding is the same as with the previous ksh93-integration project:
a patch/micro release of Solaris delivering through OS/Net
Stability levels are as described below.

Additional materials (man pages and diffs) can be found in the
'materials' subdirectory.



This project is an amendment to the Korn Shell 93 Integration project
update 1 ARC case (PSARC/2008/094) specifying the following additional
interfaces:
1) Update of ksh93 from upstream release ast-ksh.2007-12-15 to
  ast-ksh-2008-05-22
1.1) Update of ksh93
1.2) New "typeset" variable storage qualifier for function-static
variables ("typeset -S varname")
1.3) New floating-point datatype "hexfloat" ("typeset -X varname)
1.4) New reserved options for "typeset".
1.5) New ksh93 math functions "ceil":
1.6) New reserved builtin "enum"



Bug/RFE Number(s):

6619428 RFE: Update ksh93 in Solaris to ast-ksh.2007-10-15 (or higher)
6601968 RFE: Add /usr/lib/shell/ksh/ as a place to store loadable
shell functions



Interface   StabilityDescription
 
-   ----  
/usr/lib/shell/ Project private  
  Commited C99 math functions



### Part 1: Update of ksh93 from upstream release ast-ksh.2007-12-15 to
ast-ksh-2008-05-22


## Part 1.1: Update of ksh93
The 1.1 portion of this project is the update of ksh93 from
ast-ksh.2007-12-15 to ast-ksh-2008-05-22 which marks the update from
ksh93 version 's+' to version 't-' (AST/ksh93 uses the (latin)
alphabet for its version number, e.g. version 'a', version 'b' etc. ;
the '+'/'-' means the stabilty status, e.g. '-' means its "alpha",
no suffix means its "stable" (e.g. ready for production usage) and
'+' means its a bugfixed stable version etc.).

Note that the team has invested _significant_ amounts of time to ensure
the quality requirements for Solaris, doing both manual and automated
testing, e.g. testing ksh93 using it's own test suite (47 tests) in 53
different locales, one time as plain script and one time as compiled
script (resulting in 4982 test runs), improving the test suite,
enabling libast-internal memory corruption checking (similar to what
libumem does), developing own additions to the test suite and running
the standards test suites agains it, too.
Manual testing includes basic usability exists covering languages like
Japanese and Chinese (UTF-8 and GB18030), testing whether the generated
binaries work as /sbin/sh and /usr/bin/ksh in the Indiana distribution
and whether the binaries can be used to build OS/Net.


## Part 1.2: New "typeset" variable storage qualifier for function
  static variables ("typeset -S varname")
The "typeset" builtin has a new option "-S" to declare a variable
as "static", similar to |static| variables in a ISO C/C++ code.


## Part 1.3: New floating-point datatype "hexfloat" ("typeset -X
  varname")
The "typeset" builtin has a new option "-X" to define a floating-point
variable which uses the same floating-point representation as
printf "%a" (hexadecimal floating point representation). The option
is intended to be used in cases where the values should be permanently
stored or passed to other applications without suffering the loss in
precision caused by a base2--->base10--->base2 conversion chain.
All shell facilities and C99/XPG6 conformant applications which accept
floating-point values accept the hexfloat format.


## Part 1.4: New reserved options for "typeset".
The typeset options "-T", "-h" and "-C" are reserved for future usage.


## Part 1.5: New ksh93 math functions "ceil":
Function name  | Prototype| see manual page
---+--+-
  ceil |float ceil(float) |   ceil(3m)
(note: the datatypes "integer" and "float" refer to the "integer" and
"float" datatypes in ksh93 scripts, not those in the "C" language)


## Part 1.6: New reserved builtin "enum"
The builtin command "enum" is hereby reserved for future usage.

A note will be added to the ksh93(1) manual page to suggest that shell
script authors should choose shell function outside the namespace used by
r

Fixing "touch" to reset ancient/weired/far-future timestamps... / was: Re: Switch SPARC GNU coreutils+bash from 32 to64bit[PSARC/2008/351SelfReview]

[Roland Mainz]

Joerg Schilling wrote:
> John Plocher  wrote:
> > [Roland and I are bouncing a proposal back and forth...]
[snip]
> We still miss touch(1) in the list of 64 bit binaries
> in order to be able to correct the time stamps for problematic files.

Well, I can fix that assuming we declare that a "bug fix", including the
use of isaexec (the first person who says we need an ARC case will a)
volunteer to fix the bug himself and will then b) be thrown in a pit
with komodo dragons (and I'll make sure they're small so they need
_days_ to finish the victim off)).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit[PSARC/2008/351SelfReview]

[Roland Mainz]

Garrett D'Amore wrote:
> Roland Mainz wrote:
> > Joseph Kowalski wrote:
> >> I'm sorry.  I was supposed to contact you after the last PSARC meeting.
> >>
> >> There is no serious fast-track specification for this case.  If we are
> >> unable to proceed with out one.  The project team needs to produce one.
> >>
> >> At a minimum, such a proposal needs a very clear list of the effected
> >> utilities and the rationale for the list of effected utilities.
> >
> > Does that mean I have to make an ARC case for _each_ _single_ _binariy_
> > I make 64bit clean ?
> 
> No.  Two points:
> 
> 1) Making a binary 64-bit "clean", but not delivering a 64-bit binary
> (still just delivering the 32-bit binary) is just a bug fix, and well
> below ARC radar.  You can do that anytime you like, subject to CTeam.

Erm... I don't want to do the sisiphos work of doing the fixing and then
have other people to undo them by accident. If I make the code 64bit
clean I'd like to have a "safeguard" in place that noone can easily
break it anymore... and the obvious "safeguard" is to just ship the
64bit binaries on platforms which only have a 64bit kernel anyway (which
applies to SPARC([1]) and the Solaris port which shall-not-be-named).

[1]=(SPARC is actually a good choice since it enforces the natural
alignment of datatypes which is needed for some ports (but not all, e.g.
AMD64 doesn't require this))

> 2) If you're going to deliver 64-bit binaries, you can provide a single
> case listing multiple binaries.

Technically - in the long term - this would include almost everything
delievred by OS/Net+SFWNV.

> If you're also removing (or converting)
> a 32-bit binary, please make that explicit in the case.

See above.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to64bit[PSARC/2008/351SelfReview]

[Roland Mainz]

Darren J Moffat wrote:
> Roland Mainz wrote:
> > The trouble is that this increases the workload of a simple "make
> > OS/Net+SFWNV 64bit clean by switching to 64bit binaries on 64bit-only
> > platforms" by a factor of at least _three_ (delivering twice the number
> > of binaries, do the matching testing _and_ implementing packaging
> > changes (and the testing for it)).
> 
> So is this important or not ?  If it is important then please provide a
> spec for what you propose and explain where the value is.

The value in this case would be that the Solaris userland could support
platforms which are 64bit only. At one point OpenSolaris.org&&Sun try to
encourage people to start new ports but on the other side such projects
have a very hard time since they have to implement artificial 32bit
support first (if this is possible, e.g. some hardware architectures
simply cannot support 32bit and we already lost a port in the initial
stages after the interested party realised that they have to do the
"cleanup" themselves) to get the userland running.
As a (nice) side-effect we get some abilities from the 64bit land, e.g.
high-resolution timestamps, the abilty to process more than 2GB of
memory within the application itself etc. ... but my main interest is to
get the code 64bit clean.

> Note I'm not saying I object I just sill haven't seen an actual spec and
> this is wasting your time and the time of ARC members by continuing to
> try and discuss this case without a proper spec.  Please work with your
> case sponsor to get a proper detailed spec in place and lets start with
> a fasttrack.

I'll try to build one with Plocher...

>  > This is technically ARC'able but no
> > longer implementable for a single contributor (at least not if we intend
> > to crawl over both trees and clean them up).
> 
> I don't see any reason why it isn't implementable by a single
> contributor and in any case that isn't relevant for the architecture review.

It is relevant for ARC to come up with a working architecture and _not_
force people or teams to boil the ocean in one large step (preferably I
want only to work on the _code_ and get it tested without dealing with
the package system at all (at least it's serious problem with SFWNV
which currently doesn't support incremental rebuilds and therefore
eating up engineering time (I'm trying to fix that but it will need some
time until I reach that point in my ToDo list))) ...

In the meantime I have another question: A couple of people pointed out
it may be the case that ARC doesn't even need to be involved in such a
project at all since the ARC'ed interfaces don't change (unless the
binaries are explicitly labelled as "32bit" in an ARC case (which is
usually not the case)) ...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351SelfReview]

[Roland Mainz]

Joseph Kowalski wrote:
> 
> I'm sorry.  I was supposed to contact you after the last PSARC meeting.
> 
> There is no serious fast-track specification for this case.  If we are
> unable to proceed with out one.  The project team needs to produce one.
> 
> At a minimum, such a proposal needs a very clear list of the effected
> utilities and the rationale for the list of effected utilities.

Does that mean I have to make an ARC case for _each_ _single_ _binariy_
I make 64bit clean ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit[PSARC/2008/351SelfReview]

[Roland Mainz]

Garrett D'Amore wrote:
> Roland Mainz wrote:
> > Please read the RFE again - it is needed for development of applications
> > which should support both 32bit+64bit systems (and just trying to "fix"
> > the matching build systems is hopeless - it's like trying to boil the
> > oceans. You can't fix each instance of autoconf&co.) and to handle cases
> > where only 32bit applications are suiteable for a task (e.g. if a plugin
> > is needed where the required libraries are only available as 32bit
> > binaries). Right now the only "workaround" is to become the "root" user
> > and move the matching binaries around.
> 
> I see a problem with this assertion:
> 
> Either:
> 
> 1) the application works on 32-bit, and hence doesn't need the 64-bit tools
> 
> 2) the application does not work on 32-bit and needs 64-bit tools
> 
> I recognize that there are *some* applications which perhaps can perform
> better on 64-bits, perhaps supporting larger data sets or somesuch.  But
> the fact that the application's behavior is inconsistent going from 32-
> to 64- without any obvious way for the end user to realize the cause of
> the difference gives me a bit of a pause.
> 
> I still think I'd recommend that the case be pruned back to delivering a
> /usr/bin/64/bash (or somesuch) which is *available* to consumers, but
> doesn't make it used by default.  Then, developers which *need* it right
> now at least have a recourse.  (Most likely, the most pressing consumers
> for this already know they need 64-bit support.)
> 
> Follow up case work can be done to investigate the possibility of use of
> isaexec() (or hopefully a better mechanism yet to be designed) to pick
> the best binary for the platform.
> 
> In other words, I think the project team shouldn't be afraid to tackle
> this problem by baby steps.

The trouble is that this increases the workload of a simple "make
OS/Net+SFWNV 64bit clean by switching to 64bit binaries on 64bit-only
platforms" by a factor of at least _three_ (delivering twice the number
of binaries, do the matching testing _and_ implementing packaging
changes (and the testing for it)). This is technically ARC'able but no
longer implementable for a single contributor (at least not if we intend
to crawl over both trees and clean them up).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351SelfReview]

[Roland Mainz]

Bart Smaalders wrote:
> Roland Mainz wrote:
> > Right now "isaexec" isn't that bad. However the extra |exec()| becomes
> > an issue in two scenarios:
> > 1. Tiny application with very small runtime (e.g. GNU "echo")
> > 2. Machine with many CPUs - |exec()| must tear down the address space
> > and makes crosscalls to all other CPUs... which will be a problem for
> > the 256CPU Niagara2+ machine... and even more a problem when I read
> > TheRegister.co.uk's article
> > (http://www.theregister.co.uk/2007/07/26/sun_2048_thread_niagara/) about
> > a possible 2048-core machine. 2048-1 crosscalls per |exec()| won't be
> > fun... ;-(
> 
> Cross calls limited to the set of CPUs the process has run on so isaexec
> isn't likely to cause a cross call storm unless there are some pathological
> context switching problems :-).

>From what I know there were several high-priority escalations on
SF15K/25K machines last year where Bourne/bash scripts were causing
system-wide performance problems just because they were
|fork()|+|exec()|'ing madly (three solutions: Get rid of the shells
completely, adjust the script code to avoid unneccesary subshells or use
ksh93 (which avoids |fork()|+|exec()| unless it is unavoideable (and
newer versions try to use |posix_spawn()|))) ... and from my own
experience a tight |exec()|-loop can nail large machines (e.g. M8000)
against the next available wall. It sounds that the "pathological case"
happens very often (the problem gets even worse when the application
uses largepages (but don't ask me _why_ this happens...)).

> > That won't work because you have usually more than two entries in the
> > output of /usr/bin/isalist (which is needed to support specially
> > optimized versions for some CPU types like Niagara1 (which doesn't
> > implement all VIS instructions in hardware (which makes the use of
> > -xarch=sparcv8a AFAIK "tricky")) or SPARC64 (which has extra
> > floating-point instructions not available on other SPARC platforms)).
> 
> Who uses this w/ isaexec today?  As far as I can tell on sun4v systems,
> we use sparcv7 and sparcv9, which would readily be served by the
> proposed mechanism.

I don't know whether we use it this way right now. However we _know_
that it would make sense to ship SSE3 (or which SSE*-thing had the fast
strcopy/compare operations) binaries if SSE3 is supported in hardware.
The same applies for VIS vs. non-VIS in cases like "mplayer"&co.

> > And see CR #6474270 ("isaexec and magical builds" -
> > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6474270) for
> > another reason why "isaexec" should remain "dynamically" and not turned
> > into a "static" switch.
> 
> They can always invoke the version they want directly.

Please read the RFE again - it is needed for development of applications
which should support both 32bit+64bit systems (and just trying to "fix"
the matching build systems is hopeless - it's like trying to boil the
oceans. You can't fix each instance of autoconf&co.) and to handle cases
where only 32bit applications are suiteable for a task (e.g. if a plugin
is needed where the required libraries are only available as 32bit
binaries). Right now the only "workaround" is to become the "root" user
and move the matching binaries around.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

[Fwd: Re: Switch SPARC GNU coreutils+bash from 32 to 64bit[PSARC/2008/351]

[Roland Mainz]

John Plocher wrote:
[snip]
> [I'm attaching Richard's original mail since it was only sent to
> opensolaris-arc and probably didn't get into the case mail log]
> 
> Richard L. Hamilton wrote:
> > I can think of three main categories of capacity limits on 32-bit that might
> > be higher on 64-bit:
> > * file size max of 2GB
> > * register size and usage
> > * available address space, and therefore amount that can be mmap()'d at once
> > (Also, IIRC stdio is supposed to handle > 256 file pointers on 64-bit.)
> >
> > AFAIK, only where the address space is an issue is 64-bit all that 
> > beneficial
> > on SPARC.
>   > ...
> > The file size limit can be lifted on 32-bit with large file support 
> > (lfcompile(5)).
> > The register usage can be altered in an often helpful way by compiling with
> > -xarch=v8plus.  Even the stdio problem could be circumvented (using AT&T
> > SFIO in place of libc stdio), although that may be an architectural no-no 
> > for
> > now.
> 
> Roland: Could you respond to Richards comments - it may be that by using
> large file support you could get the behavior you want - on both x86 and
> SPARC.

Largefile support wasn't the goal since 32bit applications in Solaris
may have largefile support, too. Mainly the ideas/goals were:
-- snip --
- No problem with dates >= 2030 (64bit |time_t|)
- High-resolution timestamps by default
- Stack would be non-executable by default
- ARG_MAX would be larger (twice the size), therefore allowing
applications to deal with much more data (note this extra memory is
mapped via |mmap()|-like calls and only reserves the size, not actually
allocate it (reserved address space != real memory usage)).
- "bash" supports arrays and it would be nice that we don't limit the
array content to 2GB
-- snip --

And just to log it here: The long-term goal is to make both OS/Net and
SFWNV "64bit clean", e.g. that we can support hardware which only
supports a 64bit mode and no 32bit support (for at least one Solaris
port we know that the porters were _forced_ to invent artificial 32bit
support to work aound the problem that OS/Net is not 64bit clean (which
is IMO an unimaginable abdomination (IMO this comes right after putting
kittens into the microwave))). I only wanted to start the journey
through the code and picked "GNU coreutils" and "bash" as easy targets
to begin this long-term work (but it turned-out that this issue has
poisoned teeth, claws, salvia, breathes fire and has 32bit-style heads
which only accept 64bit proposals as diet).

[snip]
> With a 64-bit system, you shouldn't be stuck with a 32-bit limit just
> because someone might have a 32 bit system that behaves differently.
> Isn't that why we developed 64 bit systems in the first place?  Isn't
> that why we are doing the OpenSolaris thing - to find and fix these kinds
> of problems?

Another issue is that OS/Net and SFWNV aren't 64bit clean which prevent
Solaris from being ported to 64bit-only hardware. The problem is _known_
since the first 64bit support was added for Solaris 7 and I _finally_
want to start a small project to crawl over the sources and kill the
problem. And IMO the _obvious_ solution to prevent this issue in the
future is to make the matching binaries 64bit by default if the kernel
is 64bit-only.

[snip]
> Last I heard, unless there was some benefit to be gained, 64-bit was usually
> _slower_ on SPARC than 32-bit (unlike x86, where all the extra registers
> are a big help), due to larger address constants meaning more data to fetch
> (and less effective use of cache?), etc.

It would be the pointer size... however the amount of data shuffeled
around for pointers depends on how the application works internally. And
in the case of "GNU coreutils" and "bash" IMO doesn't matter since a)
the difference between 32bit and 64bit is _barely_ measureable and b)
AFAIK noone cared about performance of these tools in the past
(otherwise such a person would've found a very obvious issue with the
build flags already (and it seems I'm the first one who really ran
benchmarks and tries to fix the problem))

> I can think of three main categories of capacity limits on 32-bit that might
> be higher on 64-bit:
> * file size max of 2GB
> * register size and usage
> * available address space, and therefore amount that can be mmap()'d at once
> (Also, IIRC stdio is supposed to handle > 256 file pointers on 64-bit.)
> 
> AFAIK, only where the address space is an issue is 64-bit all that beneficial
> on SPARC.

Erm... that's not correct. The ideas where more like this:
-- snip --
- No problem with dates >= 2030 (64bit |time_t|)
- High-resolution timestamps by default
- Stack would be non-executable by default
- ARG_MAX would be larger (twice the size), therefore allowing
applications to deal with much more data (note this extra memory is
mapped via |mmap()|-like calls and only reserves the size, not actually
allocate it (reserved address space != real memory usage)).
- "bash" supports arrays and it would be nice tha

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

Bart Smaalders wrote:
> Garrett D'Amore wrote:
> > My personal preference, for now, would be to deliver *both* 32- and
> > 64-bit versions of the applications for both SPARC and x86, but deliver
> > the 64-bit versions in a separate path (/usr/bin/sparcv9 and
> > /usr/bin/amd64 or somesuch), so that end users can just adjust their
> > $PATH appropriately if they know they need or want 64-bit capable
> > programs (I think for the vast majority of users, this will not be
> > necessary.)
> >
> > This has the added advantage of minimizing the ARC "impact" of this, by
> > not changing the "default" version of the tools right now.  (And doesn't
> > preclude making such a change later, particularly if at some point we
> > decide we can stop supporting 32-bit-only processors, which I imagine we
> > might do some number of years in the future.)
> 
> Why should we continue to deliver a 32 bit version of a command, if today
> we always exec the 64 bit varient?  Anything that goes through isaexec
> today should be 64 bit only on SPARC.  The fact that we still use isaexec on
> SPARC at all is a bug.

Slightly offtopic: One case was ksh93 where we _explicitly_ ship 32bit
and 64bit vesions on both SPARC and x86 via "isaexec" (even against
gdamore's counsel) because ksh93 supports a plugin API (and these plugin
may need other libraries which may only be available as 32bit versions),
ksh93's ability to handle _vast_ amounts of data in variable trees and
arrays (therefore the 64bit version being the default on 64bit
platforms) and because we want to make APIs like libshell and libast
open in the future (first for ARC contracts and then gradually
increasing the stability level) and that requires both 32bit+64bit
versions, too. However this was a _special_ case and we did lots of
research and benchmarking for this topic to make sure we don't run into
any problems.
This 32bit+64bit support on 64bit-only platforms should IMO really be
the exception and not the default.

> x86 will for now need to deliver both 32 and 64 bit binaries.  If you're
> worried
> about the performance issues involved w/ isaexec, then let's design
> something
> better but have the user adjust their path is a complete non-starter and
> we still need isaexec for the correctness cases (proc tools, etc).

Right now "isaexec" isn't that bad. However the extra |exec()| becomes
an issue in two scenarios:
1. Tiny application with very small runtime (e.g. GNU "echo")
2. Machine with many CPUs - |exec()| must tear down the address space
and makes crosscalls to all other CPUs... which will be a problem for
the 256CPU Niagara2+ machine... and even more a problem when I read
TheRegister.co.uk's article
(http://www.theregister.co.uk/2007/07/26/sun_2048_thread_niagara/) about
a possible 2048-core machine. 2048-1 crosscalls per |exec()| won't be
fun... ;-(

> As an example, to replace most instances of isaexec, we could create
> /usr/bin/isaexec
> as a mount point, and then during boot mount /usr/bin/i86 or
> /usr/bin/amd64 on
> /usr/bin/isaexec as a loopback mount.  We also change the isaexec
> hardlinks in /usr/bin to
> be symbolic links pointing to /usr/bin/isaexec/{cmdname} no more
> runtime isaexec
> overhead and no fiddling w/ user paths.

That won't work because you have usually more than two entries in the
output of /usr/bin/isalist (which is needed to support specially
optimized versions for some CPU types like Niagara1 (which doesn't
implement all VIS instructions in hardware (which makes the use of
-xarch=sparcv8a AFAIK "tricky")) or SPARC64 (which has extra
floating-point instructions not available on other SPARC platforms)).
And see CR #6474270 ("isaexec and magical builds" -
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6474270) for
another reason why "isaexec" should remain "dynamically" and not turned
into a "static" switch.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

Garrett D'Amore wrote:
> Bart Smaalders wrote:
> > Darren J Moffat wrote:
> >> I'm derailing this automatic approval.  I want at least a fast-track
> >> that explains why the asymetry between SPARC and x86 is actually
> >> useful in addition to what Joe asked for.
> >
> > Because we support x86 machines that don't implement 64 bit support?
> > The same reason we ship a 32 bit and a 64 bit x86 kernel?

But |isaexec()| won't help you since it cuts stuff like |ARG_MAX| to the
32bit limit. And the original idea was to do the change _only_ for
machines which are native 64bit platforms (e.g. SPARC and two Solaris
ports which shall-not-be-named here) and not to try to boil the ocean by
enforcing the change for all platforms.

> >> I think this should actually be a full case rather than a fast-track
> >> given this would be a significant architectural change.  As Joe said
> >> the GNU part is irrelevant here what is relevant is the change in
> >> direction.
> >
> > Whether a particular executable is delivered as a 32 bit binary, 64
> > bit binary
> > or shell script/java/python seems more of an implementation detail
> > rather
> > than an architecture change.
> >
> > I assume that the proposal here is to deliver 64 bit versions of these
> > commands
> > on sparc, and both 32 and 64 bit versions on x86.
> >
> > My question here is whether there is a plan about what to do w/ 32bit
> > only (x86) systems in regards to whatever limit this change avoids
> 
> I think the above question (the behavioral difference) is one that
> really needs to be addressed.  It may be that the answer is that we
> don't care about the limit on 32-bit only systems, but what are we going
> to do for 64-bit-capable x86 boxes?  I am pretty unhappy with the
> current isaexec() solution  - the extra fork/exec combo seems
> "unfortunate", particularly for utilities that might be executed heavily
> in embedded scripts.  (The shell, and coreutils, seems to fall into this
> category.)

Please remove the shells from the list. As I explained earlier (and in
the original ksh93-integration ARC case, and via IRC, and via email
etc.) the extra |exec()| is usually (except for cases like $
/usr/bin/ksh93 -c 'true' #) canceled-out when the shell uses builtin
commands or avoids calling |fork()| for subshells (that was at least the
case of ksh93). "bash" may slightly different but it's still something
which will IMO not hurt compared to _other_ stuff in "bash" which
currently goes badly wrong (e.g. default build options). Just by fixing
that we win more than you loose via "isaexec".

> My personal preference, for now, would be to deliver *both* 32- and
> 64-bit versions of the applications for both SPARC and x86, but deliver
> the 64-bit versions in a separate path (/usr/bin/sparcv9 and
> /usr/bin/amd64 or somesuch), so that end users can just adjust their
> $PATH appropriately if they know they need or want 64-bit capable
> programs (I think for the vast majority of users, this will not be
> necessary.)

Groan... and noone will find these binaries. And /usr/bin/${MACH64}/ is
not an ARC'ed interface either...
... what about a compromise: Use 64bit versions of these utilities on
64bit-only kernels by default and use /usr/bin/64/ on platoforms like
Solaris/x86. Would that be acceptable for you?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

Nicolas Williams wrote:
> On Mon, Jun 02, 2008 at 12:14:12PM -0700, Garrett D'Amore wrote:
> > Yes, isaexec is needed for correctness, but its not used on any
> > performance critical paths, and remains, IMO, an ugly hack rather than
> > an elegant solution.
> 
> isaexec handling could move into the kernel if it had to to avoid the
> additional exec().

|isaexec()| cannot be moved into the kernel unless you can implement
CR#6474270 ("isaexec and magical builds" -
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6474270) from
the kernel, too.

[snip]
> > (I'm also looking for a way to allow this case to proceed forward,
> > without requiring it to block on a larger set of architectural problems
> > that the project team is probably ill-equipped to solve.)
> 
> It's a full case now.

And I disagree. The original idea was to get rid of some limits in some
tools, not boil the ocean or start becoming a politician (and half of
the emails I got via PM strongly indicate that the mutations on top of
the original idea are political issues which should be handled via
ogb-discuss at opensolaris.org and not PSARC).

> > (As far as "something better", I think I prefer something along the
> > lines of how AFS  handles it, where $VARIABLE could be used within a
> > symbolic link -- the kernel expands the variable when following the
> > symlink.  But we shouldn't try to design the solution here, I think.)
> 
> The ELF exec handler in the kernel could find the conditional logic to
> evaluate and tokens to expand, and just do it.  This would be much safer
> than modifying the semantics of symlinks.  And isaexec would remain, but
> the extra exec() would be avoided.

As said it wouldn't work (see above). And the |exec()| overhead for
_shell_ interpreters like "ksh93" is usually canceled out by using
builtin commands and avoiding that subshells |fork()| which more or less
eliminates gdamore's complains about isaexec in this case (and "yes" -
we did benchmark that). For "bash" it may be slighty different since it
|fork()|+|exec()|s a lot more than ksh93 but I still think it won't harm
users a lot.

> But one step at a time.  This smacks me as premature optimization (even
> though the extra exec() is wasteful -- the point it is that optimizing
> isaexec now is premature for this discussion).

Right. And please don't implement anything which prevents us from
implementing CR#6474270 ("isaexec and magical builds").



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Full case ? / was: Re: Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

Nicolas Williams wrote:
> On Mon, Jun 02, 2008 at 09:40:46PM +0200, Roland Mainz wrote:
> > Nicolas Williams wrote:
> > > On Mon, Jun 02, 2008 at 12:14:12PM -0700, Garrett D'Amore wrote:
> > [snip]
> > > It's a full case now.
> >
> > Who or what decided that ? IMO this still fast-track.
> 
> Darren Moffat derailed it.

And I disagree with Darren. Originally his was a simple engineering
issue until other people mutated the RFE. The original idea to get some
improvements (by shipping GNU coreutils and "bash" as 64bit applications
by default on 64bit-only platforms (which means SPARC and two Solaris
ports which shall-not-be-named right now)), including:
-- snip --
- No problem with dates >= 2030 (64bit |time_t|)
- High-resolution timestamps by default
- Stack would be non-executable by default
- ARG_MAX would be larger (twice the size), therefore allowing
applications to deal with much more data (note this extra memory is
mapped via |mmap()|-like calls and only reserves the size, not actually
allocate it (reserved address space != real memory usage)).
- "bash" supports arrays and it would be nice that we don't limit the
array content to 2GB
-- snip --

Please tell me why this requires a FULL ARC case. This isn't normal
anymore.

Darren: Can you please revoke the derail ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Full case ? / was: Re: Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

Nicolas Williams wrote:
> On Mon, Jun 02, 2008 at 12:14:12PM -0700, Garrett D'Amore wrote:
[snip]
> It's a full case now.

Who or what decided that ? IMO this still fast-track.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

Alan Coopersmith wrote:
> Joseph Kowalski wrote:
> > I don't see why /usr/gnu/bin/awk should get preference over /usr/bin/awk
> > just because its in gnucore.
> 
> How about because it's already known to be 64-bit clean since the GNU
> utilities run on platforms that are 64-bit only, while the Solaris
> versions may or may not be ready to go?

OS/Net utilities are definately not 64bit clean (nor are the OS/Net
build tools) which forces Solaris ports to always implement 32bit
support (whether they want it or not). And the issue is known since at
least when the 64bit support was developed for Solaris 7 and nothing was
done yet to fix it.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

James Carlson wrote:
> Joseph Kowalski writes:
> > John Plocher wrote:
> > > Joseph Kowalski wrote:
> > >> Sorry, I want a fast-track.
> > >
> > > What are the architectural (as opposed to C-Team, Design or RE) issues?
> > >>
> > >> 1)   I believe this is an incomplete project.
> > >
> > > This sounds to me like "go boil the ocean" scope creep.
> >
> > Sometimes the burners need to be turned on.
> 
> A less dramatic (but just as effective) answer would be to work out
> the details of a new Big Rule: "as only 64-bit kernels are supported,
> all new utilities should be compiled as 64-bit objects on SPARC;
> existing utilities should be transitioned when practical."

Technicially this was the original long-term goal for SPARC and the
well-known Solaris port to stuff in a build flag in usr/src/Makefile.sfw
which says "kernel is always 64bit so build tools with this flag 64bit
only".



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

> Joseph Kowalski wrote:
> John Plocher wrote:
> > Joseph Kowalski wrote:
> >> Sorry, I want a fast-track.
> >
> > What are the architectural (as opposed to C-Team, Design or RE)
> > issues?
> >
> >> 1)   I believe this is an incomplete project.
> >
> > This sounds to me like "go boil the ocean" scope creep.
> 
> Sometimes the burners need to be turned on.
> 
> > Roland has identified several utilities that would benefit from
> > being 64 bit on 64 bit OSs, and is willing to do the work to
> > "convert" them.
> 
> OK.
> 
> Then he (you) should do a "self review" for those specific items.
> 
> I don't see why /usr/gnu/bin/awk should get preference over
> /usr/bin/awk
> just because its in gnucore.

Because the OS/Net codebase is in a _much_ worse shape than GNU
coreutils and "bash". GNU coreutils and "bash" are 64bit clean while the
OS/Net utilities are not. This is the reason why a well-known Solaris
port is _FORCED_ to implement artificial 32bit support on 64bit-only
hardware. This can be cleaned-up, too - but only as an official project
with full-time engineers (e.g. it's far beyond a simple weekend work).
I'm technically willing todo it if someone can convince manangment that
this is the right way to go and resources should be invested to make
OS/Net 64bit clean.

[snip]
> >> 2)   We have a 32-bit user land (only) because 32-bit utilities on
> >> SPARC
> >>  weren't any faster than 64-bits (so why support both?).
> >
> > In this case, the 32 bit versions have capacity limits that are
> > affecting Roland and he wishes to remove those limits.  This
> > is not at all a performance issue, but a make it work better one.
> 
> Back to my awk example, if the problem is in /usr/gnu/bin/awk,
> it seems very poor form (and a source of "huh?" bugs) to not make
> the equivalent change to /usr/bin/awk.

As said I am willing to do it, assuming I am assigned to do that work
(estimated time for the basic tools in usr/src/cmd/ is one man-year (one
engineer working a year on making it 64bit clean+testing)). I really
need to get money for food, wife, baby etc. and it doesn't happen for
doing non-official work the whole time.

> Then again, since we can't make these changes for x86, these
> seems like an even worse idea - "CR 699: awk works on sparc,
> but not on x86".
> 
> Anyway, I've asked that this become a fast-track.  Please make
> it so.

I don't see a reason for this.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

Switch SPARC GNU coreutils+bash from 32 to 64bit [PSARC/2008/351Self Review]

[Roland Mainz]

Joseph Kowalski wrote:
> John Plocher wrote:
> > Template Version: @(#)sac_nextcase 1.66 04/17/08 SMI
> > This information is Copyright 2008 Sun Microsystems
> > 1. Introduction
> > 1.1. Project/Component Working Name:
> >Switch SPARC GNU coreutils+bash from 32 to 64bit
> > 1.2. Name of Document Author/Supplier:
> >Author:  Roland Mainz
> > 1.3  Date of This Document:
> >   30 May, 2008
> > 4. Technical Description
> >
> >   gisburn: Does it require an ARC case if we switch the GNU coreutils
> >from 32bit to 64bit on SPARC only (no changes in interfaces 
> > and
> >delivered files)?
> >   plocher: It doesn't sound like this has any architectural 
> > implications...
> >   gisburn: Not that I am aware of.
> >   plocher: so, I'd say that this was the entire arc review  :-)
> >
> > 6. Resources and Schedule
> > 6.4. Steering Committee requested information
> >   6.4.1. Consolidation C-team Name:
> >   SFW
> > 6.5. ARC review type: Automatic
> > 6.6. ARC Exposure: open
> 
> Sorry, I want a fast-track.

Could you please write a two-page justification with minimum 2000
words, please ?

Actually I hoped to get this done this weekend - which will be likely
the only free time to get it done in the next couple of weeks. If it
can't be done this way then the likely only course of action is to
withdraw the case since noone can work on it (because there is no time
and the nerves of the person who actually wanted to work on it are
completely ruined and likely noone will assign official engineering time
to such an "unimportant" project (which is together with the build flag
cleanup and the fixing of the dbx -check access bugs in coreutils mainly
"janitor" work)).

BTW: A while ago there was the question: Why is Sun so slow with
adopting changes for packages or adding new stuff to SFW ? Maybe one
part of the answer is: Because even small things are discussed to death,
hell and beyond (just an experience of the other ARC case where I had to
write 229 emails, two hours of IRC discussion and two phone calls,
effectively ruining two working days). For Linux they just look at the
change, document it and DO IT (and that's the reason we're using
"auto-approved+automatic" for this ARC case).
 
> 1)   I believe this is an incomplete project.  What's good for GNU coreutils
>   is also good for the rest of the utilities.  The source of the
> utilities doesn't
>   matter.
> 
>   Project completeness is always an arc concern.

Why do you want that I want to do _all_ utilities in one piece ? It's
technically impossible to do so for a volunteer, even for me. I can
handle the utilities (including testing, code review, RTI etc.)
step-by-step but certainly not all in one step.

> 2)   We have a 32-bit user land (only) because 32-bit utilities on SPARC
>   weren't any faster than 64-bits (so why support both?).  Since we have
>   unsupported any 32-bit SPARC hardware, the change could be made
>   now.  However, validation needs to be done that this doesn't result in
>   performance regression.  (Should be faster on an x86 - its all about
>   the number of registers, not the data path.)  This is a C-team (PAC?)
>   issue, but it should be considered.

We had that discussion already. The issue is: Most of the tools in GNU
coreutils are so small that the statistical noise generated by the
|fork()|+|exec()|-overhead is bigger than any regressions in
performance. From ksh93 we already know that the difference is barely
measureable on an Ultra1/143MHZ machine running Solaris 8... and somehow
I doubt it changed a lot for Solaris 11. And IMO the benefits of having
updated build flags (e.g. C99/XPG6 conformance), having no 256FD limit,
a larger ARG_MAX and other increased limits greatly outwheights any
concerns about statistical noise in the performance data (I've tested so
far "bash" and it's not worse than ksh93). However I don't have time to
test each indivisual tool for performance data - I can do the test suite
tests and manual testing required for putback - but doing anything
beyond requires to make this a fully-blown internal project (for which
there aren't engineering resources).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 Integration Update 1 Amendments1[PSARC/2008/344FastTracktimeout06/03/2008]

[Roland Mainz]

> Joseph Kowalski wrote:
> 
> >From David Korn, as referenced by Garrett, forwarded by Roland.
> 
> > I expect that the new features are subject to change
> > based on feedback (bugs and ideas).  Once there has
> > been enough feedback that I feel comfortable I remove the -
> > suffix.
> >
> That sounds like a resounding "yes" - new features can be altered
> in the "t-" to "t" transition.
> 
> (BTW: I wouldn't expect it to be any other way.  We make the
> same allowance for Beta releases.  We just shouldn't be placing
> components with Beta [non-]guarantees in potentially FCS
> products.)

Note that this is a general statement. To check "yes" for _this_ case
covering ksh93 version "t-", "t" and "t+" two things must be "true":
1. We need a good idea for adding new stuff (we don't have such a thing
right now, nearly all effort mainly goes into the new type system,
bugfixing and hunting down global variables as preparation for thread
support etc. and the contentious thing - the "enum" builtin - was the
_first_ one since 15 years)
2. Somehow magically a large chunk of "free time" needs to fall out of
the heaven, crawl out of hell or is somehow else put into existance
(Sauron's ring is unfortunately lost... ;-( )

Since neither [1] or [2] are true we can mark the "no" checkbox for this
time and move along, right ?



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 Integration Update 1 Amendments1[PSARC/2008/344FastTracktimeout 06/03/2008]

[Roland Mainz]

Joseph Kowalski wrote:
> Garrett D'Amore wrote:
> > I've seen mail from David Korn (not CC'd to PSARC, unfortunately)
> > which I think cleared this up unambiguously.  Check the "no" box, and
> > lets move forward.  :-)
> Can we forward this to PSARC (actually the case, so it is captured)?
> 
> Then we can truly move forward

I've attached his email to this one...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)
-- next part --
An embedded message was scrubbed...
From: David Korn 
Subject: Re: [ksh93-integration-discuss] 2008/344 [ksh93 Integration Update 
1
Amendments 1]
Date: Thu, 29 May 2008 17:17:01 -0400
Size: 5333
URL: 

PSARC 2008/261 EOF: Sound Blaster Pro driver sbpro

[Roland Mainz]

Garrett D'Amore wrote:
> Roland Mainz wrote:
> > Garrett D'Amore wrote:
> >> This case was approved yesterday.
> >
> > Erm... did you check whether this driver may be needed by software
> > emulators ? For example the only sound driver which AFAIK works in Bochs
> > with Solaris as guest is "sbpro" (see
> > http://www.koders.com/cpp/fid1239096CB8FECB28659D1411619E101856D78C07.aspx
> > ; and did you check "qemu", "VMware", "Parallels" etc. yet ?).
> 
> Yes, for some of these, sbpro is required.  But the mainstream VMs have
> moved on to newer emulations:
> 
> Notably:
> 
> * VirtualBox: emulates AC'97
> * Parallels: emulates AC'97
> * VMware: emulates AudioPCI, audio not supported on Solaris today -- may
> add AudioPCI driver for it in the future
> * VirtualPC: Microsoft product, does not support Solaris (at least
> officially)
> * qemu: May require sbpro or audiopci -- some evidence that it may also
> support ac'97.  (Vanishingly small market share.)

Erm... we have a QEmu projet at opensolaris.org - did you ask them
whether they like the removal ?

> * bochs: Does anyone actually *care*?  It requires OSS in any case, so
> would not represent a regression.  (Notably the sb16 emulation in bochs
> is very close to unusable due to heave CPU requirements.)

That's why I added --enable-idle-hack to Bochs long ago to deal with the
CPU hogging, together with all the other fixes to make Solaris bootable
on Bochs (e.g. SPARC as host, x86 as guest) ...



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

PSARC 2008/261 EOF: Sound Blaster Pro driver sbpro

[Roland Mainz]

Garrett D'Amore wrote:
> 
> This case was approved yesterday.

Erm... did you check whether this driver may be needed by software
emulators ? For example the only sound driver which AFAIK works in Bochs
with Solaris as guest is "sbpro" (see
http://www.koders.com/cpp/fid1239096CB8FECB28659D1411619E101856D78C07.aspx
; and did you check "qemu", "VMware", "Parallels" etc. yet ?).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)

[ksh93-integration-discuss] ksh93 Integration Update 1 Amendments1[PSARC/2008/344FastTracktimeout 06/03/2008]

[Roland Mainz]

Garrett D'Amore wrote:
> John Plocher wrote:
> > I think it is time for an updated spec/issue roll up.
> >
> > Here is where I think things stand - please correct any
> > misunderstandings:
[snip]
> >   2) Relationship of "t-, t, t+" versioning scheme and C-Team
> >  integration rules.
> >
> >  Commentary:
> >   The upstream convention seems to be "no incompatible changes",
> > "run API conformance test suites rigorously to ensure no
> > regressions" and "moderate new feature additions based on
> > [- +] release state"
> >
> > The boogie man is whether or not incompatible changes would show
> > up in a "-" release, or if bugs would show up there that would
> > cause equivalent breakage.  Given the external team's almost
> > obsessive focus on compatibility, I for one am not concerned
> > about this at all.
> >
> >  Resolution:
> > Not a problem.
> 
> Agreed.  Can the project team confirm your statement that no other such
> additions are planned?

As said on IRC: For ksh93 version "t" and "t+" there won't AFAIK be any
new keywords. Beyond that we don't have any plans for new keywords right
now, however I can't rule it out that we may come up with new ideas in
the next couple of years. Note: _YEARS_. For ksh93t we're pretty much
occupied with getting bugfixes and testing done (which includes many
sleepless nights and I even turned my current main work related to
automating license extraction into a giant ksh93 testbed (erm...
technically the only two suiteable languages with efficient variable
tree support were JAVA and ksh93) to find more bugs (as a result we
developed testcases like
http://svn.genunix.org/repos/on/branches/ksh93/gisburn/scripts/tests/test_vartree002.sh
or
http://svn.genunix.org/repos/on/branches/ksh93/gisburn/scripts/tests/test_staticvariables.sh
to hunt down the remaining bugs)).

And before someone complains: The reserved "-T" and "-h" flags for
"typeset" (marked as "reserved for future usage" in this case) are for a
new type system, e.g. user-defineable types. It's there right now in the
shipping ksh93 binary and the update will stabilize it but we are going
to ARC that later in detail (likely when ksh93 version "t+" comes out),
not now (and it won't add new keywords and the matching flags for
"typeset" are "reserved" in this ARC case).

> >   3) Creation of /usr/lib/shell as Project Private
[snip]
> > Create /usr/lib/shell as a Volatile playground with the
> > following initial expectations:
> >
> > A place to bundle shell function libraries _toegether_ in
> > one common base directory, have them share functions
> > via ${BASEDIR}/sh/ if the functions are in the POSIX
> > shell syntax and in interpreter-specfic (e.g.
> > ${BASEDIR}/zsh/..., ${BASEDIR}/ksh/...) if they use
> > extended syntax... and handle the modules (one module
> > may contain multiple functions) and functions in a
> > DNS-like hieracy and allow pattern to be used as
> > selectors.
> >
> > The intent is that shells will contain a builtin
> > variable which points to the base directory so that
> > consuming shell scripts do not need to know the
> > absolute location where the library files are stored.
> 
> I would prefer to have the above slightly better specified... including,
> what the variable name is, and how consumers will be able to use this.
> (Including perhaps some examples of the hierarchy, and references to
> what the file content is -- parsed scripts, precompiled shellcode,
> shared objects, what?)

The issue is that this is a new idea and I've crafted-up such details...
but if we ARC that now we just end-up with a theoretically perfect
design which turns out to be a pain to use in real-world applications.
That's why I had the idea to make it "private" (as a lesson learned I'll
ask John Plocher next time to check an ARC case before posting it - it
seems getting the interface stabilty level wrong ("Private" vs.
"Volatile") quickly leads to hell and beyond...)) - to make sure we can
test the design in the real world and make adjustments and _then_ ARC
it.
I don't want to end-up with a design like SMF's UTF-8 property strings -
which are _theoretically_ a good idea (I'm explictly excluding the
Unicode issues around Han unification (which could lead to the
conclusion that using UTF-8 for property storage in SMF wasn't a good
idea either)) until you realise that adding UTF-8 data to non-UTF-8
encoded multibyte data results in an invalid character sequence which no
utilty can read at character level (therefore using any property values
with charcters outside ASCII is currently impossible with SMF - the
consuming tools/shell scripts just choke and collapse when they try to
deal with it) ... perfect theoretical design, zero practical value.

> Also, how will we get things like zsh or bash to
> use these?  (See David Comay's response and commentary on zsh's

[ksh93-integration-discuss] ksh93 Integration Update 1 Amendments1 [PSARC/2008/344FastTracktimeout 06/03/2008]

[Roland Mainz]

John Plocher wrote:
> Chris Pickett wrote:
> >>  Last but not least what is a shell ? would /usr/lib/shell/python be
> >>  allowed ?
> >
> > Don't. Be. Silly.
> > http://en.wikipedia.org/wiki/UNIX_shell
> >
> > Unless python starts calling itself a shell it isn't one.
> 
> What "libraries written in a scripting language" have to do
> with "shell" is exactly the point.  Python is a scripting
> language, it has reusable libraries, people are reputed to
> have written useful things in it, and, like Tom Christianson
> and Perl, I'm sure someone has actually tried using the Python
> interpreter as their login shell at least once :-)
> 
> Given Roland's suggested usage model (a place to put compiled
> ksh93 scripting language libraries for use by others), the name
> he chose for the directory (.../shell) seems to be too generic,
> and thus out of place.

Point was: Bundle the shell function libraries _toegether_ in one common
base directory, have them share functions via ${BASEDIR}/sh/ if the
functions are in the POSIX shell syntax and in interpreter-specfic (e.g.
${BASEDIR}/zsh/..., ${BASEDIR}/ksh/...) if they use extended syntax...
and handle the modules (one module may contain multiple functions) and
functions in a DNS-like hieracy and allow pattern to be used as
selectors. 

Why is this /usr/lib/shell/ directory _that_ difficult_ to get via
ARC ? Somehow I'm starting to regret to have exposed too much detail and
now we're _bickering_ (sorry for that word but I don't know any better
word for it) the details down to a sub-atomic level to hell.



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.mainz at nrubsig.org
  \__\/\/__/  MPEG specialist, C&&JAVA&&Sun&&Unix programmer
  /O /==\ O\  TEL +49 641 7950090
 (;O/ \/ \O;)