David.Comay at Sun.COM wrote:
>
> >> 1. How exactly are the Mamfiles used? Or is this just files
> >> included with the ATT distribution for their build system.
> >
> > Yes, it's coming from the upstream sources... remember we don't want to
> > fork() the upstream sources and that includes random stripping, file
> > deletion, renaming of binaries just because someone doesn't like that
> > name (we had that discussion for "shcomp" a while ago) and all the other
> > "funny" ideas what could be done to the codebase (like making all
> > upstream sources "cstyle" clean etc.) ...
>
> I was merely asking what these files are and not asking for them to be
> changed, removed, etc. In general for ON, we don't like to see files
> checked in which serve no purpose in the build process unless they're
> there to document something about the source (for example, a README
> which explains the source layout).
In the case of the Mamfiles it's IMO "documentation". We check-in these
files and keep them updated to have a reference point for the matching
OS/Net Makfiles. I know that this is not perfect but it's at least
economical for those who work on the current tree (e.g. April and me).
Currently we can deploy updates within less than an hour (excluding
testing and a full OS/Net build) and randomly removing files will only
cost more time since the more or less automated procedure will no longer
work and then patches need to be applied by hand which takes far longer.
> I'm not trying to reopen this ON-versus-SFW debate here but the
> discussion around these files is precisely because you're trying to
> integrate this into ON. Consolidations such as SFW work much more in a
> pass-through mode - the source comes from upstream, undergoes a small
> amount of change (if any at all) and then is compiled and packaged via
> some sort of mechanism (perhaps a Sun-supplied master Makefile and the
> open source project's own ./configure script).
Grumpf... part of the decision to put ksh93 into OS/Net is that it is
(or better: should be) the successor of the old korn shell which was in
OS/Net, too. We'd like to run the work under the same rules to make sure
noone can throw the "(shell-)flavor of the day"-stones after us once we
run the ARC case for the "/usr/bin/ksh to ksh93"-migration.
> >> 2. I agree with Jim Carlson about the setting of SHELL in the
> >> various ksh-related Makefiles. At this time, I think using the
> >> crufty old /bin/sh is the right thing to use for consistency,
> >
> > "Consistency" is IMO no good argumment in this case. We use ksh to
> > simplify things and make the code smaller and better understandable.
> > IMHO a clean design using ksh is better than a painfull and unreadable
> > amount of hacked bourne shell code garbage.
>
> That certainly is a good argument if bourne shell implementation is
> unreadable. But there seem to be some Makefiles where I didn't see any
> ksh93 constructs at all, or minimal ones at that. For example, why the
> changes to libcmd's Makefiles to use /bin/ksh?
Because libcmd's Makefile "include"s things like "Makefile.libastl10n"
and "Makefile.astinclude". I know that this functionality could be
re-done using SHELL=/bin/sh which then calls scripts which use
"#!/usr/bin/ksh" and do the same job - the question is whether this is
really neccesary when the same could be done in the Makefile's itself
with less effort.
BTW: Somewhere in my Drafts/ folder is now an email to gatekeepers to
outline why I'd like to establish ksh93 as 2nd shell option (only as
second and I don't like to introduce things like SHELL=/usr/bin/csh or
SHELL=/usr/bin/bash later, don't worry...) in OS/Net...
> >> unless some compelling reason can be shown where a ksh (or
> >> ksh93) construct greatly simplies things.
> >
> > At least the Makefile.testshell part won't work without ksh and the
> > include files and l10n generation depends on a lesser extend on ksh
> > (which means that all Makefiles in
> > usr/src/lib/lib(shell|cmd|dll|pp|ast)/ depend on it). Both could be
> > re-implenented using the plain bourne shell at the expense of making the
> > code really difficult to understand and unreadable (I already agreed to
> > adjust the directory creation in usr/src/lib/libast/ at the expense of
> > cleanness in the code and I am not happy about it (it's less
> > understandable and slower) ... ;-( ).
> >
> > I would really strongly perfer to keep SHELL=/bin/ksh - it may even
> > provide the "proof" that we can (later) to a tree-global switch to a
> > POSIX shell without causing any harm.
>
> For the ksh93 specific Makefiles, I'm OK with leaving that in although
> my suggestion is to only put it in where you're actually required to
> use ksh93.
Erm... there is a problem with that. Right now there is no "ksh93" in
Solaris (excluding dtksh which lacks repairs and maintaince since at
least a decade (e.g. it is still based on ksh93 version 'd' (even an
alpha version of that which IMO should never be used on a production
system))), that's why we use /usr/bin/ksh instead for now (as
intermediate step) and do the switch to /usr/bin/ksh93 once ksh93
becomes available (e.g. likely with the patch which enables the l10n
message catalog generation again). Together with the switch to
SHELL=/usr/bin/ksh93 some of the (ksh93) Makefiles get adjusted to use
new features (like string concatenators etc.). Therefore we cannot
provide a full "precedent" in this case - right now we can only show
limited usage of SHELL=/usr/bin/ksh (including "POSIX shell" style
syntax and some shell builts)
> >> 3. With respect to the *.so library links and the the lint
> >> library, I think including these in an internal package is fine
> >> but I do not believe they should be actually included in any
> >> metacluster at this time. From my understanding of the
> >> contents of the SUNWastdev packages, that also holds true for
> >> it as well.
> >
> > Erm, the primary purpose of the "SUNWastdev" package is to deliver AST
> > development tools to /usr/ast/bin/. It was not intended to become a
> > dumping ground for everything.
>
> Thanks for the explanation. I went back to PSARC 2007/035 and it does
> seem that this package should contain just the components presented in
> that case.
Did you read the explanation what the "SUNWastdev" package is used for
in the future ?
> But I do have one question for SUNWastdev. If the *.so library links
> for libshell, libast, libdll, and libcmd (and libpp, for that matter)
> are not supplied, is this package actually usable by anyone? Let's
> leave out the ON developers but rather focus on the end-developer who
> is wishing to use the AST development tools. Do these tools depend on
> linking against the currently project private libraries?
Yes, they link against libast and some against libpp.
> Put another
> way, should this package me an internal-only package right now because
> the end-developer cannot use it given the commitment level that was
> ARCed?
What do you mean with "internal-only package" ? And see my question
above about the future target of the package (like the other ksh93 and
AST components we'd like to put them into their correct places from the
beginning... anything else (like moving files etc. around) will just
generate more paperwork which will consume much time (at least I am a
volunteer and (unfortunately) have to work for food on other stuff
(which means I don't like to spend time on paperwork unless it's really
mandatory))).
> >> When the commitment level of these things are
> >> raised in the future, then it makes sense to include them in a
> >> package like SUNWcslr, SUNWarc, a metacluster, etc.
> >>
> >> Please understand that this isn't a criticism or knock against
> >> ksh93 - the same is true of internal tools, libraries, etc as
> >> well.
> >
> > I don't take this as "criticism" or "knocking against ksh93" (nor do I
> > think that you or James have anything against ksh93 etc.). I know the
> > rules... but I was hoping for some kind of "escape route" since many
> > people are curious and like to play around with the new libraries.
> > Unfortunately the rules do not seem allow some kind of curiousity... ;-(
>
> People who want to explore their curiosity in this space can do so by
> downloading the ON source tree and compiling their programs within
> there. But it does a disservice to other using the end-user
> distribution if they begin using those interfaces and then they change
> in an incompatible way.
I wish the rest of the world would share that view... ;-/
> >> And in those cases, have you added the
> >> appropriate Copyright to each of those files?
> >
> > What do you mean with "appropriate Copyright" (note: I am not a
> > lawyer... (and usually I try to hide under my desk when the license
> > flamewars start...)) ? AFAIK we do not need (and should not need) to add
> > any CDDL license to the AST codebase since we have a permission to
> > contribute any Solaris code to upstream (Mike Kupfer can explain that
> > AFAIK better) under their own license ("CPL") which AFAIK covers any
> > tree-local modifications. The new tests and demo code has a CDDL
> > license.
>
> I know you don't want to deal with licenses so my suggestion is to
> contact Bonnie Corwin and explain precisely what files you're changing
> and in what way. It's my understanding that the usual thing is
>
> Don't add a Sun copyright to upstream files that you're not
> changing.
I am not doing that (such a change would be listed in the
usr/src/lib/libshell/misc/*.diff files).
> Do any a Sun copyright to any files being created that aren't
> coming from upstream (for example, OpenSolaris specific
> Makefiles.)
AFAIK that's already done.
> Do add a Sun copyright to files which have a significant
> change. The definition of the latter is sometimes hard to
> define but changing it to make it compile or work under
> OpenSolaris clearly falls in the "add" category. In any case,
> please check with Bonnie.
What about a file where we added an |#include|-statement to include
another (CDDL-)licensed chunk of code ? The |#include|-solution was used
to avoid the situation that we have to add a license template to the
AT&T sources...
> >> usr/src/lib/libast/amd64/Makefile
> >>
> >> Line 31 - Is there a reason you're only using the minor part
> >> here rather than the whole $(RELEASE) string? Is there some
> >> standard here that ksh93 uses with different OS versions?
> >
> > See may reply to James Carlson's email
> > (http://mail.opensolaris.org/pipermail/ksh93-integration-discuss/2007-February/002225.html).
> > Basically the upstream code uses this as a platform identifer and we use
> > the same algorithm to provide the same value in this case (the code
> > doesn't build without a value and I don't want to pass a hardcoded
> > value).
>
> I saw the response but it still sems short-sighted on including part of
> the `uname -r` value but not the rest.
Erm, what should I do in this case ? We can't really add a random value
here. I am just emulating the behaviour of the upstream build (without
hardcoding values).
> What happens if we ever produce
> a SunOS 6.0?
I hope you'll call it "Solaris" then and drop the SunOS part (yes, yes,
I know... SunOS = core OS, Solaris = whole product) ... :-)
> Will ksh93 identify itself then as "sol0.i386", etc?
Uhm... I guess it will end-up in such a situation - but in that case
we'll have to do some porting work in the upstream sources anyway and
this would give us the opportunity to solve this riddle, too... :-)
BTW: If you do something like Solaris 6.0 please make the tools in
/usr/bin/ at least XPG6 conformant... and read
http://svn.genunix.org/repos/on/branches/ksh93/gisburn/todo_list_gisburn.txt
, section "OpenSolaris-related ToDo items" (somewhere there is another
"rants+ToDo" list with more items).
> >> usr/src/lib/libcmd/common/mapfile-vers
> >>
> >> This is showing up only under "Old". Are you removing it?
> >
> > "mapfile-vers" should be in the libraries's base directory to avoid that
> > this may get lost during source updates. The common/ subdir should for
> > the upstream sources only (only exception is the additions to the demo
> > code and test suite (e.g. the test to watch over the getconf
> > compatibilty (usr/demo/ksh/tests/sun_solaris_getconf.sh)) ).
>
> I didn't explain my question well. Are you planning on moving
> usr/src/lib/libcmd/common/mapfile-vers to
> usr/src/lib/libcmd/mapfile-vers and updating the file? Perhaps this is
> a question for April since it's a Teamware operation.
Erm, yes. I am following the suggestion of Roger Faulkner (if I recall
the name correctly) who suggested to put the files in the base
directory. At least the following libraries do it the same way (this is
from our B51 tree):
-- snip --
libast/mapfile-vers
libavl/mapfile-vers
libcmd/mapfile-vers
libdevice/mapfile-vers
libdevid/mapfile-vers
libdevinfo/mapfile-vers
libdll/mapfile-vers
libdscp/mapfile-vers
libgss/mapfile-vers
libipp/mapfile-vers
libldap5/mapfile-vers
libnisdb/mapfile-vers
libnvpair/mapfile-vers
libpam/mapfile-vers
libpicl/mapfile-vers
libpicltree/mapfile-vers
libpp/mapfile-vers
librcm/mapfile-vers
libresolv/mapfile-vers
libshell/mapfile-vers
libsysevent/mapfile-vers
libtnf/mapfile-vers
libtnfctl/mapfile-vers
libtnfprobe/mapfile-vers
passwdutil/mapfile-vers
rpcsec_gss/mapfile-vers
sasl_plugins/mapfile-vers
-- snip --
AFAIK there are more examples in other directories for libraries not
build in usr/src/lib/, too.
> I don't quite understand why you're making the connection, though, with
> upstream and the common directory. If you look at
> usr/src/lib/README.mapfiles, you will see that it specifies the usual
> place for "mapfile-vers" is under the "common" subdirectory. The fact
> that most of the files under that come from an external, upstream
> source seems irrelevant to me.
Erm, if the file is in common/ it shows up during the update process and
needs to be handled explicitly... over and over again... ;-(
> > General note on the wordexp.c diff: The new version of the workexp() is
> > a cloned version of the old code and then modified to work with ksh93
> > and therefore has cloned all the cstyle and other coding layout bits
> > from the original. The idea of copying the whole function was to avoid
> > having a zillion of #ifdef/#else/#endif in there which would make the
> > code more or less unreadable or normal human beings.
> > /opt/onbld/bin/cstyle however doesn't complain about the current vesion.
> > [snip (moving this to a seperate email)]
>
> Understood but I think it's still worthwhile to clean up the additional
> "cloned" code.
>
> I didn't see another thread started on this but in addition to the
> cstyle changes, I did ask about the following but wasn't clear what the
> answers were.
See
http://mail.opensolaris.org/pipermail/ksh93-integration-discuss/2007-February/002273.html
- it was in my Drafts/-folder for over a week since I was occupied
elsewhere... ;-(
[snip]
> >> usr/src/lib/libpp/mapfile-vers
> >>
> >> Line 29 - I don't think this comment is really necessary
> >> (almost all of the other mapfiles are also generated by hand.)
> >
> > Erm, all other AST mapfile-vers stuff is created more or less
> > automatically and then verified by hand (exceptions include libpp and
> > libshell (where I had to export all |b_()| functions to avoid a
> > screw-up)).
>
> Sorry, what I meant was that all the other ON mapfiles are also
> generated by hand and calling it out here didn't make sense.
Well, in the case of the AST/ksh mapfile-vers files it's the reverse
condition - we have an almost automated way to create the files and
usr/src/lib/libpp/mapfile-vers is an exception...
> >> usr/src/lib/libshell/common/data/solaris_cmdlist.h
> >>
> >> Line 26 - The file is missing a #pragma ident.
> >
> > Uhm... this file "pretends" to be an AST source (well, it's added by the
> > "ksh93_solaris_builtin_patch.diff") and AFAIK doesn't need a "#pragma
> > ident" (AFAIK this is some kind of border case... ;-/ ).
>
> I think every header file that's checked in should include the "#pragma
> ident" unless this file is unmodified from upstream.
Fixed.
> >> usr/src/lib/libshell/misc/buildksh93.ksh
> >> usr/src/lib/libshell/misc/buildksh93.readme
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[snip]
>
> Sorry, modulo getting the build machine's environment setup correctly
> (for example, perhaps adding SUNWastdev to the machine), why isn't
> building ksh93 as simply as "cd usr/src/cmd/ksh ; make"? Yes, there
> are dependent libraries but I'm confused as to how this script is
> helpful as part of building ON.
>
> Or is the script that's used to generate the files which are eventually
> *checked in* to ON?
Right. Please read the script and look at the original AST build system
now it works - "buildksh93.ksh" does some "modifications" to the build
setup to make sure we find some features in libraries which are normally
not detected automatically and enforce things like XPG6/C99 to avoid the
limitations of a normal build (which both dramatically affects
performance and other details).
BTW: "perl" in OS/Net does the same - see
http://mail.opensolaris.org/pipermail/ksh93-integration-discuss/2007-February/002244.html
> If it's the latter, then I see this as being in the same boat as the
> *.diff files. They do deserve a home, perhaps on the ksh93 OpenSolaris
> project page on opensolaris.org or genunix.or but it seems we're
> talking about files that aren't useful for the building of ON itself.
Grumpf... please read
http://mail.opensolaris.org/pipermail/ksh93-integration-discuss/2007-February/002268.html
The "buildksh93.ksh" script (and the *.diff files) are expected to
change for each single update. Using the wrong version of the script
with the wrong upstream sources is more or less fatal since regressions
may creep into the includes and therefore into the resulting OS/Net
codebase.
> >> usr/src/lib/libshell/misc/ksh93_solaris_builtin_patch.diff
> >> usr/src/lib/libshell/misc/ksh93_solaris_builtin_patch.readme
> >>
> >> I don't believe these files should be putback to the ON gate.
> >> It makes sense to keep them on a project-specific website.
> >
> > Umpf...
> > ... this is per-putback information. It is going to change per putback.
> > How should an external project side track this and all branches made
> > from the main OS/Net tree ? And there is no gurantee that the project
> > webpage will be available in ten or more years which will generate a
> > serious problem for the maintainers. We already had that problem for
> > dtksh and except the generic "we try to avoid that mistake in the future
> > for OpenSolaris" noone has proposed a working alternative which tracks
> > the progress of ksh93 in OS/Net including all branches.
>
> Again, please don't draw any conclusions on what happened with dtksh.
> That wasn't part of ON and is, I think, not a representative example to
> draw conclusions from.
It is still the worst case we could hit and I really like to see some
kind of "protection" from such a case. And putting it into the
opensolaris.org webpage is the option I would only use after cutting off
some of my fingers - the thing doesn't even have some kind of version
control behind it and deleting files by accident may result in a
non-recoverable situation. Another problem is that there is no way to
keep both trees in sync (as I said using the wrong version of the script
with the wrong version of the upstream sources will generate
"unpredictable" results (which is an understatement; for example if you
use the current "buildksh93.ksh" on Solaris 10 will result in the loss
of all the librt functions since we no longer pass librt to the linker
flags (librt has been folded into libc during Nevada development) which
affects precise sub-second timing in ksh93)).
> What I think is a better example is the source
> which has an active upstream.
>
> I think we're going to have to agree to disagree here on the value of
> incorporating these into ON itself.
Please read
http://mail.opensolaris.org/pipermail/ksh93-integration-discuss/2007-February/002244.html
- IMO this delivers the "precedence" for adding the "buildksh93.ksh"
script to the codebase.
> >> usr/src/pkgdefs/SUNWcsu/prototype_sparc
> >>
> >> Lines 50-51 - What is the reason for shipping a 32-bit version
> >> on SPARC? Can ksh93 be used to read 32-bit core files or
> >> processes? :-)
> >
> > There are several reasons including:
> > - ksh93 supports (loadable) binary plugins which may itself rely on
> > other shared libraries which may not be available as 64bit versions. In
> > those cases a 32bit ksh93 is needed. I am waiting for a sponsor for
> > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6474270
> > ("isaexec and magical builds") (see request-sponsor queue) to make
> > "isaexec" adjustable for applications which need such a functionality
> > (via restricting the list of ISAs based on accept and/or reject filter
> > environment variables).
>
> So how does a SPARC user use ksh93 in that case? Do they code their
> scripts to use /usr/bin/sparcv7/ksh93 directly?
No, the idea was to implement Sun-Bug #6474270 ("isaexec and magical
builds") and set an environment variable which tells isaexec which ISA
should be used. I was hoping to get this work done in parallel but
somehow the request is still waiting in the request-sponsor list (see
http://opensolaris.org/os/bug_reports/request_sponsor/) ... ;-(
> BTW, what is the API for creating those binary plugins? Is it an open
> API as covered by the PSARC case?
Uhm... yes and no. The ksh93(1) manual page describes the usage but
doesn't go into all the details. Basically the prototype is
|b_<name_of_command>(int ac, char *av[], void *context);|. Simple
commands can work with that while more complex ones which depend on
modifying the shell's state have to access |context| (which is a
|Shell_t *|) and the libshell API which uses |Shell_t *| as arguments is
not ARC'ed.
> > - We need libshell&co. around for future consumers like the various
> > tools currently wrapped in "alias.sh" (this includes things like
> > /usr/bin/test etc.), "shcomp" (shell script compiler), /usr/bin/sleep
> > (using a 64bit binary for this is IMO an overkill), /usr/bin/printf etc.
> > and obmitting the 32bit shell would not be wise in this case (for
> > example: how else can we test the libraries then ?).
>
> Yes, but those components are not part of this case, correct?
Well, we heavily stripped the original ARC case to make the first
putback self-contained, e.g. it should not affect anything else except
adding ksh93+libraries (we even stripped "shcomp" since it would deliver
a new kernel module (to recognize compiled shell code (similar to
javaexec))) - but ARC 2006/550 specifies these files and the usage of
isaexec (from
http://www.opensolaris.org/os/community/arc/testbed/caselog/2006/550/onepager/):
-- snip --
Interface Description
--------- -----------
/usr/bin/ksh93 the korn shell
/usr/bin/pfksh93 profile shell
/usr/bin/rksh93 restricted shell
which will be hard links to /usr/lib/isaexec; isaexec will execute
the corresponding 32-bit binary in /usr/bin/{sparcv7,i86} or
the 64-bit binary/usr/bin/{sparcv9,amd64}, depending on the
architecture.
The isaexec links will allow the 64-bit version of ksh93
to be executed by default on 64-bit platforms.
-- snip --
As I said this is done intentional, even on SPARC (Garret bickered
already about the use of isaexec... ;-( ) ...
... you're not coming up with the argumentation that we have to remove
parts which are not in use yet and re-add them later with the part which
starts using it, right ?
----
Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 7950090
(;O/ \/ \O;)
