My putback of
6798660 Cadmium .NOT file processing problem with CWD relative file paths
Contributed by Richard Lowe
6785284 Mapfile versioning rules need to be more visible to gatelings
6800164 Standard file exclusion mechanism needed for Cadmium tools
introduces some changes into the gate tools, intended to make it
harder for object versioning errors to be integrated into Solaris.
Read on if any of the following apply to you:
- You maintain a build server
- You install your own copy of the build tools (you'll need to
update them to get this fix)
- You expect to ever modify a linker mapfile within the
OSnet consolidation.
- You do nightly builds specifying the -r flag (check_rtime)
------------------------------------------------------------------------------
The symbols within Solaris objects are versioned. This plays
a key role in how we maintain backward compatibility. Errors in
versioning can be quite painful. Unlike most errors, which can
usually be repaired in a following putback, versioning errors that
are exposed by a public release (patch, backport to a shipping
release, OpenSolaris, etc) are not easily undone. Patching up these
situations is tricky, takes time from many people, and leaves a permanent
mark on the product.
And yet, errors do happen. We have observed that most of
these errors are innocent, due to people not understanding:
- What the rules for versioning are
- What's at stake if they make an error
- That they should ask for help, and from whom
I say these are innocent errors, because although this information exists,
it has been buried, allowing people to integrate incorrect changes without
ever having a hint of the trouble ahead. With 6785284, I am seeking to
prevent this class of error. There are four changes involved:
1) Every file in ON with a name that matches the pattern
'*mapfile*' now contains a standard header comment, as follows:
#
# MAPFILE HEADER START
#
# WARNING: STOP NOW. DO NOT MODIFY THIS FILE.
# Object versioning must comply with the rules detailed in
#
# usr/src/lib/README.mapfiles
#
# You should not be making modifications here until you've read the most current
# copy of that file. If you need help, contact a gatekeeper for guidance.
#
# MAPFILE HEADER END
#
The intent of this comment is to grab the attention of
the unwitting gateling who does not yet know or understand
object versioning, but who is about to attempt to make
a change. By putting the message at the top of the file
they have to edit to make their change, we hope to catch
most of them before they go further.
Also note that the message does not tell you how to do
versioning, but simply points you at the one true source
for that information. This gives us the flexibility to evolve
the rules by modifying one file (usr/src/README.mapfiles).
2) There is a new Mercurial/Cadmium command, 'hg mapfilechk',
that is run against any file in ON matching the pattern
'*mapfile*'. This tool operates much as cddlchk does, and
in fact shares a common implementation under the covers.
mapfilechk ensures that the comments described in (1) are
not altered or removed, and that new mapfiles introduced
into ON do not integrate without one.
There is also a command line version of mapfilechk, found in
usr/src/tools/scripts, that will examine any files you point
it at. This works similarly to the cddlchk command found in
the same directory.
mapfilechk is included by 'hg nits' and 'hg pbchk'.
3) There is a new standard file exclusion mechanism for cadmium,
based on the preexisting .NOT mechanism. You will find a new
directory at the top level of your workspace named 'exception_lists',
containing an exclusion file for 'hg mapfilechk'. This exclusion
file prevents 'hg mapfilechk' from examining certain files in ON
that match the name '*mapfile*', but which are either not mapfiles,
or which are mapfiles that we know do not (and will not) specify
symbol versioning.
4) The check_rtime script, which is run by nightly when the -r
flag is set, now records the versions and symbols tied to each
version, for all the ELF objects it processes. This means that
adding a new symbol to an object, moving a symbol from one version
to another, or changing the versioning details of any ON object
will cause nightly to report a change. The gatekeepers will note
these changes post-integration, and you can expect to receive
an email query from them verifying your intentions if you have
not already given them a heads up (which is recommended).
We highly recommend running nightly with -r as a standard practice.
You should note that these messages do not necessarily reflect
an error in your code. They simply indicate that a change has
occurred, and provide a final opportunity to verify that you
made the change properly.
Note: The first time you rebuild an existing workspace after
updating to these new tools, you may see a very large set of
differences. See below for details.
Tools Need To Be Updated
------------------------
If you install SUNWonbld on your own build servers, you should
update them. Similarly, if you maintain a mirror of the tools from
the gate, that will need to be updated.
Note that this putback does not break the ability of the older
tools to build the gate. However, until you update you will be
missing functionality.
One Time Potential for Large Output from 'nightly -r'
-----------------------------------------------------
When nightly is run with the -r flag, it uses the check_rtime tool
to generate two files:
x86:
usr/src/runtime-i386.out
usr/src/runtime-i386.ref
sparc:
usr/src/runtime-sparc.out
usr/src/runtime-sparc.ref
When you do repeated builds within a single workspace, nightly reports
changes in these files in the
==== Diff ELF runtime attributes (since last build) ====
section of your build mail. With the changes to check_rtime in this
fix, the size of these files grows from ~270K to ~1.8MB. If you
pull these changes into an existing workspace and rebuild using the
new check_rtime, the first time you do so, your build email will
contain ~25000 lines of output, all of which will include one of
the following patterns:
VERDEF=XXX
VERSION=XXX, SYMBOL=YYY
This is harmless, and if it happens, you can safely ignore it.
You can prevent it by removing the two files shown above from your
workspace, after you update your gate tools, and before the next
time you run nightly.
This large output will not occur in subsequent builds. Once
you get past the first one, any diffs in the build report with one
of these lines will reflect a change you've made to the public
interface of an ELF object. Such output will be proportional
to the size of your change (generally, tiny).
------------------------------------------------------------------------------
Example 1: mapfilechk
---------------------
In this example, I have intentionally corrupted the mapfile
comment in usr/src/cmd/sgs/libld/common/mapfile-vers:
% hg mapfilechk usr/src/cmd/sgs/libld/common/mapfile-vers
Mapfile comment check:
Error: Invalid line in MAPFILE block in file
usr/src/cmd/sgs/libld/common/mapfile-vers
at line 32, should be
'WARNING: STOP NOW. DO NOT MODIFY THIS FILE.'
is
'XXXWARNING: STOP NOW. DO NOT MODIFY THIS FILE.'
It is also part of 'nits' and 'pbchk', so that is probably
the context in which you will see it:
% hg nits
CDDL block check:
Copyright check:
C style check:
Header format check:
Java style check:
Mapfile comment check:
Error: Invalid line in MAPFILE block in file
usr/src/cmd/sgs/libld/common/mapfile-vers
at line 32, should be
'WARNING: STOP NOW. DO NOT MODIFY THIS FILE.'
is
'XXXWARNING: STOP NOW. DO NOT MODIFY THIS FILE.'
File permission check:
Keywords check:
Example 2: check_rtime
----------------------
In this example, I did a full nightly build using the nightly
'-r' option. I then modified libc to add a function named foo(),
and assigned it to the Solaris 9 version (SUNW_2.21.3). This is
an obviously invalid change. After doing an incremental build in
the same workspace, my nightly build mail included the following:
==== Diff ELF runtime attributes (since last build) ====
1567a1568
> ./lib/amd64/libc.so.1: VERSION=SUNW_1.21.3, SYMBOL=foo
6537a6539
> ./lib/libc.so.1: VERSION=SUNW_1.21.3, SYMBOL=foo
(note that there are 32-bit and 64-bit versions of libc).
