As explained a bit in bug 6929667 which I putback yesterday, the
way we deal with the upstream tarballs in the hg gate is a bit
different from teamware.
We no longer check in the upstream tarballs to the source repo.
This is very inefficient, and a poor match of tools - you can't
effectively diff a compressed tarball, so even if only one line
changed upstream we'd carry around a new complete copy of the
full tarball for every version we've ever built, and the size of
our Nevada gates was getting large with all the SCCS history of
these tarballs.
Also, we didn't include the tarballs when exporting the gate to
opensolaris.org either in the hg repository or the snapshot
tarballs, so our gate was built differently by internal users than
external.
We do however need to always be able to build any version, without
relying on upstream sites being online or still having a copy of
the version we built with, and to verify that the tarball used is
the same one we expect it to be.
The solution I've developed for this is to set up a simple, separate
archive of tarballs internally, and changed the builds to
automatically download tarballs when needed and verify their checksums
against the published/expected values using the MD5 & SHA1 algorithms.
(These were chosen since they are well supported in Solaris, and are
what X.Org publishes in release announcements.)
When downloads happen:
- You can manually download with "make download" in any open-src
subdirectory.
- You can manually download the entire tree still with the top-level
"download-tarballs" script, which is mostly a wrapper around
"make download". The -c option to this script deletes the
entire tarballs directory first and redownloads everything.
- If the tarball is missing, make will automatically download it
when it gets to the "make source" step of the build that needs
to unpack it.
Where downloads come from:
- If TARBALLS_ARCHIVE is set when you run make, it will use that
as the site to download all the needed tarballs from.
If the "domainname" command on your machine returns anything
ending in ".sun.com", then the default TARBALLS_ARCHIVE
is set to the internal tarball archive/mirror site.
- If TARBALLS_ARCHIVE is not set, and the domainname auto-detection
doesn't match your domain, then downloads come from the original
open source upstream locations. If tarballs disappear from those
sites, we can copy them from our internal archive to the set we
mirror on opensolaris.org.
- If you need to use a HTTP proxy/cache to reach whichever site you
download from, set the "http_proxy" environment variable
(all lower case, as documented in the "info wget" manual).
Checksum verification:
- Two new variables have been added to all module Makefiles that have
upstream sources:
# Checksums for upstream tarball
TARBALL_MD5 = 001d780829f936e34851ef7cd37b4dfd
TARBALL_SHA1 = d4b3966d8eeb79514daad3eeae0debf65d16edca
For most X.Org releases, you can just copy these from the release
announcement - for example, the above for lib/libX11 were copied
from the .bz2 lines of:
http://lists.freedesktop.org/archives/xorg-announce/2009-October/001184.html
If you're updating a module that doesn't have those in their release
announcement, you can generate them yourself using the "md5sum" and
"sha1sum" programs (or "digest -a sha1" and "digest -a md5").
- tarballs can be manually verified against their checksums by running
"make check-checksums" - the makefile dependencies will cause the
"make download" actions to occur if the tarball does not exist.
If the checksums match, it will be silent - if they fail, errors
will be printed and reported as failures to "make"
- The builds automatically run the "check-checksums" rule as part of
the "make source" rules to unpack the source from the tarballs.
Updating modules to new versions:
As an example, I'm updating the xserver/xorg module from 1.7.4 to 1.7.5,
because I've seen the announcement on xorg-announce, which you can see at:
http://lists.freedesktop.org/archives/xorg-announce/2010-February/001258.html
I've edited the Makefile to change the version number and pasted in the SHA1
& MD5 sums from the announcement.
To download a local copy for testing my build, I do:
cd open-src/xserver/xorg
make download TARBALL_ARCHIVE=''
(to override the default TARBALL_ARCHIVE that doesn't have it yet)
I verify I pasted the checksums correctly by doing:
make check-checksums
and get no output because they are correct.
As always, if you have questions or hit problems, please ask.
--
-Alan Coopersmith- alan.coopersmith at sun.com
Oracle Solaris Platform Engineering: X Window System
