Adding -dev to CC: in case someone has any meaningful input or has
already tackled this problem...
On Aug 14, 2005, at 8:21 AM, Grobian wrote:
Introduction
============
Recently, once again we were confronted with a package marked as
ppc-macos stable, while it didn't compile at all, let alone run.
It is
believed more of these packages are in portage, and need to be
found and
fixed. Keeping the cause of why they are marked stable up to another
discussion, and out of the scope of this discussion, I will focus
on how
to track these packages down and report them to us.
In the secondary line, all 'unstable' packages, marked ~ppc-macos
should
be tested as well, since they can be faulty as well. Since for OSX
much
is in ~ppc-macos, many users consider it a normal procedure to
switch to
the unstable side of portage, hence some extra need for careful
testing
of ~ppc-macos also.
Proposed Global Structure
=========================
Testing should be done on a regular basis, both push and pull based.
This means that the testing machine would start testing packages
itself
if it is out of work, and on the other hand starts testing packages as
soon as they are being added/changed in CVS. It may need no great
imagination to see that the latter 'push-based' activity has priority
over the 'pull-based' work.
I'm not sure direct interaction with CVS would be needed, usually
only takes a short time for cvs commits to hit the rsync mirrors
(hence the volatile nature of the tree)
Starting over, will for the test machine mean that it starts cleaning
out its world file. Cleaning this file out to a bare minimum is an
important aspect of getting a test environment that reflects the
situation on new user's machines. If an ebuild uses a package without
having it in it's DEPEND, this may get noticed only when starting on a
clean machine. This, however, will add a big delay in testing as many
packages will need to be built prior the right package can be
installed.
The testing machine will have a queue file, which it reads packages to
emerge from. If the queue file is empty, i.e. when there is no push
based work, the machine will generate work by starting to compile
uncompiled packages, or emptying the tree.
Because ~ppc-macos and ppc-macos packages interfere with each other
-- a ~ppc-macos package overwrites a ppc-macos package -- both
stable and unstable have to be dealt with separately, i.e. they
should both have their own environment either via two separate
machines, or through the use of a chroot jail.
I think seperate chroots are definitely the way to go. We can just
store a 'pristine' chroot in iso or dmg or whatever on the build
server and copy when needed.
Queues
------
In order not to drag in a full DBMS (in the end Portage already is
one) queues are just simple flat files consisting of absolute
package names, one per line. Table wise locking granularity is
handled by the OS as one process opens the file in write mode.
Consumers -- the testing box in this case -- read the first line
and delete it, while producers simply add one line (or more) to the
end of the file.
The queue itself, is more a set than a list. This means that
packages that are in the queue, should be unique. If a package is
added that is already in the queue, it is dropped such that the
original queue position of the package is maintained.
Maybe a 'proper' dbms wouldn't be such a bad idea, could also store
build logs, timestamps, etc. there and make it easier for multiple
build hosts to push/pull from a centralized server.
CVS Producer
------------
To catch up automatically with changes made to the tree, it is
necessary to act upon any commit to the tree for an ebuild file. A
possibility to do this would be via processing of CVS commit
messages, sent out as email by the CVS server. It is a task of the
producer to find out whether the ebuild found applies to the
testing machine (ppc-macos) and add the package/ebuild to the queue.
Consumer (testing process)
--------------------------
The test machine reads a line from the queue, and basically
executes 'emerge ${PACKAGE}'. However, before doing this, first it
figures out which use flags can be used (emerge -pv) and which
dependencies will be pulled (emerge -pt). If portage returns the
message all ebuilds that could satisfy X have been masked, the
emerge is cancelled, the line is removed from the queue and an
email message will be sent out.
All dependencies are put in the right order and emerged as normal
packages, that is: all dependencies are pushed at the front of the
queue, thereby keeping uniqueness of the queue and removing
duplicates that appear later on in the queue. After this, the
consumer is restarted and reads again from the queue. This should
result in usually merging only one package at a time, and as such
quite isolated cases, which should improve the error email
notification service.
Compile testing a package is supposed to be a thorough test that
tries all possible combinations of the package's USE flags. As
this might be somewhat endless as some packages are rather big and
have zillions of USE flags, it may be necessary to have a special
"don't do it" file.
Since all dependencies were put at the front of the queue, there
should normally be no dependencies that the package pulls.
If compilation fails for a certain USE-flag combination this is
reported by sending out an email, and compilation of the next USE-
flag combination is attempted.
When everything goes fine, no email notification is being sent
out. A convenient log structure would, however, make it possible
to see which packages and USE-flag combinations successfully passed
through. Providing this log via a web-page would be a useful
thing. Again backing this with a DBMS to allow easy searching,
versioning and stuff is considered to be overhead, though crafting
logs in SQL's "INSERT INTO" format might enable another machine to
display the output data. Perhaps the communication methods needs a
section on itself.
Recap and Conclusion
====================
By setting up a testing system, it is possible to greatly improve
the Quality of Service of the portage tree for an architecture by
exhaustive testing of both packages already in there, as well as
packages added or modified. Automated testing should not release
developers from testing themselves, but should help in pointing out
problems that may arise on moving grounds such as portage where
packages are constantly updated and dependencies might get broken.
ToDo
====
- Not only check dependencies of the respective package, but also
consider packages that depend on the respective package, thus
rebuilding all packages that depend on the package to check if
anything is broken by the update.
- Is there a gleptomaniac in the room? This would be useful for
x86 also, of course. In that case it may be necessary to make sure
the packages are split over multiple machines.
- The message system needs more customisation options, especially
backing things by a DBMS would allow for many nice bugzilla-like
preferences for email generation as well as web-based versioned
info/report pages
- To make the system even bigger, a central DBMS powered server
might take a leading role and ... {editor note: wait, stop it right
now, you're going too fast right now}
By The Way
==========
- Kito offers his lil' chico as machine for this automated testing
initiative.
- Comments are welcome, as well as expressions of worry on my
mental state.
- Implementation of described system will need some better
specified system and needs some coding (the dirty work) in some
language...
--
Fabian Groffen
eBuild && Porting
Gentoo for Mac OS X
--
gentoo-osx@gentoo.org mailing list
--
gentoo-dev@gentoo.org mailing list
