Vladimir Ivanov wrote: > The more I go further with build&testing infrastructure and think about it > the more the following overall picture is drawing in my mind. > Let me share it to see the whole picture and what I see should be next in > build and testing infrastructure: > > Goal: Harmony quality > Objectives: > - Set-up regular building & build publishing process
Yep > - Set-up testing process, which: > - Provides code integrity (Harmony can be built at each moment of > time), regressions are seen immediately. Yep. We now have a "proprietary" one running at IBM, for which we are grateful. The buildtest subproject in Harmony is meant to replace this at some point and we all will run it. > - Done by both at Harmony hosting site and by community members. > - Is easy to set-up and run. > - Results of testing on community member's platforms can be > published. > - History of test failures, build quality can be tracked. Yes > > Builds. > Lets consider the following: > 1. Regular (daily, weekly?) building of JRE and HDK and tests. daily snapshot > Goal: being regularly published, one can just download and run / test them > without need to download sources, tools and building Harmony themselves. > What we need to be built and published? – I suppose we can start with: > - Harmony API + DRLVM > - Harmony API + J9 > - Harmony API + JCHVM > - API unit tests I would suggest : HDK (classlib + unit tests) JDK/JRE (classlib + DRLVM) and once JCHEVM uses either the classlib or the glue layer is finished, a JDK/JRE with that. We won't create anything w/ J9 in it (as that's IBMs proprietary code) and anyone who wishes to use it can take the HDK and drop J9 right in very easily. That said, I do think we still want people to be independently running (as part fo the test loop) Classlib + J9 because testing with the broad variety of VMs available to us will is useful. > > Do we want to provide capabilities for people to install "build tool" and > have Harmony built regularly on their platforms? – I guess yes. Yes - that is the point of the buildtest/ project. > > ****I can (and plan) do a simple ant script for building and archiving > binaries (If no objection). We already have a script for building the HDK and JRE. The publishing part should be fairly straightforward. I imagine we'll have volunteers that produce the designated nightly snapshots for a given platform so we can get the widest variety of OS/chipset as possible. > Once it is done, I can start publishing the binaries more or less regularly > at some site and provide links at our wiki pages – is it worth doing? We would put them in the snapshot directory that we use now. http://people.apache.org/dist/incubator/harmony/snapshots/ > > 2. We establish continuous building to make sure we have consistent > workspace. Cruise Control is OK for this purpose. > What we build: > - Harmony API > - DRLVM > - JCHVM > > Do we want to provide capabilities for people to install and have "Cruise > Control – based tool" running on their platforms? – I guess yes (correct me > if I'm wrong), in case if somebody is interested in keeping Harmony > buildable and runnable on his favorite platform (FreeBSD, for example). That's what the buildtest subproject is for. Have you looked at it? > > Testing. > I would propose to consider the following levels of testing. > 1. Pre-integration testing – quickly executed set of tests to be run > to make sure a patch does not worsen Harmony quality. > Goal: for a community member or committer to check that the patch does not > worsen Harmony quality. > > ****This is already discussed, agreed and implemented for both API and VM. > Do we need something more to be done here? I hope not. > > 2. Code integrity testing – done by regular often building of Harmony > components by Cruise Control. > Goal: to make sure no "compilation errors" were introduced by recent > commits, if introduced, then, notification along with list of recent > commits > is sent to all. > > ****This is what is done recently by Geir by Cruise Control script. Yes, I am doing it, but the goal isn't to have only one person doing it. The goal of the buildtest/ subproject https://svn.apache.org/repos/asf/incubator/harmony/enhanced/buildtest/trunk is to help *anyone* that wants to help to easily checkout the system, have it auto-setup as much as possible, and then just run. What remains to be worked out is the email and notification aggregation, so that we can have a "dashboard" or matrix of all the platforms being tested, with their current status and history. This is an important thing to get done, and I'm hoping someone volunteers. > > 3. Smoke testing – a number of tests should be run over received via > Cruise Control build. > Goal: to make sure no regressions were introduced by recent commits, the > built runtime can do its basic operations. > If regression is introduced, then, notification along with list of recent > commits is sent to all. > ****This is what is done recently by Geir in Cruise Control script. Right > > 4. Full testing - the whole set of Harmony tests are run on regular > builds, results are published. > Goal: to see what happened with Harmony quality on the whole set of > automated tests, see new bugs, see quality of Harmony runtimes. > Users should be able to do this kind of testing on their specific platforms > and publish results on Harmony web site. > ****This script (prototype) I implemented, see issue 984. I thought that the classlib tests are run as well w/ the CC script right now, but will check. > > 5. Application testing – people enable applications, try already > enabled applications on fresh builds, publish results. > Goal: to see what happened with Harmony quality on various applications > people like. > We can start from having a wiki page (which partially exists in some form) > on which community publishes results of running applications on various > Harmony builds. > ****Do we need here more? Maybe eventually we'll want to automate. We certainly want to get Gump runs going too for this. > 6. [optional] Reliability testing – either special reliability tests > or some applications are executed during long time (from one regular build > to another). > Goal: to make sure we have reliable, stable, long running runtime. > For future. > > thanks, Vladimir > --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
