The more I go further with build&testing infrastructure and think about it
the more the following overall picture is drawing in my mind.
Let me share it to see the whole picture and what I see should be next in
build and testing infrastructure:
Goal: Harmony quality
Objectives:
- Set-up regular building & build publishing process
- Set-up testing process, which:
- Provides code integrity (Harmony can be built at each moment of
time), regressions are seen immediately.
- Done by both at Harmony hosting site and by community members.
- Is easy to set-up and run.
- Results of testing on community member's platforms can be
published.
- History of test failures, build quality can be tracked.
Builds.
Lets consider the following:
1. Regular (daily, weekly?) building of JRE and HDK and tests.
Goal: being regularly published, one can just download and run / test them
without need to download sources, tools and building Harmony themselves.
What we need to be built and published? – I suppose we can start with:
- Harmony API + DRLVM
- Harmony API + J9
- Harmony API + JCHVM
- API unit tests
Do we want to provide capabilities for people to install "build tool" and
have Harmony built regularly on their platforms? – I guess yes.
****I can (and plan) do a simple ant script for building and archiving
binaries (If no objection).
Once it is done, I can start publishing the binaries more or less regularly
at some site and provide links at our wiki pages – is it worth doing?
2. We establish continuous building to make sure we have consistent
workspace. Cruise Control is OK for this purpose.
What we build:
- Harmony API
- DRLVM
- JCHVM
Do we want to provide capabilities for people to install and have "Cruise
Control – based tool" running on their platforms? – I guess yes (correct me
if I'm wrong), in case if somebody is interested in keeping Harmony
buildable and runnable on his favorite platform (FreeBSD, for example).
Testing.
I would propose to consider the following levels of testing.
1. Pre-integration testing – quickly executed set of tests to be run
to make sure a patch does not worsen Harmony quality.
Goal: for a community member or committer to check that the patch does not
worsen Harmony quality.
****This is already discussed, agreed and implemented for both API and VM.
Do we need something more to be done here?
2. Code integrity testing – done by regular often building of Harmony
components by Cruise Control.
Goal: to make sure no "compilation errors" were introduced by recent
commits, if introduced, then, notification along with list of recent commits
is sent to all.
****This is what is done recently by Geir by Cruise Control script.
3. Smoke testing – a number of tests should be run over received via
Cruise Control build.
Goal: to make sure no regressions were introduced by recent commits, the
built runtime can do its basic operations.
If regression is introduced, then, notification along with list of recent
commits is sent to all.
****This is what is done recently by Geir in Cruise Control script.
4. Full testing - the whole set of Harmony tests are run on regular
builds, results are published.
Goal: to see what happened with Harmony quality on the whole set of
automated tests, see new bugs, see quality of Harmony runtimes.
Users should be able to do this kind of testing on their specific platforms
and publish results on Harmony web site.
****This script (prototype) I implemented, see issue 984.
5. Application testing – people enable applications, try already
enabled applications on fresh builds, publish results.
Goal: to see what happened with Harmony quality on various applications
people like.
We can start from having a wiki page (which partially exists in some form)
on which community publishes results of running applications on various
Harmony builds.
****Do we need here more?
6. [optional] Reliability testing – either special reliability tests
or some applications are executed during long time (from one regular build
to another).
Goal: to make sure we have reliable, stable, long running runtime.
For future.
thanks, Vladimir
