Pavel, All, Let me add one "pro" for the second approach: it works already. Vladimir's scripts daily upload test results to http://harmonytest.org.
With best regards, Alexei Fedotov, Intel Java & XML Engineering >-----Original Message----- >From: Tim Ellison [mailto:[EMAIL PROTECTED] >Sent: Thursday, November 16, 2006 12:37 PM >To: harmony-dev@incubator.apache.org >Subject: Re: [drlvm][unit] 100% of class library tests pass > >Pavel Ozhdikhin wrote: >> We have to evolving systems - classlib and DRLVM. To check commits to >> classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise >it's >> impossible to use DRLVM for pre-commit testing - you never know whether >> your >> test fail because of your patch or due to latest changes in DRLVM. >> >> I remember the time when DRLVM and Jitrino actively evolved - for some >time >> JIT had to use an older version of DRLVM which could pass all commit >> criteria because newer versions suffered from regressions. And finally we >> came to comon strict commit criteria which prevented regressions in both >VM >> and JIT. >> >> To avoid regressions using DRLVM in classlib testing I see 3 possible >> solutions: >> >> 1. Use one fixed DRLVM version which can pass 100% HUT test. Update this >> version from time to time. >> Pros: + Less time to run DRLVM pre-commit tests >> + Classlib does not suffer from regressions in DRLVM >> Cons: - DRLVM will suffer from regressions >> - Classlib can not use the latest DRLVM >> - Need additional efforts to regain stability on DRLVM >> when we want to update the version for classlib testing >> >> 2. Add HUT to CruiseControl testing on DRLVM and rollback commits causing >> regressions >> Pros: + Less time to run DRLVM pre-commit tests >> + Classlib can use the latest DRLVM >> Cons: - Classlib can suffer from DRLVM regressions (time lag before >> rollback) >> - It is not always clear which commit caused a regression >> - Rollbacks are costly >> >> 3. Add HUT to the commit criteria for DRLVM >> Pros: + Classlib always can use the latest DRLVM >> + DRLVM has no regressions regarding to HUT >> Cons: - More time to run DRLVM pre-commit tests (I was told that HUT >> take 25 minutes running in single JVM mode) >> >> I think that preventing a problem is better than solving it afterwards. >So, >> I personally would choose the 3rd approach, don't mind against the second >> and dislike the first one. Probably some combination of these is >possible. > >While I appreciate the desire to keep things stable, I think it is >unreasonable to ask developers to run the entire test suite each time. >As we have seen in the classlib code, running targeted tests before >commit and leaving the build machines to run continuous tests ensures >that we are productive and are notified of breakages in time to easily >back out a patch and re-evaluate. > >With the amount of machine time we have running harmony tests on >different cpu's/os's/compilers/etc we are getting better coverage than >any individual could be expected to provide. > >Which is a long way of saying I think option (2) above is best -- and >relies on the bid machines letting us know if things break, and the >commitment from all of us to go straight in and fix it. > >Regards, >Tim > >-- > >Tim Ellison ([EMAIL PROTECTED]) >IBM Java technology centre, UK.
