On the 0x223 day of Apache Harmony Alexey Varlamov wrote: > 2006/11/16, Tim Ellison <[EMAIL PROTECTED]>: > > Pavel Ozhdikhin wrote: > > > We have to evolving systems - classlib and DRLVM. To check commits to > > > classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise it's > > > impossible to use DRLVM for pre-commit testing - you never know whether > > > your > > > test fail because of your patch or due to latest changes in DRLVM. > > > > > > I remember the time when DRLVM and Jitrino actively evolved - for some > > > time > > > JIT had to use an older version of DRLVM which could pass all commit > > > criteria because newer versions suffered from regressions. And finally we > > > came to comon strict commit criteria which prevented regressions in both > > > VM > > > and JIT. > > > > > > To avoid regressions using DRLVM in classlib testing I see 3 possible > > > solutions: > > > > > > 1. Use one fixed DRLVM version which can pass 100% HUT test. Update this > > > version from time to time. > > > Pros: + Less time to run DRLVM pre-commit tests > > > + Classlib does not suffer from regressions in DRLVM > > > Cons: - DRLVM will suffer from regressions > > > - Classlib can not use the latest DRLVM > > > - Need additional efforts to regain stability on DRLVM > > > when we want to update the version for classlib testing > > > > > > 2. Add HUT to CruiseControl testing on DRLVM and rollback commits causing > > > regressions > > > Pros: + Less time to run DRLVM pre-commit tests > > > + Classlib can use the latest DRLVM > > > Cons: - Classlib can suffer from DRLVM regressions (time lag before > > > rollback) > > > - It is not always clear which commit caused a regression > > > - Rollbacks are costly > > > > > > 3. Add HUT to the commit criteria for DRLVM > > > Pros: + Classlib always can use the latest DRLVM > > > + DRLVM has no regressions regarding to HUT > > > Cons: - More time to run DRLVM pre-commit tests (I was told that HUT > > > take 25 minutes running in single JVM mode) > > > > > > I think that preventing a problem is better than solving it afterwards. > > > So, > > > I personally would choose the 3rd approach, don't mind against the second > > > and dislike the first one. Probably some combination of these is possible. > > > > While I appreciate the desire to keep things stable, I think it is > > unreasonable to ask developers to run the entire test suite each time. > > As we have seen in the classlib code, running targeted tests before > > commit and leaving the build machines to run continuous tests ensures > > that we are productive and are notified of breakages in time to easily > > back out a patch and re-evaluate. > > > > With the amount of machine time we have running harmony tests on > > different cpu's/os's/compilers/etc we are getting better coverage than > > any individual could be expected to provide. > > > Which is a long way of saying I think option (2) above is best -- and > > relies on the bid machines letting us know if things break, and the > > commitment from all of us to go straight in and fix it. > > I can't say it better. Thank you Tim :) > Maybe just to reinforce: > 1) We have absolutely stable model VM for classlib verification - j9 > it's name. Therefore I really don't think DRLVM can affect classlib's > progress disruptively. > 2) Yes, there are times when some component advances in leaps as > against accustomed smooth evolution. I do believe we'll be able to > manage such cases individually, w/o overburdening everyday activities.
I am for (2) too. But a small correction: rollback is not always reasonable. We can explicitly agree if we do rollback or fix ASAP (as we did with TM and launcher). I do not like (1) because interfaces are evolving. And I am not feeling like this process would stop in a short time. (3) is just too long to run as pre-commit, IMHO. Though, we might select some most bug-catching tests from HUT to run as pre-commit. I have nothing concrete to suggest now. We might return to this idea in future, when we have a longer history. -- Egor Pasko
