We have to evolving systems - classlib and DRLVM. To check commits to
classlib we need a stable DRLVM which can pass 100% of HUT. Otherwise it's
impossible to use DRLVM for pre-commit testing - you never know whether your
test fail because of your patch or due to latest changes in DRLVM.
I remember the time when DRLVM and Jitrino actively evolved - for some time
JIT had to use an older version of DRLVM which could pass all commit
criteria because newer versions suffered from regressions. And finally we
came to comon strict commit criteria which prevented regressions in both VM
and JIT.
To avoid regressions using DRLVM in classlib testing I see 3 possible
solutions:
1. Use one fixed DRLVM version which can pass 100% HUT test. Update this
version from time to time.
Pros: + Less time to run DRLVM pre-commit tests
+ Classlib does not suffer from regressions in DRLVM
Cons: - DRLVM will suffer from regressions
- Classlib can not use the latest DRLVM
- Need additional efforts to regain stability on DRLVM
when we want to update the version for classlib testing
2. Add HUT to CruiseControl testing on DRLVM and rollback commits causing
regressions
Pros: + Less time to run DRLVM pre-commit tests
+ Classlib can use the latest DRLVM
Cons: - Classlib can suffer from DRLVM regressions (time lag before
rollback)
- It is not always clear which commit caused a regression
- Rollbacks are costly
3. Add HUT to the commit criteria for DRLVM
Pros: + Classlib always can use the latest DRLVM
+ DRLVM has no regressions regarding to HUT
Cons: - More time to run DRLVM pre-commit tests (I was told that HUT
take 25 minutes running in single JVM mode)
I think that preventing a problem is better than solving it afterwards. So,
I personally would choose the 3rd approach, don't mind against the second
and dislike the first one. Probably some combination of these is possible.
thanks,
Pavel
On 11/16/06, Alexey Varlamov <[EMAIL PROTECTED]> wrote:
2006/11/16, Mikhail Loenko <[EMAIL PROTECTED]>:
> why not?
But what benefit it would bring? build test in DRLVM takes too much
time already, I'm afraid people will just stop using it :(
This is analogous to enforcing full testing in classlib for every
change regardless of module. Evidently this is too expensive.
>
> 2006/11/16, Geir Magnusson Jr. <[EMAIL PROTECTED]>:
> >
> >
> > Pavel Ozhdikhin wrote:
> > > On 11/16/06, Geir Magnusson Jr. <[EMAIL PROTECTED]> wrote:
> > >>
> > >> Be sure to not miss anyone :) This was a great community effort,
with
> > >> everyone pitching in.
> > >>
> > >> DRLVM is now a full peer to J9 in Harmony testing. :) We still
need
> > >> to use J9 (and another VM that happens to work with our
classlibrary),
> > >> as a sanity check, but we should from now on use DRLVM in our CI
testing
> > >> framework.
> > >
> > >
> > > On the other hand, to make sure DRLVM has no regressions regarding
> > > to Classlibrary Unit Tests it makes sense to add these tests to the
"test"
> > > target of DRLVM build.
> > > What do people think?
> >
> > Adding classlib unit tests to DRLVM test target? No thanks :)
> >
> > geir
> >
> > >
> > > Thanks,
> > > Pavel
> > >
> > >
> > >
> > >> geir
> > >>
> > >>
> > >> Alexei Fedotov wrote:
> > >> > Oops, I've missed:
> > >> > * Andrew Zhang for reviewing class library patches and helpful
> > >> discussions
> > >> >
> > >> > On 11/16/06, Alexei Fedotov <[EMAIL PROTECTED]> wrote:
> > >> >>
> > >> >> Folks,
> > >> >> According to http://harmonytest.org, today 100% of class library
unit
> > >> >> tests pass on DRLVM. Thank you all! It takes 44 days for the
great
> > >> >> team we are. Thanks for your thoughtful, diligent work and deep
> > >> >> inspiration. Kudos to you for the following (and not limited to
this):
> > >> >>
> > >> >> * Alexey Varlamov and Elena for driving the whole process
> > >> >> * Anton and Vladimir Ivanov for automating test runs
> > >> >> * Geir and Gregory for checking and committing related DRLVM
patches
> > >> >> * Paulex, Tim, Nathan, Stepan and Mikhail Loenko for checking
and
> > >> >> committing related class library patches
> > >> >> * Alexey Petrenko for becoming ICU expert and writing good JIRA
issue
> > >> >> resolution guidelines
> > >> >> * Alexei Zakharov for resolving class library test issues
> > >> >> * Ilya Okomin, Denis Kishenko, Oleg Khaschansky and Alexey
Ivanov for
> > >> >> fixing class library and tests* Ivan, Egor, Mikhail Fursov,
Nikolay
> > >> >> Sidelnikov and Alexander Astapchuk for making execution engines
work
> > >> >> * Tatiana and Maxim for filing JIRA issues about test failures
> > >> >> * Nikolay Kuznetsov for completing thread interruption handling
and
> > >> >> reverting consequences of park/unpark integration
> > >> >> * Pavel Afremov for fixing exception handling
> > >> >> * Boris Kuznetsov for intelligent fixing of security tests
> > >> >> * Rana and Salikh for evaluating and discussing problems,
reviewing
> > >> >> and trying DRLVM patches
> > >> >> * Pavel Pervov and Evgueni for help with DRLVM patches
> > >> >> * Artem for discovering and fixing weird Windows* behavior
> > >> >> * My wife for bringing hot tea to the computer during sleepless
nights
> > >> >>
> > >> >> There are still open issues with reliability, multiprocessor and
other
> > >> >> special configurations, so the page
> > >> >> http://wiki.apache.org/harmony/Unit_Tests_Pass_on_DRLVM remains
> > >> >> active. But this shouldn't prevent us from including class
library
> > >> >> testing into Harmony "zero regression" policy. What do you
think?
> > >> >>
> > >> >> --
> > >> >> Thank you,
> > >> >> Alexei
> > >> >
> > >> >
> > >> >
> > >>
> > >
> >
>
