There was a submission that enabled finer control of failing tests (even by platform etc.)
I may be wrong but commenting out tests usually means that they never get fixed; even putting them into exclude clauses in the ant script is too hidden for me -- I prefer to see the exclusions and failures made available very clearly. Regards, Tim Alexei Zakharov wrote: > Hi, > +1 for (3), but I think it will be better to define suite() method and > enumerate passing tests there rather than to comment out the code. > > 2006/6/27, Richard Liang <[EMAIL PROTECTED]>: >> Hello Vladimir, >> >> +1 to option 3) . We shall comment the failed test cases out and add >> FIXME to remind us to diagnose the problems later. ;-) >> >> Vladimir Ivanov wrote: >> > I see your point. >> > But I feel that we can miss regression in non-tested code if we exclude >> > TestCases. >> > Now, for example we miss testing of >> java.lang.Class/Process/Thread/String >> > and some other classes. >> > >> > While we have failing tests and don't want to pay attention to these >> > failures we can: >> > 1) Leave things as is – do not run TestCases with failing tests. >> > 2) Split passing/failing TestCase into separate "failing TestCase" and >> > "passing TestCase" and exclude "failing TestCases". When test or >> > implementation is fixed we move tests from failing TestCase to passing >> > TestCase. >> > 3) Comment failing tests in TestCases. It is better to run 58 tests >> > instead >> > of 0 for String. >> > 4) Run all TestCases, then, compare test run results with the 'list of >> > known >> > failures' and see whether new failures appeared. This, I think, is >> better >> > then 1, 2 and 3, but, overhead is that we support 2 lists - list of >> known >> > failing tests and exclude list where we put crashing tests. >> > >> > Thanks, Vladimir >> > On 6/26/06, Tim Ellison <[EMAIL PROTECTED]> wrote: >> >> >> >> Mikhail Loenko wrote: >> >> > Hi Vladimir, >> >> > >> >> > IMHO the tests are to verify that an update does not introduce any >> >> > regression. So there are two options: remember which exactly >> tests may >> >> fail >> >> > and remember that all tests must pass. I believe the latter one is >> >> a bit >> >> > easier and safer. >> >> >> >> +1 >> >> >> >> Tim >> >> >> >> > Thanks, >> >> > Mikhail >> >> > >> >> > 2006/6/26, Vladimir Ivanov <[EMAIL PROTECTED]>: >> >> >> Hi, >> >> >> Working with tests I noticed that we are excluding some tests just >> >> >> because >> >> >> several tests from single TestCase fail. >> >> >> >> >> >> For example, the TestCase 'tests.api.java.lang.StringTest' has 60 >> >> >> tests and >> >> >> only 2 of them fails. But the build excludes the whole TestCase >> >> and we >> >> >> just >> >> >> miss testing of java.lang.String implementation. >> >> >> >> >> >> Do we really need to exclude TestCases in 'ant test' target? >> >> >> >> >> >> My suggestion is: do not exclude any tests until it crashes VM. >> >> >> If somebody needs a list of tests that always passed a separated >> >> >> target can >> >> >> be added to build. >> >> >> >> >> >> Do you think we should add target 'test-all' to the build? >> >> >> Thanks, Vladimir >> >> >> >> >> >> >> >> > >> >> > >> --------------------------------------------------------------------- >> >> > Terms of use : http://incubator.apache.org/harmony/mailing.html >> >> > To unsubscribe, e-mail: [EMAIL PROTECTED] >> >> > For additional commands, e-mail: >> [EMAIL PROTECTED] >> >> > >> >> > >> >> >> >> -- >> >> >> >> Tim Ellison ([EMAIL PROTECTED]) >> >> IBM Java technology centre, UK. >> >> >> >> --------------------------------------------------------------------- >> >> Terms of use : http://incubator.apache.org/harmony/mailing.html >> >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> >> For additional commands, e-mail: [EMAIL PROTECTED] >> >> >> >> >> > >> >> -- >> Richard Liang >> China Software Development Lab, IBM > > > -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
