By groups annotation, we can group them by arch.<arch id>, type.impl, os.<os id>, vm.<vm id>. These groups indicate that we have different behaviors but it is allowed. Again, we can keep track of them without moving them to other directories by TestNG.
2008/6/12 Nathan Beyer <[EMAIL PROTECTED]>: > I think having a mechanism for exclusions of failing tests is bad, > especially any mechanism that allows excluding tests to sit hidden within > working tests. I think tests that are excluded because they are failing and > aren't going to be fixed need to be moved out of the running tests and > dropped in an separate folder like "src/test/failing" so that they stand > out. > > -Nathan > > On Wed, Jun 11, 2008 at 6:46 AM, Regis <[EMAIL PROTECTED]> wrote: > >> >> Junit 4 has @ignore annotation which of target is method, which can ignore >> failing methods. >> But some modules, such as luni, have different exclude lists on different >> os or vm, I haven't >> found any ways to do this using @ignore annotation. While TestNG's @group >> annotation >> can do this easily, just define groups for different exclude lists. >> >> Best Regards, >> Regis. >> >> >> Alexei Zakharov wrote: >> >>> As far as I understand in spite of the fact there were no >>> TestNG-related discussions since 2006 the problem is still relevant. >>> There are big exclude lists in some classlib modules still, and many >>> tests are excluded only because of a couple of failing methods. >>> Frankly speaking I'm not familiar with new feature introduced in Junit >>> 4.4. Are there any enhancements that can help to resolve this >>> exclude-whole-class-because-of-one-bad-method issue? >>> >>> Thanks, >>> Alexei >>> >>> 2008/6/11, Regis <[EMAIL PROTECTED]>: >>> >>>> Nathan Beyer wrote: >>>> >>>>> On Fri, Jun 6, 2008 at 1:08 AM, Regis <[EMAIL PROTECTED]> wrote: >>>>> >>>>> >>>>> Hi, >>>>>> >>>>>> Matcher and Assumptions are great ideas! Thanks Nathan. >>>>>> They would be very helpful for our new test cases. But I notice that >>>>>> Junit 4.4 doesn't support group which is very important feature for >>>>>> both old tests and new tests. We can partition our test suite and run >>>>>> them separately. It's make our tests more flexible and configurable, >>>>>> and it's the main reason we discuss to migrate to TestNG long time ago. >>>>>> >>>>>> >>>>> Don't we partition our tests already? Isn't that what the 'api' and >>>>> 'impl' >>>>> folders are about? >>>>> >>>>> Yes, but it's not enough. We have discussed and created a wiki page[1] >>>> about >>>> how >>>> to configuration and group harmony tests. The page is a little old, but I >>>> think the problems >>>> it tried to resolve still exist now. The partitions are not only include >>>> 'api' and 'impl', but also >>>> include partition of different os, architecture, partition of broken >>>> tests >>>> and level of tests. >>>> folder structure or exclude files can't help in this complex situation, >>>> so >>>> we need some tools >>>> to help us to deal with this, i think TestNG is suitable. If JUnit 4.4 >>>> can >>>> do it, i will vote to JUnit, >>>> update to a new version is always easier than switch to a new tool after >>>> all. >>>> >>>> [1] http://wiki.apache.org/harmony/Testing_Convention >>>> >>>> Best Regards, >>>> Regis. >>>> >>>> >>>> -Nathan >>>>> >>>>> >>>>> >>>>> Best Regards, >>>>>> Regis. >>>>>> >>>>>> >>>>>> Nathan Beyer wrote: >>>>>> >>>>>> >>>>>> That discussion was a very long time ago. Is there still value in >>>>>>> >>>>>> TestNG? >>>> >>>>> I'd prefer to move to JUnit 4.4. All of our current tests will >>>>>>> >>>>>> continue to >>>> >>>>> work and new tests can be implemented using the latest conventions and >>>>>>> older >>>>>>> tests can be updated as we get to them. JUnit 4.4 is a far cry from >>>>>>> >>>>>> 4.0. >>>> >>>>> Here's the things I think would be create for our use and testing in >>>>>>> general >>>>>>> - Matchers and the 'assertThat' - much more readable code and readable >>>>>>> failure messages >>>>>>> - Assumptions and the 'assumeThat' - allows methods to add statements >>>>>>> >>>>>> that >>>> >>>>> guarantee that preconditions for the test are correct; this allows >>>>>>> >>>>>> tests >>>> >>>>> to >>>>>>> fail such that you know it's an environment issue and not an actual >>>>>>> >>>>>> test >>>> >>>>> failure >>>>>>> >>>>>>> If you're not familiar with matchers, check out this quick tutorial - >>>>>>> http://code.google.com/p/hamcrest/wiki/Tutorial. >>>>>>> >>>>>>> -Nathan >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 5, 2008 at 10:21 PM, Sean Qiu <[EMAIL PROTECTED]> >>>>>>> >>>>>> wrote: >>>> >>>>> Hi, all. >>>>>>> >>>>>>> We had discussed the migration to testNG before and got some >>>>>>>> >>>>>>> conclusions >>>> >>>>> for >>>>>>>> grouping[1] >>>>>>>> including how to deal with boot path test[2]. Am i missing >>>>>>>> >>>>>>> something? >>>> >>>>> Is it still in our schedule? I think it's valueable to Harmony. >>>>>>>> I volunteer to carry out this job if no one objects. Any other >>>>>>>> volunteers? >>>>>>>> >>>>>>>> IMHO, we can only add some ant tasks to integrate testng at the >>>>>>>> beginning. >>>>>>>> So our original junit tests can still work at the mean time when >>>>>>>> migrating. >>>>>>>> When one module's migration task is finished, we can judge the >>>>>>>> >>>>>>> result >>>> >>>>> to dertermine whether we should go on for other modules. >>>>>>>> >>>>>>>> Maybe we can create a branch for luni to start this work, shall we? >>>>>>>> therefore there won't be any impact on other's development. >>>>>>>> Once it is completed in the branch, we could merge it back to our >>>>>>>> >>>>>>> trunk. >>>> >>>>> Does it make sense? >>>>>>>> >>>>>>>> Any sugestions or comments are welcomed. Thanks very much. >>>>>>>> >>>>>>>> [1] >>>>>>>> >>>>>>> http://wiki.apache.org/harmony/Testing_Convention >>>> >>>>> [2] >>>>>>>> >>>>>>>> >>>>>>>> >>>> http://www.mail-archive.com/[EMAIL PROTECTED]/msg12413.html >>>> >>>>> [3] >>>>>>>> >>>>>>> http://testng.org/doc/documentation-main.html#annotations >>>> >>>>> -- >>>>>>>> Best Regards >>>>>>>> Sean, Xiao Xia Qiu >>>>>>>> China Software Development Lab, IBM >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>> > -- Best Regards Sean, Xiao Xia Qiu China Software Development Lab, IBM
