>-----Original Message-----
>From: Andrew Zhang [mailto:[EMAIL PROTECTED]
>Sent: Monday, September 04, 2006 6:18 PM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [classlib][TestNG] groups of Harmony test
>
>On 9/4/06, Richard Liang <[EMAIL PROTECTED]> wrote:
>>
>> On 9/4/06, Mikhail Loenko <[EMAIL PROTECTED]> wrote:
>> > Well, my question was for what particular reason? for example?
>> >
>> > Tio verify correctness of bug-fixing IMHO all the unit,
intergration,
>> api, and
>> > regression tests should be run
>>
>> Running all tests are always good to verify our code quality. And I
>> think this is what we have been doing. But what will we do if it
takes
>> us 24 hours to run all Harmony tests? Anyway, running regression
tests
>> could provide some confidence to our bug-fixing. We may consider it
as
>> another option. :-)
>
>
>I prefer to run all tests in one module instead. :) Although it can not
>guarentee all tests would pass, it's less likey to break build system
>frequently. If the fix causes other module fails, it shows the lack of
>tests in its module. And we should add the corresponding test in the
module.
>
>Currently, Harmony regression test is a test for certain Harmony jira
issue.
>So IMHO, running all regression tests for certain issue doesn't make
sense.
>
>But whether using annotation to mark regression test is another story.
At
>least, it doesn't have any disadvantages compared with comment way,
does it?
Agree. We may introduce a special annotation for this purpose.
Regards,
Alexey.
>
>
>Best regards,
>> Richard
>>
>> >
>> > Thanks,
>> > Mikhail
>> >
>> > 2006/9/4, Richard Liang <[EMAIL PROTECTED]>:
>> > >
>> > >
>> > > Mikhail Loenko wrote:
>> > > > Hi Vladimir
>> > > >
>> > > > Could you please decribe for what purpose it will be used?
>> > > >
>> > > > I mean why one might have to either exclude or run only
regression
>> tests?
>> > >
>> > > If running all tests takes up much time, running all regression
test
>> > > (for one particular version) may be a convenient way to verify
the
>> > > correctness of bug-fixing.
>> > >
>> > > Best regards,
>> > > Richard.
>> > >
>> > > >
>> > > > Thanks,
>> > > > Mikhail
>> > > >
>> > > > 2006/8/30, Vladimir Ivanov <[EMAIL PROTECTED]>:
>> > > >> On 8/30/06, Richard Liang <[EMAIL PROTECTED]> wrote:
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > Vladimir Ivanov wrote:
>> > > >> > > Also some tag for regression tests should be added.
>> > > >> > Yes. Do you think we could annotate regression test as
>> > > >> > *level.regression*? Thanks a lot.
>> > > >>
>> > > >>
>> > > >> Yes, I do. While tests can have more than one group it will
enough.
>> > > >> thanks, Vladimir
>> > > >>
>> > > >>
>> > > >> Richard
>> > > >> > > thanks, Vladimir
>> > > >> > >
>> > > >> > >
>> > > >> > > On 8/28/06, Richard Liang <[EMAIL PROTECTED]>
wrote:
>> > > >> > >>
>> > > >> > >>
>> > > >> > >>
>> > > >> > >> Richard Liang wrote:
>> > > >> > >> > Hello All,
>> > > >> > >> >
>> > > >> > >> > Now let's talk about the TestNG groups. I have read the
>> related
>> > > >> > >> > threads which posted by George, Vladimir Ivanov and
Alexei
>> > > >> Zakharov.
>> > > >> > >> > All of them are good discussion about TestNG groups.
>> > > >> > >> >
>> > > >> > >> > IMHO, we may define Harmony test groups according the
>> following 4
>> > > >> > >> > dimensions:
>> > > >> > >> >
>> > > >> > >> > 1) [Platform] os.any, os.<platform id>
>> > > >> > >> > *os.any* - group of tests which pass on any platform.
IMHO,
>> > > >> most of
>> > > >> > >> > our tests should be in this group.
>> > > >> > >> > *os.<platform id>* - group of tests which are designed
for
>> one
>> > > >> > >> > specific platform. A test may be in more than one of
the
>> > > >> groups. e.g
>> > > >> > .,
>> > > >> > >> > @Test(groups={"os.win.IA32", "os.linux.IA32"})
>> > > >> > >> >
>> > > >> > >> > ** os.any and os.<platform id> are mutually
exclusive,
>> that
>> > > >> is,
>> > > >> > >> > tests in os.any group should not be in os.win.IA32.
>> > > >> > >> >
>> > > >> > >> > 2) [Test state] state.broken, state.broken.<platform
id>
>> > > >> > >> > *state.broken* - group of tests which fail on every
>> platform,
>> > > >> because
>> > > >> > >> > of bugs of tests or implementation. We need to fix the
bugs
>> of
>> > > >> tests
>> > > >> > >> > or implementation to make them pass.
>> > > >> > >> > *state.broken.<platform id>* - groups of test which
only
>> fail
>> > > >> on one
>> > > >> > >> > specific platform. A test may be in more than one of
the
>> > > >> groups. e.g
>> > > >> > .,
>> > > >> > >> > @Test(groups={"state.broken.linux.IA32", "
>> os.broken.linux.IA64"})
>> > > >> > >> >
>> > > >> > >> > **state.broken.<platform id> group may be used as a
>> > > >> convenient
>> > > >> > way
>> > > >> > >> > to indicate that a test is platform-specific. e.g., If
we
>> > > >> support 10
>> > > >> > >> > platforms, and one test are designed for 9 platforms
except
>> for
>> > > >> > MacOS,
>> > > >> > >> > instead of list 9 os.<platform id>, we can just use
>> > > >> > state.broken.MacOS
>> > > >> > >> >
>> > > >> > >> > 3) [Test type] type.api , type.impl
>> > > >> > >> > *type.api* - group of tests which are tests for APIs in
the
>> Java
>> > > >> > >> > Specification
>> > > >> > >> > *type.impl* - groups of tests which are tests for
>> > > >> Harmony-specific
>> > > >> > >> > implementation
>> > > >> > >> >
>> > > >> > >> > ** type.api and type.impl are also mutually
exclusive.
>> > > >> > >> >
>> > > >> > >> > 4) [Test Level] level.unit, level.integration,
level.system,
>> > > >> > >> > level.stress, etc. (Levels of Test refer to the
increase in
>> > > >> > complexity
>> > > >> > >> > as moving through test cycle. )
>> > > >> > >> > ** A test may be in more than one of the groups.
>> > > >> > >> > ** In fact, some tests such as System tests are the
>> > > >> verification
>> > > >> > of
>> > > >> > >> > the entire system. Maybe we'll put them into a
separate
>> project.
>> > > >> > >> > e.g., harmony/enhanced/SVT (System Verification Test).
>> > > >> > >> >
>> > > >> > >> > If we want to run all the unit test for APIs on
windows, we
>> > > >> may use
>> > > >> > >> > TestNG groups to select the tests:
>> > > >> > >> > <groups>
>> > > >> > >> > <run>
>> > > >> > >> > <include name="os.any" />
>> > > >> > >> > <include name="type.api" />
>> > > >> > >> > <include name="os.win.IA32" />
>> > > >> > >> > <exclude name=" state.broken" />
>> > > >> > >> > <exclude name="state.broken.win.IA32" />
>> > > >> > >> > </run>
>> > > >> > >> > </groups>
>> > > >> > >> >
>> > > >> > >> Hello All,
>> > > >> > >>
>> > > >> > >> I'm sorry. It seems that the example does not work. I
will
>try
>> to
>> > > >> > figure
>> > > >> > >> another example soon. ;-)
>> > > >> > >>
>> > > >> > >> Best regards,
>> > > >> > >> Richard
>> > > >> > >> >
>> > > >> > >> > Well, I think our most of existing tests are in the
groups
>> of
>> > > >> > >> > {"os.any", "type.api", "level.unit"}, and I have asked
>> TestNG
>> > > >> to add
>> > > >> > a
>> > > >> > >> > new option "-groups" for its JUnitConverter which allow
us
>> to
>> > > >> specify
>> > > >> > >> > the test groups when migrate from JUnit test to TestNG
test.
>> > > >> > >> >
>> > > >> > >> > Thanks for reading so far, and I will highly appreciate
>your
>> > > >> comments
>> > > >> > >> > or suggestion. ;-)
>> > > >> > >> >
>> > > >> > >>
>> > > >> > >> --
>> > > >> > >> Richard Liang
>> > > >> > >> China Software Development Lab, IBM
>> > > >> > >>
>> > > >> > >>
>> > > >> > >>
>> > > >> > >>
>> > > >>
>> ---------------------------------------------------------------------
>> > > >> > >> Terms of use :
>> http://incubator.apache.org/harmony/mailing.html
>> > > >> > >> To unsubscribe, e-mail:
>> > > >> [EMAIL PROTECTED]
>> > > >> > >> For additional commands, e-mail:
>> > > >> [EMAIL PROTECTED]
>> > > >> > >>
>> > > >> > >>
>> > > >> > >
>> > > >> >
>> > > >> > --
>> > > >> > Richard Liang
>> > > >> > China Software Development Lab, IBM
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> >
>> ---------------------------------------------------------------------
>> > > >> > Terms of use :
http://incubator.apache.org/harmony/mailing.html
>> > > >> > To unsubscribe, e-mail:
>> [EMAIL PROTECTED]
>> > > >> > For additional commands, e-mail:
>> [EMAIL PROTECTED]
>> > > >> >
>> > > >> >
>> > > >>
>> > > >>
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > > > To unsubscribe, e-mail: harmony-dev-
>[EMAIL PROTECTED]
>> > > > For additional commands, e-mail:
>> [EMAIL PROTECTED]
>> > > >
>> > > >
>> > >
>> > > --
>> > > Richard Liang
>> > > China Software Development Lab, IBM
>> > >
>> > >
>> > >
>> > >
---------------------------------------------------------------------
>> > > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > > To unsubscribe, e-mail:
[EMAIL PROTECTED]
>> > > For additional commands, e-mail: harmony-dev-
>[EMAIL PROTECTED]
>> > >
>> > >
>> >
>> >
---------------------------------------------------------------------
>> > Terms of use : http://incubator.apache.org/harmony/mailing.html
>> > To unsubscribe, e-mail:
[EMAIL PROTECTED]
>> > For additional commands, e-mail:
[EMAIL PROTECTED]
>> >
>> >
>>
>>
>> --
>> Richard Liang
>> China Software Development Lab, IBM
>>
>> ---------------------------------------------------------------------
>> Terms of use : http://incubator.apache.org/harmony/mailing.html
>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>> For additional commands, e-mail:
[EMAIL PROTECTED]
>>
>>
>
>
>--
>Andrew Zhang
>China Software Development Lab, IBM
--
Alexey A. Ivanov
Intel Middleware Product Division
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
