Stepan Mishura wrote:
On 8/25/06, Richard Liang wrote:
Hello All,
Now let's talk about the TestNG groups. I have read the related threads
which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them
are good discussion about TestNG groups.
IMHO, we may define Harmony test groups according the following 4
dimensions:
1) [Platform] os.any, os.<platform id>
*os.any* - group of tests which pass on any platform. IMHO, most of our
tests should be in this group.
*os.<platform id>* - group of tests which are designed for one specific
platform. A test may be in more than one of the groups. e.g.,
@Test(groups={"os.win.IA32", "os.linux.IA32"})
** os.any and os.<platform id> are mutually exclusive, that is,
tests in os.any group should not be in os.win.IA32.
2) [Test state] state.broken, state.broken.<platform id>
*state.broken* - group of tests which fail on every platform, because
of bugs of tests or implementation. We need to fix the bugs of tests or
implementation to make them pass.
*state.broken.<platform id>* - groups of test which only fail on one
specific platform. A test may be in more than one of the groups. e.g.,
@Test(groups={"state.broken.linux.IA32", "os.broken.linux.IA64"})
**state.broken.<platform id> group may be used as a convenient way
to indicate that a test is platform-specific. e.g., If we support 10
platforms, and one test are designed for 9 platforms except for MacOS,
instead of list 9 os.<platform id>, we can just use state.broken.MacOS
If a test is marked as *state.broken.MacOS* then it sounds like the
test/implementation should be fixed. IMO we should use tag
os.<platform id>
to define explicitly valid platforms for the test so in this
particular case
we should use 9 os.<platform id>s
Thank you, Stepan. This sounds reasonable, we should explicitly list
the valid platforms.
Best regards,
Richard
3) [Test type] type.api, type.impl
*type.api* - group of tests which are tests for APIs in the Java
Specification
*type.impl* - groups of tests which are tests for Harmony-specific
implementation
** type.api and type.impl are also mutually exclusive.
4) [Test Level] level.unit, level.integration, level.system,
level.stress, etc. (Levels of Test refer to the increase in complexity
as moving through test cycle. )
** A test may be in more than one of the groups.
** In fact, some tests such as System tests are the verification of
the entire system. Maybe we'll put them into a separate project. e.g.,
harmony/enhanced/SVT (System Verification Test).
Mixing different types of testing into one test-file doesn't look good
for
me. I'd separate such tests by placing into different
directories/packages.
Thanks,
Stepan.
If we want to run all the unit test for APIs on windows, we may use
TestNG groups to select the tests:
<groups>
<run>
<include name="os.any" />
<include name="type.api" />
<include name="os.win.IA32" />
<exclude name="state.broken" />
<exclude name="state.broken.win.IA32" />
</run>
</groups>
Well, I think our most of existing tests are in the groups of {"os.any",
"type.api", "level.unit"}, and I have asked TestNG to add a new option
"-groups" for its JUnitConverter which allow us to specify the test
groups when migrate from JUnit test to TestNG test.
Thanks for reading so far, and I will highly appreciate your comments or
suggestion. ;-)
--
Richard Liang
China Software Development Lab, IBM
------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Richard Liang
China Software Development Lab, IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
