Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Alexei Zakharov]

Stepan, All,


This is not a good argument for using os.any. Even more it may sound like
we are going to choose wrong tool - why we have to add os.any to 99% of
classlib tests? just because the harness can not do without it?


Well, we have to add the new stuff to these 99% anyway, I mean the
annotation itself. And the copy-pasting of longer block doesn't seem
to be harder than copy-pasting of short block to me.


As I understand arguments in the old thread - TestNG makes us to use 
os.any annotation otherwise we have to do a lot of tricks to run tests,
right?


This was the one of the arguments.


So a test's annotation should point out that the test is a special one.


I was thinking about a little bit different approach. I will try to
explain. The idea was to use annotations as a *single* place for the
complete information about test: its type, target OS(es) and etc.
Something like inlined test descriptor. Not a red flag. In case
someone is looking at the test at the first time he will get a
complete information just by reading the annotation located at the
same place with the java code. If the annotation itself is not obvious
then the developer is obliged to go to some page with annotation
descriptions, read it carefully and etc. And it seems we get two
places with information about single test: the test's source file and
the page with decoding rules for annotations. This IMHO reduces the
value of the main TestNG benefit - to have all information about the
test in single place. But if we are ok to have several places - we may
use Junit TestSuites instead of TestNG. No refactoring is needed at
all.

To put this another way, these efforts doesn't seem to be so
unnecessary to me. Efforts are required to make the life better
sometimes IMHO. Does this make sense to anyone?


Thanks for the reading,

2006/9/12, Stepan Mishura [EMAIL PROTECTED]:

On 9/11/06, Alexei Zakharov  wrote:

 Hi all,

  One more note (seems it already was said sorry if I repeat): the test
  without any marks should be run in all configurations (i.e. we have
  'default' group but declaration of this group may be missed).

 I'd like to point your attention on the previous discussion about
 default groups :

 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]

 I am still for using os.any since it is more self-descriptive and
 the build script will be simpler with os.any.



This is not a good argument for using os.any. Even more it may sound like
we are going to choose wrong tool - why we have to add os.any to 99% of
classlib tests? just because the harness can not do without it?


It will be nice to
 hear more arguments for using defaults because it seems the arguments
 that were gathered in that old thread hasn't been taken into account
 by participants of this thread.



As I understand arguments in the old thread - TestNG makes us to use 
os.any annotation otherwise we have to do a lot of tricks to run tests,
right?

IMO a test annotation should be used as red flag for developer, for
example
* state.broken - hey, i'm broken fix me please
* os.win - i'm valid only for Windows, don't try to run me on Linux

So a test's annotation should point out that the test is a special one. But
if most of tests will contain a similar block of annotations then nobody
will looked at them. Does this make sense?

Thanks,
Stepan.

Thanks,

 2006/9/5, Vladimir Ivanov [EMAIL PROTECTED]:
  One more note (seems it already was said sorry if I repeat): the test
  without any marks should be run in all configurations (i.e. we have
  'default' group but declaration of this group may be missed).
 
   thanks, Vladimir
 
 
  On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:
  
OK, let's return back to the usage model.
   If I understood it correctly, before the commit of any changes each
   developer run *all* tests (at least all which we have now) on all
 available
   to him platforms. In this context seems we don't need in any 'level'
 group
   (while 'stress' tests require reasonable time to pass).
   Seems, that platform group also can be deleted (at present time we
 have
   10 platform-dependent tests and this amount should not increase
   dramatically so the platform-detection can be included to the each
 such
   test).
   Also cpu groups can be deleted (while we have not cpu-dependent
 test).
   At the end we need only state groups to support test exclusion on
 the
   'one-element' level (while we have unresolved entries in the current
 exclude
   list).
  
   So, after small update of unit (aka integration, aka regression etc)
 tests
   and resolution of all entries in the exclude list we don't need any
 groups
   and pure JUnit covers all our needs :)
  
   On the other side, if we define some groups it will nice to define
 *all*
   reasonable groups at the begin of the process.
  
thanks, Vladimir
  
   By the way, our regression tests are 'classic' regression tests that
   demonstrate some issues 

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Stepan Mishura]

On 9/11/06, Alexei Zakharov  wrote:


Hi all,

 One more note (seems it already was said sorry if I repeat): the test
 without any marks should be run in all configurations (i.e. we have
 'default' group but declaration of this group may be missed).

I'd like to point your attention on the previous discussion about
default groups :

http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

I am still for using os.any since it is more self-descriptive and
the build script will be simpler with os.any.




This is not a good argument for using os.any. Even more it may sound like
we are going to choose wrong tool - why we have to add os.any to 99% of
classlib tests? just because the harness can not do without it?


It will be nice to

hear more arguments for using defaults because it seems the arguments
that were gathered in that old thread hasn't been taken into account
by participants of this thread.




As I understand arguments in the old thread - TestNG makes us to use 
os.any annotation otherwise we have to do a lot of tricks to run tests,
right?

IMO a test annotation should be used as red flag for developer, for
example
* state.broken - hey, i'm broken fix me please
* os.win - i'm valid only for Windows, don't try to run me on Linux

So a test's annotation should point out that the test is a special one. But
if most of tests will contain a similar block of annotations then nobody
will looked at them. Does this make sense?

Thanks,
Stepan.

Thanks,


2006/9/5, Vladimir Ivanov [EMAIL PROTECTED]:
 One more note (seems it already was said sorry if I repeat): the test
 without any marks should be run in all configurations (i.e. we have
 'default' group but declaration of this group may be missed).

  thanks, Vladimir


 On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:
 
   OK, let's return back to the usage model.
  If I understood it correctly, before the commit of any changes each
  developer run *all* tests (at least all which we have now) on all
available
  to him platforms. In this context seems we don't need in any 'level'
group
  (while 'stress' tests require reasonable time to pass).
  Seems, that platform group also can be deleted (at present time we
have
  10 platform-dependent tests and this amount should not increase
  dramatically so the platform-detection can be included to the each
such
  test).
  Also cpu groups can be deleted (while we have not cpu-dependent
test).
  At the end we need only state groups to support test exclusion on
the
  'one-element' level (while we have unresolved entries in the current
exclude
  list).
 
  So, after small update of unit (aka integration, aka regression etc)
tests
  and resolution of all entries in the exclude list we don't need any
groups
  and pure JUnit covers all our needs :)
 
  On the other side, if we define some groups it will nice to define
*all*
  reasonable groups at the begin of the process.
 
   thanks, Vladimir
 
  By the way, our regression tests are 'classic' regression tests that
  demonstrate some issues which were not resolved by initial code. But
it
  provides less coverage than 'regression tests' + unit tests, of cause.
 
   On 9/5/06, Richard Liang [EMAIL PROTECTED]  wrote:
  
   On 9/5/06, Alex Blewitt  [EMAIL PROTECTED] wrote:
On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
 On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
 
  If you've got fast and slow tests, then have a group for fast
and
   slow
  tests. Then you can choose to just run the fast tests, and any
  automated build system can handle running the slow tests.

 IMHO, fast or slow may not be the key point. The question is
   whether we
 have any requirements to run only the regression tests.
   
No, probably not the key point, but (a) groups don't have to be
mutually exclusive (so you can decorate it with whatever groups
you
want)
  
   I agree. For example, os.win and os.linux are not mutually
exclusive.
  
   Thanks a lot.
  
   and (b) it might be useful for an automated build system to run
fast tests first, followed by slow (or non-fast) tests.
  
   That makes sense through we have not clear requirement currently.
  
Mind you, I don't know what's going to happen with an automated
   test'n'build
system; so it might not make sense to do it at this point.
  
   Really? ;-) We could also discuss whether it's feasible to move to
   TestNG. As you may know, there are already several threads about
   TestNG  JUnit. Here I just review the open questions one by one so
   that we have sufficient preparation.
  
  
   [1]http://mail-
archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
  
   [2]http://mail-
archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
  
   [3]http://mail-
archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
  
  
   Best regards,
   Richard
  
   
Alex.
   

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

On 9/12/06, Stepan Mishura [EMAIL PROTECTED] wrote:

On 9/11/06, Alexei Zakharov  wrote:

 Hi all,

  One more note (seems it already was said sorry if I repeat): the test
  without any marks should be run in all configurations (i.e. we have
  'default' group but declaration of this group may be missed).

 I'd like to point your attention on the previous discussion about
 default groups :

 
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]

 I am still for using os.any since it is more self-descriptive and
 the build script will be simpler with os.any.



This is not a good argument for using os.any. Even more it may sound like
we are going to choose wrong tool - why we have to add os.any to 99% of
classlib tests? just because the harness can not do without it?



Yes. And we have found a way to avoid os.any ;-)



It will be nice to
 hear more arguments for using defaults because it seems the arguments
 that were gathered in that old thread hasn't been taken into account
 by participants of this thread.



As I understand arguments in the old thread - TestNG makes us to use 
os.any annotation otherwise we have to do a lot of tricks to run tests,
right?

IMO a test annotation should be used as red flag for developer, for
example
* state.broken - hey, i'm broken fix me please
* os.win - i'm valid only for Windows, don't try to run me on Linux

So a test's annotation should point out that the test is a special one. But
if most of tests will contain a similar block of annotations then nobody
will looked at them. Does this make sense?


Yes. It really makes sense. We do not want to involve developers in
unnecessary effort.



Thanks,
Stepan.

Thanks,

 2006/9/5, Vladimir Ivanov [EMAIL PROTECTED]:
  One more note (seems it already was said sorry if I repeat): the test
  without any marks should be run in all configurations (i.e. we have
  'default' group but declaration of this group may be missed).
 
   thanks, Vladimir
 
 
  On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:
  
OK, let's return back to the usage model.
   If I understood it correctly, before the commit of any changes each
   developer run *all* tests (at least all which we have now) on all
 available
   to him platforms. In this context seems we don't need in any 'level'
 group
   (while 'stress' tests require reasonable time to pass).
   Seems, that platform group also can be deleted (at present time we
 have
   10 platform-dependent tests and this amount should not increase
   dramatically so the platform-detection can be included to the each
 such
   test).
   Also cpu groups can be deleted (while we have not cpu-dependent
 test).
   At the end we need only state groups to support test exclusion on
 the
   'one-element' level (while we have unresolved entries in the current
 exclude
   list).
  
   So, after small update of unit (aka integration, aka regression etc)
 tests
   and resolution of all entries in the exclude list we don't need any
 groups
   and pure JUnit covers all our needs :)
  
   On the other side, if we define some groups it will nice to define
 *all*
   reasonable groups at the begin of the process.
  
thanks, Vladimir
  
   By the way, our regression tests are 'classic' regression tests that
   demonstrate some issues which were not resolved by initial code. But
 it
   provides less coverage than 'regression tests' + unit tests, of cause.
  
On 9/5/06, Richard Liang [EMAIL PROTECTED]  wrote:
   
On 9/5/06, Alex Blewitt  [EMAIL PROTECTED] wrote:
 On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
  On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
  
   If you've got fast and slow tests, then have a group for fast
 and
slow
   tests. Then you can choose to just run the fast tests, and any
   automated build system can handle running the slow tests.
 
  IMHO, fast or slow may not be the key point. The question is
whether we
  have any requirements to run only the regression tests.

 No, probably not the key point, but (a) groups don't have to be
 mutually exclusive (so you can decorate it with whatever groups
 you
 want)
   
I agree. For example, os.win and os.linux are not mutually
 exclusive.
   
Thanks a lot.
   
and (b) it might be useful for an automated build system to run
 fast tests first, followed by slow (or non-fast) tests.
   
That makes sense through we have not clear requirement currently.
   
 Mind you, I don't know what's going to happen with an automated
test'n'build
 system; so it might not make sense to do it at this point.
   
Really? ;-) We could also discuss whether it's feasible to move to
TestNG. As you may know, there are already several threads about
TestNG  JUnit. Here I just review the open questions one by one so
that we have sufficient preparation.
   
   
[1]http://mail-
 

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Alexei Zakharov]

Hi all,


One more note (seems it already was said sorry if I repeat): the test
without any marks should be run in all configurations (i.e. we have
'default' group but declaration of this group may be missed).


I'd like to point your attention on the previous discussion about
default groups :
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

I am still for using os.any since it is more self-descriptive and
the build script will be simpler with os.any. It will be nice to
hear more arguments for using defaults because it seems the arguments
that were gathered in that old thread hasn't been taken into account
by participants of this thread.

Thanks,

2006/9/5, Vladimir Ivanov [EMAIL PROTECTED]:

One more note (seems it already was said sorry if I repeat): the test
without any marks should be run in all configurations (i.e. we have
'default' group but declaration of this group may be missed).

 thanks, Vladimir


On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:

  OK, let's return back to the usage model.
 If I understood it correctly, before the commit of any changes each
 developer run *all* tests (at least all which we have now) on all available
 to him platforms. In this context seems we don't need in any 'level' group
 (while 'stress' tests require reasonable time to pass).
 Seems, that platform group also can be deleted (at present time we have
 10 platform-dependent tests and this amount should not increase
 dramatically so the platform-detection can be included to the each such
 test).
 Also cpu groups can be deleted (while we have not cpu-dependent test).
 At the end we need only state groups to support test exclusion on the
 'one-element' level (while we have unresolved entries in the current exclude
 list).

 So, after small update of unit (aka integration, aka regression etc) tests
 and resolution of all entries in the exclude list we don't need any groups
 and pure JUnit covers all our needs :)

 On the other side, if we define some groups it will nice to define *all*
 reasonable groups at the begin of the process.

  thanks, Vladimir

 By the way, our regression tests are 'classic' regression tests that
 demonstrate some issues which were not resolved by initial code. But it
 provides less coverage than 'regression tests' + unit tests, of cause.

  On 9/5/06, Richard Liang [EMAIL PROTECTED]  wrote:
 
  On 9/5/06, Alex Blewitt  [EMAIL PROTECTED] wrote:
   On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:

 If you've got fast and slow tests, then have a group for fast and
  slow
 tests. Then you can choose to just run the fast tests, and any
 automated build system can handle running the slow tests.
   
IMHO, fast or slow may not be the key point. The question is
  whether we
have any requirements to run only the regression tests.
  
   No, probably not the key point, but (a) groups don't have to be
   mutually exclusive (so you can decorate it with whatever groups you
   want)
 
  I agree. For example, os.win and os.linux are not mutually exclusive.
 
  Thanks a lot.
 
  and (b) it might be useful for an automated build system to run
   fast tests first, followed by slow (or non-fast) tests.
 
  That makes sense through we have not clear requirement currently.
 
   Mind you, I don't know what's going to happen with an automated
  test'n'build
   system; so it might not make sense to do it at this point.
 
  Really? ;-) We could also discuss whether it's feasible to move to
  TestNG. As you may know, there are already several threads about
  TestNG  JUnit. Here I just review the open questions one by one so
  that we have sufficient preparation.
 
 
  
[1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
 
  
[2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
 
  
[3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
 
 
  Best regards,
  Richard
 
  
   Alex.
  




--
Alexei Zakharov,
Intel Middleware Product Division

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

On 9/11/06, Alexei Zakharov [EMAIL PROTECTED] wrote:

Hi all,

 One more note (seems it already was said sorry if I repeat): the test
 without any marks should be run in all configurations (i.e. we have
 'default' group but declaration of this group may be missed).

I'd like to point your attention on the previous discussion about
default groups :
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

I am still for using os.any since it is more self-descriptive and
the build script will be simpler with os.any. It will be nice to
hear more arguments for using defaults because it seems the arguments
that were gathered in that old thread hasn't been taken into account
by participants of this thread.


I have not any strong objection about os.any. And actually I had
ever proposed to define the default group because we could not
include tests with annotation @Test which belong to no groups. Now it
isn't a problem as we already have a solution for them. To facilitate
writing test cases, we annotate the unit tests which are designed to
pass on all platforms (os + arch) with @Test.

If we use os.any and arch.any, then the default annotation would
be @Test(groups={os.any, arch.any})

Could any other give more comments? Thanks a lot.

Best regards,
Richard





Thanks,

2006/9/5, Vladimir Ivanov [EMAIL PROTECTED]:
 One more note (seems it already was said sorry if I repeat): the test
 without any marks should be run in all configurations (i.e. we have
 'default' group but declaration of this group may be missed).

  thanks, Vladimir


 On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:
 
   OK, let's return back to the usage model.
  If I understood it correctly, before the commit of any changes each
  developer run *all* tests (at least all which we have now) on all available
  to him platforms. In this context seems we don't need in any 'level' group
  (while 'stress' tests require reasonable time to pass).
  Seems, that platform group also can be deleted (at present time we have
  10 platform-dependent tests and this amount should not increase
  dramatically so the platform-detection can be included to the each such
  test).
  Also cpu groups can be deleted (while we have not cpu-dependent test).
  At the end we need only state groups to support test exclusion on the
  'one-element' level (while we have unresolved entries in the current exclude
  list).
 
  So, after small update of unit (aka integration, aka regression etc) tests
  and resolution of all entries in the exclude list we don't need any groups
  and pure JUnit covers all our needs :)
 
  On the other side, if we define some groups it will nice to define *all*
  reasonable groups at the begin of the process.
 
   thanks, Vladimir
 
  By the way, our regression tests are 'classic' regression tests that
  demonstrate some issues which were not resolved by initial code. But it
  provides less coverage than 'regression tests' + unit tests, of cause.
 
   On 9/5/06, Richard Liang [EMAIL PROTECTED]  wrote:
  
   On 9/5/06, Alex Blewitt  [EMAIL PROTECTED] wrote:
On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
 On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
 
  If you've got fast and slow tests, then have a group for fast and
   slow
  tests. Then you can choose to just run the fast tests, and any
  automated build system can handle running the slow tests.

 IMHO, fast or slow may not be the key point. The question is
   whether we
 have any requirements to run only the regression tests.
   
No, probably not the key point, but (a) groups don't have to be
mutually exclusive (so you can decorate it with whatever groups you
want)
  
   I agree. For example, os.win and os.linux are not mutually exclusive.
  
   Thanks a lot.
  
   and (b) it might be useful for an automated build system to run
fast tests first, followed by slow (or non-fast) tests.
  
   That makes sense through we have not clear requirement currently.
  
Mind you, I don't know what's going to happen with an automated
   test'n'build
system; so it might not make sense to do it at this point.
  
   Really? ;-) We could also discuss whether it's feasible to move to
   TestNG. As you may know, there are already several threads about
   TestNG  JUnit. Here I just review the open questions one by one so
   that we have sufficient preparation.
  
  
   
[1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
  
   
[2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
  
   
[3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
  
  
   Best regards,
   Richard
  
   
Alex.
   



--
Alexei Zakharov,
Intel Middleware Product Division

-
Terms of use : 

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Andrew Zhang]

On 9/11/06, Richard Liang [EMAIL PROTECTED] wrote:


On 9/11/06, Alexei Zakharov [EMAIL PROTECTED] wrote:
 Hi all,

  One more note (seems it already was said sorry if I repeat): the test
  without any marks should be run in all configurations (i.e. we have
  'default' group but declaration of this group may be missed).

 I'd like to point your attention on the previous discussion about
 default groups :

http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

 I am still for using os.any since it is more self-descriptive and
 the build script will be simpler with os.any. It will be nice to
 hear more arguments for using defaults because it seems the arguments
 that were gathered in that old thread hasn't been taken into account
 by participants of this thread.

I have not any strong objection about os.any. And actually I had
ever proposed to define the default group because we could not
include tests with annotation @Test which belong to no groups. Now it
isn't a problem as we already have a solution for them. To facilitate
writing test cases, we annotate the unit tests which are designed to
pass on all platforms (os + arch) with @Test.

If we use os.any and arch.any, then the default annotation would
be @Test(groups={os.any, arch.any})

Could any other give more comments? Thanks a lot.



Either is ok. One is more descriptive while the other is more convenient.

If we have no problem to write test.xml with default group(@Test), then I
prefer this option a little. Thanks!

Best regards,

Richard




 Thanks,

 2006/9/5, Vladimir Ivanov [EMAIL PROTECTED]:
  One more note (seems it already was said sorry if I repeat): the test
  without any marks should be run in all configurations (i.e. we have
  'default' group but declaration of this group may be missed).
 
   thanks, Vladimir
 
 
  On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:
  
OK, let's return back to the usage model.
   If I understood it correctly, before the commit of any changes each
   developer run *all* tests (at least all which we have now) on all
available
   to him platforms. In this context seems we don't need in any 'level'
group
   (while 'stress' tests require reasonable time to pass).
   Seems, that platform group also can be deleted (at present time we
have
   10 platform-dependent tests and this amount should not increase
   dramatically so the platform-detection can be included to the each
such
   test).
   Also cpu groups can be deleted (while we have not cpu-dependent
test).
   At the end we need only state groups to support test exclusion on
the
   'one-element' level (while we have unresolved entries in the current
exclude
   list).
  
   So, after small update of unit (aka integration, aka regression etc)
tests
   and resolution of all entries in the exclude list we don't need any
groups
   and pure JUnit covers all our needs :)
  
   On the other side, if we define some groups it will nice to define
*all*
   reasonable groups at the begin of the process.
  
thanks, Vladimir
  
   By the way, our regression tests are 'classic' regression tests that
   demonstrate some issues which were not resolved by initial code. But
it
   provides less coverage than 'regression tests' + unit tests, of
cause.
  
On 9/5/06, Richard Liang [EMAIL PROTECTED]  wrote:
   
On 9/5/06, Alex Blewitt  [EMAIL PROTECTED] wrote:
 On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
  On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
  
   If you've got fast and slow tests, then have a group for
fast and
slow
   tests. Then you can choose to just run the fast tests, and
any
   automated build system can handle running the slow tests.
 
  IMHO, fast or slow may not be the key point. The question is
whether we
  have any requirements to run only the regression tests.

 No, probably not the key point, but (a) groups don't have to be
 mutually exclusive (so you can decorate it with whatever groups
you
 want)
   
I agree. For example, os.win and os.linux are not mutually
exclusive.
   
Thanks a lot.
   
and (b) it might be useful for an automated build system to run
 fast tests first, followed by slow (or non-fast) tests.
   
That makes sense through we have not clear requirement currently.
   
 Mind you, I don't know what's going to happen with an automated
test'n'build
 system; so it might not make sense to do it at this point.
   
Really? ;-) We could also discuss whether it's feasible to move to
TestNG. As you may know, there are already several threads about
TestNG  JUnit. Here I just review the open questions one by one
so
that we have sufficient preparation.
   
   
[1]http://mail-
archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
   
[2]http://mail-
archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
   

Re: [classlib][TestNG] groups of Harmony test

[Paulex Yang]

Vladimir Ivanov wrote:

Do we really need in it? At present time tests were designed for win/unix
only.


I thought at least we have interest to port Harmony to these platforms[1].

[1] http://incubator.apache.org/harmony/roadmap.html#Porting%20Matrix

--
Paulex Yang
China Software Development Lab
IBM



-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Vladimir Ivanov]

On 9/6/06, Paulex Yang [EMAIL PROTECTED] wrote:


Vladimir Ivanov wrote:
 Do we really need in it? At present time tests were designed for
win/unix
 only.

I thought at least we have interest to port Harmony to these platforms[1].

[1] http://incubator.apache.org/harmony/roadmap.html#Porting%20Matrix



Yes, but I don't sure that we will have unique java API tests for all of
them. In any case, if we define groups set it should include names for all
defined platforms.

thanks, Vladimir


--

Paulex Yang
China Software Development Lab
IBM



-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

On 9/6/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:

On 9/6/06, Paulex Yang [EMAIL PROTECTED] wrote:

 Vladimir Ivanov wrote:
  Do we really need in it? At present time tests were designed for
 win/unix
  only.
 
 I thought at least we have interest to port Harmony to these platforms[1].

 [1] http://incubator.apache.org/harmony/roadmap.html#Porting%20Matrix


Yes, but I don't sure that we will have unique java API tests for all of
them. In any case, if we define groups set it should include names for all
defined platforms.



We proposed to use os+arch to define platform.

Best regards,
Richard


 thanks, Vladimir


--
 Paulex Yang
 China Software Development Lab
 IBM



 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]







--
Richard Liang
China Software Development Lab, IBM

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Vladimir Ivanov]

On 9/5/06, Andrew Zhang [EMAIL PROTECTED] wrote:


 If test fails on linux than (for example):
 - the test/ implementation should be fixed or


What if the behaviour is different on different platforms?



Seems, that it may be just 2 different tests.

- the test should define platform and report 'passed' if it does not

 support current platform


How to define platform? Get platform information in test code?



Yes. It is just one line of code. For example,  from Support_Exec.java
class:
boolean onUnix = File.separatorChar == '/';

thanks, Vladimir

In this context seems we don't need in any 'level' group

   (while 'stress' tests require reasonable time to pass).
   Seems, that platform group also can be deleted (at present time we
  have
   10 platform-dependent tests and this amount should not increase
   dramatically so the platform-detection can be included to the each
 such
   test).
   Also cpu groups can be deleted (while we have not cpu-dependent
 test).
   At the end we need only state groups to support test exclusion on
 the
   'one-element' level (while we have unresolved entries in the current
   exclude
   list).
  
   So, after small update of unit (aka integration, aka regression etc)
  tests
   and resolution of all entries in the exclude list we don't need any
  groups
   and pure JUnit covers all our needs :)
  
   On the other side, if we define some groups it will nice to define
 *all*
   reasonable groups at the begin of the process.
 
 
  Yes. We should figure out all possible groups. And it can be
 consolidated
  during applying TestNG.


 Agree.
 thanks, Vladimir

 thanks, Vladimir
  
   By the way, our regression tests are 'classic' regression tests that
   demonstrate some issues which were not resolved by initial code. But
 it
   provides less coverage than 'regression tests' + unit tests, of
cause.
  
   On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:
   
On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
 On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
  On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
  
   If you've got fast and slow tests, then have a group for
fast
  and
slow
   tests. Then you can choose to just run the fast tests, and
any
   automated build system can handle running the slow tests.
 
  IMHO, fast or slow may not be the key point. The question is
   whether
we
  have any requirements to run only the regression tests.

 No, probably not the key point, but (a) groups don't have to be
 mutually exclusive (so you can decorate it with whatever groups
 you
 want)
   
I agree. For example, os.win and os.linux are not mutually
 exclusive.
   
Thanks a lot.
   
and (b) it might be useful for an automated build system to run
 fast tests first, followed by slow (or non-fast) tests.
   
That makes sense through we have not clear requirement currently.
   
 Mind you, I don't know what's going to happen with an automated
test'n'build
 system; so it might not make sense to do it at this point.
   
Really? ;-) We could also discuss whether it's feasible to move to
TestNG. As you may know, there are already several threads about
TestNG  JUnit. Here I just review the open questions one by one
so
that we have sufficient preparation.
   
   
[1]http://mail-
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
   
[2]http://mail-
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
   
[3]http://mail-
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
   
   
Best regards,
Richard
   

 Alex.


  -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail:
 [EMAIL PROTECTED]
 For additional commands, e-mail:
  [EMAIL PROTECTED]


   
   
--
Richard Liang
China Software Development Lab, IBM
   
   
 -
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
 [EMAIL PROTECTED]
   
   
  
  
 
 
  --
  Andrew Zhang
  China Software Development Lab, IBM
 
 




--
Andrew Zhang
China Software Development Lab, IBM

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Andrew Zhang]

On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:


On 9/5/06, Andrew Zhang [EMAIL PROTECTED] wrote:

  If test fails on linux than (for example):
  - the test/ implementation should be fixed or


 What if the behaviour is different on different platforms?


Seems, that it may be just 2 different tests.

- the test should define platform and report 'passed' if it does not
  support current platform


 How to define platform? Get platform information in test code?


Yes. It is just one line of code. For example,  from Support_Exec.java
class:
boolean onUnix = File.separatorChar == '/';



Yes, it does work sometimes, but ...

How to differentiate AIX, solaris, linux, unix, windows and etc...

If there's a public API or can be retrieved from system property, it may
work.

thanks, Vladimir


In this context seems we don't need in any 'level' group
(while 'stress' tests require reasonable time to pass).
Seems, that platform group also can be deleted (at present time
we
   have
10 platform-dependent tests and this amount should not increase
dramatically so the platform-detection can be included to the each
  such
test).
Also cpu groups can be deleted (while we have not cpu-dependent
  test).
At the end we need only state groups to support test exclusion
on
  the
'one-element' level (while we have unresolved entries in the
current
exclude
list).
   
So, after small update of unit (aka integration, aka regression
etc)
   tests
and resolution of all entries in the exclude list we don't need
any
   groups
and pure JUnit covers all our needs :)
   
On the other side, if we define some groups it will nice to define
  *all*
reasonable groups at the begin of the process.
  
  
   Yes. We should figure out all possible groups. And it can be
  consolidated
   during applying TestNG.
 
 
  Agree.
  thanks, Vladimir
 
  thanks, Vladimir
   
By the way, our regression tests are 'classic' regression tests
that
demonstrate some issues which were not resolved by initial code.
But
  it
provides less coverage than 'regression tests' + unit tests, of
 cause.
   
On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:

 On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
  On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
   On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
   
If you've got fast and slow tests, then have a group for
 fast
   and
 slow
tests. Then you can choose to just run the fast tests, and
 any
automated build system can handle running the slow tests.
  
   IMHO, fast or slow may not be the key point. The question
is
whether
 we
   have any requirements to run only the regression tests.
 
  No, probably not the key point, but (a) groups don't have to
be
  mutually exclusive (so you can decorate it with whatever
groups
  you
  want)

 I agree. For example, os.win and os.linux are not mutually
  exclusive.

 Thanks a lot.

 and (b) it might be useful for an automated build system to run
  fast tests first, followed by slow (or non-fast) tests.

 That makes sense through we have not clear requirement
currently.

  Mind you, I don't know what's going to happen with an
automated
 test'n'build
  system; so it might not make sense to do it at this point.

 Really? ;-) We could also discuss whether it's feasible to move
to
 TestNG. As you may know, there are already several threads about
 TestNG  JUnit. Here I just review the open questions one by one
 so
 that we have sufficient preparation.


 [1]http://mail-
   
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]

 [2]http://mail-
   
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]

 [3]http://mail-
   
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]


 Best regards,
 Richard

 
  Alex.
 
 
  
-
  Terms of use :
http://incubator.apache.org/harmony/mailing.html
  To unsubscribe, e-mail:
  [EMAIL PROTECTED]
  For additional commands, e-mail:
   [EMAIL PROTECTED]
 
 


 --
 Richard Liang
 China Software Development Lab, IBM


  -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail:
 [EMAIL PROTECTED]
 For additional commands, e-mail:
  [EMAIL PROTECTED]


   
   
  
  
   --
   Andrew Zhang
   China Software Development Lab, IBM
  
  
 
 


 --
 Andrew Zhang
 China Software Development Lab, IBM







--
Andrew Zhang
China Software Development Lab, IBM

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Vladimir Ivanov]

On 9/5/06, Andrew Zhang [EMAIL PROTECTED] wrote:
...


  How to define platform? Get platform information in test code?


 Yes. It is just one line of code. For example,  from Support_Exec.java
 class:
 boolean onUnix = File.separatorChar == '/';


Yes, it does work sometimes, but ...

How to differentiate AIX, solaris, linux, unix, windows and etc...



Do we really need in it? At present time tests were designed for win/unix
only.
In any case, I don't against the groups but if we define 'general' groups
set it should include 'regression' group too.




If there's a public API or can be retrieved from system property, it may
work.



The public API does not specify exact names for platforms (java is platform
independent :) ) so these ways are equals (but for impl tests it will works
fine).

thanks, Vladimir



thanks, Vladimir

 In this context seems we don't need in any 'level' group
 (while 'stress' tests require reasonable time to pass).
 Seems, that platform group also can be deleted (at present
time
 we
have
 10 platform-dependent tests and this amount should not increase
 dramatically so the platform-detection can be included to the
each
   such
 test).
 Also cpu groups can be deleted (while we have not
cpu-dependent
   test).
 At the end we need only state groups to support test exclusion
 on
   the
 'one-element' level (while we have unresolved entries in the
 current
 exclude
 list).

 So, after small update of unit (aka integration, aka regression
 etc)
tests
 and resolution of all entries in the exclude list we don't need
 any
groups
 and pure JUnit covers all our needs :)

 On the other side, if we define some groups it will nice to
define
   *all*
 reasonable groups at the begin of the process.
   
   
Yes. We should figure out all possible groups. And it can be
   consolidated
during applying TestNG.
  
  
   Agree.
   thanks, Vladimir
  
   thanks, Vladimir

 By the way, our regression tests are 'classic' regression tests
 that
 demonstrate some issues which were not resolved by initial code.
 But
   it
 provides less coverage than 'regression tests' + unit tests, of
  cause.

 On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:
 
  On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
   On 04/09/06, Richard Liang [EMAIL PROTECTED]
wrote:
On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:

 If you've got fast and slow tests, then have a group for
  fast
and
  slow
 tests. Then you can choose to just run the fast tests,
and
  any
 automated build system can handle running the slow
tests.
   
IMHO, fast or slow may not be the key point. The
question
 is
 whether
  we
have any requirements to run only the regression tests.
  
   No, probably not the key point, but (a) groups don't have to
 be
   mutually exclusive (so you can decorate it with whatever
 groups
   you
   want)
 
  I agree. For example, os.win and os.linux are not mutually
   exclusive.
 
  Thanks a lot.
 
  and (b) it might be useful for an automated build system to
run
   fast tests first, followed by slow (or non-fast) tests.
 
  That makes sense through we have not clear requirement
 currently.
 
   Mind you, I don't know what's going to happen with an
 automated
  test'n'build
   system; so it might not make sense to do it at this point.
 
  Really? ;-) We could also discuss whether it's feasible to
move
 to
  TestNG. As you may know, there are already several threads
about
  TestNG  JUnit. Here I just review the open questions one by
one
  so
  that we have sufficient preparation.
 
 
  [1]http://mail-

   
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
 
  [2]http://mail-

   
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
 
  [3]http://mail-

   
  
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
 
 
  Best regards,
  Richard
 
  
   Alex.
  
  
   
 -
   Terms of use :
 http://incubator.apache.org/harmony/mailing.html
   To unsubscribe, e-mail:
   [EMAIL PROTECTED]
   For additional commands, e-mail:
[EMAIL PROTECTED]
  
  
 
 
  --
  Richard Liang
  China Software Development Lab, IBM
 
 
  
-
  Terms of use :
http://incubator.apache.org/harmony/mailing.html
  To unsubscribe, e-mail:
  [EMAIL PROTECTED]
  For additional commands, e-mail:
   [EMAIL PROTECTED]
 
 


   
   
--
Andrew Zhang
China Software Development Lab, IBM
   
   

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:

On 9/5/06, Andrew Zhang [EMAIL PROTECTED] wrote:
...

   How to define platform? Get platform information in test code?
 
 
  Yes. It is just one line of code. For example,  from Support_Exec.java
  class:
  boolean onUnix = File.separatorChar == '/';


 Yes, it does work sometimes, but ...

 How to differentiate AIX, solaris, linux, unix, windows and etc...


Do we really need in it? At present time tests were designed for win/unix
only.


Yes. In the future, Harmony will support more platforms. We may have
more platform-dependent test cases.



In any case, I don't against the groups but if we define 'general' groups
set it should include 'regression' group too.



 If there's a public API or can be retrieved from system property, it may
 work.


The public API does not specify exact names for platforms (java is platform
independent :) ) so these ways are equals (but for impl tests it will works
fine).

 thanks, Vladimir


 thanks, Vladimir
 
  In this context seems we don't need in any 'level' group
  (while 'stress' tests require reasonable time to pass).
  Seems, that platform group also can be deleted (at present
 time
  we
 have
  10 platform-dependent tests and this amount should not increase
  dramatically so the platform-detection can be included to the
 each
such
  test).
  Also cpu groups can be deleted (while we have not
 cpu-dependent
test).
  At the end we need only state groups to support test exclusion
  on
the
  'one-element' level (while we have unresolved entries in the
  current
  exclude
  list).
 
  So, after small update of unit (aka integration, aka regression
  etc)
 tests
  and resolution of all entries in the exclude list we don't need
  any
 groups
  and pure JUnit covers all our needs :)
 
  On the other side, if we define some groups it will nice to
 define
*all*
  reasonable groups at the begin of the process.


 Yes. We should figure out all possible groups. And it can be
consolidated
 during applying TestNG.
   
   
Agree.
thanks, Vladimir
   
thanks, Vladimir
 
  By the way, our regression tests are 'classic' regression tests
  that
  demonstrate some issues which were not resolved by initial code.
  But
it
  provides less coverage than 'regression tests' + unit tests, of
   cause.
 
  On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:
  
   On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
On 04/09/06, Richard Liang [EMAIL PROTECTED]
 wrote:
 On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
 
  If you've got fast and slow tests, then have a group for
   fast
 and
   slow
  tests. Then you can choose to just run the fast tests,
 and
   any
  automated build system can handle running the slow
 tests.

 IMHO, fast or slow may not be the key point. The
 question
  is
  whether
   we
 have any requirements to run only the regression tests.
   
No, probably not the key point, but (a) groups don't have to
  be
mutually exclusive (so you can decorate it with whatever
  groups
you
want)
  
   I agree. For example, os.win and os.linux are not mutually
exclusive.
  
   Thanks a lot.
  
   and (b) it might be useful for an automated build system to
 run
fast tests first, followed by slow (or non-fast) tests.
  
   That makes sense through we have not clear requirement
  currently.
  
Mind you, I don't know what's going to happen with an
  automated
   test'n'build
system; so it might not make sense to do it at this point.
  
   Really? ;-) We could also discuss whether it's feasible to
 move
  to
   TestNG. As you may know, there are already several threads
 about
   TestNG  JUnit. Here I just review the open questions one by
 one
   so
   that we have sufficient preparation.
  
  
   [1]http://mail-
 

   
  
 
 archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
  
   [2]http://mail-
 

   
  
 
 archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
  
   [3]http://mail-
 

   
  
 
 archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL 
PROTECTED]
  
  
   Best regards,
   Richard
  
   
Alex.
   
   

  -
Terms of use :
  http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
 [EMAIL PROTECTED]
   
   
  
  
   --
   Richard Liang
   China Software Development Lab, IBM
  
  
   
 

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

Mikhail Loenko wrote:

Hi Vladimir

Could you please decribe for what purpose it will be used?

I mean why one might have to either exclude or run only regression tests?


If running all tests takes up much time, running all regression test 
(for one particular version) may be a convenient way to verify the 
correctness of bug-fixing.


Best regards,
Richard.



Thanks,
Mikhail

2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:

On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:



 Vladimir Ivanov wrote:
  Also some tag for regression tests should be added.
 Yes. Do you think we could annotate regression test as
 *level.regression*? Thanks a lot.


Yes, I do. While tests can have more than one group it will enough.
 thanks, Vladimir


Richard
  thanks, Vladimir
 
 
  On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:
 
 
 
  Richard Liang wrote:
   Hello All,
  
   Now let's talk about the TestNG groups. I have read the related
   threads which posted by George, Vladimir Ivanov and Alexei 
Zakharov.

   All of them are good discussion about TestNG groups.
  
   IMHO, we may define Harmony test groups according the following 4
   dimensions:
  
   1) [Platform] os.any, os.platform id
   *os.any* - group of tests which pass on any platform. IMHO, 
most of

   our tests should be in this group.
   *os.platform id* - group of tests which are designed for one
   specific platform. A test may be in more than one of the 
groups. e.g

 .,
   @Test(groups={os.win.IA32, os.linux.IA32})
  
  ** os.any and os.platform id are mutually exclusive, that 
is,

   tests in os.any group should not be in os.win.IA32.
  
   2) [Test state] state.broken, state.broken.platform id
   *state.broken* - group of tests which fail on every platform, 
because
   of bugs of tests or implementation. We need to fix the bugs of 
tests

   or implementation to make them pass.
   *state.broken.platform id* - groups of test which only fail 
on one
   specific platform. A test may be in more than one of the 
groups. e.g

 .,
   @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})
  
   **state.broken.platform id group may be used as a 
convenient

 way
   to indicate that a test is platform-specific. e.g., If we 
support 10

   platforms, and one test are designed for 9 platforms except for
 MacOS,
   instead of list 9 os.platform id, we can just use
 state.broken.MacOS
  
   3) [Test type] type.api , type.impl
   *type.api* - group of tests which are tests for APIs in the Java
   Specification
   *type.impl* - groups of tests which are tests for 
Harmony-specific

   implementation
  
   ** type.api and type.impl are also mutually exclusive.
  
   4) [Test Level] level.unit, level.integration, level.system,
   level.stress, etc. (Levels of Test refer to the increase in
 complexity
   as moving through test cycle. )
  ** A test may be in more than one of the groups.
  ** In fact, some tests such as System tests are the 
verification

 of
   the entire system.  Maybe we'll put them into a separate project.
   e.g., harmony/enhanced/SVT (System Verification Test).
  
   If we want to run all the unit test for APIs on windows, we 
may use

   TestNG groups to select the tests:
  groups
  run
  include name=os.any /
  include name=type.api /
  include name=os.win.IA32 /
  exclude name= state.broken /
  exclude name=state.broken.win.IA32 /
  /run
  /groups
  
  Hello All,
 
  I'm sorry. It seems that the example does not work. I will try to
 figure
  another example soon. ;-)
 
  Best regards,
  Richard
  
   Well, I think our most of existing tests are in the groups of
   {os.any, type.api, level.unit}, and I have asked TestNG 
to add

 a
   new option -groups for its JUnitConverter which allow us to 
specify

   the test groups when migrate from JUnit test to TestNG test.
  
   Thanks for reading so far, and I will highly appreciate your 
comments

   or suggestion.  ;-)
  
 
  --
  Richard Liang
  China Software Development Lab, IBM
 
 
 
  
-

  Terms of use : http://incubator.apache.org/harmony/mailing.html
  To unsubscribe, e-mail: 
[EMAIL PROTECTED]
  For additional commands, e-mail: 
[EMAIL PROTECTED]

 
 
 

 --
 Richard Liang
 China Software Development Lab, IBM



 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]






-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]




--
Richard Liang
China Software Development Lab, IBM 

Re: [classlib][TestNG] groups of Harmony test

[Vladimir Ivanov]

On 9/4/06, Richard Liang [EMAIL PROTECTED] wrote:




Mikhail Loenko wrote:
 Hi Vladimir

 Could you please decribe for what purpose it will be used?

 I mean why one might have to either exclude or run only regression
tests?

If running all tests takes up much time, running all regression test
(for one particular version) may be a convenient way to verify the
correctness of bug-fixing.



Thanks Richard. It is exactly what I want to say :)
On the other hand may be all proposed groups need similar explanation (smt.
like use-case for group)?

thanks, Vladimir

Best regards,

Richard.


 Thanks,
 Mikhail

 2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:
 On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:
 
 
 
  Vladimir Ivanov wrote:
   Also some tag for regression tests should be added.
  Yes. Do you think we could annotate regression test as
  *level.regression*? Thanks a lot.


 Yes, I do. While tests can have more than one group it will enough.
  thanks, Vladimir


 Richard
   thanks, Vladimir
  
  
   On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:
  
  
  
   Richard Liang wrote:
Hello All,
   
Now let's talk about the TestNG groups. I have read the related
threads which posted by George, Vladimir Ivanov and Alexei
 Zakharov.
All of them are good discussion about TestNG groups.
   
IMHO, we may define Harmony test groups according the following
4
dimensions:
   
1) [Platform] os.any, os.platform id
*os.any* - group of tests which pass on any platform. IMHO,
 most of
our tests should be in this group.
*os.platform id* - group of tests which are designed for one
specific platform. A test may be in more than one of the
 groups. e.g
  .,
@Test(groups={os.win.IA32, os.linux.IA32})
   
   ** os.any and os.platform id are mutually exclusive, that
 is,
tests in os.any group should not be in os.win.IA32.
   
2) [Test state] state.broken, state.broken.platform id
*state.broken* - group of tests which fail on every platform,
 because
of bugs of tests or implementation. We need to fix the bugs of
 tests
or implementation to make them pass.
*state.broken.platform id* - groups of test which only fail
 on one
specific platform. A test may be in more than one of the
 groups. e.g
  .,
@Test(groups={state.broken.linux.IA32, os.broken.linux.IA64
})
   
**state.broken.platform id group may be used as a
 convenient
  way
to indicate that a test is platform-specific. e.g., If we
 support 10
platforms, and one test are designed for 9 platforms except for
  MacOS,
instead of list 9 os.platform id, we can just use
  state.broken.MacOS
   
3) [Test type] type.api , type.impl
*type.api* - group of tests which are tests for APIs in the Java
Specification
*type.impl* - groups of tests which are tests for
 Harmony-specific
implementation
   
** type.api and type.impl are also mutually exclusive.
   
4) [Test Level] level.unit, level.integration, level.system,
level.stress, etc. (Levels of Test refer to the increase in
  complexity
as moving through test cycle. )
   ** A test may be in more than one of the groups.
   ** In fact, some tests such as System tests are the
 verification
  of
the entire system.  Maybe we'll put them into a separate
project.
e.g., harmony/enhanced/SVT (System Verification Test).
   
If we want to run all the unit test for APIs on windows, we
 may use
TestNG groups to select the tests:
   groups
   run
   include name=os.any /
   include name=type.api /
   include name=os.win.IA32 /
   exclude name= state.broken /
   exclude name=state.broken.win.IA32 /
   /run
   /groups
   
   Hello All,
  
   I'm sorry. It seems that the example does not work. I will try to
  figure
   another example soon. ;-)
  
   Best regards,
   Richard
   
Well, I think our most of existing tests are in the groups of
{os.any, type.api, level.unit}, and I have asked TestNG
 to add
  a
new option -groups for its JUnitConverter which allow us to
 specify
the test groups when migrate from JUnit test to TestNG test.
   
Thanks for reading so far, and I will highly appreciate your
 comments
or suggestion.  ;-)
   
  
   --
   Richard Liang
   China Software Development Lab, IBM
  
  
  
  
 -
   Terms of use : http://incubator.apache.org/harmony/mailing.html
   To unsubscribe, e-mail:
 [EMAIL PROTECTED]
   For additional commands, e-mail:
 [EMAIL PROTECTED]
  
  
  
 
  --
  Richard Liang
  China Software Development Lab, IBM
 
 
 
  -
  Terms of use : http://incubator.apache.org/harmony/mailing.html
  To unsubscribe, e-mail: [EMAIL PROTECTED]
  For additional commands, e-mail:
[EMAIL 

Re: Re: [classlib][TestNG] groups of Harmony test

[Alex Blewitt]

IMNSO it doesn't make sense to arbitrarily partition the tests based
on a moniker, such as 'integration test', 'unit test', 'regression
test' etc. For one thing, developers are generally not good at
agreeing on the difference between them :-)

If you've got fast and slow tests, then have a group for fast and slow
tests. Then you can choose to just run the fast tests, and any
automated build system can handle running the slow tests.

Alex.

On 04/09/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:

On 9/4/06, Richard Liang [EMAIL PROTECTED] wrote:



 Mikhail Loenko wrote:
  Hi Vladimir
 
  Could you please decribe for what purpose it will be used?
 
  I mean why one might have to either exclude or run only regression
 tests?

 If running all tests takes up much time, running all regression test
 (for one particular version) may be a convenient way to verify the
 correctness of bug-fixing.


Thanks Richard. It is exactly what I want to say :)
On the other hand may be all proposed groups need similar explanation (smt.
like use-case for group)?

 thanks, Vladimir

Best regards,
 Richard.

 
  Thanks,
  Mikhail
 
  2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:
  On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:
  
  
  
   Vladimir Ivanov wrote:
Also some tag for regression tests should be added.
   Yes. Do you think we could annotate regression test as
   *level.regression*? Thanks a lot.
 
 
  Yes, I do. While tests can have more than one group it will enough.
   thanks, Vladimir
 
 
  Richard
thanks, Vladimir
   
   
On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:
   
   
   
Richard Liang wrote:
 Hello All,

 Now let's talk about the TestNG groups. I have read the related
 threads which posted by George, Vladimir Ivanov and Alexei
  Zakharov.
 All of them are good discussion about TestNG groups.

 IMHO, we may define Harmony test groups according the following
 4
 dimensions:

 1) [Platform] os.any, os.platform id
 *os.any* - group of tests which pass on any platform. IMHO,
  most of
 our tests should be in this group.
 *os.platform id* - group of tests which are designed for one
 specific platform. A test may be in more than one of the
  groups. e.g
   .,
 @Test(groups={os.win.IA32, os.linux.IA32})

** os.any and os.platform id are mutually exclusive, that
  is,
 tests in os.any group should not be in os.win.IA32.

 2) [Test state] state.broken, state.broken.platform id
 *state.broken* - group of tests which fail on every platform,
  because
 of bugs of tests or implementation. We need to fix the bugs of
  tests
 or implementation to make them pass.
 *state.broken.platform id* - groups of test which only fail
  on one
 specific platform. A test may be in more than one of the
  groups. e.g
   .,
 @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64
 })

 **state.broken.platform id group may be used as a
  convenient
   way
 to indicate that a test is platform-specific. e.g., If we
  support 10
 platforms, and one test are designed for 9 platforms except for
   MacOS,
 instead of list 9 os.platform id, we can just use
   state.broken.MacOS

 3) [Test type] type.api , type.impl
 *type.api* - group of tests which are tests for APIs in the Java
 Specification
 *type.impl* - groups of tests which are tests for
  Harmony-specific
 implementation

 ** type.api and type.impl are also mutually exclusive.

 4) [Test Level] level.unit, level.integration, level.system,
 level.stress, etc. (Levels of Test refer to the increase in
   complexity
 as moving through test cycle. )
** A test may be in more than one of the groups.
** In fact, some tests such as System tests are the
  verification
   of
 the entire system.  Maybe we'll put them into a separate
 project.
 e.g., harmony/enhanced/SVT (System Verification Test).

 If we want to run all the unit test for APIs on windows, we
  may use
 TestNG groups to select the tests:
groups
run
include name=os.any /
include name=type.api /
include name=os.win.IA32 /
exclude name= state.broken /
exclude name=state.broken.win.IA32 /
/run
/groups

Hello All,
   
I'm sorry. It seems that the example does not work. I will try to
   figure
another example soon. ;-)
   
Best regards,
Richard

 Well, I think our most of existing tests are in the groups of
 {os.any, type.api, level.unit}, and I have asked TestNG
  to add
   a
 new option -groups for its JUnitConverter which allow us to
  specify
 the test groups when migrate from JUnit test to TestNG test.

 Thanks for reading so far, and I will highly appreciate your
  comments
 or 

RE: [classlib][TestNG] groups of Harmony test

[Ivanov, Alexey A]

-Original Message-
From: Richard Liang [mailto:[EMAIL PROTECTED]
Sent: Monday, September 04, 2006 10:56 AM
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib][TestNG] groups of Harmony test



Mikhail Loenko wrote:
 Hi Vladimir

 Could you please decribe for what purpose it will be used?

 I mean why one might have to either exclude or run only regression
tests?

If running all tests takes up much time, running all regression test
(for one particular version) may be a convenient way to verify the
correctness of bug-fixing.

I can be used for informational purposes as well. If the test fails, you
need simply re-open the associated bug provided the bug number can be
easily found.

Regards,
Alexey.


Best regards,
Richard.


 Thanks,
 Mikhail

 2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:
 On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:
 
 
 
  Vladimir Ivanov wrote:
   Also some tag for regression tests should be added.
  Yes. Do you think we could annotate regression test as
  *level.regression*? Thanks a lot.


 Yes, I do. While tests can have more than one group it will enough.
  thanks, Vladimir


 Richard
   thanks, Vladimir
  
  
   On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:
  
  
  
   Richard Liang wrote:
Hello All,
   
Now let's talk about the TestNG groups. I have read the
related
threads which posted by George, Vladimir Ivanov and Alexei
 Zakharov.
All of them are good discussion about TestNG groups.
   
IMHO, we may define Harmony test groups according the
following 4
dimensions:
   
1) [Platform] os.any, os.platform id
*os.any* - group of tests which pass on any platform. IMHO,
 most of
our tests should be in this group.
*os.platform id* - group of tests which are designed for
one
specific platform. A test may be in more than one of the
 groups. e.g
  .,
@Test(groups={os.win.IA32, os.linux.IA32})
   
   ** os.any and os.platform id are mutually exclusive,
that
 is,
tests in os.any group should not be in os.win.IA32.
   
2) [Test state] state.broken, state.broken.platform id
*state.broken* - group of tests which fail on every platform,
 because
of bugs of tests or implementation. We need to fix the bugs
of
 tests
or implementation to make them pass.
*state.broken.platform id* - groups of test which only fail
 on one
specific platform. A test may be in more than one of the
 groups. e.g
  .,
@Test(groups={state.broken.linux.IA32,
os.broken.linux.IA64})
   
**state.broken.platform id group may be used as a
 convenient
  way
to indicate that a test is platform-specific. e.g., If we
 support 10
platforms, and one test are designed for 9 platforms except
for
  MacOS,
instead of list 9 os.platform id, we can just use
  state.broken.MacOS
   
3) [Test type] type.api , type.impl
*type.api* - group of tests which are tests for APIs in the
Java
Specification
*type.impl* - groups of tests which are tests for
 Harmony-specific
implementation
   
** type.api and type.impl are also mutually exclusive.
   
4) [Test Level] level.unit, level.integration, level.system,
level.stress, etc. (Levels of Test refer to the increase in
  complexity
as moving through test cycle. )
   ** A test may be in more than one of the groups.
   ** In fact, some tests such as System tests are the
 verification
  of
the entire system.  Maybe we'll put them into a separate
project.
e.g., harmony/enhanced/SVT (System Verification Test).
   
If we want to run all the unit test for APIs on windows, we
 may use
TestNG groups to select the tests:
   groups
   run
   include name=os.any /
   include name=type.api /
   include name=os.win.IA32 /
   exclude name= state.broken /
   exclude name=state.broken.win.IA32 /
   /run
   /groups
   
   Hello All,
  
   I'm sorry. It seems that the example does not work. I will try
to
  figure
   another example soon. ;-)
  
   Best regards,
   Richard
   
Well, I think our most of existing tests are in the groups of
{os.any, type.api, level.unit}, and I have asked TestNG
 to add
  a
new option -groups for its JUnitConverter which allow us to
 specify
the test groups when migrate from JUnit test to TestNG test.
   
Thanks for reading so far, and I will highly appreciate your
 comments
or suggestion.  ;-)
   
  
   --
   Richard Liang
   China Software Development Lab, IBM
  
  
  
  

-
   Terms of use : http://incubator.apache.org/harmony/mailing.html
   To unsubscribe, e-mail:
 [EMAIL PROTECTED]
   For additional commands, e-mail:
 [EMAIL PROTECTED]
  
  
  
 
  --
  Richard Liang
  China Software Development Lab, IBM

Re: Re: [classlib][TestNG] groups of Harmony test

[Mikhail Loenko]

+1

2006/9/4, Alex Blewitt [EMAIL PROTECTED]:

IMNSO it doesn't make sense to arbitrarily partition the tests based
on a moniker, such as 'integration test', 'unit test', 'regression
test' etc. For one thing, developers are generally not good at
agreeing on the difference between them :-)

If you've got fast and slow tests, then have a group for fast and slow
tests. Then you can choose to just run the fast tests, and any
automated build system can handle running the slow tests.

Alex.

On 04/09/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:
 On 9/4/06, Richard Liang [EMAIL PROTECTED] wrote:
 
 
 
  Mikhail Loenko wrote:
   Hi Vladimir
  
   Could you please decribe for what purpose it will be used?
  
   I mean why one might have to either exclude or run only regression
  tests?
 
  If running all tests takes up much time, running all regression test
  (for one particular version) may be a convenient way to verify the
  correctness of bug-fixing.


 Thanks Richard. It is exactly what I want to say :)
 On the other hand may be all proposed groups need similar explanation (smt.
 like use-case for group)?

  thanks, Vladimir

 Best regards,
  Richard.
 
  
   Thanks,
   Mikhail
  
   2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:
   On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:
   
   
   
Vladimir Ivanov wrote:
 Also some tag for regression tests should be added.
Yes. Do you think we could annotate regression test as
*level.regression*? Thanks a lot.
  
  
   Yes, I do. While tests can have more than one group it will enough.
thanks, Vladimir
  
  
   Richard
 thanks, Vladimir


 On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:



 Richard Liang wrote:
  Hello All,
 
  Now let's talk about the TestNG groups. I have read the related
  threads which posted by George, Vladimir Ivanov and Alexei
   Zakharov.
  All of them are good discussion about TestNG groups.
 
  IMHO, we may define Harmony test groups according the following
  4
  dimensions:
 
  1) [Platform] os.any, os.platform id
  *os.any* - group of tests which pass on any platform. IMHO,
   most of
  our tests should be in this group.
  *os.platform id* - group of tests which are designed for one
  specific platform. A test may be in more than one of the
   groups. e.g
.,
  @Test(groups={os.win.IA32, os.linux.IA32})
 
 ** os.any and os.platform id are mutually exclusive, that
   is,
  tests in os.any group should not be in os.win.IA32.
 
  2) [Test state] state.broken, state.broken.platform id
  *state.broken* - group of tests which fail on every platform,
   because
  of bugs of tests or implementation. We need to fix the bugs of
   tests
  or implementation to make them pass.
  *state.broken.platform id* - groups of test which only fail
   on one
  specific platform. A test may be in more than one of the
   groups. e.g
.,
  @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64
  })
 
  **state.broken.platform id group may be used as a
   convenient
way
  to indicate that a test is platform-specific. e.g., If we
   support 10
  platforms, and one test are designed for 9 platforms except for
MacOS,
  instead of list 9 os.platform id, we can just use
state.broken.MacOS
 
  3) [Test type] type.api , type.impl
  *type.api* - group of tests which are tests for APIs in the Java
  Specification
  *type.impl* - groups of tests which are tests for
   Harmony-specific
  implementation
 
  ** type.api and type.impl are also mutually exclusive.
 
  4) [Test Level] level.unit, level.integration, level.system,
  level.stress, etc. (Levels of Test refer to the increase in
complexity
  as moving through test cycle. )
 ** A test may be in more than one of the groups.
 ** In fact, some tests such as System tests are the
   verification
of
  the entire system.  Maybe we'll put them into a separate
  project.
  e.g., harmony/enhanced/SVT (System Verification Test).
 
  If we want to run all the unit test for APIs on windows, we
   may use
  TestNG groups to select the tests:
 groups
 run
 include name=os.any /
 include name=type.api /
 include name=os.win.IA32 /
 exclude name= state.broken /
 exclude name=state.broken.win.IA32 /
 /run
 /groups
 
 Hello All,

 I'm sorry. It seems that the example does not work. I will try to
figure
 another example soon. ;-)

 Best regards,
 Richard
 
  Well, I think our most of existing tests are in the groups of
  {os.any, type.api, level.unit}, and I have asked TestNG
   to add
a
  new option -groups for its 

Re: [classlib][TestNG] groups of Harmony test

[Andrew Zhang]

On 9/4/06, Richard Liang [EMAIL PROTECTED] wrote:


On 9/4/06, Mikhail Loenko [EMAIL PROTECTED] wrote:
 Well, my question was for what particular reason? for example?

 Tio verify correctness of bug-fixing IMHO all the unit, intergration,
api, and
 regression tests should be run

Running all tests are always good to verify our code quality. And I
think this is what we have been doing. But what will we do if it takes
us 24 hours to run all Harmony tests? Anyway, running regression tests
could provide some confidence to our bug-fixing. We may consider it as
another option. :-)



I prefer to run all tests in one module instead. :) Although it can not
guarentee all tests would pass, it's less likey to break build system
frequently.  If the fix causes other module fails, it shows the lack of
tests in its module. And we should add the corresponding test in the module.

Currently, Harmony regression test is a test for certain Harmony jira issue.
So IMHO, running all regression tests for certain issue doesn't make sense.

But whether using annotation to mark regression test is another story. At
least, it doesn't have any disadvantages compared with comment way, does it?


Best regards,

Richard


 Thanks,
 Mikhail

 2006/9/4, Richard Liang [EMAIL PROTECTED]:
 
 
  Mikhail Loenko wrote:
   Hi Vladimir
  
   Could you please decribe for what purpose it will be used?
  
   I mean why one might have to either exclude or run only regression
tests?
 
  If running all tests takes up much time, running all regression test
  (for one particular version) may be a convenient way to verify the
  correctness of bug-fixing.
 
  Best regards,
  Richard.
 
  
   Thanks,
   Mikhail
  
   2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:
   On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:
   
   
   
Vladimir Ivanov wrote:
 Also some tag for regression tests should be added.
Yes. Do you think we could annotate regression test as
*level.regression*? Thanks a lot.
  
  
   Yes, I do. While tests can have more than one group it will enough.
thanks, Vladimir
  
  
   Richard
 thanks, Vladimir


 On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:



 Richard Liang wrote:
  Hello All,
 
  Now let's talk about the TestNG groups. I have read the
related
  threads which posted by George, Vladimir Ivanov and Alexei
   Zakharov.
  All of them are good discussion about TestNG groups.
 
  IMHO, we may define Harmony test groups according the
following 4
  dimensions:
 
  1) [Platform] os.any, os.platform id
  *os.any* - group of tests which pass on any platform. IMHO,
   most of
  our tests should be in this group.
  *os.platform id* - group of tests which are designed for
one
  specific platform. A test may be in more than one of the
   groups. e.g
.,
  @Test(groups={os.win.IA32, os.linux.IA32})
 
 ** os.any and os.platform id are mutually exclusive,
that
   is,
  tests in os.any group should not be in os.win.IA32.
 
  2) [Test state] state.broken, state.broken.platform id
  *state.broken* - group of tests which fail on every
platform,
   because
  of bugs of tests or implementation. We need to fix the bugs
of
   tests
  or implementation to make them pass.
  *state.broken.platform id* - groups of test which only
fail
   on one
  specific platform. A test may be in more than one of the
   groups. e.g
.,
  @Test(groups={state.broken.linux.IA32, 
os.broken.linux.IA64})
 
  **state.broken.platform id group may be used as a
   convenient
way
  to indicate that a test is platform-specific. e.g., If we
   support 10
  platforms, and one test are designed for 9 platforms except
for
MacOS,
  instead of list 9 os.platform id, we can just use
state.broken.MacOS
 
  3) [Test type] type.api , type.impl
  *type.api* - group of tests which are tests for APIs in the
Java
  Specification
  *type.impl* - groups of tests which are tests for
   Harmony-specific
  implementation
 
  ** type.api and type.impl are also mutually exclusive.
 
  4) [Test Level] level.unit, level.integration, level.system,
  level.stress, etc. (Levels of Test refer to the increase in
complexity
  as moving through test cycle. )
 ** A test may be in more than one of the groups.
 ** In fact, some tests such as System tests are the
   verification
of
  the entire system.  Maybe we'll put them into a separate
project.
  e.g., harmony/enhanced/SVT (System Verification Test).
 
  If we want to run all the unit test for APIs on windows, we
   may use
  TestNG groups to select the tests:
 groups
 run
 include name=os.any /
 include name=type.api /
 include name=os.win.IA32 /
 exclude name= 

Re: Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

On 9/4/06, Andrew Zhang [EMAIL PROTECTED] wrote:

On 9/4/06, Richard Liang [EMAIL PROTECTED] wrote:

 I sent my reply this afternoon, but I have not received it. (it seems
 there
 is something wrong with my smtp server). So I send it again. :-)

 On 9/4/06, Alex Blewitt [EMAIL PROTECTED] wrote:
 
  IMNSO it doesn't make sense to arbitrarily partition the tests based
  on a moniker, such as 'integration test', 'unit test', 'regression
  test' etc. For one thing, developers are generally not good at
  agreeing on the difference between them :-)



 This is really a problem, however it might be simpler than we imagine. We
 are open to any discussion. ;-) Anyway, developers are required to write
 unit tests.


 If you've got fast and slow tests, then have a group for fast and slow
  tests. Then you can choose to just run the fast tests, and any
  automated build system can handle running the slow tests.


 IMHO, fast or slow may not be the key point. The question is whether we
 have any requirements to run only the regression tests.


I think your first proposal looks fine. Platform + os + state + api/impl
information are enough for all tests. We have to assure all tests pass on
every platform.



Thanks a lot. Well, I agree :-). TestNG annotations are most helpful
to us in the management of unit tests that are prone to change. For
example, some unit tests suddenly break on a particular platform. or
we may want to exclude some tests temporarily.

Here I just try to list all the possible facets, so that we could
have more thorough discussion.

Best regards,
Richard


For quick run, developers can use their own short approach, like run single
test in IDE... It's not business of TestNG. :-)


Best regards,
 Richard

 Alex.
 
  On 04/09/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:
   On 9/4/06, Richard Liang [EMAIL PROTECTED] wrote:
   
   
   
Mikhail Loenko wrote:
 Hi Vladimir

 Could you please decribe for what purpose it will be used?

 I mean why one might have to either exclude or run only regression
tests?
   
If running all tests takes up much time, running all regression test
(for one particular version) may be a convenient way to verify the
correctness of bug-fixing.
  
  
   Thanks Richard. It is exactly what I want to say :)
   On the other hand may be all proposed groups need similar explanation
  (smt.
   like use-case for group)?
  
thanks, Vladimir
  
   Best regards,
Richard.
   

 Thanks,
 Mikhail

 2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:
 On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:
 
 
 
  Vladimir Ivanov wrote:
   Also some tag for regression tests should be added.
  Yes. Do you think we could annotate regression test as
  *level.regression*? Thanks a lot.


 Yes, I do. While tests can have more than one group it will
 enough.
  thanks, Vladimir


 Richard
   thanks, Vladimir
  
  
   On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:
  
  
  
   Richard Liang wrote:
Hello All,
   
Now let's talk about the TestNG groups. I have read the
  related
threads which posted by George, Vladimir Ivanov and Alexei
 Zakharov.
All of them are good discussion about TestNG groups.
   
IMHO, we may define Harmony test groups according the
  following
4
dimensions:
   
1) [Platform] os.any, os.platform id
*os.any* - group of tests which pass on any platform.
 IMHO,
 most of
our tests should be in this group.
*os.platform id* - group of tests which are designed for
  one
specific platform. A test may be in more than one of the
 groups. e.g
  .,
@Test(groups={os.win.IA32, os.linux.IA32})
   
   ** os.any and os.platform id are mutually exclusive,
  that
 is,
tests in os.any group should not be in os.win.IA32.
   
2) [Test state] state.broken, state.broken.platform id
*state.broken* - group of tests which fail on every
  platform,
 because
of bugs of tests or implementation. We need to fix the
 bugs
  of
 tests
or implementation to make them pass.
*state.broken.platform id* - groups of test which only
  fail
 on one
specific platform. A test may be in more than one of the
 groups. e.g
  .,
@Test(groups={state.broken.linux.IA32, 
  os.broken.linux.IA64
})
   
**state.broken.platform id group may be used as a
 convenient
  way
to indicate that a test is platform-specific. e.g., If we
 support 10
platforms, and one test are designed for 9 platforms
 except
  for
  MacOS,
instead of list 9 os.platform id, we can just use
  state.broken.MacOS
   
3) [Test type] type.api , type.impl
*type.api* - group of tests which are tests for APIs in

RE: [classlib][TestNG] groups of Harmony test

[Ivanov, Alexey A]

-Original Message-
From: Andrew Zhang [mailto:[EMAIL PROTECTED]
Sent: Monday, September 04, 2006 6:18 PM
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib][TestNG] groups of Harmony test

On 9/4/06, Richard Liang [EMAIL PROTECTED] wrote:

 On 9/4/06, Mikhail Loenko [EMAIL PROTECTED] wrote:
  Well, my question was for what particular reason? for example?
 
  Tio verify correctness of bug-fixing IMHO all the unit,
intergration,
 api, and
  regression tests should be run

 Running all tests are always good to verify our code quality. And I
 think this is what we have been doing. But what will we do if it
takes
 us 24 hours to run all Harmony tests? Anyway, running regression
tests
 could provide some confidence to our bug-fixing. We may consider it
as
 another option. :-)


I prefer to run all tests in one module instead. :) Although it can not
guarentee all tests would pass, it's less likey to break build system
frequently.  If the fix causes other module fails, it shows the lack of
tests in its module. And we should add the corresponding test in the
module.

Currently, Harmony regression test is a test for certain Harmony jira
issue.
So IMHO, running all regression tests for certain issue doesn't make
sense.

But whether using annotation to mark regression test is another story.
At
least, it doesn't have any disadvantages compared with comment way,
does it?

Agree. We may introduce a special annotation for this purpose.


Regards,
Alexey.



Best regards,
 Richard

 
  Thanks,
  Mikhail
 
  2006/9/4, Richard Liang [EMAIL PROTECTED]:
  
  
   Mikhail Loenko wrote:
Hi Vladimir
   
Could you please decribe for what purpose it will be used?
   
I mean why one might have to either exclude or run only
regression
 tests?
  
   If running all tests takes up much time, running all regression
test
   (for one particular version) may be a convenient way to verify
the
   correctness of bug-fixing.
  
   Best regards,
   Richard.
  
   
Thanks,
Mikhail
   
2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:
On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:



 Vladimir Ivanov wrote:
  Also some tag for regression tests should be added.
 Yes. Do you think we could annotate regression test as
 *level.regression*? Thanks a lot.
   
   
Yes, I do. While tests can have more than one group it will
enough.
 thanks, Vladimir
   
   
Richard
  thanks, Vladimir
 
 
  On 8/28/06, Richard Liang [EMAIL PROTECTED]
wrote:
 
 
 
  Richard Liang wrote:
   Hello All,
  
   Now let's talk about the TestNG groups. I have read the
 related
   threads which posted by George, Vladimir Ivanov and
Alexei
Zakharov.
   All of them are good discussion about TestNG groups.
  
   IMHO, we may define Harmony test groups according the
 following 4
   dimensions:
  
   1) [Platform] os.any, os.platform id
   *os.any* - group of tests which pass on any platform.
IMHO,
most of
   our tests should be in this group.
   *os.platform id* - group of tests which are designed
for
 one
   specific platform. A test may be in more than one of
the
groups. e.g
 .,
   @Test(groups={os.win.IA32, os.linux.IA32})
  
  ** os.any and os.platform id are mutually
exclusive,
 that
is,
   tests in os.any group should not be in os.win.IA32.
  
   2) [Test state] state.broken, state.broken.platform
id
   *state.broken* - group of tests which fail on every
 platform,
because
   of bugs of tests or implementation. We need to fix the
bugs
 of
tests
   or implementation to make them pass.
   *state.broken.platform id* - groups of test which
only
 fail
on one
   specific platform. A test may be in more than one of
the
groups. e.g
 .,
   @Test(groups={state.broken.linux.IA32, 
 os.broken.linux.IA64})
  
   **state.broken.platform id group may be used as a
convenient
 way
   to indicate that a test is platform-specific. e.g., If
we
support 10
   platforms, and one test are designed for 9 platforms
except
 for
 MacOS,
   instead of list 9 os.platform id, we can just use
 state.broken.MacOS
  
   3) [Test type] type.api , type.impl
   *type.api* - group of tests which are tests for APIs in
the
 Java
   Specification
   *type.impl* - groups of tests which are tests for
Harmony-specific
   implementation
  
   ** type.api and type.impl are also mutually
exclusive.
  
   4) [Test Level] level.unit, level.integration,
level.system,
   level.stress, etc. (Levels of Test refer to the
increase in
 complexity
   as moving through test cycle. )
  ** A test may be in more than one of the groups.
  ** In fact, some tests such as System tests are the
verification
 of
   the entire system.  Maybe we'll put them

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

On 9/4/06, Andrew Zhang [EMAIL PROTECTED] wrote:

On 9/4/06, Richard Liang [EMAIL PROTECTED] wrote:

 On 9/4/06, Mikhail Loenko [EMAIL PROTECTED] wrote:
  Well, my question was for what particular reason? for example?
 
  Tio verify correctness of bug-fixing IMHO all the unit, intergration,
 api, and
  regression tests should be run

 Running all tests are always good to verify our code quality. And I
 think this is what we have been doing. But what will we do if it takes
 us 24 hours to run all Harmony tests? Anyway, running regression tests
 could provide some confidence to our bug-fixing. We may consider it as
 another option. :-)


I prefer to run all tests in one module instead. :) Although it can not
guarentee all tests would pass, it's less likey to break build system
frequently.  If the fix causes other module fails, it shows the lack of
tests in its module. And we should add the corresponding test in the module.



Running all tests for one module is practical for most cases (except luni).



Currently, Harmony regression test is a test for certain Harmony jira issue.
So IMHO, running all regression tests for certain issue doesn't make sense.


Yes, Harmony regression test is not the so-called regression test
which aim to verify that no unwanted changes were introduced to one
part of the system as a
result of making changes to another part of the system.

I agree that we may not want to run all the regression test. If
there is no objection I will remove level.regression



But whether using annotation to mark regression test is another story. At
least, it doesn't have any disadvantages compared with comment way, does it?



We may use @Test(description=Regression: Harmony-).



Best regards,
 Richard

 
  Thanks,
  Mikhail
 
  2006/9/4, Richard Liang [EMAIL PROTECTED]:
  
  
   Mikhail Loenko wrote:
Hi Vladimir
   
Could you please decribe for what purpose it will be used?
   
I mean why one might have to either exclude or run only regression
 tests?
  
   If running all tests takes up much time, running all regression test
   (for one particular version) may be a convenient way to verify the
   correctness of bug-fixing.
  
   Best regards,
   Richard.
  
   
Thanks,
Mikhail
   
2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:
On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:



 Vladimir Ivanov wrote:
  Also some tag for regression tests should be added.
 Yes. Do you think we could annotate regression test as
 *level.regression*? Thanks a lot.
   
   
Yes, I do. While tests can have more than one group it will enough.
 thanks, Vladimir
   
   
Richard
  thanks, Vladimir
 
 
  On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:
 
 
 
  Richard Liang wrote:
   Hello All,
  
   Now let's talk about the TestNG groups. I have read the
 related
   threads which posted by George, Vladimir Ivanov and Alexei
Zakharov.
   All of them are good discussion about TestNG groups.
  
   IMHO, we may define Harmony test groups according the
 following 4
   dimensions:
  
   1) [Platform] os.any, os.platform id
   *os.any* - group of tests which pass on any platform. IMHO,
most of
   our tests should be in this group.
   *os.platform id* - group of tests which are designed for
 one
   specific platform. A test may be in more than one of the
groups. e.g
 .,
   @Test(groups={os.win.IA32, os.linux.IA32})
  
  ** os.any and os.platform id are mutually exclusive,
 that
is,
   tests in os.any group should not be in os.win.IA32.
  
   2) [Test state] state.broken, state.broken.platform id
   *state.broken* - group of tests which fail on every
 platform,
because
   of bugs of tests or implementation. We need to fix the bugs
 of
tests
   or implementation to make them pass.
   *state.broken.platform id* - groups of test which only
 fail
on one
   specific platform. A test may be in more than one of the
groups. e.g
 .,
   @Test(groups={state.broken.linux.IA32, 
 os.broken.linux.IA64})
  
   **state.broken.platform id group may be used as a
convenient
 way
   to indicate that a test is platform-specific. e.g., If we
support 10
   platforms, and one test are designed for 9 platforms except
 for
 MacOS,
   instead of list 9 os.platform id, we can just use
 state.broken.MacOS
  
   3) [Test type] type.api , type.impl
   *type.api* - group of tests which are tests for APIs in the
 Java
   Specification
   *type.impl* - groups of tests which are tests for
Harmony-specific
   implementation
  
   ** type.api and type.impl are also mutually exclusive.
  
   4) [Test Level] level.unit, level.integration, level.system,
   level.stress, etc. (Levels of Test refer to the increase in
 

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Alex Blewitt]

On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:

On 9/4/06, Alex Blewitt [EMAIL PROTECTED] wrote:

 If you've got fast and slow tests, then have a group for fast and slow
 tests. Then you can choose to just run the fast tests, and any
 automated build system can handle running the slow tests.

IMHO, fast or slow may not be the key point. The question is whether we
have any requirements to run only the regression tests.


No, probably not the key point, but (a) groups don't have to be
mutually exclusive (so you can decorate it with whatever groups you
want) and (b) it might be useful for an automated build system to run
fast tests first, followed by slow (or non-fast) tests. Mind you, I
don't know what's going to happen with an automated test'n'build
system; so it might not make sense to do it at this point.

Alex.

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:

On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
 On 9/4/06, Alex Blewitt [EMAIL PROTECTED] wrote:
 
  If you've got fast and slow tests, then have a group for fast and slow
  tests. Then you can choose to just run the fast tests, and any
  automated build system can handle running the slow tests.

 IMHO, fast or slow may not be the key point. The question is whether we
 have any requirements to run only the regression tests.

No, probably not the key point, but (a) groups don't have to be
mutually exclusive (so you can decorate it with whatever groups you
want)


I agree. For example, os.win and os.linux are not mutually exclusive.

Thanks a lot.

and (b) it might be useful for an automated build system to run

fast tests first, followed by slow (or non-fast) tests.


That makes sense through we have not clear requirement currently.


Mind you, I don't know what's going to happen with an automated test'n'build
system; so it might not make sense to do it at this point.


Really? ;-) We could also discuss whether it's feasible to move to
TestNG. As you may know, there are already several threads about
TestNG  JUnit. Here I just review the open questions one by one so
that we have sufficient preparation.


[1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]
[2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]
[3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

Best regards,
Richard



Alex.

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]





--
Richard Liang
China Software Development Lab, IBM

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Vladimir Ivanov]

OK, let's return back to the usage model.
If I understood it correctly, before the commit of any changes each
developer run *all* tests (at least all which we have now) on all available
to him platforms. In this context seems we don't need in any 'level' group
(while 'stress' tests require reasonable time to pass).
Seems, that platform group also can be deleted (at present time we have
10 platform-dependent tests and this amount should not increase
dramatically so the platform-detection can be included to the each such
test).
Also cpu groups can be deleted (while we have not cpu-dependent test).
At the end we need only state groups to support test exclusion on the
'one-element' level (while we have unresolved entries in the current exclude
list).

So, after small update of unit (aka integration, aka regression etc) tests
and resolution of all entries in the exclude list we don't need any groups
and pure JUnit covers all our needs :)

On the other side, if we define some groups it will nice to define *all*
reasonable groups at the begin of the process.

thanks, Vladimir

By the way, our regression tests are 'classic' regression tests that
demonstrate some issues which were not resolved by initial code. But it
provides less coverage than 'regression tests' + unit tests, of cause.

On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:


On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
 On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
  On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
  
   If you've got fast and slow tests, then have a group for fast and
slow
   tests. Then you can choose to just run the fast tests, and any
   automated build system can handle running the slow tests.
 
  IMHO, fast or slow may not be the key point. The question is whether
we
  have any requirements to run only the regression tests.

 No, probably not the key point, but (a) groups don't have to be
 mutually exclusive (so you can decorate it with whatever groups you
 want)

I agree. For example, os.win and os.linux are not mutually exclusive.

Thanks a lot.

and (b) it might be useful for an automated build system to run
 fast tests first, followed by slow (or non-fast) tests.

That makes sense through we have not clear requirement currently.

 Mind you, I don't know what's going to happen with an automated
test'n'build
 system; so it might not make sense to do it at this point.

Really? ;-) We could also discuss whether it's feasible to move to
TestNG. As you may know, there are already several threads about
TestNG  JUnit. Here I just review the open questions one by one so
that we have sufficient preparation.


[1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

[2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

[3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]


Best regards,
Richard


 Alex.

 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]




--
Richard Liang
China Software Development Lab, IBM

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Alexey Varlamov]

2006/9/4, Richard Liang [EMAIL PROTECTED]:



Alex Blewitt wrote:
 IMNSO it doesn't make sense to arbitrarily partition the tests based
 on a moniker, such as 'integration test', 'unit test', 'regression
 test' etc. For one thing, developers are generally not good at
 agreeing on the difference between them :-)

This is really a problem, however it might be simpler than we imagine.
We are open to any discussion. ;-) Anyway, developers are required to
write unit tests.


 If you've got fast and slow tests, then have a group for fast and slow
 tests. Then you can choose to just run the fast tests, and any
 automated build system can handle running the slow tests.
IMHO, fast or slow may not be the key point. The question is whether
we have any requirements to run only the regression tests.


I believe we have not. If a testcase was added to prevent regression,
it basically means that there was a hole in test coverage for some
reason. Provided that such holes are scattered randomly through the
given module, what for we may want to run such a sieve test suite?
I can think of the sole reason, that regression tests may
*potentially* highlight weak spots in the code (or design) which are
more prone to fail during code evolution. But still I see no reason to
run only regression tests ignoring others. I'd rather second Alex in
fast/slow grouping orthogonal to regressions.

As for information purposes, Mikhail (?) suggested good idea -
explicitly specify no. of the issue in descripton comment (or
annotation).

[snip]

--
Alexey

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Andrew Zhang]

On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:


OK, let's return back to the usage model.
If I understood it correctly, before the commit of any changes each
developer run *all* tests (at least all which we have now) on all
available
to him platforms.



Yes. But as you mentioned, what's all available?

If a test passes on windows while fails on linux, is it available to
windows?

If it is, how will we control it? TestNG groups.

In this context seems we don't need in any 'level' group

(while 'stress' tests require reasonable time to pass).
Seems, that platform group also can be deleted (at present time we have
10 platform-dependent tests and this amount should not increase
dramatically so the platform-detection can be included to the each such
test).
Also cpu groups can be deleted (while we have not cpu-dependent test).
At the end we need only state groups to support test exclusion on the
'one-element' level (while we have unresolved entries in the current
exclude
list).

So, after small update of unit (aka integration, aka regression etc) tests
and resolution of all entries in the exclude list we don't need any groups
and pure JUnit covers all our needs :)

On the other side, if we define some groups it will nice to define *all*
reasonable groups at the begin of the process.



Yes. We should figure out all possible groups. And it can be consolidated
during applying TestNG.

thanks, Vladimir


By the way, our regression tests are 'classic' regression tests that
demonstrate some issues which were not resolved by initial code. But it
provides less coverage than 'regression tests' + unit tests, of cause.

On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:

 On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
  On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
   On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
   
If you've got fast and slow tests, then have a group for fast and
 slow
tests. Then you can choose to just run the fast tests, and any
automated build system can handle running the slow tests.
  
   IMHO, fast or slow may not be the key point. The question is
whether
 we
   have any requirements to run only the regression tests.
 
  No, probably not the key point, but (a) groups don't have to be
  mutually exclusive (so you can decorate it with whatever groups you
  want)

 I agree. For example, os.win and os.linux are not mutually exclusive.

 Thanks a lot.

 and (b) it might be useful for an automated build system to run
  fast tests first, followed by slow (or non-fast) tests.

 That makes sense through we have not clear requirement currently.

  Mind you, I don't know what's going to happen with an automated
 test'n'build
  system; so it might not make sense to do it at this point.

 Really? ;-) We could also discuss whether it's feasible to move to
 TestNG. As you may know, there are already several threads about
 TestNG  JUnit. Here I just review the open questions one by one so
 that we have sufficient preparation.


 [1]http://mail-
archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]

 [2]http://mail-
archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]

 [3]http://mail-
archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]


 Best regards,
 Richard

 
  Alex.
 
  -
  Terms of use : http://incubator.apache.org/harmony/mailing.html
  To unsubscribe, e-mail: [EMAIL PROTECTED]
  For additional commands, e-mail: [EMAIL PROTECTED]
 
 


 --
 Richard Liang
 China Software Development Lab, IBM

 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]







--
Andrew Zhang
China Software Development Lab, IBM

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Vladimir Ivanov]

One more note (seems it already was said sorry if I repeat): the test
without any marks should be run in all configurations (i.e. we have
'default' group but declaration of this group may be missed).

thanks, Vladimir


On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:


 OK, let's return back to the usage model.
If I understood it correctly, before the commit of any changes each
developer run *all* tests (at least all which we have now) on all available
to him platforms. In this context seems we don't need in any 'level' group
(while 'stress' tests require reasonable time to pass).
Seems, that platform group also can be deleted (at present time we have
10 platform-dependent tests and this amount should not increase
dramatically so the platform-detection can be included to the each such
test).
Also cpu groups can be deleted (while we have not cpu-dependent test).
At the end we need only state groups to support test exclusion on the
'one-element' level (while we have unresolved entries in the current exclude
list).

So, after small update of unit (aka integration, aka regression etc) tests
and resolution of all entries in the exclude list we don't need any groups
and pure JUnit covers all our needs :)

On the other side, if we define some groups it will nice to define *all*
reasonable groups at the begin of the process.

 thanks, Vladimir

By the way, our regression tests are 'classic' regression tests that
demonstrate some issues which were not resolved by initial code. But it
provides less coverage than 'regression tests' + unit tests, of cause.

 On 9/5/06, Richard Liang [EMAIL PROTECTED]  wrote:

 On 9/5/06, Alex Blewitt  [EMAIL PROTECTED] wrote:
  On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
   On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
   
If you've got fast and slow tests, then have a group for fast and
 slow
tests. Then you can choose to just run the fast tests, and any
automated build system can handle running the slow tests.
  
   IMHO, fast or slow may not be the key point. The question is
 whether we
   have any requirements to run only the regression tests.
 
  No, probably not the key point, but (a) groups don't have to be
  mutually exclusive (so you can decorate it with whatever groups you
  want)

 I agree. For example, os.win and os.linux are not mutually exclusive.

 Thanks a lot.

 and (b) it might be useful for an automated build system to run
  fast tests first, followed by slow (or non-fast) tests.

 That makes sense through we have not clear requirement currently.

  Mind you, I don't know what's going to happen with an automated
 test'n'build
  system; so it might not make sense to do it at this point.

 Really? ;-) We could also discuss whether it's feasible to move to
 TestNG. As you may know, there are already several threads about
 TestNG  JUnit. Here I just review the open questions one by one so
 that we have sufficient preparation.


 
[1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

 
[2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

 
[3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]


 Best regards,
 Richard

 
  Alex.
 
  -
  Terms of use : http://incubator.apache.org/harmony/mailing.html
  To unsubscribe, e-mail: [EMAIL PROTECTED]
  For additional commands, e-mail: [EMAIL PROTECTED]
 
 


 --
 Richard Liang
 China Software Development Lab, IBM

 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Vladimir Ivanov]

On 9/5/06, Andrew Zhang [EMAIL PROTECTED] wrote:


On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:

 OK, let's return back to the usage model.
 If I understood it correctly, before the commit of any changes each
 developer run *all* tests (at least all which we have now) on all
 available
 to him platforms.


Yes. But as you mentioned, what's all available?

If a test passes on windows while fails on linux, is it available to
windows?

If it is, how will we control it? TestNG groups.



If developer has only windows box than he should run tests on windows only.
If test fails on linux than (for example):
- the test/ implementation should be fixed or
- the test should define platform and report 'passed' if it does not
support current platform

In this context seems we don't need in any 'level' group

 (while 'stress' tests require reasonable time to pass).
 Seems, that platform group also can be deleted (at present time we
have
 10 platform-dependent tests and this amount should not increase
 dramatically so the platform-detection can be included to the each such
 test).
 Also cpu groups can be deleted (while we have not cpu-dependent test).
 At the end we need only state groups to support test exclusion on the
 'one-element' level (while we have unresolved entries in the current
 exclude
 list).

 So, after small update of unit (aka integration, aka regression etc)
tests
 and resolution of all entries in the exclude list we don't need any
groups
 and pure JUnit covers all our needs :)

 On the other side, if we define some groups it will nice to define *all*
 reasonable groups at the begin of the process.


Yes. We should figure out all possible groups. And it can be consolidated
during applying TestNG.



Agree.
thanks, Vladimir

thanks, Vladimir


 By the way, our regression tests are 'classic' regression tests that
 demonstrate some issues which were not resolved by initial code. But it
 provides less coverage than 'regression tests' + unit tests, of cause.

 On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:
 
  On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
   On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:

 If you've got fast and slow tests, then have a group for fast
and
  slow
 tests. Then you can choose to just run the fast tests, and any
 automated build system can handle running the slow tests.
   
IMHO, fast or slow may not be the key point. The question is
 whether
  we
have any requirements to run only the regression tests.
  
   No, probably not the key point, but (a) groups don't have to be
   mutually exclusive (so you can decorate it with whatever groups you
   want)
 
  I agree. For example, os.win and os.linux are not mutually exclusive.
 
  Thanks a lot.
 
  and (b) it might be useful for an automated build system to run
   fast tests first, followed by slow (or non-fast) tests.
 
  That makes sense through we have not clear requirement currently.
 
   Mind you, I don't know what's going to happen with an automated
  test'n'build
   system; so it might not make sense to do it at this point.
 
  Really? ;-) We could also discuss whether it's feasible to move to
  TestNG. As you may know, there are already several threads about
  TestNG  JUnit. Here I just review the open questions one by one so
  that we have sufficient preparation.
 
 
  [1]http://mail-

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
 
  [2]http://mail-

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
 
  [3]http://mail-

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
 
 
  Best regards,
  Richard
 
  
   Alex.
  
  
-
   Terms of use : http://incubator.apache.org/harmony/mailing.html
   To unsubscribe, e-mail: [EMAIL PROTECTED]
   For additional commands, e-mail:
[EMAIL PROTECTED]
  
  
 
 
  --
  Richard Liang
  China Software Development Lab, IBM
 
  -
  Terms of use : http://incubator.apache.org/harmony/mailing.html
  To unsubscribe, e-mail: [EMAIL PROTECTED]
  For additional commands, e-mail: [EMAIL PROTECTED]
 
 




--
Andrew Zhang
China Software Development Lab, IBM

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Andrew Zhang]

On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:


On 9/5/06, Andrew Zhang [EMAIL PROTECTED] wrote:

 On 9/5/06, Vladimir Ivanov [EMAIL PROTECTED] wrote:
 
  OK, let's return back to the usage model.
  If I understood it correctly, before the commit of any changes each
  developer run *all* tests (at least all which we have now) on all
  available
  to him platforms.


 Yes. But as you mentioned, what's all available?

 If a test passes on windows while fails on linux, is it available to
 windows?

 If it is, how will we control it? TestNG groups.


If developer has only windows box than he should run tests on windows
only.
If test fails on linux than (for example):
- the test/ implementation should be fixed or



What if the behaviour is different on different platforms?

- the test should define platform and report 'passed' if it does not

support current platform



How to define platform? Get platform information in test code?

In this context seems we don't need in any 'level' group

  (while 'stress' tests require reasonable time to pass).
  Seems, that platform group also can be deleted (at present time we
 have
  10 platform-dependent tests and this amount should not increase
  dramatically so the platform-detection can be included to the each
such
  test).
  Also cpu groups can be deleted (while we have not cpu-dependent
test).
  At the end we need only state groups to support test exclusion on
the
  'one-element' level (while we have unresolved entries in the current
  exclude
  list).
 
  So, after small update of unit (aka integration, aka regression etc)
 tests
  and resolution of all entries in the exclude list we don't need any
 groups
  and pure JUnit covers all our needs :)
 
  On the other side, if we define some groups it will nice to define
*all*
  reasonable groups at the begin of the process.


 Yes. We should figure out all possible groups. And it can be
consolidated
 during applying TestNG.


Agree.
thanks, Vladimir

thanks, Vladimir
 
  By the way, our regression tests are 'classic' regression tests that
  demonstrate some issues which were not resolved by initial code. But
it
  provides less coverage than 'regression tests' + unit tests, of cause.
 
  On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:
  
   On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
 On 9/4/06, Alex Blewitt [EMAIL PROTECTED]  wrote:
 
  If you've got fast and slow tests, then have a group for fast
 and
   slow
  tests. Then you can choose to just run the fast tests, and any
  automated build system can handle running the slow tests.

 IMHO, fast or slow may not be the key point. The question is
  whether
   we
 have any requirements to run only the regression tests.
   
No, probably not the key point, but (a) groups don't have to be
mutually exclusive (so you can decorate it with whatever groups
you
want)
  
   I agree. For example, os.win and os.linux are not mutually
exclusive.
  
   Thanks a lot.
  
   and (b) it might be useful for an automated build system to run
fast tests first, followed by slow (or non-fast) tests.
  
   That makes sense through we have not clear requirement currently.
  
Mind you, I don't know what's going to happen with an automated
   test'n'build
system; so it might not make sense to do it at this point.
  
   Really? ;-) We could also discuss whether it's feasible to move to
   TestNG. As you may know, there are already several threads about
   TestNG  JUnit. Here I just review the open questions one by one so
   that we have sufficient preparation.
  
  
   [1]http://mail-
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
  
   [2]http://mail-
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
  
   [3]http://mail-
 

archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL PROTECTED]
  
  
   Best regards,
   Richard
  
   
Alex.
   
   
 -
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail:
[EMAIL PROTECTED]
For additional commands, e-mail:
 [EMAIL PROTECTED]
   
   
  
  
   --
   Richard Liang
   China Software Development Lab, IBM
  
  
-
   Terms of use : http://incubator.apache.org/harmony/mailing.html
   To unsubscribe, e-mail: [EMAIL PROTECTED]
   For additional commands, e-mail:
[EMAIL PROTECTED]
  
  
 
 


 --
 Andrew Zhang
 China Software Development Lab, IBM







--
Andrew Zhang
China Software Development Lab, IBM

Re: Re: Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

On 9/5/06, Richard Liang [EMAIL PROTECTED] wrote:

On 9/5/06, Alex Blewitt [EMAIL PROTECTED] wrote:
 On 04/09/06, Richard Liang [EMAIL PROTECTED] wrote:
  On 9/4/06, Alex Blewitt [EMAIL PROTECTED] wrote:
  
   If you've got fast and slow tests, then have a group for fast and slow
   tests. Then you can choose to just run the fast tests, and any
   automated build system can handle running the slow tests.
 
  IMHO, fast or slow may not be the key point. The question is whether we
  have any requirements to run only the regression tests.

 No, probably not the key point, but (a) groups don't have to be
 mutually exclusive (so you can decorate it with whatever groups you
 want)

I agree. For example, os.win and os.linux are not mutually exclusive.

Thanks a lot.

and (b) it might be useful for an automated build system to run
 fast tests first, followed by slow (or non-fast) tests.

That makes sense through we have not clear requirement currently.

 Mind you, I don't know what's going to happen with an automated test'n'build
 system; so it might not make sense to do it at this point.

Really? ;-) We could also discuss whether it's feasible to move to
TestNG. As you may know, there are already several threads about
TestNG  JUnit. Here I just review the open questions one by one so
that we have sufficient preparation.



Alex, I must say sorry if I misunderstood you. I mean here we could
discuss anything related to Harmony testing. ;-)

Best regards,
Richard




[1]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]
[2]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]
[3]http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200607.mbox/[EMAIL
 PROTECTED]

Best regards,
Richard


 Alex.

 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]




--
Richard Liang
China Software Development Lab, IBM




--
Richard Liang
China Software Development Lab, IBM

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Mikhail Loenko]

Hi Vladimir

Could you please decribe for what purpose it will be used?

I mean why one might have to either exclude or run only regression tests?

Thanks,
Mikhail

2006/8/30, Vladimir Ivanov [EMAIL PROTECTED]:

On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:



 Vladimir Ivanov wrote:
  Also some tag for regression tests should be added.
 Yes. Do you think we could annotate regression test as
 *level.regression*? Thanks a lot.


Yes, I do. While tests can have more than one group it will enough.
 thanks, Vladimir


Richard
  thanks, Vladimir
 
 
  On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:
 
 
 
  Richard Liang wrote:
   Hello All,
  
   Now let's talk about the TestNG groups. I have read the related
   threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
   All of them are good discussion about TestNG groups.
  
   IMHO, we may define Harmony test groups according the following 4
   dimensions:
  
   1) [Platform] os.any, os.platform id
   *os.any* - group of tests which pass on any platform. IMHO, most of
   our tests should be in this group.
   *os.platform id* - group of tests which are designed for one
   specific platform. A test may be in more than one of the groups. e.g
 .,
   @Test(groups={os.win.IA32, os.linux.IA32})
  
  ** os.any and os.platform id are mutually exclusive, that is,
   tests in os.any group should not be in os.win.IA32.
  
   2) [Test state] state.broken, state.broken.platform id
   *state.broken* - group of tests which fail on every platform, because
   of bugs of tests or implementation. We need to fix the bugs of tests
   or implementation to make them pass.
   *state.broken.platform id* - groups of test which only fail on one
   specific platform. A test may be in more than one of the groups. e.g
 .,
   @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})
  
   **state.broken.platform id group may be used as a convenient
 way
   to indicate that a test is platform-specific. e.g., If we support 10
   platforms, and one test are designed for 9 platforms except for
 MacOS,
   instead of list 9 os.platform id, we can just use
 state.broken.MacOS
  
   3) [Test type] type.api , type.impl
   *type.api* - group of tests which are tests for APIs in the Java
   Specification
   *type.impl* - groups of tests which are tests for Harmony-specific
   implementation
  
   ** type.api and type.impl are also mutually exclusive.
  
   4) [Test Level] level.unit, level.integration, level.system,
   level.stress, etc. (Levels of Test refer to the increase in
 complexity
   as moving through test cycle. )
  ** A test may be in more than one of the groups.
  ** In fact, some tests such as System tests are the verification
 of
   the entire system.  Maybe we'll put them into a separate project.
   e.g., harmony/enhanced/SVT (System Verification Test).
  
   If we want to run all the unit test for APIs on windows, we may use
   TestNG groups to select the tests:
  groups
  run
  include name=os.any /
  include name=type.api /
  include name=os.win.IA32 /
  exclude name= state.broken /
  exclude name=state.broken.win.IA32 /
  /run
  /groups
  
  Hello All,
 
  I'm sorry. It seems that the example does not work. I will try to
 figure
  another example soon. ;-)
 
  Best regards,
  Richard
  
   Well, I think our most of existing tests are in the groups of
   {os.any, type.api, level.unit}, and I have asked TestNG to add
 a
   new option -groups for its JUnitConverter which allow us to specify
   the test groups when migrate from JUnit test to TestNG test.
  
   Thanks for reading so far, and I will highly appreciate your comments
   or suggestion.  ;-)
  
 
  --
  Richard Liang
  China Software Development Lab, IBM
 
 
 
  -
  Terms of use : http://incubator.apache.org/harmony/mailing.html
  To unsubscribe, e-mail: [EMAIL PROTECTED]
  For additional commands, e-mail: [EMAIL PROTECTED]
 
 
 

 --
 Richard Liang
 China Software Development Lab, IBM



 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]






-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

Stepan Mishura wrote:

On 8/25/06, Richard Liang wrote:


Hello All,

Now let's talk about the TestNG groups. I have read the related threads
which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them
are good discussion about TestNG groups.

IMHO, we may define Harmony test groups according the following 4
dimensions:

1) [Platform] os.any, os.platform id
*os.any* - group of tests which pass on any platform. IMHO, most of our
tests should be in this group.
*os.platform id* - group of tests which are designed for one specific
platform. A test may be in more than one of the groups. e.g.,
@Test(groups={os.win.IA32, os.linux.IA32})

   ** os.any and os.platform id are mutually exclusive, that is,
tests in os.any group should not be in os.win.IA32.

2) [Test state] state.broken, state.broken.platform id
*state.broken* - group of tests which fail on every platform, because
of bugs of tests or implementation. We need to fix the bugs of tests or
implementation to make them pass.
*state.broken.platform id* - groups of test which only fail on one
specific platform. A test may be in more than one of the groups. e.g.,
@Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})

**state.broken.platform id group may be used as a convenient way
to indicate that a test is platform-specific. e.g., If we support 10
platforms, and one test are designed for 9 platforms except for MacOS,
instead of list 9 os.platform id, we can just use state.broken.MacOS



If a test is marked as *state.broken.MacOS* then it sounds like the
test/implementation should be fixed. IMO we should use tag 
os.platform id
to define explicitly valid platforms for the test so in this 
particular case

we should use 9 os.platform ids



Thank you,  Stepan.  This sounds reasonable, we should explicitly list 
the valid platforms.


Best regards,
Richard



3) [Test type] type.api, type.impl

*type.api* - group of tests which are tests for APIs in the Java
Specification
*type.impl* - groups of tests which are tests for Harmony-specific
implementation

** type.api and type.impl are also mutually exclusive.

4) [Test Level] level.unit, level.integration, level.system,
level.stress, etc. (Levels of Test refer to the increase in complexity
as moving through test cycle. )
   ** A test may be in more than one of the groups.
   ** In fact, some tests such as System tests are the verification of
the entire system.  Maybe we'll put them into a separate project. e.g.,
harmony/enhanced/SVT (System Verification Test).



Mixing different types of testing into one test-file doesn't look good 
for
me. I'd separate such tests by placing into different 
directories/packages.


Thanks,
Stepan.

If we want to run all the unit test for APIs on windows, we may use

TestNG groups to select the tests:
   groups
   run
   include name=os.any /
   include name=type.api /
   include name=os.win.IA32 /
   exclude name=state.broken /
   exclude name=state.broken.win.IA32 /
   /run
   /groups


Well, I think our most of existing tests are in the groups of {os.any,
type.api, level.unit}, and I have asked TestNG to add a new option
-groups for its JUnitConverter which allow us to specify the test
groups when migrate from JUnit test to TestNG test.

Thanks for reading so far, and I will highly appreciate your comments or
suggestion.  ;-)

--
Richard Liang
China Software Development Lab, IBM





--
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



--
Richard Liang
China Software Development Lab, IBM 




-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Vladimir Ivanov]

On 8/30/06, Richard Liang [EMAIL PROTECTED] wrote:




Vladimir Ivanov wrote:
 Also some tag for regression tests should be added.
Yes. Do you think we could annotate regression test as
*level.regression*? Thanks a lot.



Yes, I do. While tests can have more than one group it will enough.
thanks, Vladimir


Richard

 thanks, Vladimir


 On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:



 Richard Liang wrote:
  Hello All,
 
  Now let's talk about the TestNG groups. I have read the related
  threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
  All of them are good discussion about TestNG groups.
 
  IMHO, we may define Harmony test groups according the following 4
  dimensions:
 
  1) [Platform] os.any, os.platform id
  *os.any* - group of tests which pass on any platform. IMHO, most of
  our tests should be in this group.
  *os.platform id* - group of tests which are designed for one
  specific platform. A test may be in more than one of the groups. e.g
.,
  @Test(groups={os.win.IA32, os.linux.IA32})
 
 ** os.any and os.platform id are mutually exclusive, that is,
  tests in os.any group should not be in os.win.IA32.
 
  2) [Test state] state.broken, state.broken.platform id
  *state.broken* - group of tests which fail on every platform, because
  of bugs of tests or implementation. We need to fix the bugs of tests
  or implementation to make them pass.
  *state.broken.platform id* - groups of test which only fail on one
  specific platform. A test may be in more than one of the groups. e.g
.,
  @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})
 
  **state.broken.platform id group may be used as a convenient
way
  to indicate that a test is platform-specific. e.g., If we support 10
  platforms, and one test are designed for 9 platforms except for
MacOS,
  instead of list 9 os.platform id, we can just use
state.broken.MacOS
 
  3) [Test type] type.api , type.impl
  *type.api* - group of tests which are tests for APIs in the Java
  Specification
  *type.impl* - groups of tests which are tests for Harmony-specific
  implementation
 
  ** type.api and type.impl are also mutually exclusive.
 
  4) [Test Level] level.unit, level.integration, level.system,
  level.stress, etc. (Levels of Test refer to the increase in
complexity
  as moving through test cycle. )
 ** A test may be in more than one of the groups.
 ** In fact, some tests such as System tests are the verification
of
  the entire system.  Maybe we'll put them into a separate project.
  e.g., harmony/enhanced/SVT (System Verification Test).
 
  If we want to run all the unit test for APIs on windows, we may use
  TestNG groups to select the tests:
 groups
 run
 include name=os.any /
 include name=type.api /
 include name=os.win.IA32 /
 exclude name= state.broken /
 exclude name=state.broken.win.IA32 /
 /run
 /groups
 
 Hello All,

 I'm sorry. It seems that the example does not work. I will try to
figure
 another example soon. ;-)

 Best regards,
 Richard
 
  Well, I think our most of existing tests are in the groups of
  {os.any, type.api, level.unit}, and I have asked TestNG to add
a
  new option -groups for its JUnitConverter which allow us to specify
  the test groups when migrate from JUnit test to TestNG test.
 
  Thanks for reading so far, and I will highly appreciate your comments
  or suggestion.  ;-)
 

 --
 Richard Liang
 China Software Development Lab, IBM



 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]




--
Richard Liang
China Software Development Lab, IBM



-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Tony Wu]

I think we can treat something like *os.any*,*type.api* as default
groups.Itwill make our logic more clear and bring convenience to
testcase writing :)
for example, an os.platformid group indicates that it is designed for
paltformid only. If there is no group like os.platformid here, this is a
*os.any* test.

so if we want to run all *api* tests on *win32* platform which is *not
broken*, we could write the testng.xml like this:
groups
  run
  include name=.* /
  exclude name=type.impl /
  exclude name=os\.(?!win\.IA32).* /
  exclude name=state.broken /
  exclude name=state.broken.win.IA32 /
  /run
  /groups


2006/8/29, Vladimir Ivanov [EMAIL PROTECTED]:


Also some tag for regression tests should be added.
thanks, Vladimir


On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:



 Richard Liang wrote:
  Hello All,
 
  Now let's talk about the TestNG groups. I have read the related
  threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
  All of them are good discussion about TestNG groups.
 
  IMHO, we may define Harmony test groups according the following 4
  dimensions:
 
  1) [Platform] os.any, os.platform id
  *os.any* - group of tests which pass on any platform. IMHO, most of
  our tests should be in this group.
  *os.platform id* - group of tests which are designed for one
  specific platform. A test may be in more than one of the groups. e.g.,
  @Test(groups={os.win.IA32, os.linux.IA32})
 
 ** os.any and os.platform id are mutually exclusive, that is,
  tests in os.any group should not be in os.win.IA32.
 
  2) [Test state] state.broken, state.broken.platform id
  *state.broken* - group of tests which fail on every platform, because
  of bugs of tests or implementation. We need to fix the bugs of tests
  or implementation to make them pass.
  *state.broken.platform id* - groups of test which only fail on one
  specific platform. A test may be in more than one of the groups. e.g.,
  @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})
 
  **state.broken.platform id group may be used as a convenient way
  to indicate that a test is platform-specific. e.g., If we support 10
  platforms, and one test are designed for 9 platforms except for MacOS,
  instead of list 9 os.platform id, we can just use state.broken.MacOS
 
  3) [Test type] type.api , type.impl
  *type.api* - group of tests which are tests for APIs in the Java
  Specification
  *type.impl* - groups of tests which are tests for Harmony-specific
  implementation
 
  ** type.api and type.impl are also mutually exclusive.
 
  4) [Test Level] level.unit, level.integration, level.system,
  level.stress, etc. (Levels of Test refer to the increase in complexity
  as moving through test cycle. )
 ** A test may be in more than one of the groups.
 ** In fact, some tests such as System tests are the verification of
  the entire system.  Maybe we'll put them into a separate project.
  e.g., harmony/enhanced/SVT (System Verification Test).
 
  If we want to run all the unit test for APIs on windows, we may use
  TestNG groups to select the tests:
 groups
 run
 include name=os.any /
 include name=type.api /
 include name=os.win.IA32 /
 exclude name= state.broken /
 exclude name=state.broken.win.IA32 /
 /run
 /groups
 
 Hello All,

 I'm sorry. It seems that the example does not work. I will try to figure
 another example soon. ;-)

 Best regards,
 Richard
 
  Well, I think our most of existing tests are in the groups of
  {os.any, type.api, level.unit}, and I have asked TestNG to add a
  new option -groups for its JUnitConverter which allow us to specify
  the test groups when migrate from JUnit test to TestNG test.
 
  Thanks for reading so far, and I will highly appreciate your comments
  or suggestion.  ;-)
 

 --
 Richard Liang
 China Software Development Lab, IBM



 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]







--
Tony Wu
China Software Development Lab, IBM

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

Tony Wu wrote:

I think we can treat something like *os.any*,*type.api* as default
groups.Itwill make our logic more clear and bring convenience to
testcase writing :)
for example, an os.platformid group indicates that it is designed for
paltformid only. If there is no group like os.platformid here, 
this is a

*os.any* test.

so if we want to run all *api* tests on *win32* platform which is *not
broken*, we could write the testng.xml like this:
groups
  run
  include name=.* /
  exclude name=type.impl /
  exclude name=os\.(?!win\.IA32).* /
  exclude name=state.broken /
  exclude name=state.broken.win.IA32 /
  /run
  /groups


Hello Tony,

It's a good idea. However, we shall define a default group which means 
os.any, type.impl, level.unit. Thanks a lot.


Best regards,
Richard



2006/8/29, Vladimir Ivanov [EMAIL PROTECTED]:


Also some tag for regression tests should be added.
thanks, Vladimir


On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:



 Richard Liang wrote:
  Hello All,
 
  Now let's talk about the TestNG groups. I have read the related
  threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
  All of them are good discussion about TestNG groups.
 
  IMHO, we may define Harmony test groups according the following 4
  dimensions:
 
  1) [Platform] os.any, os.platform id
  *os.any* - group of tests which pass on any platform. IMHO, most of
  our tests should be in this group.
  *os.platform id* - group of tests which are designed for one
  specific platform. A test may be in more than one of the groups. 
e.g.,

  @Test(groups={os.win.IA32, os.linux.IA32})
 
 ** os.any and os.platform id are mutually exclusive, that is,
  tests in os.any group should not be in os.win.IA32.
 
  2) [Test state] state.broken, state.broken.platform id
  *state.broken* - group of tests which fail on every platform, 
because

  of bugs of tests or implementation. We need to fix the bugs of tests
  or implementation to make them pass.
  *state.broken.platform id* - groups of test which only fail on one
  specific platform. A test may be in more than one of the groups. 
e.g.,

  @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})
 
  **state.broken.platform id group may be used as a 
convenient way

  to indicate that a test is platform-specific. e.g., If we support 10
  platforms, and one test are designed for 9 platforms except for 
MacOS,
  instead of list 9 os.platform id, we can just use 
state.broken.MacOS

 
  3) [Test type] type.api , type.impl
  *type.api* - group of tests which are tests for APIs in the Java
  Specification
  *type.impl* - groups of tests which are tests for Harmony-specific
  implementation
 
  ** type.api and type.impl are also mutually exclusive.
 
  4) [Test Level] level.unit, level.integration, level.system,
  level.stress, etc. (Levels of Test refer to the increase in 
complexity

  as moving through test cycle. )
 ** A test may be in more than one of the groups.
 ** In fact, some tests such as System tests are the 
verification of

  the entire system.  Maybe we'll put them into a separate project.
  e.g., harmony/enhanced/SVT (System Verification Test).
 
  If we want to run all the unit test for APIs on windows, we may use
  TestNG groups to select the tests:
 groups
 run
 include name=os.any /
 include name=type.api /
 include name=os.win.IA32 /
 exclude name= state.broken /
 exclude name=state.broken.win.IA32 /
 /run
 /groups
 
 Hello All,

 I'm sorry. It seems that the example does not work. I will try to 
figure

 another example soon. ;-)

 Best regards,
 Richard
 
  Well, I think our most of existing tests are in the groups of
  {os.any, type.api, level.unit}, and I have asked TestNG to 
add a
  new option -groups for its JUnitConverter which allow us to 
specify

  the test groups when migrate from JUnit test to TestNG test.
 
  Thanks for reading so far, and I will highly appreciate your 
comments

  or suggestion.  ;-)
 

 --
 Richard Liang
 China Software Development Lab, IBM



 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]









--
Richard Liang
China Software Development Lab, IBM 




-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

Richard Liang wrote:



Tony Wu wrote:

I think we can treat something like *os.any*,*type.api* as default
groups.Itwill make our logic more clear and bring convenience to
testcase writing :)
for example, an os.platformid group indicates that it is designed for
paltformid only. If there is no group like os.platformid here, 
this is a

*os.any* test.

so if we want to run all *api* tests on *win32* platform which is *not
broken*, we could write the testng.xml like this:
groups
  run
  include name=.* /
  exclude name=type.impl /
  exclude name=os\.(?!win\.IA32).* /
  exclude name=state.broken /
  exclude name=state.broken.win.IA32 /
  /run
  /groups


Hello Tony,

It's a good idea. However, we shall define a default group which 
means os.any, type.impl, level.unit. Thanks a lot.


And even the default group is not necessary by removing the include 
name=.* / 


 groups
 run
 exclude name=type.impl /
 exclude name=os\.(?!win\.IA32).* /
 exclude name=state.broken /
 exclude name=state.broken.win.IA32 /
 /run
 /groups


Richard

Best regards,
Richard



2006/8/29, Vladimir Ivanov [EMAIL PROTECTED]:


Also some tag for regression tests should be added.
thanks, Vladimir


On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:



 Richard Liang wrote:
  Hello All,
 
  Now let's talk about the TestNG groups. I have read the related
  threads which posted by George, Vladimir Ivanov and Alexei 
Zakharov.

  All of them are good discussion about TestNG groups.
 
  IMHO, we may define Harmony test groups according the following 4
  dimensions:
 
  1) [Platform] os.any, os.platform id
  *os.any* - group of tests which pass on any platform. IMHO, most of
  our tests should be in this group.
  *os.platform id* - group of tests which are designed for one
  specific platform. A test may be in more than one of the groups. 
e.g.,

  @Test(groups={os.win.IA32, os.linux.IA32})
 
 ** os.any and os.platform id are mutually exclusive, that is,
  tests in os.any group should not be in os.win.IA32.
 
  2) [Test state] state.broken, state.broken.platform id
  *state.broken* - group of tests which fail on every platform, 
because
  of bugs of tests or implementation. We need to fix the bugs of 
tests

  or implementation to make them pass.
  *state.broken.platform id* - groups of test which only fail on 
one
  specific platform. A test may be in more than one of the groups. 
e.g.,

  @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})
 
  **state.broken.platform id group may be used as a 
convenient way
  to indicate that a test is platform-specific. e.g., If we 
support 10
  platforms, and one test are designed for 9 platforms except for 
MacOS,
  instead of list 9 os.platform id, we can just use 
state.broken.MacOS

 
  3) [Test type] type.api , type.impl
  *type.api* - group of tests which are tests for APIs in the Java
  Specification
  *type.impl* - groups of tests which are tests for Harmony-specific
  implementation
 
  ** type.api and type.impl are also mutually exclusive.
 
  4) [Test Level] level.unit, level.integration, level.system,
  level.stress, etc. (Levels of Test refer to the increase in 
complexity

  as moving through test cycle. )
 ** A test may be in more than one of the groups.
 ** In fact, some tests such as System tests are the 
verification of

  the entire system.  Maybe we'll put them into a separate project.
  e.g., harmony/enhanced/SVT (System Verification Test).
 
  If we want to run all the unit test for APIs on windows, we may use
  TestNG groups to select the tests:
 groups
 run
 include name=os.any /
 include name=type.api /
 include name=os.win.IA32 /
 exclude name= state.broken /
 exclude name=state.broken.win.IA32 /
 /run
 /groups
 
 Hello All,

 I'm sorry. It seems that the example does not work. I will try to 
figure

 another example soon. ;-)

 Best regards,
 Richard
 
  Well, I think our most of existing tests are in the groups of
  {os.any, type.api, level.unit}, and I have asked TestNG to 
add a
  new option -groups for its JUnitConverter which allow us to 
specify

  the test groups when migrate from JUnit test to TestNG test.
 
  Thanks for reading so far, and I will highly appreciate your 
comments

  or suggestion.  ;-)
 

 --
 Richard Liang
 China Software Development Lab, IBM



 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: 
[EMAIL PROTECTED]












--
Richard Liang
China Software Development Lab, IBM 




-
Terms of use : 

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

Vladimir Ivanov wrote:

Also some tag for regression tests should be added.
Yes. Do you think we could annotate regression test as 
*level.regression*? Thanks a lot.


Richard

thanks, Vladimir


On 8/28/06, Richard Liang [EMAIL PROTECTED] wrote:




Richard Liang wrote:
 Hello All,

 Now let's talk about the TestNG groups. I have read the related
 threads which posted by George, Vladimir Ivanov and Alexei Zakharov.
 All of them are good discussion about TestNG groups.

 IMHO, we may define Harmony test groups according the following 4
 dimensions:

 1) [Platform] os.any, os.platform id
 *os.any* - group of tests which pass on any platform. IMHO, most of
 our tests should be in this group.
 *os.platform id* - group of tests which are designed for one
 specific platform. A test may be in more than one of the groups. e.g.,
 @Test(groups={os.win.IA32, os.linux.IA32})

** os.any and os.platform id are mutually exclusive, that is,
 tests in os.any group should not be in os.win.IA32.

 2) [Test state] state.broken, state.broken.platform id
 *state.broken* - group of tests which fail on every platform, because
 of bugs of tests or implementation. We need to fix the bugs of tests
 or implementation to make them pass.
 *state.broken.platform id* - groups of test which only fail on one
 specific platform. A test may be in more than one of the groups. e.g.,
 @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})

 **state.broken.platform id group may be used as a convenient way
 to indicate that a test is platform-specific. e.g., If we support 10
 platforms, and one test are designed for 9 platforms except for MacOS,
 instead of list 9 os.platform id, we can just use state.broken.MacOS

 3) [Test type] type.api , type.impl
 *type.api* - group of tests which are tests for APIs in the Java
 Specification
 *type.impl* - groups of tests which are tests for Harmony-specific
 implementation

 ** type.api and type.impl are also mutually exclusive.

 4) [Test Level] level.unit, level.integration, level.system,
 level.stress, etc. (Levels of Test refer to the increase in complexity
 as moving through test cycle. )
** A test may be in more than one of the groups.
** In fact, some tests such as System tests are the verification of
 the entire system.  Maybe we'll put them into a separate project.
 e.g., harmony/enhanced/SVT (System Verification Test).

 If we want to run all the unit test for APIs on windows, we may use
 TestNG groups to select the tests:
groups
run
include name=os.any /
include name=type.api /
include name=os.win.IA32 /
exclude name= state.broken /
exclude name=state.broken.win.IA32 /
/run
/groups

Hello All,

I'm sorry. It seems that the example does not work. I will try to figure
another example soon. ;-)

Best regards,
Richard

 Well, I think our most of existing tests are in the groups of
 {os.any, type.api, level.unit}, and I have asked TestNG to add a
 new option -groups for its JUnitConverter which allow us to specify
 the test groups when migrate from JUnit test to TestNG test.

 Thanks for reading so far, and I will highly appreciate your comments
 or suggestion.  ;-)


--
Richard Liang
China Software Development Lab, IBM



-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]






--
Richard Liang
China Software Development Lab, IBM 




-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

Paulex Yang wrote:

Richard Liang wrote:

Hello All,

Now let's talk about the TestNG groups. I have read the related 
threads which posted by George, Vladimir Ivanov and Alexei Zakharov. 
All of them are good discussion about TestNG groups.


IMHO, we may define Harmony test groups according the following 4 
dimensions:


1) [Platform] os.any, os.platform id
*os.any* - group of tests which pass on any platform. IMHO, most of 
our tests should be in this group.
*os.platform id* - group of tests which are designed for one 
specific platform. A test may be in more than one of the groups. 
e.g., @Test(groups={os.win.IA32, os.linux.IA32})


   ** os.any and os.platform id are mutually exclusive, that is, 
tests in os.any group should not be in os.win.IA32.
I suggest to separate the CPU arch with OS, they are actually 
different things, and we probably will have tests written for all 
64bits arch, or for all Linux OS.


Thanks a lot, Paulex. I agree. Separating os and arch make our groups 
more flexible. I will update the grouping proposal soon to integrate 
your and others' suggestion.


Best regards,
Richard



2) [Test state] state.broken, state.broken.platform id
*state.broken* - group of tests which fail on every platform, because 
of bugs of tests or implementation. We need to fix the bugs of tests 
or implementation to make them pass.
*state.broken.platform id* - groups of test which only fail on one 
specific platform. A test may be in more than one of the groups. 
e.g., @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})


**state.broken.platform id group may be used as a convenient 
way to indicate that a test is platform-specific. e.g., If we support 
10 platforms, and one test are designed for 9 platforms except for 
MacOS, instead of list 9 os.platform id, we can just use 
state.broken.MacOS


3) [Test type] type.api, type.impl
*type.api* - group of tests which are tests for APIs in the Java 
Specification
*type.impl* - groups of tests which are tests for Harmony-specific 
implementation


** type.api and type.impl are also mutually exclusive.

4) [Test Level] level.unit, level.integration, level.system, 
level.stress, etc. (Levels of Test refer to the increase in 
complexity as moving through test cycle. )

   ** A test may be in more than one of the groups.
   ** In fact, some tests such as System tests are the verification 
of the entire system.  Maybe we'll put them into a separate project. 
e.g., harmony/enhanced/SVT (System Verification Test).


If we want to run all the unit test for APIs on windows, we may use 
TestNG groups to select the tests:

   groups
   run
   include name=os.any /
   include name=type.api /
   include name=os.win.IA32 /
   exclude name=state.broken /
   exclude name=state.broken.win.IA32 /
   /run
   /groups


Well, I think our most of existing tests are in the groups of 
{os.any, type.api, level.unit}, and I have asked TestNG to add 
a new option -groups for its JUnitConverter which allow us to 
specify the test groups when migrate from JUnit test to TestNG test.


Thanks for reading so far, and I will highly appreciate your comments 
or suggestion.  ;-)







--
Richard Liang
China Software Development Lab, IBM 




-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Geir Magnusson Jr.]

I'm just catching up after a few days of travel.  Have you begun 
sketching this out on the wiki?


geir


Richard Liang wrote:

Hello All,

Now let's talk about the TestNG groups. I have read the related threads 
which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them 
are good discussion about TestNG groups.


IMHO, we may define Harmony test groups according the following 4 
dimensions:


1) [Platform] os.any, os.platform id
*os.any* - group of tests which pass on any platform. IMHO, most of our 
tests should be in this group.
*os.platform id* - group of tests which are designed for one specific 
platform. A test may be in more than one of the groups. e.g., 
@Test(groups={os.win.IA32, os.linux.IA32})


   ** os.any and os.platform id are mutually exclusive, that is, tests 
in os.any group should not be in os.win.IA32.


2) [Test state] state.broken, state.broken.platform id
*state.broken* - group of tests which fail on every platform, because of 
bugs of tests or implementation. We need to fix the bugs of tests or 
implementation to make them pass.
*state.broken.platform id* - groups of test which only fail on one 
specific platform. A test may be in more than one of the groups. e.g., 
@Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})


**state.broken.platform id group may be used as a convenient way 
to indicate that a test is platform-specific. e.g., If we support 10 
platforms, and one test are designed for 9 platforms except for MacOS, 
instead of list 9 os.platform id, we can just use state.broken.MacOS


3) [Test type] type.api, type.impl
*type.api* - group of tests which are tests for APIs in the Java 
Specification
*type.impl* - groups of tests which are tests for Harmony-specific 
implementation


** type.api and type.impl are also mutually exclusive.

4) [Test Level] level.unit, level.integration, level.system, 
level.stress, etc. (Levels of Test refer to the increase in complexity 
as moving through test cycle. )

   ** A test may be in more than one of the groups.
   ** In fact, some tests such as System tests are the verification of 
the entire system.  Maybe we'll put them into a separate project. e.g., 
harmony/enhanced/SVT (System Verification Test).


If we want to run all the unit test for APIs on windows, we may use 
TestNG groups to select the tests:

   groups
   run
   include name=os.any /
   include name=type.api /
   include name=os.win.IA32 /
   exclude name=state.broken /
   exclude name=state.broken.win.IA32 /
   /run
   /groups


Well, I think our most of existing tests are in the groups of {os.any, 
type.api, level.unit}, and I have asked TestNG to add a new option 
-groups for its JUnitConverter which allow us to specify the test 
groups when migrate from JUnit test to TestNG test.


Thanks for reading so far, and I will highly appreciate your comments or 
suggestion.  ;-)




-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: [classlib][TestNG] groups of Harmony test

[Ivanov, Alexey A]

-Original Message-
From: Richard Liang [mailto:[EMAIL PROTECTED]
Sent: Friday, August 25, 2006 7:18 PM
To: harmony-dev@incubator.apache.org
Subject: [classlib][TestNG] groups of Harmony test

Hello All,

Now let's talk about the TestNG groups. I have read the related threads
which posted by George, Vladimir Ivanov and Alexei Zakharov. All of
them
are good discussion about TestNG groups.

IMHO, we may define Harmony test groups according the following 4
dimensions:

1) [Platform] os.any, os.platform id
 *os.any* - group of tests which pass on any platform. IMHO, most of
our
tests should be in this group.
 *os.platform id* - group of tests which are designed for one
specific
platform. A test may be in more than one of the groups. e.g.,
@Test(groups={os.win.IA32, os.linux.IA32})

** os.any and os.platform id are mutually exclusive, that is,
tests in os.any group should not be in os.win.IA32.

2) [Test state] state.broken, state.broken.platform id
 *state.broken* - group of tests which fail on every platform, because
of bugs of tests or implementation. We need to fix the bugs of tests or
implementation to make them pass.
 *state.broken.platform id* - groups of test which only fail on one
specific platform. A test may be in more than one of the groups. e.g.,
@Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})

 **state.broken.platform id group may be used as a convenient way
to indicate that a test is platform-specific. e.g., If we support 10
platforms, and one test are designed for 9 platforms except for MacOS,
instead of list 9 os.platform id, we can just use state.broken.MacOS

3) [Test type] type.api, type.impl
 *type.api* - group of tests which are tests for APIs in the Java
Specification
 *type.impl* - groups of tests which are tests for Harmony-specific
implementation

 ** type.api and type.impl are also mutually exclusive.

4) [Test Level] level.unit, level.integration, level.system,
level.stress, etc. (Levels of Test refer to the increase in complexity
as moving through test cycle. )
** A test may be in more than one of the groups.
** In fact, some tests such as System tests are the verification of
the entire system.  Maybe we'll put them into a separate project. e.g.,
harmony/enhanced/SVT (System Verification Test).

5) [Environment] env.display, env.headless
   To distinguish AWT and Swing tests which need a display to run, and
those which don't, as Mark proposed [1].

Regards,
Alexey.

[1]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mb
ox/[EMAIL PROTECTED]


If we want to run all the unit test for APIs on windows, we may use
TestNG groups to select the tests:
groups
run
include name=os.any /
include name=type.api /
include name=os.win.IA32 /
exclude name=state.broken /
exclude name=state.broken.win.IA32 /
/run
/groups


Well, I think our most of existing tests are in the groups of
{os.any,
type.api, level.unit}, and I have asked TestNG to add a new option
-groups for its JUnitConverter which allow us to specify the test
groups when migrate from JUnit test to TestNG test.

Thanks for reading so far, and I will highly appreciate your comments
or
suggestion.  ;-)

--
Richard Liang
China Software Development Lab, IBM



-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

--
Alexey A. Ivanov
Intel Middleware Product Division

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Andrew Zhang]

On 8/28/06, Ivanov, Alexey A [EMAIL PROTECTED] wrote:



-Original Message-
From: Richard Liang [mailto:[EMAIL PROTECTED]
Sent: Friday, August 25, 2006 7:18 PM
To: harmony-dev@incubator.apache.org
Subject: [classlib][TestNG] groups of Harmony test

Hello All,

Now let's talk about the TestNG groups. I have read the related threads
which posted by George, Vladimir Ivanov and Alexei Zakharov. All of
them
are good discussion about TestNG groups.

IMHO, we may define Harmony test groups according the following 4
dimensions:

1) [Platform] os.any, os.platform id
 *os.any* - group of tests which pass on any platform. IMHO, most of
our
tests should be in this group.
 *os.platform id* - group of tests which are designed for one
specific
platform. A test may be in more than one of the groups. e.g.,
@Test(groups={os.win.IA32, os.linux.IA32})

** os.any and os.platform id are mutually exclusive, that is,
tests in os.any group should not be in os.win.IA32.

2) [Test state] state.broken, state.broken.platform id
 *state.broken* - group of tests which fail on every platform, because
of bugs of tests or implementation. We need to fix the bugs of tests or
implementation to make them pass.
 *state.broken.platform id* - groups of test which only fail on one
specific platform. A test may be in more than one of the groups. e.g.,
@Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})

 **state.broken.platform id group may be used as a convenient way
to indicate that a test is platform-specific. e.g., If we support 10
platforms, and one test are designed for 9 platforms except for MacOS,
instead of list 9 os.platform id, we can just use state.broken.MacOS

3) [Test type] type.api, type.impl
 *type.api* - group of tests which are tests for APIs in the Java
Specification
 *type.impl* - groups of tests which are tests for Harmony-specific
implementation

 ** type.api and type.impl are also mutually exclusive.

4) [Test Level] level.unit, level.integration, level.system,
level.stress, etc. (Levels of Test refer to the increase in complexity
as moving through test cycle. )
** A test may be in more than one of the groups.
** In fact, some tests such as System tests are the verification of
the entire system.  Maybe we'll put them into a separate project. e.g.,
harmony/enhanced/SVT (System Verification Test).

5) [Environment] env.display, env.headless
  To distinguish AWT and Swing tests which need a display to run, and
those which don't, as Mark proposed [1].



Will display option be passed manually as an argument to TestNG, or detected
automatically  when running test?

Regards,

Alexey.

[1]
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mb
ox/[EMAIL PROTECTED]


If we want to run all the unit test for APIs on windows, we may use
TestNG groups to select the tests:
groups
run
include name=os.any /
include name=type.api /
include name=os.win.IA32 /
exclude name=state.broken /
exclude name=state.broken.win.IA32 /
/run
/groups


Well, I think our most of existing tests are in the groups of
{os.any,
type.api, level.unit}, and I have asked TestNG to add a new option
-groups for its JUnitConverter which allow us to specify the test
groups when migrate from JUnit test to TestNG test.

Thanks for reading so far, and I will highly appreciate your comments
or
suggestion.  ;-)

--
Richard Liang
China Software Development Lab, IBM



-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

--
Alexey A. Ivanov
Intel Middleware Product Division

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]





--
Andrew Zhang
China Software Development Lab, IBM

RE: [classlib][TestNG] groups of Harmony test

[Ivanov, Alexey A]

-Original Message-
From: Andrew Zhang [mailto:[EMAIL PROTECTED]
Sent: Monday, August 28, 2006 1:06 PM
To: harmony-dev@incubator.apache.org
Subject: Re: [classlib][TestNG] groups of Harmony test

On 8/28/06, Ivanov, Alexey A [EMAIL PROTECTED] wrote:


 -Original Message-
 From: Richard Liang [mailto:[EMAIL PROTECTED]
 Sent: Friday, August 25, 2006 7:18 PM
 To: harmony-dev@incubator.apache.org
 Subject: [classlib][TestNG] groups of Harmony test
 
 Hello All,
 
 Now let's talk about the TestNG groups. I have read the related
threads
 which posted by George, Vladimir Ivanov and Alexei Zakharov. All of
 them
 are good discussion about TestNG groups.
 
 IMHO, we may define Harmony test groups according the following 4
 dimensions:
 
 1) [Platform] os.any, os.platform id
  *os.any* - group of tests which pass on any platform. IMHO, most of
 our
 tests should be in this group.
  *os.platform id* - group of tests which are designed for one
 specific
 platform. A test may be in more than one of the groups. e.g.,
 @Test(groups={os.win.IA32, os.linux.IA32})
 
 ** os.any and os.platform id are mutually exclusive, that is,
 tests in os.any group should not be in os.win.IA32.
 
 2) [Test state] state.broken, state.broken.platform id
  *state.broken* - group of tests which fail on every platform,
because
 of bugs of tests or implementation. We need to fix the bugs of tests
or
 implementation to make them pass.
  *state.broken.platform id* - groups of test which only fail on
one
 specific platform. A test may be in more than one of the groups.
e.g.,
 @Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})
 
  **state.broken.platform id group may be used as a convenient
way
 to indicate that a test is platform-specific. e.g., If we support 10
 platforms, and one test are designed for 9 platforms except for
MacOS,
 instead of list 9 os.platform id, we can just use
state.broken.MacOS
 
 3) [Test type] type.api, type.impl
  *type.api* - group of tests which are tests for APIs in the Java
 Specification
  *type.impl* - groups of tests which are tests for Harmony-specific
 implementation
 
  ** type.api and type.impl are also mutually exclusive.
 
 4) [Test Level] level.unit, level.integration, level.system,
 level.stress, etc. (Levels of Test refer to the increase in
complexity
 as moving through test cycle. )
 ** A test may be in more than one of the groups.
 ** In fact, some tests such as System tests are the verification
of
 the entire system.  Maybe we'll put them into a separate project.
e.g.,
 harmony/enhanced/SVT (System Verification Test).

 5) [Environment] env.display, env.headless
   To distinguish AWT and Swing tests which need a display to run, and
 those which don't, as Mark proposed [1].


Will display option be passed manually as an argument to TestNG, or
detected automatically when running test?

I think it should be passed as an argument to TestNG.
Ant script can be setup to detect the state by default, with the ability
to override it from the command line.


Regards,
Alexey.


Regards,
 Alexey.

 [1]

http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200608.mb
 ox/[EMAIL PROTECTED]

 
 If we want to run all the unit test for APIs on windows, we may use
 TestNG groups to select the tests:
 groups
 run
 include name=os.any /
 include name=type.api /
 include name=os.win.IA32 /
 exclude name=state.broken /
 exclude name=state.broken.win.IA32 /
 /run
 /groups
 
 
 Well, I think our most of existing tests are in the groups of
 {os.any,
 type.api, level.unit}, and I have asked TestNG to add a new
option
 -groups for its JUnitConverter which allow us to specify the test
 groups when migrate from JUnit test to TestNG test.
 
 Thanks for reading so far, and I will highly appreciate your
comments
 or
 suggestion.  ;-)
 
 --
 Richard Liang
 China Software Development Lab, IBM
 
 
 

-
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail:
[EMAIL PROTECTED]

 --
 Alexey A. Ivanov
 Intel Middleware Product Division

 -
 Terms of use : http://incubator.apache.org/harmony/mailing.html
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail:
[EMAIL PROTECTED]




--
Andrew Zhang
China Software Development Lab, IBM

--
Alexey A. Ivanov
Intel Middleware Product Division

-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Re: [classlib][TestNG] groups of Harmony test

[Richard Liang]

Richard Liang wrote:

Hello All,

Now let's talk about the TestNG groups. I have read the related 
threads which posted by George, Vladimir Ivanov and Alexei Zakharov. 
All of them are good discussion about TestNG groups.


IMHO, we may define Harmony test groups according the following 4 
dimensions:


1) [Platform] os.any, os.platform id
*os.any* - group of tests which pass on any platform. IMHO, most of 
our tests should be in this group.
*os.platform id* - group of tests which are designed for one 
specific platform. A test may be in more than one of the groups. e.g., 
@Test(groups={os.win.IA32, os.linux.IA32})


   ** os.any and os.platform id are mutually exclusive, that is, 
tests in os.any group should not be in os.win.IA32.


2) [Test state] state.broken, state.broken.platform id
*state.broken* - group of tests which fail on every platform, because 
of bugs of tests or implementation. We need to fix the bugs of tests 
or implementation to make them pass.
*state.broken.platform id* - groups of test which only fail on one 
specific platform. A test may be in more than one of the groups. e.g., 
@Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})


**state.broken.platform id group may be used as a convenient way 
to indicate that a test is platform-specific. e.g., If we support 10 
platforms, and one test are designed for 9 platforms except for MacOS, 
instead of list 9 os.platform id, we can just use state.broken.MacOS


3) [Test type] type.api, type.impl
*type.api* - group of tests which are tests for APIs in the Java 
Specification
*type.impl* - groups of tests which are tests for Harmony-specific 
implementation


** type.api and type.impl are also mutually exclusive.

4) [Test Level] level.unit, level.integration, level.system, 
level.stress, etc. (Levels of Test refer to the increase in complexity 
as moving through test cycle. )

   ** A test may be in more than one of the groups.
   ** In fact, some tests such as System tests are the verification of 
the entire system.  Maybe we'll put them into a separate project. 
e.g., harmony/enhanced/SVT (System Verification Test).


If we want to run all the unit test for APIs on windows, we may use 
TestNG groups to select the tests:

   groups
   run
   include name=os.any /
   include name=type.api /
   include name=os.win.IA32 /
   exclude name=state.broken /
   exclude name=state.broken.win.IA32 /
   /run
   /groups


Hello All,

I'm sorry. It seems that the example does not work. I will try to figure 
another example soon. ;-)


Best regards,
Richard


Well, I think our most of existing tests are in the groups of 
{os.any, type.api, level.unit}, and I have asked TestNG to add a 
new option -groups for its JUnitConverter which allow us to specify 
the test groups when migrate from JUnit test to TestNG test.


Thanks for reading so far, and I will highly appreciate your comments 
or suggestion.  ;-)




--
Richard Liang
China Software Development Lab, IBM 




-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

[classlib][TestNG] groups of Harmony test

[Richard Liang]

Hello All,

Now let's talk about the TestNG groups. I have read the related threads 
which posted by George, Vladimir Ivanov and Alexei Zakharov. All of them 
are good discussion about TestNG groups.


IMHO, we may define Harmony test groups according the following 4 
dimensions:


1) [Platform] os.any, os.platform id
*os.any* - group of tests which pass on any platform. IMHO, most of our 
tests should be in this group.
*os.platform id* - group of tests which are designed for one specific 
platform. A test may be in more than one of the groups. e.g., 
@Test(groups={os.win.IA32, os.linux.IA32})


   ** os.any and os.platform id are mutually exclusive, that is, 
tests in os.any group should not be in os.win.IA32.


2) [Test state] state.broken, state.broken.platform id
*state.broken* - group of tests which fail on every platform, because 
of bugs of tests or implementation. We need to fix the bugs of tests or 
implementation to make them pass.
*state.broken.platform id* - groups of test which only fail on one 
specific platform. A test may be in more than one of the groups. e.g., 
@Test(groups={state.broken.linux.IA32, os.broken.linux.IA64})


**state.broken.platform id group may be used as a convenient way 
to indicate that a test is platform-specific. e.g., If we support 10 
platforms, and one test are designed for 9 platforms except for MacOS, 
instead of list 9 os.platform id, we can just use state.broken.MacOS


3) [Test type] type.api, type.impl
*type.api* - group of tests which are tests for APIs in the Java 
Specification
*type.impl* - groups of tests which are tests for Harmony-specific 
implementation


** type.api and type.impl are also mutually exclusive.

4) [Test Level] level.unit, level.integration, level.system, 
level.stress, etc. (Levels of Test refer to the increase in complexity 
as moving through test cycle. )

   ** A test may be in more than one of the groups.
   ** In fact, some tests such as System tests are the verification of 
the entire system.  Maybe we'll put them into a separate project. e.g., 
harmony/enhanced/SVT (System Verification Test).


If we want to run all the unit test for APIs on windows, we may use 
TestNG groups to select the tests:

   groups
   run
   include name=os.any /
   include name=type.api /
   include name=os.win.IA32 /
   exclude name=state.broken /
   exclude name=state.broken.win.IA32 /
   /run
   /groups


Well, I think our most of existing tests are in the groups of {os.any, 
type.api, level.unit}, and I have asked TestNG to add a new option 
-groups for its JUnitConverter which allow us to specify the test 
groups when migrate from JUnit test to TestNG test.


Thanks for reading so far, and I will highly appreciate your comments or 
suggestion.  ;-)


--
Richard Liang
China Software Development Lab, IBM 




-
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]