Re: [classlib] do we need a global exclude list?
Geir Magnusson Jr wrote: Stepan Mishura wrote: Did anybody think of creating a 'global' (i.e. shared by all modules) exclude list or every module will have its own exclude list? Or Harmony tests will always pass and we don't need it at all :-) That would be the goal :) I see at least the following benefits of creating 'global' exclude list: all know issues are kept in one well known place (they don't spread between several private lists) That's true, but I always imagined that people would be working in the modules anyway, so there isn't much gain. Modules should be as self-sufficient as we can make them, so devolving tests and their exclude lists into a module makes sense to me too. They can be linked into a global view quite easily (read on...) also it is easier to create an exclude list for a target platform, for example, linux.exclude or win.exclude. Yes, that could be. Interesting idea. The mechanism that George contributed in HARMONY-57 uses a decorator to implement an exclusion list on regular JUnit tests. (I happen to know that George is off enjoying himself for the next few days, so I hope he doesn't mind me describing it here.) The exclusion list is implemented as a (declarative) XML file read when the tests run -- in HARMONY-57 you can see one in Harmony_Tests/src/test/resources/config/jcltest-excludes.xml Picking an entry at random from there: ... hy:type id=tests.api.java.io.FileTest hy:exclude id=test_Constructor_String_String_112270 shouldfix=true hy:reasonUndiagnosed failure/hy:reason /hy:exclude hy:exclude id=test_Constructor_File_String_112270 shouldfix=true hy:reasonUndiagnosed failure/hy:reason /hy:exclude hy:exclude id=test_deleteOnExit shouldfix=true hy:reasonNeeds investigation.../hy:reason /hy:exclude /hy:type ... You can see how it works -- some tests are excluded because they fail and should be fixed, others (not shown here) can be excluded because they don't make sense on a particular platform, VM, etc. Applying a style sheet makes it easy to read the exclusions list in glorious technicolor, either in an individual module or as a combined global view. Take a look at the incoming contribution and see if it fits your needs. Regards, Tim However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? geir Thoughts? Thanks, Stepan Mishura Intel Middleware Products Division -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
Re: [classlib] do we need a global exclude list?
On 2/16/06, Geir Magnusson Jr geir at pobox.com wrote: Stepan Mishura wrote: Did anybody think of creating a 'global' (i.e. shared by all modules) exclude list or every module will have its own exclude list? Or Harmony tests will always pass and we don't need it at all :-) That would be the goal :) I see at least the following benefits of creating 'global' exclude list: all know issues are kept in one well known place (they don't spread between several private lists) That's true, but I always imagined that people would be working in the modules anyway, so there isn't much gain. also it is easier to create an exclude list for a target platform, for example, linux.exclude or win.exclude. Yes, that could be. Interesting idea. However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? I think we won't restrict freedom too much if we ask update only one list. And I believe it won't be updated too often that may cause frequent conflict resolution. Also the global exclude list may be useful to understand impact of a bug, for example, we may organize its entries in the following way: !-- HARMONY-NNN, summary : a bug in luni module -- exclude name=java/lang/SomeTest.java/ exclude name=java/security/SomeTest.java/ exclude name=java/nio/SomeTest.java/ ... exclude name=java/text/SomeTest.java/ !-- HARMONY-ZZZ, summary : a bug in auth module -- exclude name=javax/security/auth/SomeTest.java/ Thanks, Stepan geir Thoughts? Thanks, Stepan Mishura Intel Middleware Products Division -- Thanks, Stepan Mishura Intel Middleware Products Division
Re: [classlib] do we need a global exclude list?
On 2/16/06, Tim Ellison t.p.ellison at gmail.com wrote: Applying a style sheet makes it easy to read the exclusions list in glorious technicolor, either in an individual module or as a combined global view. I see. And global view will be very helpful Thanks, Stepan On 2/16/06, Tim Ellison t.p.ellison at gmail.com wrote: Geir Magnusson Jr wrote: Stepan Mishura wrote: Did anybody think of creating a 'global' (i.e. shared by all modules) exclude list or every module will have its own exclude list? Or Harmony tests will always pass and we don't need it at all :-) That would be the goal :) I see at least the following benefits of creating 'global' exclude list: all know issues are kept in one well known place (they don't spread between several private lists) That's true, but I always imagined that people would be working in the modules anyway, so there isn't much gain. Modules should be as self-sufficient as we can make them, so devolving tests and their exclude lists into a module makes sense to me too. They can be linked into a global view quite easily (read on...) also it is easier to create an exclude list for a target platform, for example, linux.exclude or win.exclude. Yes, that could be. Interesting idea. The mechanism that George contributed in HARMONY-57 uses a decorator to implement an exclusion list on regular JUnit tests. (I happen to know that George is off enjoying himself for the next few days, so I hope he doesn't mind me describing it here.) The exclusion list is implemented as a (declarative) XML file read when the tests run -- in HARMONY-57 you can see one in Harmony_Tests/src/test/resources/config/jcltest-excludes.xml Picking an entry at random from there: ... hy:type id=tests.api.java.io.FileTest hy:exclude id=test_Constructor_String_String_112270 shouldfix=true hy:reasonUndiagnosed failure/hy:reason /hy:exclude hy:exclude id=test_Constructor_File_String_112270 shouldfix=true hy:reasonUndiagnosed failure/hy:reason /hy:exclude hy:exclude id=test_deleteOnExit shouldfix=true hy:reasonNeeds investigation.../hy:reason /hy:exclude /hy:type ... You can see how it works -- some tests are excluded because they fail and should be fixed, others (not shown here) can be excluded because they don't make sense on a particular platform, VM, etc. Applying a style sheet makes it easy to read the exclusions list in glorious technicolor, either in an individual module or as a combined global view. Take a look at the incoming contribution and see if it fits your needs. Regards, Tim However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? geir Thoughts? Thanks, Stepan Mishura Intel Middleware Products Division -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK. -- Thanks, Stepan Mishura Intel Middleware Products Division
Re: [classlib] do we need a global exclude list?
+1 Richard Liang China Software Development Lab, IBM Stepan Mishura wrote: Did anybody think of creating a 'global' (i.e. shared by all modules) exclude list or every module will have its own exclude list? Or Harmony tests will always pass and we don't need it at all :-) I see at least the following benefits of creating 'global' exclude list: all know issues are kept in one well known place (they don't spread between several private lists), also it is easier to create an exclude list for a target platform, for example, linux.exclude or win.exclude. Thoughts? Thanks, Stepan Mishura Intel Middleware Products Division
Re: [classlib] do we need a global exclude list?
On 2/16/06, Tim Ellison [EMAIL PROTECTED] wrote: That's true, but I always imagined that people would be working in the modules anyway, so there isn't much gain. Modules should be as self-sufficient as we can make them, so devolving tests and their exclude lists into a module makes sense to me too. As far as tests in one module can fail because of bugs in other modules, I think the modules are not self-sufficient. Thanks, Mikhail
Re: [classlib] do we need a global exclude list?
Could there be a switch to over-ride the global exclude list entry/entries? This way a module by default would use a global exclude list and also have the freedom to ignore it and use the module exclude list. Geir Magnusson Jr wrote: However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? Stepan Mishura wrote: Did anybody think of creating a 'global' (i.e. shared by all modules) exclude list or every module will have its own exclude list? Or Harmony tests will always pass and we don't need it at all :-) That would be the goal :) I see at least the following benefits of creating 'global' exclude list: all know issues are kept in one well known place (they don't spread between several private lists) That's true, but I always imagined that people would be working in the modules anyway, so there isn't much gain. also it is easier to create an exclude list for a target platform, for example, linux.exclude or win.exclude. Yes, that could be. Interesting idea. However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? geir Thoughts? Thanks, Stepan Mishura Intel Middleware Products Division -- Karan Singh
Re: [classlib] do we need a global exclude list?
Geir Magnusson Jr wrote: Tim Ellison wrote: The exclusion list is implemented as a (declarative) XML file read when the tests run -- in HARMONY-57 you can see one in Harmony_Tests/src/test/resources/config/jcltest-excludes.xml Picking an entry at random from there: ... hy:type id=tests.api.java.io.FileTest hy:exclude id=test_Constructor_String_String_112270 shouldfix=true hy:reasonUndiagnosed failure/hy:reason /hy:exclude hy:exclude id=test_Constructor_File_String_112270 shouldfix=true hy:reasonUndiagnosed failure/hy:reason /hy:exclude hy:exclude id=test_deleteOnExit shouldfix=true hy:reasonNeeds investigation.../hy:reason /hy:exclude /hy:type ... You can see how it works -- some tests are excluded because they fail and should be fixed, others (not shown here) can be excluded because they don't make sense on a particular platform, VM, etc. Applying a style sheet makes it easy to read the exclusions list in glorious technicolor, either in an individual module or as a combined global view. That's cute. And if these were local to each module, in a well-known place, we still could present a global picture with a top level thingy that generates a list... Yep, probably putting the exclusion list into src/test/resources in each module, and like you say, easy to combine. We could also add a platform element to make exclusions platform specific... Already there ;-) xsd:element name=exclude xsd:complexType xsd:sequence xsd:element ref=reason minOccurs=0 maxOccurs=1/ /xsd:sequence xsd:attribute name=id type=idtype default=all / xsd:attribute name=platform type=platformlisttype default=all / xsd:attribute name=shouldfix type=booleanstringtype default=true / /xsd:complexType /xsd:element where xsd:simpleType name=platformstringtype xsd:restriction base=xsd:string xsd:enumeration value=win.IA32/ xsd:enumeration value=linux.IA32/ xsd:enumeration value=all/ /xsd:restriction /xsd:simpleType but of course we can do whatever we want. Regards, Tim -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
Re: [classlib] do we need a global exclude list?
karan malhi wrote: Could there be a switch to over-ride the global exclude list entry/entries? This way a module by default would use a global exclude list and also have the freedom to ignore it and use the module exclude list. I don't know what having two exclude lists would mean - a test should be excluded for a specific reason, and therefore I can't see why one would be in the global list, but not in the module list (and vice versa...) geir Geir Magnusson Jr wrote: However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? Stepan Mishura wrote: Did anybody think of creating a 'global' (i.e. shared by all modules) exclude list or every module will have its own exclude list? Or Harmony tests will always pass and we don't need it at all :-) That would be the goal :) I see at least the following benefits of creating 'global' exclude list: all know issues are kept in one well known place (they don't spread between several private lists) That's true, but I always imagined that people would be working in the modules anyway, so there isn't much gain. also it is easier to create an exclude list for a target platform, for example, linux.exclude or win.exclude. Yes, that could be. Interesting idea. However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? geir Thoughts? Thanks, Stepan Mishura Intel Middleware Products Division
Re: [classlib] do we need a global exclude list?
karan malhi wrote: Could there be a switch to over-ride the global exclude list entry/entries? This way a module by default would use a global exclude list and also have the freedom to ignore it and use the module exclude list. In the incoming HARMONY-57 contribution the list is 'global', but you can override the exclusion list by specifying a replacement's location in a java property. In future, I see the global list as simply a dynamic amalgam of the module lists in the target. If we put the exclude lists in a standard location, (in the repo and in the tests jar file(s)) then they can be found by the test framework. Regards, Tim Geir Magnusson Jr wrote: However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? Stepan Mishura wrote: Did anybody think of creating a 'global' (i.e. shared by all modules) exclude list or every module will have its own exclude list? Or Harmony tests will always pass and we don't need it at all :-) That would be the goal :) I see at least the following benefits of creating 'global' exclude list: all know issues are kept in one well known place (they don't spread between several private lists) That's true, but I always imagined that people would be working in the modules anyway, so there isn't much gain. also it is easier to create an exclude list for a target platform, for example, linux.exclude or win.exclude. Yes, that could be. Interesting idea. However, I did imagine that we'd give the modules a bit of freedom and independence for testing - a global exclude list might impact that? geir Thoughts? Thanks, Stepan Mishura Intel Middleware Products Division -- Tim Ellison ([EMAIL PROTECTED]) IBM Java technology centre, UK.
[classlib] do we need a global exclude list?
Did anybody think of creating a 'global' (i.e. shared by all modules) exclude list or every module will have its own exclude list? Or Harmony tests will always pass and we don't need it at all :-) I see at least the following benefits of creating 'global' exclude list: all know issues are kept in one well known place (they don't spread between several private lists), also it is easier to create an exclude list for a target platform, for example, linux.exclude or win.exclude. Thoughts? Thanks, Stepan Mishura Intel Middleware Products Division
