Hi Vladimir,
Cons:
1. More complex algorithms of grouping / selecting tests for execution: harness (being it Junit, TestNG or my own script) must understand "word" mark-up instead of just simple walking directories.
BTW, after brief browsing of the JUnit 4.0 documentation I didn't find any extra *grouping* capabilities (comparing to JUnit 3.8.1). Only old TestSuites. Does anybody have more information about this? ...
3. If test characteristics changed we should fix test file, instead of just moving file, that is easier to track then changes.
I don't think test characteristics (do you mean groups?) can be changed by itself. I mean Linux -specific test can't immediately become Windows-specific, it needs someone to update the code. This someone can also update the metadata that comes along with the code, it is easy. The only exception I can think about is when some test suddenly starts failing. But in this case it is also a good idea to go and look at the source code of failing tests. And again this will be the place there the metadata lives – it is easy so change it at once.
Other pros/cons?
4. If someone needs the number of failing tests or platform-specific tests he will need to run the test harness to collect this information since the metadata is scattered about the source tree.
I think using metadata in tests vs. physical test location wins. thanks, Vladimir
Well, currently I think the metadata approach better suits the needs of our project. Regards, 2006/7/24, Vladimir Ivanov <[EMAIL PROTECTED]>:
Let me add a couple of words. The problem: in Harmony unit test suite we need to mark-up somehow and manage: - Platform-specific unit tests - Harmony implementation test vs. API tests - Broken buggy tests - Tests failing due to bug in implementation - [Potentially in future] type of test – unit, functional, stress, reliability, etc. - [Potentially in future] regression test - Each of the listed above can be: - OS-specific - Processor architecture-specific Metadata vs. directory layout pros: 1. Directory structure simplification: we have all tests for the given class in one directory, there is no need to have numerous Win/Linux/ia32/ia64/etc. subdirectories. 2. Lossless test exclusion – we do not exclude the whole TestCase, just small tests. 3. If we decide to keep all excluded tests in a separate TestCase *Fail* Test.java classes, then, having metadata, there is no need in moving tests from one file to another when we want to un-exclude tests (this is risky operation which can lead to loosing tests). 4. Easy to collect statistics – how many / which tests are: "broken buggy tests", Harmony specific, Win-specific, etc. 5. Easy to do massive changes if necessary, while no need to re-name Win/Linux/ia32/etc. directories or file or move files. 6. History of all test changes are stored along with the test file. Cons: 1. More complex algorithms of grouping / selecting tests for execution: harness (being it Junit, TestNG or my own script) must understand "word" mark-up instead of just simple walking directories. 2. People should learn allowed mark-up (list of groups) instead of just intuitively put tests into specific directory or TestCase. 3. If test characteristics changed we should fix test file, instead of just moving file, that is easier to track then changes. Other pros/cons? I think using metadata in tests vs. physical test location wins. thanks, Vladimir On 7/7/06, Tim Ellison <[EMAIL PROTECTED]> wrote: > > May I tactfully suggest that we get this back to a discussion of the > pros and cons of JUnit test suites and/or TestNG metadata vs. directory > layout. > > It sounds like we all want to resolve that problem asap. > > Regards, > Tim > > George Harley wrote: > > Mark Hindess wrote: > >> On 6 July 2006 at 18:05, George Harley > >> <[EMAIL PROTECTED]> wrote: > >> > >>> Mark Hindess wrote: > >>> > >>>> On 6 July 2006 at 12:55, George Harley < > [EMAIL PROTECTED]> > >>>> wrote: > >>>> > >>>>> Hi Mark, > >>>>> > >>>>> From what I can tell this JIRA hasn't really achieved much apart > >>>>> from pushing code around the repository and breaking at least one > >>>>> patch (HARMONY-755). > >>>>> > >>>> Well, obviously that wasn't my motivation! ;-) > >>>> > >>>> > >>> Hi Mark, > >>> > >>> No one was saying it was. BTW, good to hear you have some motivation > :-) > >>> > >>> > >>> > >>>> >From the description, it was clear (to me anyway) that the patch > >>>> was to > >>>> enable the use of platform-specific test code. While the directories > >>>> for the platform-specific test code are currently empty, I'm certain > >>>> Paulex plans to rectify this pretty soon. > >>>> > >>>> > >>> Creating empty directories is not the issue here. The patch also > >>> entailed moving a whole bunch of other files around the source tree > >>> for reasons that are currently being discussed in the dev list. > >>> > >>> > >>>> I think Paulex was correct to separate the process of allowing for > >>>> platform-specific tests (HARMONY-782) from any JIRA containing new > >>>> tests. > >>>> > >>> The "process" of allowing for new platform-specific tests is > >>> precisely what is being currently discussed on the dev-list in the > >>> referenced thread. > >>> > >> > >> I thought it was categorisation of tests in general. > >> > >> > > > > Hi Mark, > > > > Since "platform-specific" is one important category of test then > > discussion and agreement on the general topic is important. > > > > > >>>> The JIRA comment by Paulex mentioned that it would break two existing > >>>> JIRA issues - HARMONY-775 and HARMONY-767. I applied the former but > >>>> the > >>>> latter was already assigned to Tim and marked 'In Progress' so I > didn't > >>>> feel it was right to steal it. However I have made the trivial > change > >>>> to the patch metadata to fix the HARMONY-767 patch. > >>>> > >>>> Unfortunately it didn't mention the HARMONY-775 patch, otherwise I > >>>> might > >>>> have checked with you first. > >>>> > >>> It was HARMONY-755. I know, now I'm just being picky :-) > >>> > >> > >> Yes. :-) > >> > >> > >>>>> It would be great if you or Paulex (and everyone in fact) could > >>>>> comment in the "[classlib] Testing conventions - a proposal" thread > >>>>> [1] about this. > >>>>> > >>>> Certainly - though this seems to me to be orthogonal to the purpose > of > >>>> the HARMONY-782 patch. > >>>> > >>> The summary of HARMONY-782 is "Relayout NIO test cases to platform > >>> dependent". That is orthogonal to the dev-list discussion on proposed > >>> test layout ??? Are you serious ?????? > >>> > >> > >> Ok so maybe not orthogonal but the JIRA (regardless of the exact title) > >> was an enabler to allow additional platform-specific tests to be added. > >> And adding new tests is something that is independent of the need to > >> restructure. Or are you saying we shouldn't create any more tests (or > >> fix existing tests) until the restructuring issue is decided? > >> > >> > > > > If adding new platform-specific tests is "independent of the need to > > restructure" then why did you restructure the NIO tests ? > > > > > > No, I am not saying that we shouldn't create any more tests. No, I am > > not saying that we should stop fixing existing ones. This is not a > > restructuring issue. If anything, this is an anti-restructuring issue. > > This is about pausing to consider a different approach to the existing > > proposal for how we manage our tests. It deserves to be considered as it > > has the potential to save us all a lot of time and effort pushing files > > around. > > > >> While I see the importance of the restructuring I'm also keen not to > >> prevent the problematic nio tests to be fixed. > >> > >> > > > > Ditto. But what is the urgency here ? > >> Are you suggesting that applying the JIRA made the state of the tests > >> any worse than it was before? (I even made an effort to ensure that > the > >> change was made in a way that was more consistent with the current > state > >> of another module - to make it easier to programmatically fix them > later > >> when the test structure issue is resolved.) > >> > >> Regards, > >> Mark. > >> > >> > > > > IMHO this is not really about just HARMONY 782 and I would be genuinely > > upset if the impression was that I was getting at you or Paulex because > > it's not true. This is about asking you, Paulex and everyone to think > > about what our tests structure is going to look like eventually, how > > much effort is going to be required to maintain its labyrinth layout, > > the amount of overhead that is going to mean for our infrastructure (Ant > > scripts, IDE metadata files etc) and whether or not we can do better. > > > > > > Best regards, > > George > > > >> > >> --------------------------------------------------------------------- > >> Terms of use : http://incubator.apache.org/harmony/mailing.html > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: [EMAIL PROTECTED] > >> > >> > >> > > > > > > --------------------------------------------------------------------- > > Terms of use : http://incubator.apache.org/harmony/mailing.html > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > -- > > Tim Ellison ([EMAIL PROTECTED]) > IBM Java technology centre, UK. > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
-- Alexei Zakharov, Intel Middleware Product Division --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
