Alex Blewitt wrote: > On 08/07/06, Geir Magnusson Jr <[EMAIL PROTECTED]> wrote: >> >> So while I like the annotations, and expect we can use them effectively, >> I have an instinctive skepticism of annotations right now because in >> general (in general in Java), I'm not convinced we've used them enough >> to grok good design patterns. > > There's really no reason to get hung up on the annotations. TestNG > works just as well with JavaDoc source comments; annotations are only > another means to that end. (They're probably a better one for the > future, but it's just an implementation detail.)
You don't understand what I meant - one of the elements of this subject is that we can punt on how we're organizing tests in a directory layout, and use annotations for that purpose, and I'm worried about going fully to that extreme. > >> Now since I still haven't read the thread fully, I'm jumping to >> conclusions, taking it to the extreme, etc etc, but my thinking in >> writing the above is that if we bury everything about our test >> 'parameter space' in annotations, some of the visible organization we >> have now w/ on-disk layout becomes invisible, and the readable >> "summaries" of aspects of testing that we'd have in an XML metadata >> document (or whatever) also are hard because you need to scan the >> sources to find all instances of annotation "X". > > I'm hoping that this would be just as applicable to using JavaDoc > variants, and that the problem's not with annotations per se. Right. > > In either case, both are grokkable with tools -- either > annotation-savy readers or a JavaDoc tag processor, and it wouldn't be > hard to configure one of those to periodically scan the codebase to > generate reports. Furthermore, as long as the annotation X is well > defined, *you* don't have to scan it -- you leave it up to TestNG to > figure it out. Maybe. Tools help, but they have to be universal, lightweight and pretty transparent, IMO. Don't force people to boot Eclipse (or Netbeans or IDEA) to just figure out the general details of a test class... > > Actually, there's a very valid benefit for using TestNG markers (= > annotations/JavaDoc) for grouping tests; the directory structure is a > tree, whereas the markers can form any slice of tests, and the sets > don't need to be strict subsets (with a tree, everything has to be a > strict subset of its parents). That means that it's possible to define > a marker IO to run all the IO tests, or a marker Win32 to run all the > Win32 tests, and both of those will contain IO-specific Win32 tests. > You can't do that in a tree structure without duplicating content > somewhere along the line (e.g. /win/io or /io/win). Neither of these > scale well, and every time you add a new dimension, you're doubling > the structure of the directory, but merely adding a new marker with > TestNG. So if you wanted to have (say) boot classpath tests vs api > tests, then you'd ahve to have /api/win/io and /boot/win/io (or > various permutations as applicable). I understand this, which is why I think a general human-readable metadata system will help, preferably one that has the data in one place, rather than scattered throughout... of course, centralization has it's downsides too... it's not an easy problem :) > > Most of the directory-based arguments seem to be along the lines of > "/api/win/io is better! No, /win/io/api is better!". Just have an > 'api', 'win', 'io' TestNG marker, and then let TestNG figure out which > ones to run. You can then even get specific, and only run the Windows > IO API tests, if you really want -- but if you don't, you get the > benefit of being able to run all IO tests (both API and boot). Or a document that has groups/suites that ant then uses... > > There doesn't seem to be any benefit to having a strict tree-like > structure to the tests when it's possible to have a multi-dimensional > matrix of all possible combinations that's managed by the tool. Right. Clearly a "directory-only" solution won't ever work well, in the same way an annotation/marker based solution won't either (I'm guessing). I think we first have to figure out what we want to achieve *irrespective* of how it will be done, and then find the right tools/process/strategy to achieve that, or create them. This is important. Lets not let the directory-tail or the annotation-tail wag the testing dog. :D geir --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
