On 7/18/06, George Harley <[EMAIL PROTECTED]> wrote:
>
> Oliver Deakin wrote:
> > George Harley wrote:
> >> <SNIP!>
> >>
> >> Here the annotation on MyTestClass applies to all of its test
methods.
> >>
> >> So what are the well-known TestNG groups that we could define
for use
> >> inside Harmony ? Here are some of my initial thoughts:
> >>
> >>
> >> * type.impl -- tests that are specific to Harmony
> >
> > So tests are implicitly API unless specified otherwise?
> >
> > I'm slightly confused by your definition of impl tests as "tests
that
> are
> > specific to Harmony". Does this mean that impl tests are only
> > those that test classes in org.apache.harmony packages?
> > I thought that impl was our way of saying "tests that need to go on
> > the bootclasspath".
> >
> > I think I just need a little clarification...
> >
>
> Hi Oliver,
>
> I was using the definition of implementation-specific tests that we
> currently have on the Harmony testing conventions web page. That is,
> implementation-specific tests are those that are dependent on some
> aspect of the Harmony implementation and would therefore not pass when
> run against the RI or other conforming implementations. It's
orthogonal
> to the classpath/bootclasspath issue.
>
>
> >> * state.broken.<platform id> -- tests bust on a specific platform
> >>
> >> * state.broken -- tests broken on every platform but we want to
> >> decide whether or not to run from our suite configuration
> >>
> >> * os.<platform id> -- tests that are to be run only on the
> >> specified platform (a test could be member of more than one of
these)
> >
> > And the defaults for these are an unbroken state and runs on any
> > platform.
> > That makes sense...
> >
> > Will the platform ids be organised in a similar way to the
platform ids
> > we've discussed before for organisation of native code [1]?
> >
>
> The actual string used to identify a particular platform can be
whatever
> we want it to be, just so long as we are consistent. So, yes, the ids
> mentioned in the referenced email would seem a good starting point. Do
> we need to include a 32-bit/64-bit identifier ?
>
>
> > So all tests are, by default, in an all-platforms (or shared) group.
> > If a test fails on all Windows platforms, it is marked with
> > state.broken.windows.
> > If a test fails on Windows but only on, say, amd hardware,
> > it is marked state.broken.windows.amd.
> >
>
> Yes. Agreed.
>
>
> > Then when you come to run tests on your windows amd machine,
> > you want to include all tests in the all-platform (shared) group,
> > os.windows and os.windows.amd, and exclude all tests in
> > the state.broken, state.broken.windows and state.broken.windows.amd
> > groups.
> >
> > Does this tally with what you were thinking?
> >
>
> Yes, that is the idea.
>
>
> >>
> >>
> >> What does everyone else think ? Does such a scheme sound
reasonable ?
> >
> > I think so - it seems to cover our current requirements. Thanks for
> > coming up with this!
> >
>
> Thanks, but I don't see it as final yet really. It would be great to
> prove the worth of this by doing a trial on one of the existing
modules,
> ideally something that contains tests that are platform-specific.
Hello George, how about doing a trial on NIO module?
So far as I know, there are several platform dependent tests in NIO
module.
:)
The assert statements are commented out in these tests, with "FIXME"
mark.
Furthurmore, I also find some platform dependent behaviours of
FileChannel.
If TestNG is applied on NIO, I will supplement new tests for
FileChannel and
fix the bug of source code.
What's your opnion? Any suggestions/comments?
Thanks!
Best regards,
> George
>
>
> > Regards,
> > Oliver
> >
> > [1]
> >
>
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200605.mbox/[EMAIL PROTECTED]
> >
> >
> >>
> >> Thanks for reading this far.
> >>
> >> Best regards,
> >> George
> >>
> >>
> >>
> >> George Harley wrote:
> >>> Hi,
> >>>
> >>> Just seen Tim's note on test support classes and it really
caught my
> >>> attention as I have been mulling over this issue for a little
while
> >>> now. I think that it is a good time for us to return to the
topic of
> >>> class library test layouts.
> >>>
> >>> The current proposal [1] sets out to segment our different
types of
> >>> test by placing them in different file locations. After looking at
> >>> the recent changes to the LUNI module tests (where the layout
> >>> guidelines were applied) I have a real concern that there are
> >>> serious problems with this approach. We have started down a
track of
> >>> just continually growing the number of test source folders as new
> >>> categories of test are identified and IMHO that is going to bring
> >>> complexity and maintenance issues with these tests.
> >>>
> >>> Consider the dimensions of tests that we have ...
> >>>
> >>> API
> >>> Harmony-specific
> >>> Platform-specific
> >>> Run on classpath
> >>> Run on bootclasspath
> >>> Behaves different between Harmony and RI
> >>> Stress
> >>> ...and so on...
> >>>
> >>>
> >>> If you weigh up all of the different possible permutations and
then
> >>> consider that the above list is highly likely to be extended as
> >>> things progress it is obvious that we are eventually heading for
> >>> large amounts of related test code scattered or possibly
duplicated
> >>> across numerous "hard wired" source directories. How
maintainable is
> >>> that going to be ?
> >>>
> >>> If we want to run different tests in different configurations then
> >>> IMHO we need to be thinking a whole lot smarter. We need to be
> >>> thinking about keeping tests for specific areas of functionality
> >>> together (thus easing maintenance); we need something quick and
> >>> simple to re-configure if necessary (pushing whole directories of
> >>> files around the place does not seem a particularly lightweight
> >>> approach); and something that is not going to potentially mess up
> >>> contributed patches when the file they patch is found to have been
> >>> recently pushed from source folder A to B.
> >>>
> >>> To connect into another recent thread, there have been some posts
> >>> lately about handling some test methods that fail on Harmony and
> >>> have meant that entire test case classes have been excluded
from our
> >>> test runs. I have also been noticing some API test methods that
pass
> >>> fine on Harmony but fail when run against the RI. Are the
different
> >>> behaviours down to errors in the Harmony implementation ? An error
> >>> in the RI implementation ? A bug in the RI Javadoc ? Only after
some
> >>> investigation has been carried out do we know for sure. That takes
> >>> time. What do we do with the test methods in the meantime ? Do we
> >>> push them round the file system into yet another new source
folder ?
> >>> IMHO we need a testing strategy that enables such "problem"
methods
> >>> to be tracked easily without disruption to the rest of the other
> tests.
> >>>
> >>> A couple of weeks ago I mentioned that the TestNG framework [2]
> >>> seemed like a reasonably good way of allowing us to both group
> >>> together different kinds of tests and permit the exclusion of
> >>> individual tests/groups of tests [3]. I would like to strongly
> >>> propose that we consider using TestNG as a means of providing the
> >>> different test configurations required by Harmony. Using a
> >>> combination of annotations and XML to capture the kinds of
> >>> sophisticated test configurations that people need, and that
allows
> >>> us to specify down to the individual method, has got to be more
> >>> scalable and flexible than where we are headed now.
> >>>
> >>> Thanks for reading this far.
> >>>
> >>> Best regards,
> >>> George
> >>>
> >>>
> >>> [1]
> >>>
>
http://incubator.apache.org/harmony/subcomponents/classlibrary/testing.html
> >>>
> >>> [2] http://testng.org
> >>> [3]
> >>>
>
http://mail-archives.apache.org/mod_mbox/incubator-harmony-dev/200606.mbox/[EMAIL PROTECTED]
