how about 'specific'? impl seems to be not very informative.
I have a concern abou proposed package naming guidelines:
package name
org.apache.harmony.security.tests.org.apache.harmony.security
is not much better then 1000-character long test name.
Thanks,
Mikhail
2006/4/26, Paulex Yang <[EMAIL PROTECTED]>:
> Oliver Deakin wrote:
> > George Harley wrote:
> >> Mikhail Loenko wrote:
> >>> Hello
> >>>
> >>> I'd like to bring this thread back.
> >>>
> >>> Number of tests is growing and it is time to put them in order.
> >>>
> >>> So far we may have:
> >>>
> >>> 1) implementation-specific tests that designed to be run from
> >>> bootclasspath
> >>> 2) implementation-specific tests that might be run from classpath
> >>> 3) implementation-specific tests that designed to be run from classpath
> >>> 4) implementation-independent tests that designed to be run from
> >>> bootclasspath
> >>> 5) implementation-independent tests that might be run from classpath
> >>> 6) implementation-independent tests that designed to be run from
> >>> classpath
> >>>
> >>> Also we seem to have the following packages, where the tests are:
> >>>
> >>> 1) the same package as implementation
> >>> 2) org.apache.harmony.tests.[the same package as implementation]
> >>> 3) tests.api.[the same package as implementation]
> >>>
> >>> I suggest that we work out step-by-step solution as we could not reach
> >>> an agreement for the general universal one
> >>>
> >>> So for the first step I suggest that we separate i-independent tests
> >>> that must or may be run from classpath
> >>>
> >>> I suggest that we put them into package
> >>> tests.module.compatible.[package of implementation being tested]
> >>>
> >>> Comments?
> >>>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>
> >> Hi Mikhail,
> >>
> >> I've just started working through the modules to merge test packages
> >> "tests.api.[same package as implementation]" and "tests.api.[same
> >> package as implementation]" into one package space. Using the class
> >> library package naming guidelines from off the web site [1], all of
> >> the tests for the text module have been consolidated under
> >> org.apache.harmony.text.tests.[package under test].
> >>
> >> Of course, the text module has only "implementation-independent tests
> >> that designed to be run from classpath". For modules that have got
> >> implementation-specific tests then I suppose we could use something
> >> like "org.apache.harmony.[module].tests.impl.[package under test]" or
> >> "org.apache.harmony.[module].tests.internal.[package under test]"
> >> etc. I've got no preference.
> >
> > I think impl is preferable over internal here, as we already use
> > internal in our implementation package names to indicate classes
> > totally internal to that bundle. To also use internal to label tests
> > that are implementation specific may cause confusion.
> >
> +1 from me.
> > Regards,
> > Oliver
> >
> >>
> >> Best regards,
> >> George
> >>
> >>
> >> [1]
> >> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
> >>
> >>
> >>
> >>>
> >>> 2006/3/24, George Harley <[EMAIL PROTECTED]>:
> >>>
> >>>> Geir Magnusson Jr wrote:
> >>>>
> >>>>> Leo Simons wrote:
> >>>>>
> >>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr wrote:
> >>>>>>
> >>>>>>> Leo Simons wrote:
> >>>>>>>
> >>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr wrote:
> >>>>>>>>
> >>>>>>>>> Pulling out of the various threads where we have been discussing,
> >>>>>>>>> can we agree on the problem :
> >>>>>>>>>
> >>>>>>>>> We have unique problems compared to other Java projects
> >>>>>>>>> because we
> >>>>>>>>> need to find a way to reliably test the things that are commonly
> >>>>>>>>> expected to be a solid point of reference - namely the core class
> >>>>>>>>> library.
> >>>>>>>>>
> >>>>>>>>> Further, we've been implicitly doing "integration testing"
> >>>>>>>>> because
> >>>>>>>>> - so far - the only way we've been testing our code has been 'in
> >>>>>>>>> situ' in the VM - not in an isolated test harness. To me, this
> >>>>>>>>> turns it into an integration test.
> >>>>>>>>>
> >>>>>>>>> Sure, we're using JUnit, but because of the fact we are
> >>>>>>>>> implmenting core java.* APIs, we aren't testing with a framework
> >>>>>>>>> that has been independently tested for correctness, like we would
> >>>>>>>>> when testing any other code.
> >>>>>>>>>
> >>>>>>>>> I hope I got that idea across - I believe that we have to go
> >>>>>>>>> beyond normal testing approaches because we don't have a normal
> >>>>>>>>> situation.
> >>>>>>>>>
> >>>>>>>> Where we define 'normal situation' as "running a test framework on
> >>>>>>>> top of
> >>>>>>>> the sun jdk and expecting any bugs to not be in that jdk". There's
> >>>>>>>> plenty
> >>>>>>>> of projects out there that have to test things without having
> >>>>>>>> such a
> >>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is
> >>>>>>>> just as
> >>>>>>>> hard as this problem we have here :-)
> >>>>>>>>
> >>>>>>> Is it the same? We need to have a running JVM+classlibarary to
> >>>>>>> test
> >>>>>>> the classlibrary code.
> >>>>>>>
> >>>>>> Well you need a working C compiler and standard C library to
> >>>>>> compile the
> >>>>>> compiler so you can compile make so you can build bash so you can
> >>>>>> run
> >>>>>> perl (which uses the standard C library functions all over the
> >>>>>> place of
> >>>>>> course) so you can run the standard C library tests so that you know
> >>>>>> that
> >>>>>> the library you used when compiling the compiler were correct so
> >>>>>> you can
> >>>>>> run the compiler tests. I don't think they actually do things that
> >>>>>> way, but
> >>>>>> it seems like basically the same problem. Having a virtual
> >>>>>> machine just
> >>>>>> makes it easier since you still assume "the native world" as a
> >>>>>> baseline,
> >>>>>> which is a lot more than "the hardware".
> >>>>>>
> >>>>> There's a difference. You can use a completely separate toolchain to
> >>>>> build, test and verify the output of the C compiler.
> >>>>>
> >>>>> In our case, we are using the thing we are testing to test itself.
> >>>>> There is no "known good" element possible right now.
> >>>>>
> >>>>> We use the classlibrary we are trying to test to execute the test
> >>>>> framework that tests the classlibrary that is running it.
> >>>>>
> >>>>> The tool is testing itself.
> >>>>>
> >>>>>
> >>>>>
> >>>>>>>>> So I think there are three things we want to do (adopting the
> >>>>>>>>> terminology that came from the discussion with Tim and Leo ) :
> >>>>>>>>>
> >>>>>>>>> 1) implementation tests
> >>>>>>>>> 2) spec/API tests (I'll bundle together)
> >>>>>>>>> 3) integration/functional tests
> >>>>>>>>>
> >>>>>>>>> I believe that for #1, the issues related to being on the
> >>>>>>>>> bootclasspath don't matter, because we aren't testing that aspect
> >>>>>>>>> of the classes (which is how they behave integrated w/ the VM and
> >>>>>>>>> security system) but rather the basic internal functioning.
> >>>>>>>>>
> >>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to
> >>>>>>>>> hear
> >>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it isn't an
> >>>>>>>>> issue :)
> >>>>>>>>>
> >>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via
> >>>>>>>>> the
> >>>>>>>>> standard same-package technique (i.e. testing a.b.C w/ a.b.CTest)
> >>>>>>>>> but we'll run into a tangle of classloader problems, I suspect,
> >>>>>>>>> becuase we want to be testing java.* code in a system that
> >>>>>>>>> already
> >>>>>>>>> has java.* code. Can anyone see a way we can do this - test the
> >>>>>>>>> classlibrary from the integration point of view - using some test
> >>>>>>>>> harness + any known-good JRE, like Sun's or IBM's?
> >>>>>>>>>
> >>>>>>>> Ew, that won't work in the end since we should assume our own JRE
> >>>>>>>> is going
> >>>>>>>> to be "known-better" :-). But it might be a nice way to
> >>>>>>>> "bootstrap"
> >>>>>>>> (eg
> >>>>>>>> we test with an external JRE until we satisfy the tests and
> >>>>>>>> then we
> >>>>>>>> switch
> >>>>>>>> to testing with an earlier build).
> >>>>>>>>
> >>>>>>> Lets be clear - even using our own "earlier build" doesn't solve
> >>>>>>> the
> >>>>>>> problem I'm describing, because as it stands now, we don't use
> >>>>>>> "earlier build" classes to test with - we use the code we want to
> >>>>>>> test as the clsaslibrary for the JRE that's running the test
> >>>>>>> framework.
> >>>>>>>
> >>>>>>> The classes that we are testing are also the classes used by the
> >>>>>>> testing framework. IOW, any of the java.* classes that JUnit
> >>>>>>> itself
> >>>>>>> needs (ex. java.util.HashMap) are exactly the same implementation
> >>>>>>> that it's testing.
> >>>>>>>
> >>>>>>> That's why I think it's subtly different than a "bootstrap and use
> >>>>>>> version - 1 to test" problem. See what I mean?
> >>>>>>>
> >>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is usable
> >>>>>> for the
> >>>>>> kind of test you're describing. At some point, fundamentally, you
> >>>>>> either trust
> >>>>>> something external (whether its the sun jdk or the intel compiler
> >>>>>> designers,
> >>>>>> at some point you do draw a line) or you find a way to bootstrap.
> >>>>>>
> >>>>> Well, we do trust the Sun JDK.
> >>>>>
> >>>>>
> >>>>>>> I'm very open to the idea that I'm missing something here, but I'd
> >>>>>>> like to know that you see the issue - that when we test, we have
> >>>>>>>
> >>>>>>> VM + "classlib to be tested" + JUnit + testcases
> >>>>>>>
> >>>>>>> where the testcases are testing the classlib the VM is running
> >>>>>>> JUnit
> >>>>>>> with.
> >>>>>>>
> >>>>>>> There never is isolation of the code being tested :
> >>>>>>>
> >>>>>>> VM + "known good classlib" + Junit + testcases
> >>>>>>>
> >>>>>>> unless we have some framework where
> >>>>>>>
> >>>>>>> VM + "known good classlib" + JUnit
> >>>>>>> + framework("classlib to be tested")
> >>>>>>> + testcases
> >>>>>>>
> >>>>>>> and it's that notion of "framework()" that I'm advocating we
> >>>>>>> explore.
> >>>>>>>
> >>>>>> I'm all for exploring it, I just fundamentally don't buy into the
> >>>>>> "known
> >>>>>> good" bit. What happens when the 'classlib to be tested' is 'known
> >>>>>> better' than the 'known good' one? How do you define "known"? How
> >>>>>> do you
> >>>>>> define "good"?
> >>>>>>
> >>>>> Known? Passed some set of tests. So it could be the Sun JDK for the
> >>>>> VM + "known good" part.
> >>>>>
> >>>>> I think you intuitively understand this. When you find a bug in code
> >>>>> you are testing, you first assume it's your code, not the framework,
> >>>>> right? In our case, our framework is actually the code we are
> >>>>> testing, so we have a bit of a logical conundrum.
> >>>>>
> >>>>>
> >>>> Hi Geir,
> >>>>
> >>>> The number of Harmony public API classes that get loaded just to
> >>>> run the
> >>>> JUnit harness is a little over 200. The majority of these are out of
> >>>> LUNI with a very low number coming from each of Security, NIO, Archive
> >>>> and Text.
> >>>>
> >>>> Sure there is a circular dependency between what we are building
> >>>> and the
> >>>> framework we are using to test it but it appears to touch on only a
> >>>> relatively small part of Harmony....IMHO.
> >>>>
> >>>> Best regards,
> >>>> George
> >>>>
> >>>>
> >>>>
> >>>>>>>> Further ideas...
> >>>>>>>>
> >>>>>>>> -> look at how the native world does testing
> >>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and
> >>>>>>>> it is
> >>>>>>>> certainly
> >>>>>>>> "messy")
> >>>>>>>> -> emulate that
> >>>>>>>>
> >>>>>>>> -> build a bigger, better specification test
> >>>>>>>> -> and somehow "prove" it is "good enough"
> >>>>>>>>
> >>>>>>>> -> build a bigger, better integration test
> >>>>>>>> -> and somehow "prove" it is "good enough"
> >>>>>>>>
> >>>>>>>> I'll admit my primary interest is the last one...
> >>>>>>>>
> >>>>>>> The problem I see with the last one is that the "parameter
> >>>>>>> space" is
> >>>>>>> *huge*.
> >>>>>>>
> >>>>>> Yeah, that's one of the things that makes it interesting.
> >>>>>> Fortunately
> >>>>>> open source does have many monkeys...
> >>>>>>
> >>>>>>
> >>>>>>> I believe that your preference for the last one comes from the
> >>>>>>> Monte-Carlo style approach that Gump uses - hope that your test
> >>>>>>> suite has enough variance that you "push" the thing being tested
> >>>>>>> through enough of the parameter space that you can be comfortable
> >>>>>>> you would have exposed the bugs. Maybe.
> >>>>>>>
> >>>>>> Ooh, now its becoming rather abstract...
> >>>>>>
> >>>>>> Well, perhaps, but more of the gump approache comes from the idea
> >>>>>> that
> >>>>>> the parameter space itself is also at some point defined in
> >>>>>> software,
> >>>>>> which may have bugs of its own. You circumvent that by making
> >>>>>> humans the
> >>>>>> parameter space (don't start about how humans are buggy. We don't
> >>>>>> want to
> >>>>>> get into existialism or faith systems when talking about unit
> >>>>>> testing do
> >>>>>> we?). The thing that gump enables is "many monkey QA" - a way for
> >>>>>> thousands
> >>>>>> of human beings to concurrently make shared assertions about
> >>>>>> software
> >>>>>> without actually needing all that much human interaction.
> >>>>>>
> >>>>>> More concretely, if harmony can run all known java software, and run
> >>>>>> it to
> >>>>>> the asserted satisfaction of all its developers, you can trust that
> >>>>>> you have
> >>>>>> covered all the /relevant/ parts of the parameter space you
> >>>>>> describe.
> >>>>>>
> >>>>> Yes. And when you can run all knownn Java software, let me know :)
> >>>>> That's my point about the parameter space being huge. Even when you
> >>>>> reduce the definition to "that of all known Java software", you still
> >>>>> have a huge problem on your hands.
> >>>>>
> >>>>>
> >>>>>> You
> >>>>>> will never get that level of trust when the assertions are made by
> >>>>>> software
> >>>>>> rather than humans. This is how open source leads to software
> >>>>>> quality.
> >>>>>>
> >>>>>> Quoting myself, 'gump is the most misunderstood piece of software,
> >>>>>> ever'.
> >>>>>>
> >>>>>> cheers,
> >>>>>>
> >>>>>> Leo
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >>> To unsubscribe, e-mail: [EMAIL PROTECTED]
> >>> For additional commands, e-mail: [EMAIL PROTECTED]
> >>>
> >>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> Terms of use : http://incubator.apache.org/harmony/mailing.html
> >> To unsubscribe, e-mail: [EMAIL PROTECTED]
> >> For additional commands, e-mail: [EMAIL PROTECTED]
> >>
> >>
> >
>
>
> --
> Paulex Yang
> China Software Development Lab
> IBM
>
>
>
> ---------------------------------------------------------------------
> Terms of use : http://incubator.apache.org/harmony/mailing.html
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
