Seems like implemantation-specific tests for java.security package do not fit well into the proposed solution
I think we do not have to stick to requirement about organization name in the tests: we already have tests in package java.* why don't have package tests.*? Thanks, Mikhail 2006/4/26, Paulex Yang <[EMAIL PROTECTED]>: > Richard Liang wrote: > > Mikhail Loenko wrote: > >> how about 'specific'? impl seems to be not very informative. > >> > >> > > +1 to Oliver's impl :-) > >> I have a concern abou proposed package naming guidelines: > >> package name > >> org.apache.harmony.security.tests.org.apache.harmony.security > >> is not much better then 1000-character long test name. > >> > >> > > +1. I think the prefix org.apache.harmony.security is unnecessary. > > "tests.impl.org.apache.harmony.security" is enough to tell people what > > the test cases belong to > I agree it is duplicated and unnecessary, but it is a general convention > for Java application to start with organization name, i.e., > "org.apache.harmony". If I understand correctly, our implementation > package's name must start with "org.apache.harmony.<module name>", so > how about this: > > org.apache.harmony.<module name>.tests.impl.<rest of package name> > > as a example, test for org.apache.harmony.security.util will be named as: > org.apache.harmony.security.tests.impl.util > > while the API test still be named as > org.apache.harmony.security.tests.api.java.security > > comments? > > > > Any comments? Thanks a lot. > >> Thanks, > >> Mikhail > >> > >> > >> 2006/4/26, Paulex Yang <[EMAIL PROTECTED]>: > >> > >>> Oliver Deakin wrote: > >>> > >>>> George Harley wrote: > >>>> > >>>>> Mikhail Loenko wrote: > >>>>> > >>>>>> Hello > >>>>>> > >>>>>> I'd like to bring this thread back. > >>>>>> > >>>>>> Number of tests is growing and it is time to put them in order. > >>>>>> > >>>>>> So far we may have: > >>>>>> > >>>>>> 1) implementation-specific tests that designed to be run from > >>>>>> bootclasspath > >>>>>> 2) implementation-specific tests that might be run from classpath > >>>>>> 3) implementation-specific tests that designed to be run from > >>>>>> classpath > >>>>>> 4) implementation-independent tests that designed to be run from > >>>>>> bootclasspath > >>>>>> 5) implementation-independent tests that might be run from classpath > >>>>>> 6) implementation-independent tests that designed to be run from > >>>>>> classpath > >>>>>> > >>>>>> Also we seem to have the following packages, where the tests are: > >>>>>> > >>>>>> 1) the same package as implementation > >>>>>> 2) org.apache.harmony.tests.[the same package as implementation] > >>>>>> 3) tests.api.[the same package as implementation] > >>>>>> > >>>>>> I suggest that we work out step-by-step solution as we could not > >>>>>> reach > >>>>>> an agreement for the general universal one > >>>>>> > >>>>>> So for the first step I suggest that we separate i-independent tests > >>>>>> that must or may be run from classpath > >>>>>> > >>>>>> I suggest that we put them into package > >>>>>> tests.module.compatible.[package of implementation being tested] > >>>>>> > >>>>>> Comments? > >>>>>> > >>>>>> Thanks, > >>>>>> Mikhail > >>>>>> > >>>>>> > >>>>> Hi Mikhail, > >>>>> > >>>>> I've just started working through the modules to merge test packages > >>>>> "tests.api.[same package as implementation]" and "tests.api.[same > >>>>> package as implementation]" into one package space. Using the class > >>>>> library package naming guidelines from off the web site [1], all of > >>>>> the tests for the text module have been consolidated under > >>>>> org.apache.harmony.text.tests.[package under test]. > >>>>> > >>>>> Of course, the text module has only "implementation-independent tests > >>>>> that designed to be run from classpath". For modules that have got > >>>>> implementation-specific tests then I suppose we could use something > >>>>> like "org.apache.harmony.[module].tests.impl.[package under test]" or > >>>>> "org.apache.harmony.[module].tests.internal.[package under test]" > >>>>> etc. I've got no preference. > >>>>> > >>>> I think impl is preferable over internal here, as we already use > >>>> internal in our implementation package names to indicate classes > >>>> totally internal to that bundle. To also use internal to label tests > >>>> that are implementation specific may cause confusion. > >>>> > >>>> > >>> +1 from me. > >>> > >>>> Regards, > >>>> Oliver > >>>> > >>>> > >>>>> Best regards, > >>>>> George > >>>>> > >>>>> > >>>>> [1] > >>>>> http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> 2006/3/24, George Harley <[EMAIL PROTECTED]>: > >>>>>> > >>>>>> > >>>>>>> Geir Magnusson Jr wrote: > >>>>>>> > >>>>>>> > >>>>>>>> Leo Simons wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>>> On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Leo Simons wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> Pulling out of the various threads where we have been > >>>>>>>>>>>> discussing, > >>>>>>>>>>>> can we agree on the problem : > >>>>>>>>>>>> > >>>>>>>>>>>> We have unique problems compared to other Java projects > >>>>>>>>>>>> because we > >>>>>>>>>>>> need to find a way to reliably test the things that are > >>>>>>>>>>>> commonly > >>>>>>>>>>>> expected to be a solid point of reference - namely the core > >>>>>>>>>>>> class > >>>>>>>>>>>> library. > >>>>>>>>>>>> > >>>>>>>>>>>> Further, we've been implicitly doing "integration testing" > >>>>>>>>>>>> because > >>>>>>>>>>>> - so far - the only way we've been testing our code has > >>>>>>>>>>>> been 'in > >>>>>>>>>>>> situ' in the VM - not in an isolated test harness. To me, > >>>>>>>>>>>> this > >>>>>>>>>>>> turns it into an integration test. > >>>>>>>>>>>> > >>>>>>>>>>>> Sure, we're using JUnit, but because of the fact we are > >>>>>>>>>>>> implmenting core java.* APIs, we aren't testing with a > >>>>>>>>>>>> framework > >>>>>>>>>>>> that has been independently tested for correctness, like we > >>>>>>>>>>>> would > >>>>>>>>>>>> when testing any other code. > >>>>>>>>>>>> > >>>>>>>>>>>> I hope I got that idea across - I believe that we have to go > >>>>>>>>>>>> beyond normal testing approaches because we don't have a > >>>>>>>>>>>> normal > >>>>>>>>>>>> situation. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> Where we define 'normal situation' as "running a test > >>>>>>>>>>> framework on > >>>>>>>>>>> top of > >>>>>>>>>>> the sun jdk and expecting any bugs to not be in that jdk". > >>>>>>>>>>> There's > >>>>>>>>>>> plenty > >>>>>>>>>>> of projects out there that have to test things without having > >>>>>>>>>>> such a > >>>>>>>>>>> "stable reference JDK" luxury.....I imagine that testing GCC is > >>>>>>>>>>> just as > >>>>>>>>>>> hard as this problem we have here :-) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> Is it the same? We need to have a running JVM+classlibarary to > >>>>>>>>>> test > >>>>>>>>>> the classlibrary code. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> Well you need a working C compiler and standard C library to > >>>>>>>>> compile the > >>>>>>>>> compiler so you can compile make so you can build bash so you can > >>>>>>>>> run > >>>>>>>>> perl (which uses the standard C library functions all over the > >>>>>>>>> place of > >>>>>>>>> course) so you can run the standard C library tests so that > >>>>>>>>> you know > >>>>>>>>> that > >>>>>>>>> the library you used when compiling the compiler were correct so > >>>>>>>>> you can > >>>>>>>>> run the compiler tests. I don't think they actually do things > >>>>>>>>> that > >>>>>>>>> way, but > >>>>>>>>> it seems like basically the same problem. Having a virtual > >>>>>>>>> machine just > >>>>>>>>> makes it easier since you still assume "the native world" as a > >>>>>>>>> baseline, > >>>>>>>>> which is a lot more than "the hardware". > >>>>>>>>> > >>>>>>>>> > >>>>>>>> There's a difference. You can use a completely separate > >>>>>>>> toolchain to > >>>>>>>> build, test and verify the output of the C compiler. > >>>>>>>> > >>>>>>>> In our case, we are using the thing we are testing to test itself. > >>>>>>>> There is no "known good" element possible right now. > >>>>>>>> > >>>>>>>> We use the classlibrary we are trying to test to execute the test > >>>>>>>> framework that tests the classlibrary that is running it. > >>>>>>>> > >>>>>>>> The tool is testing itself. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>>>> So I think there are three things we want to do (adopting the > >>>>>>>>>>>> terminology that came from the discussion with Tim and Leo ) : > >>>>>>>>>>>> > >>>>>>>>>>>> 1) implementation tests > >>>>>>>>>>>> 2) spec/API tests (I'll bundle together) > >>>>>>>>>>>> 3) integration/functional tests > >>>>>>>>>>>> > >>>>>>>>>>>> I believe that for #1, the issues related to being on the > >>>>>>>>>>>> bootclasspath don't matter, because we aren't testing that > >>>>>>>>>>>> aspect > >>>>>>>>>>>> of the classes (which is how they behave integrated w/ the > >>>>>>>>>>>> VM and > >>>>>>>>>>>> security system) but rather the basic internal functioning. > >>>>>>>>>>>> > >>>>>>>>>>>> I'm not sure how to approach this, but I'll try. I'd love to > >>>>>>>>>>>> hear > >>>>>>>>>>>> how Sun, IBM or BEA deals with this, or be told why it > >>>>>>>>>>>> isn't an > >>>>>>>>>>>> issue :) > >>>>>>>>>>>> > >>>>>>>>>>>> Implementation tests : I'd like to see us be able to do #1 via > >>>>>>>>>>>> the > >>>>>>>>>>>> standard same-package technique (i.e. testing a.b.C w/ > >>>>>>>>>>>> a.b.CTest) > >>>>>>>>>>>> but we'll run into a tangle of classloader problems, I > >>>>>>>>>>>> suspect, > >>>>>>>>>>>> becuase we want to be testing java.* code in a system that > >>>>>>>>>>>> already > >>>>>>>>>>>> has java.* code. Can anyone see a way we can do this - test > >>>>>>>>>>>> the > >>>>>>>>>>>> classlibrary from the integration point of view - using > >>>>>>>>>>>> some test > >>>>>>>>>>>> harness + any known-good JRE, like Sun's or IBM's? > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> Ew, that won't work in the end since we should assume our > >>>>>>>>>>> own JRE > >>>>>>>>>>> is going > >>>>>>>>>>> to be "known-better" :-). But it might be a nice way to > >>>>>>>>>>> "bootstrap" > >>>>>>>>>>> (eg > >>>>>>>>>>> we test with an external JRE until we satisfy the tests and > >>>>>>>>>>> then we > >>>>>>>>>>> switch > >>>>>>>>>>> to testing with an earlier build). > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> Lets be clear - even using our own "earlier build" doesn't solve > >>>>>>>>>> the > >>>>>>>>>> problem I'm describing, because as it stands now, we don't use > >>>>>>>>>> "earlier build" classes to test with - we use the code we > >>>>>>>>>> want to > >>>>>>>>>> test as the clsaslibrary for the JRE that's running the test > >>>>>>>>>> framework. > >>>>>>>>>> > >>>>>>>>>> The classes that we are testing are also the classes used by the > >>>>>>>>>> testing framework. IOW, any of the java.* classes that JUnit > >>>>>>>>>> itself > >>>>>>>>>> needs (ex. java.util.HashMap) are exactly the same > >>>>>>>>>> implementation > >>>>>>>>>> that it's testing. > >>>>>>>>>> > >>>>>>>>>> That's why I think it's subtly different than a "bootstrap > >>>>>>>>>> and use > >>>>>>>>>> version - 1 to test" problem. See what I mean? > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> Yeah yeah, I was already way beyond thinking "just" JUnit is > >>>>>>>>> usable > >>>>>>>>> for the > >>>>>>>>> kind of test you're describing. At some point, fundamentally, you > >>>>>>>>> either trust > >>>>>>>>> something external (whether its the sun jdk or the intel compiler > >>>>>>>>> designers, > >>>>>>>>> at some point you do draw a line) or you find a way to bootstrap. > >>>>>>>>> > >>>>>>>>> > >>>>>>>> Well, we do trust the Sun JDK. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>>> I'm very open to the idea that I'm missing something here, > >>>>>>>>>> but I'd > >>>>>>>>>> like to know that you see the issue - that when we test, we have > >>>>>>>>>> > >>>>>>>>>> VM + "classlib to be tested" + JUnit + testcases > >>>>>>>>>> > >>>>>>>>>> where the testcases are testing the classlib the VM is running > >>>>>>>>>> JUnit > >>>>>>>>>> with. > >>>>>>>>>> > >>>>>>>>>> There never is isolation of the code being tested : > >>>>>>>>>> > >>>>>>>>>> VM + "known good classlib" + Junit + testcases > >>>>>>>>>> > >>>>>>>>>> unless we have some framework where > >>>>>>>>>> > >>>>>>>>>> VM + "known good classlib" + JUnit > >>>>>>>>>> + framework("classlib to be tested") > >>>>>>>>>> + testcases > >>>>>>>>>> > >>>>>>>>>> and it's that notion of "framework()" that I'm advocating we > >>>>>>>>>> explore. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> I'm all for exploring it, I just fundamentally don't buy into the > >>>>>>>>> "known > >>>>>>>>> good" bit. What happens when the 'classlib to be tested' is > >>>>>>>>> 'known > >>>>>>>>> better' than the 'known good' one? How do you define "known"? How > >>>>>>>>> do you > >>>>>>>>> define "good"? > >>>>>>>>> > >>>>>>>>> > >>>>>>>> Known? Passed some set of tests. So it could be the Sun JDK > >>>>>>>> for the > >>>>>>>> VM + "known good" part. > >>>>>>>> > >>>>>>>> I think you intuitively understand this. When you find a bug > >>>>>>>> in code > >>>>>>>> you are testing, you first assume it's your code, not the > >>>>>>>> framework, > >>>>>>>> right? In our case, our framework is actually the code we are > >>>>>>>> testing, so we have a bit of a logical conundrum. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>> Hi Geir, > >>>>>>> > >>>>>>> The number of Harmony public API classes that get loaded just to > >>>>>>> run the > >>>>>>> JUnit harness is a little over 200. The majority of these are > >>>>>>> out of > >>>>>>> LUNI with a very low number coming from each of Security, NIO, > >>>>>>> Archive > >>>>>>> and Text. > >>>>>>> > >>>>>>> Sure there is a circular dependency between what we are building > >>>>>>> and the > >>>>>>> framework we are using to test it but it appears to touch on only a > >>>>>>> relatively small part of Harmony....IMHO. > >>>>>>> > >>>>>>> Best regards, > >>>>>>> George > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>>>>> Further ideas... > >>>>>>>>>>> > >>>>>>>>>>> -> look at how the native world does testing > >>>>>>>>>>> (hint: it usually has #ifdefs, uses perl along the way, and > >>>>>>>>>>> it is > >>>>>>>>>>> certainly > >>>>>>>>>>> "messy") > >>>>>>>>>>> -> emulate that > >>>>>>>>>>> > >>>>>>>>>>> -> build a bigger, better specification test > >>>>>>>>>>> -> and somehow "prove" it is "good enough" > >>>>>>>>>>> > >>>>>>>>>>> -> build a bigger, better integration test > >>>>>>>>>>> -> and somehow "prove" it is "good enough" > >>>>>>>>>>> > >>>>>>>>>>> I'll admit my primary interest is the last one... > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> The problem I see with the last one is that the "parameter > >>>>>>>>>> space" is > >>>>>>>>>> *huge*. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> Yeah, that's one of the things that makes it interesting. > >>>>>>>>> Fortunately > >>>>>>>>> open source does have many monkeys... > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> I believe that your preference for the last one comes from the > >>>>>>>>>> Monte-Carlo style approach that Gump uses - hope that your test > >>>>>>>>>> suite has enough variance that you "push" the thing being tested > >>>>>>>>>> through enough of the parameter space that you can be > >>>>>>>>>> comfortable > >>>>>>>>>> you would have exposed the bugs. Maybe. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> Ooh, now its becoming rather abstract... > >>>>>>>>> > >>>>>>>>> Well, perhaps, but more of the gump approache comes from the idea > >>>>>>>>> that > >>>>>>>>> the parameter space itself is also at some point defined in > >>>>>>>>> software, > >>>>>>>>> which may have bugs of its own. You circumvent that by making > >>>>>>>>> humans the > >>>>>>>>> parameter space (don't start about how humans are buggy. We don't > >>>>>>>>> want to > >>>>>>>>> get into existialism or faith systems when talking about unit > >>>>>>>>> testing do > >>>>>>>>> we?). The thing that gump enables is "many monkey QA" - a way for > >>>>>>>>> thousands > >>>>>>>>> of human beings to concurrently make shared assertions about > >>>>>>>>> software > >>>>>>>>> without actually needing all that much human interaction. > >>>>>>>>> > >>>>>>>>> More concretely, if harmony can run all known java software, > >>>>>>>>> and run > >>>>>>>>> it to > >>>>>>>>> the asserted satisfaction of all its developers, you can trust > >>>>>>>>> that > >>>>>>>>> you have > >>>>>>>>> covered all the /relevant/ parts of the parameter space you > >>>>>>>>> describe. > >>>>>>>>> > >>>>>>>>> > >>>>>>>> Yes. And when you can run all knownn Java software, let me > >>>>>>>> know :) > >>>>>>>> That's my point about the parameter space being huge. Even > >>>>>>>> when you > >>>>>>>> reduce the definition to "that of all known Java software", you > >>>>>>>> still > >>>>>>>> have a huge problem on your hands. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> You > >>>>>>>>> will never get that level of trust when the assertions are > >>>>>>>>> made by > >>>>>>>>> software > >>>>>>>>> rather than humans. This is how open source leads to software > >>>>>>>>> quality. > >>>>>>>>> > >>>>>>>>> Quoting myself, 'gump is the most misunderstood piece of > >>>>>>>>> software, > >>>>>>>>> ever'. > >>>>>>>>> > >>>>>>>>> cheers, > >>>>>>>>> > >>>>>>>>> Leo > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>> --------------------------------------------------------------------- > >>>>>> > >>>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>>>> For additional commands, e-mail: > >>>>>> [EMAIL PROTECTED] > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> --------------------------------------------------------------------- > >>>>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>>> For additional commands, e-mail: > >>>>> [EMAIL PROTECTED] > >>>>> > >>>>> > >>>>> > >>> -- > >>> Paulex Yang > >>> China Software Development Lab > >>> IBM > >>> > >>> > >>> > >>> --------------------------------------------------------------------- > >>> Terms of use : http://incubator.apache.org/harmony/mailing.html > >>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>> For additional commands, e-mail: [EMAIL PROTECTED] > >>> > >>> > >>> > >> > >> --------------------------------------------------------------------- > >> Terms of use : http://incubator.apache.org/harmony/mailing.html > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: [EMAIL PROTECTED] > >> > >> > >> > > > > > > > -- > Paulex Yang > China Software Development Lab > IBM > > > > --------------------------------------------------------------------- > Terms of use : http://incubator.apache.org/harmony/mailing.html > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- Terms of use : http://incubator.apache.org/harmony/mailing.html To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
