Mikhail Loenko wrote:
Seems like implemantation-specific tests for java.security package do not
fit well into the proposed solution
I think we do not have to stick to requirement about
organization name in the tests: we already have tests in package java.*
why don't have package tests.*?
Well, this is another issue. If the organization name is not necessary
as prefix, then I'm fine with the tests.<package name>.
Thanks,
Mikhail
2006/4/26, Paulex Yang <[EMAIL PROTECTED]>:
Richard Liang wrote:
Mikhail Loenko wrote:
how about 'specific'? impl seems to be not very informative.
+1 to Oliver's impl :-)
I have a concern abou proposed package naming guidelines:
package name
org.apache.harmony.security.tests.org.apache.harmony.security
is not much better then 1000-character long test name.
+1. I think the prefix org.apache.harmony.security is unnecessary.
"tests.impl.org.apache.harmony.security" is enough to tell people what
the test cases belong to
I agree it is duplicated and unnecessary, but it is a general convention
for Java application to start with organization name, i.e.,
"org.apache.harmony". If I understand correctly, our implementation
package's name must start with "org.apache.harmony.<module name>", so
how about this:
org.apache.harmony.<module name>.tests.impl.<rest of package name>
as a example, test for org.apache.harmony.security.util will be named as:
org.apache.harmony.security.tests.impl.util
while the API test still be named as
org.apache.harmony.security.tests.api.java.security
comments?
Any comments? Thanks a lot.
Thanks,
Mikhail
2006/4/26, Paulex Yang <[EMAIL PROTECTED]>:
Oliver Deakin wrote:
George Harley wrote:
Mikhail Loenko wrote:
Hello
I'd like to bring this thread back.
Number of tests is growing and it is time to put them in order.
So far we may have:
1) implementation-specific tests that designed to be run from
bootclasspath
2) implementation-specific tests that might be run from classpath
3) implementation-specific tests that designed to be run from
classpath
4) implementation-independent tests that designed to be run from
bootclasspath
5) implementation-independent tests that might be run from classpath
6) implementation-independent tests that designed to be run from
classpath
Also we seem to have the following packages, where the tests are:
1) the same package as implementation
2) org.apache.harmony.tests.[the same package as implementation]
3) tests.api.[the same package as implementation]
I suggest that we work out step-by-step solution as we could not
reach
an agreement for the general universal one
So for the first step I suggest that we separate i-independent tests
that must or may be run from classpath
I suggest that we put them into package
tests.module.compatible.[package of implementation being tested]
Comments?
Thanks,
Mikhail
Hi Mikhail,
I've just started working through the modules to merge test packages
"tests.api.[same package as implementation]" and "tests.api.[same
package as implementation]" into one package space. Using the class
library package naming guidelines from off the web site [1], all of
the tests for the text module have been consolidated under
org.apache.harmony.text.tests.[package under test].
Of course, the text module has only "implementation-independent tests
that designed to be run from classpath". For modules that have got
implementation-specific tests then I suppose we could use something
like "org.apache.harmony.[module].tests.impl.[package under test]" or
"org.apache.harmony.[module].tests.internal.[package under test]"
etc. I've got no preference.
I think impl is preferable over internal here, as we already use
internal in our implementation package names to indicate classes
totally internal to that bundle. To also use internal to label tests
that are implementation specific may cause confusion.
+1 from me.
Regards,
Oliver
Best regards,
George
[1]
http://incubator.apache.org/harmony/subcomponents/classlibrary/pkgnaming.html
2006/3/24, George Harley <[EMAIL PROTECTED]>:
Geir Magnusson Jr wrote:
Leo Simons wrote:
On Wed, Mar 22, 2006 at 08:02:44AM -0500, Geir Magnusson Jr
wrote:
Leo Simons wrote:
On Wed, Mar 22, 2006 at 07:15:28AM -0500, Geir Magnusson Jr
wrote:
Pulling out of the various threads where we have been
discussing,
can we agree on the problem :
We have unique problems compared to other Java projects
because we
need to find a way to reliably test the things that are
commonly
expected to be a solid point of reference - namely the core
class
library.
Further, we've been implicitly doing "integration testing"
because
- so far - the only way we've been testing our code has
been 'in
situ' in the VM - not in an isolated test harness. To me,
this
turns it into an integration test.
Sure, we're using JUnit, but because of the fact we are
implmenting core java.* APIs, we aren't testing with a
framework
that has been independently tested for correctness, like we
would
when testing any other code.
I hope I got that idea across - I believe that we have to go
beyond normal testing approaches because we don't have a
normal
situation.
Where we define 'normal situation' as "running a test
framework on
top of
the sun jdk and expecting any bugs to not be in that jdk".
There's
plenty
of projects out there that have to test things without having
such a
"stable reference JDK" luxury.....I imagine that testing GCC is
just as
hard as this problem we have here :-)
Is it the same? We need to have a running JVM+classlibarary to
test
the classlibrary code.
Well you need a working C compiler and standard C library to
compile the
compiler so you can compile make so you can build bash so you can
run
perl (which uses the standard C library functions all over the
place of
course) so you can run the standard C library tests so that
you know
that
the library you used when compiling the compiler were correct so
you can
run the compiler tests. I don't think they actually do things
that
way, but
it seems like basically the same problem. Having a virtual
machine just
makes it easier since you still assume "the native world" as a
baseline,
which is a lot more than "the hardware".
There's a difference. You can use a completely separate
toolchain to
build, test and verify the output of the C compiler.
In our case, we are using the thing we are testing to test itself.
There is no "known good" element possible right now.
We use the classlibrary we are trying to test to execute the test
framework that tests the classlibrary that is running it.
The tool is testing itself.
So I think there are three things we want to do (adopting the
terminology that came from the discussion with Tim and Leo ) :
1) implementation tests
2) spec/API tests (I'll bundle together)
3) integration/functional tests
I believe that for #1, the issues related to being on the
bootclasspath don't matter, because we aren't testing that
aspect
of the classes (which is how they behave integrated w/ the
VM and
security system) but rather the basic internal functioning.
I'm not sure how to approach this, but I'll try. I'd love to
hear
how Sun, IBM or BEA deals with this, or be told why it
isn't an
issue :)
Implementation tests : I'd like to see us be able to do #1 via
the
standard same-package technique (i.e. testing a.b.C w/
a.b.CTest)
but we'll run into a tangle of classloader problems, I
suspect,
becuase we want to be testing java.* code in a system that
already
has java.* code. Can anyone see a way we can do this - test
the
classlibrary from the integration point of view - using
some test
harness + any known-good JRE, like Sun's or IBM's?
Ew, that won't work in the end since we should assume our
own JRE
is going
to be "known-better" :-). But it might be a nice way to
"bootstrap"
(eg
we test with an external JRE until we satisfy the tests and
then we
switch
to testing with an earlier build).
Lets be clear - even using our own "earlier build" doesn't solve
the
problem I'm describing, because as it stands now, we don't use
"earlier build" classes to test with - we use the code we
want to
test as the clsaslibrary for the JRE that's running the test
framework.
The classes that we are testing are also the classes used by the
testing framework. IOW, any of the java.* classes that JUnit
itself
needs (ex. java.util.HashMap) are exactly the same
implementation
that it's testing.
That's why I think it's subtly different than a "bootstrap
and use
version - 1 to test" problem. See what I mean?
Yeah yeah, I was already way beyond thinking "just" JUnit is
usable
for the
kind of test you're describing. At some point, fundamentally, you
either trust
something external (whether its the sun jdk or the intel compiler
designers,
at some point you do draw a line) or you find a way to bootstrap.
Well, we do trust the Sun JDK.
I'm very open to the idea that I'm missing something here,
but I'd
like to know that you see the issue - that when we test, we have
VM + "classlib to be tested" + JUnit + testcases
where the testcases are testing the classlib the VM is running
JUnit
with.
There never is isolation of the code being tested :
VM + "known good classlib" + Junit + testcases
unless we have some framework where
VM + "known good classlib" + JUnit
+ framework("classlib to be tested")
+ testcases
and it's that notion of "framework()" that I'm advocating we
explore.
I'm all for exploring it, I just fundamentally don't buy into the
"known
good" bit. What happens when the 'classlib to be tested' is
'known
better' than the 'known good' one? How do you define "known"? How
do you
define "good"?
Known? Passed some set of tests. So it could be the Sun JDK
for the
VM + "known good" part.
I think you intuitively understand this. When you find a bug
in code
you are testing, you first assume it's your code, not the
framework,
right? In our case, our framework is actually the code we are
testing, so we have a bit of a logical conundrum.
Hi Geir,
The number of Harmony public API classes that get loaded just to
run the
JUnit harness is a little over 200. The majority of these are
out of
LUNI with a very low number coming from each of Security, NIO,
Archive
and Text.
Sure there is a circular dependency between what we are building
and the
framework we are using to test it but it appears to touch on only a
relatively small part of Harmony....IMHO.
Best regards,
George
Further ideas...
-> look at how the native world does testing
(hint: it usually has #ifdefs, uses perl along the way, and
it is
certainly
"messy")
-> emulate that
-> build a bigger, better specification test
-> and somehow "prove" it is "good enough"
-> build a bigger, better integration test
-> and somehow "prove" it is "good enough"
I'll admit my primary interest is the last one...
The problem I see with the last one is that the "parameter
space" is
*huge*.
Yeah, that's one of the things that makes it interesting.
Fortunately
open source does have many monkeys...
I believe that your preference for the last one comes from the
Monte-Carlo style approach that Gump uses - hope that your test
suite has enough variance that you "push" the thing being tested
through enough of the parameter space that you can be
comfortable
you would have exposed the bugs. Maybe.
Ooh, now its becoming rather abstract...
Well, perhaps, but more of the gump approache comes from the idea
that
the parameter space itself is also at some point defined in
software,
which may have bugs of its own. You circumvent that by making
humans the
parameter space (don't start about how humans are buggy. We don't
want to
get into existialism or faith systems when talking about unit
testing do
we?). The thing that gump enables is "many monkey QA" - a way for
thousands
of human beings to concurrently make shared assertions about
software
without actually needing all that much human interaction.
More concretely, if harmony can run all known java software,
and run
it to
the asserted satisfaction of all its developers, you can trust
that
you have
covered all the /relevant/ parts of the parameter space you
describe.
Yes. And when you can run all knownn Java software, let me
know :)
That's my point about the parameter space being huge. Even
when you
reduce the definition to "that of all known Java software", you
still
have a huge problem on your hands.
You
will never get that level of trust when the assertions are
made by
software
rather than humans. This is how open source leads to software
quality.
Quoting myself, 'gump is the most misunderstood piece of
software,
ever'.
cheers,
Leo
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail:
[EMAIL PROTECTED]
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Paulex Yang
China Software Development Lab
IBM
---------------------------------------------------------------------
Terms of use : http://incubator.apache.org/harmony/mailing.html
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
