George Harley1 wrote:
Hi,
This is not the goal of unit tests. They do not test API, they test the
code,
including package-access members etc. That is why unit tests must reside
in the same package as the classes.
This is radical stuff. I have never heard the argument before that unit
tests *must* share the same package name as the type under test. One
important reason why many think it is a bad idea to take this approach is
that the resulting unit test code becomes too closely coupled with the
code under test.
Huh? It's a common practice - it gives the unit test the ability to
really test the implementation because it has package level priv.
Think about it ...
If unit test code can access every package-access member and method in a
given type then the test code will eventually reach a point where it
contains a lot of brittle assumptions about internal implementation.
It can, yes. OTOH, it is testing internal private methods, so yes,
there's going to be those assumptions built in.
Consequently, when implementation changes occur, the corresponding test
code will break. Note : I'm not talking about API changes breaking the
test code here, I'm talking about changes to protected and package-access
stuff which - by definition - the developer has chosen to keep non-public.
Right - and the unit tests test that code. How else do you test it in a
fine-grained way? Only via public methods?
So as the internals change the unit tests break. And further internal
changes make the unit tests break. And ultimately maintaining the tests
with close-coupling to the internals of the type under test becomes a bit
of a pain and their overall value in the test harness diminishes because
minor refactorings to the type-under-test now take so darned long to carry
out.
But it gives you very fine-grained coverage, if you want that.
geir
Best regards,
George
________________________________________
George C. Harley
Mikhail Loenko <[EMAIL PROTECTED]>
16/01/2006 11:58
Please respond to
harmony-dev@incubator.apache.org
To
harmony-dev@incubator.apache.org
cc
Subject
Re: Test suite layout
On 1/16/06, Tim Ellison <[EMAIL PROTECTED]> wrote:
...
If your unit tests are intended to test API, then they should be calling
the API in the same manner that an application will call the API. Just
This is not the goal of unit tests. They do not test API, they test the
code,
including package-access members etc. That is why unit tests must reside
in the same package as the classes.
Other kinds of tests like compatibility or functional tests call just
public and protected methods so they can be in different packages.
Thanks,
Mikhail Loenko
Intel Middleware Products Division
look at the number of places where we check to see if the classloader ==
null to see where this matters.
Some of them may depend on security manager component if
you like to run a test in some security restricted environment. But it
is not a big issue: the security manager component can be configured
dynamically and you should install your custom security manager before
running a test.
Yes, we should do that on the application classloader to test security
manager functionality.
Regards,
Tim
What do you think?
Thanks,
Stepan Mishura
Intel Middleware Products Division
-----Original Message-----
From: Tim Ellison [mailto:[EMAIL PROTECTED]
Sent: Thursday, January 12, 2006 8:04 PM
To: harmony-dev@incubator.apache.org
Subject: Re: Test suite layout
Geir Magnusson Jr wrote:
Tim Ellison wrote:
<snip>
We would have written it as java.io.tests, but the java.<whatever>
namespace is reserved, so the formula is simply
<package>.<type> -> org.apache.harmony.tests.<package>.<type>Test
Ug - then you have the problem of not being in the namespace as what
you
are testing.
THat's why people use parallel trees - so your test code is
physically
separate but you have freedom of package access.
For 'normal' application code then you can do this, but since we are
writing the java packages themselves then you come unstuck because the
java packages have to run on the bootclasspath, and the tests on the
application classpath.
You don't want to run all your tests on the bootclasspath (because
then
it would not be subject to the same security sandboxing as
applications
using the APIs you are testing); and you cannot put java.<whatever> on
the application classpath because the VM will catch you out (you'd get
a
linkage error IIRC).
This makes it clear what is being tested, and where to add new tests
etc.
So would
test/org/apache/harmony/io/TestFoo.java
(to test something in org.apache.harmony.io, and arguable to test the
Foo.java class in there. (or TestFoo.java - it's early - no coffee
yet)
Not sure what you are saying here... For java.<whatever> packages we
need a prefix on the test packages to keep the VM happy, but for
org.apache.harmony packages we can have either pre- or post-.
I'd actually prefer a postfix of .tests for non-API packages, though I
can understand if people object to the inconsistency; so
org.apache.harmony.tests.java.io.FileTest.java <- test API
org.apache.harmony.io.tests.FileImpltest.java <- test public methods
in our IO impl'
Simiarly
test/java/util/TestMap.java
Then within the test class itself the methods are named after the
method
under test, with a familar JNI-style encoding, so we have things
like:
org.apache.harmony.tests.java.io.FileTest contains
public void test_ConstructorLjava_io_FileLjava_lang_String() {
...
}
and
org.apache.harmony.tests.java.lang.FloatTest contains
public void test_compareToLjava_lang_Float() {
...
}
...or whatever the convention is for JUnit. I think that's one of
the
nice things about TestNG, is that it's annotated, so you have the
freedom there.
If the test is added due to a regression, then it is put into the
right
place in the test suite, and flagged with a comment (i.e. a
reference to
the Harmony JIRA number).
Yes - and I'd even advocate a parallel directory there too so that
it's
clear that the regressions are different, but whatever. The only
snag
there is name collision with the classes.
I thought we'd agreed that 'regression' was not a useful
classification
within the test suite layout ...
I think that a simple comment is enough. If we want to get cute,
maybe
a javadoc tag so we can manage mechanically in the future.
ok -- do you have a usecase in mind?
Regards,
Tim
George Harley1 wrote:
Hi,
I think that regression tests should be marked in some way.
Agreed. But can we please *resist* the temptation to do this by
incorporating JIRA issue numbers into test case names (e.g. calling
unit test methods test_26() or test_JIRA_26()). I've seen this kind
of approach adopted in a couple of projects and, in my experience,
it
often leads to the scattering of duplicated test code around the
test
harness.
Better, methinks, to either create a new test method with a
meaningful name or else augment an existing method - whatever makes
more sense for the particular issue. Then marking certain code as
being for regression test purposes could be done in comments that
include the URL of the JIRA issue. Perhaps an agreed tag like
"JIRA"
or "BUG" etc could be used as an eye-catcher as well ?
e.g.
// BUG http://issues.apache.org/jira/browse/HARMONY-26
My 2 Euro Cents.
Best regards, George
________________________________________
George C. Harley
"Mishura, Stepan M" <[EMAIL PROTECTED]> 12/01/2006 04:56
Please respond to
harmony-dev@incubator.apache.org
To
<harmony-dev@incubator.apache.org>
cc
Subject
RE: regression test suite
Hello,
Tim Ellison wrote:
[snip]
What is the useful distinction for regression tests being kept
separate?
I can see that you may distinguish unit and 'system-level' tests
just
because of the difference in frameworks etc. required, but why do
I
care
if the test was written due to a JIRA issue or test-based
development
or
someone who get's kicks out of writing tests to break the code?
I agree that separating regression tests doesn't make sense.
However I think that regression tests should be marked in some way.
This will signal a developer that a test was created to track
already
known issue. IMHO, a regression test should point out to a bug
report
and a bug report (after resolving a bug) should contain a reference
to
corresponding regression test in repository.
Thanks,
Stepan Mishura
Intel Middleware Products Division
--
Tim Ellison ([EMAIL PROTECTED])
IBM Java technology centre, UK.
--
Tim Ellison ([EMAIL PROTECTED])
IBM Java technology centre, UK.
