David W. Van Couvering wrote:
Hi, Kristian, thanks for your questions. My one overriding thought is
we should take this in incremental steps -- do all of these questions
need to be answered before we can rewrite a single old canon-based test
to a JUnit test? Can some of these questions be deferred?
Yes, they can be deferred. I did not intend to say that we should try to
fix/implement all issues/features at in one step.
My main point of concern, is that it is too hard to write JUnit tests
now, because there is so little information available. Just have a look
at the number of JUnit tests that have been added to the repository - it
sure ain't many! The conversion process, which I understand is fully
based on "it's my itch" initiatives, is also moving along very, very slowly.
Kristian Waagan wrote:
Hello,
I have wanted to rewrite a old canon based test to a JUnit test for a
while, but each time I get started I always run into issues that we do
not have guidelines for yet. So, I have a few questions I would like
some feedback on.
I know there are a lot of issues to consider, far more then the one I
mention here, but I hope to get a feeling of the general direction we
want to move in.
1) Should we support specific ordering of tests (within a single class)?
This feature is wanted if a number of tests operate on shared data and
each test method can be seen as a step in the test. If the order is
mixed up, the intermediate results will not be as expected and all the
tests may fail.
In general, assuming anything about the execution order of tests is
strongly disencouraged. However, since we are testing a database
system, we might want to allow for this. This feature can influence
the both the size of the test methods and the execution time for the
test/suite.
I thought if you name your tests correctly (e.g. test_010, test_020,
etc.) then they are run in that order. Or is that not the case because
all method names on a class are hashed?
This is not the case. Tests are not ordered by method names. I do not
know the implementation details about ordering when using reflection. I
do know that when adding one test at a time in suite(), they are stored
in a Vector and thus ordered (and executed in order). But please note
that this is not a documented feature of JUnit.
Assuming that's not the case, it seems to me that if you have specific
ordering requirements, you could use multiple test classes, create a
suite of those classes, and use the TestSetup mechanism you describe
below to do any setup work prior to running the suite.
Might work. In my opinion, the best thing is to write each test to be
totally independent, but I am looking for ways to handle tests that are
dependent on each other because I suspect that need will arise.
2) Should we support setting up a shared test fixture for a set of tests?
This is a common issue with JUnit, and there are mechanisms to handle
it. For instance, we could let tests that require this to wrap itself
in a TestSetup instance (as described in the JUNit FAQ). Letting
DerbyJUnitTest extend TestSetup is another solution, but it might have
unwanted side-effects.
I think there is a lot of value to having something like this (see above
for an example).
3) Many tests just want a database to work on. Other tests want to
create their own database due to specific requirements. How do we make
this easy for the developer?
I think if you have a TestSetup mechanism, then a suite of tests can
depend on the fact that a database has been created. Creating a
database is pretty darn easy already; are there particular things that
are repetetive or error-prone?
No. As long as you have the driver name and the connection string
(including the correct protocol and database path/name), it is very
easy. This information must be passed on to the test setup class when
suite() is run, and with the current harness implementation (with
respect to JUnit tests) this information is not available then. There
are easy workarounds, but they are not written down anywhere...
This is actually a matter of question 6.
4) Should it be possible to run single JUnit test/suite from the
command line by simply invoking the main method?
Personally, I think this should be possible. This require us to remove
the current use of main for running a test in the old test harness,
which I believe is doable with a few changes (in the old harness).
I think this is essential for debugging and incremental testing. +1 on
this idea.
5) Is the ultimate goal to end up with a pure JUnit based test harness?
If this is the goal, we should design for this and accept some extra
work for plugging JUnit tests into the old test harness.
I can see us getting there over time, but this is a Very Big Effort. Are
there particular things we have to decide upon now in order for this to
be possible in the future?
I don't know of any, but someone else might.
As long as we code the tests in typical JUnit spirit, we should be good.
As soon as we start deviating from "standard" behavior, people won't be
able to run the tests in third-party tools (for instance NetBeans).
We might not be able to allow for this anyway, though.
6) Where, and how, do we want to store settings for a test?
In a database! :)
I personally think a properties file is easier to work with than an XML
file, *as long as* we don't load these in as System properties, gads,
that's a nightmare.
7) How is it determined which framework(s) a JUnit test should be run
in? (i.e. embedded, client, JCC)
I think you should have ways to run a particular suite under a
particular framework, as you do today.
I agree. You could also have a way to run under all available
frameworks, but then a bit more is required by the harness (or manual
work - starting servers).
If you have other opinions or comments related to JUnit testing,
please put them forth :)
I think ther are two separate questions:
- Running tests under JUnit with a JUnit harness
- Rewriting existing tests to no longer use canons
I think we need to make sure we discuss these separately. Existing
tests can run under JUnit with canons just fine (that's what we're
already doing). I think that's could be big performance improvement
right there (I remember Dan showing the difference in performance).
Hmm, I thought we were running JUnit tests under the old harness. That
is, the harness automatically generates an empty canon.
Anyway, I agree that if our current harness gets the job done, there is
no need to rush for a new harness capable of running only JUnit tests.
But I mean we need to produce enough information to enable people to
write JUnit tests quickly and without great suffering ;)
--
Kristian
Rewriting tests to no longer use canons (and thus .sql files) is a Much
Bigger Task and I suggest that be separated out, and we should discuss
its merits/demerits separately.
I hope we are able to get a discussion on all or some of these issues,
and that it results in a Wiki page describing JUnit testing for Derby.
I created a page here: http://wiki.apache.org/db-derby/DerbyJUnitTesting
I am sure you will get a discussion. In terms of it "resulting" in a
Wiki page, I think that is up to whoever has the itch to do that (it
sounds like you already do).
All the best,
David
--
Kristian