Re: Features of the JUnit test execution harness
Myrna van Lunteren wrote: On 2/2/06, *Andreas Korneliussen* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: I think the work currently done on DERBY-874 was mainly to improve the DerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw that away. However I do propose to change DerbyJUnitTest to move out everything about configuration into a separate class. cool. thx for the reply. I now noticed that the wiki says all suggestions are to be put on the list, so here I go rather than plopping them directly on the wiki: Great feature list. What I think is important is that we provide a common mechanism to get access to the configuration of the test. If someone then need to do some framework/version specific logic, they have a mechanism to get i.e the framework enum, and can write logic based on that. I think the following could qualify as 'more details' to the jvm, framework, version specific logic: 1. jvm-specific: 1.1. not all parameters are consistent for all jvms. Think here of jit settings / configurations, memory settings. For j2ME testing, that jvm doesn't come with a DriverManager implementation, so already from the start you know you have to go with DataSources. So I guess what you are saying is that if the test framework provides a common mechanism to give a Connection to a derby database, it should go through a DataSource, instead of using DriverManager ? 1.2. Different versions of a vendor's jvm also may have slightly different implementations resulting in slightly different behavior - e.g. of the order of rows, for instance, or rounding of certain numeric values. 1.3. Some behavior will be only supported by later versions... 2. version specific. This really falls back to the discussion on ...(can't find right now, raman's working on it, I think)... re mixed version testing. I think the conclusion was that the harness needs a way to cope with results from newer servers and clients - if they differ from results with same versions as the harness. 3. framework specific The tests needs to be able to cope with the following 3.1. different client drivers (e.g. DerbyClient, IBM Universal JDBC Driver) 3.2. server may need to be started by harness, or not 3.3. server may be on the localhost, or be running on a remote machine. certain individual tests may not be able to run in with this mechanism... 3.4 should be able to have the harness start the server in a differrent jvm. 4. one thing the current harness has no way of doing is to cope with different OSs. For instance, sometimes there are slight differences in behaviour of the same jvm version on different OSs. Like slightly different error messages (although this may be irrelevant if we're not gathering comparing output). I think the following details would be useful (in addition to the above and item 1 on the wiki): - there must be a way to skip individual tests without causing an error but with an informational message for certain configurations. eg. absence of optional jars (specifically thinking of db2jcc.jar), unsupported functionality with older jvms..., or when there is a problem that's being worked on, or that's been referred to some other organization ( e.g. in the case of jvm bugs, OS bugs...). - some way to compare runtimestatistics. Currently this is done by comparing the output, I have a hard time thinking of another mechanism. I am not sure which runtimestatistics you think of. Which output ? Output from a SQL statement ? Thanks --Andreas
Re: Features of the JUnit test execution harness
So I guess what you are saying is that if the test framework provides a common mechanism to give a Connection to a derby database, it should go through a DataSource, instead of using DriverManager ? I think we will need both mechanisms to get connection - using DataSource and DriverManager. J2ME/CDC/Foundation Profile is based on JSR169 for JDBC, which does not have java.sql.DriverManager class. So, here we need a mechanism to get a connection using javax.sql.DataSource. Using DataSource to get connection will not work in case of j9 vms (not foundation profile). In this case, database_enabler.jar is used for JDBC support and this does not have javax.sql.DataSource class. I'd appreciate if someone else familiar in this area can confirm this. Thanks, Deepa
Re: Features of the JUnit test execution harness
Deepa Remesh wrote: So I guess what you are saying is that if the test framework provides a common mechanism to give a Connection to a derby database, it should go through a DataSource, instead of using DriverManager ? I think we will need both mechanisms to get connection - using DataSource and DriverManager. J2ME/CDC/Foundation Profile is based on JSR169 for JDBC, which does not have java.sql.DriverManager class. So, here we need a mechanism to get a connection using javax.sql.DataSource. Using DataSource to get connection will not work in case of j9 vms (not foundation profile). In this case, database_enabler.jar is used for JDBC support and this does not have javax.sql.DataSource class. I'd appreciate if someone else familiar in this area can confirm this. Correct, the other j9 profile is implementing the standard JDBC 2.1 api and does not supply the optional JDBC 2.0 extensions which include DataSource etc. Derby currently does support this environment and so the test harness needs to as well. In addition we will need to test getting connections from DriverManager, so it needs to be supported in some way. Ideally the majority of tests should be written to be indepedent of how a connection is obtained, which is how it is today. Then all of the tests could be run with a conneciton from a DataSource, the DriverManager, a Driver, a PooledConnection or a XAConnection. Dan.
Re: Features of the JUnit test execution harness
Deepa Remesh wrote: So I guess what you are saying is that if the test framework provides a common mechanism to give a Connection to a derby database, it should go through a DataSource, instead of using DriverManager ? I think we will need both mechanisms to get connection - using DataSource and DriverManager. J2ME/CDC/Foundation Profile is based on JSR169 for JDBC, which does not have java.sql.DriverManager class. So, here we need a mechanism to get a connection using javax.sql.DataSource. Using DataSource to get connection will not work in case of j9 vms (not foundation profile). In this case, database_enabler.jar is used for JDBC support and this does not have javax.sql.DataSource class. I'd appreciate if someone else familiar in this area can confirm this. Thanks, Deepa I'm not familiar with this area, but wouldn't it in general be best to have a single method in the harness that returns a valid connection? If the test is running in a J2ME/CDC/Foundation environment, it uses DataSource to get a connection, if it is in a j9 environment then it should use whatever is supported there. For most tests, how the connection is obtained is irrelevant, as long as it works! I see that a few tests, for instance a test that is testing the different ways to get connections, would have the need to override such a standard method, but that's something that can be added when these tests are to be written in JUnit-style. However, depending on JVM and (un)available jars, some of the test cases in such a suite would have to be skipped anyway. Instead of learning the harness how to get connections in X different ways, I would consider just extracting the required connection information (driver/framework prefix, database name, server, port etc) from the harness and then code how I get the connection in my test *if this is required*. The harness should only have to know about a minimal set of ways to get connections. With minimal set, I mean it has to know about one way to get a connection for all jvms/platforms (that we support or have the itch for). It does not need to know all the ways for all jvms/platforms. From the previous mails, the harness would have to know how to use DataSource and DriverManager. Are there jvms/platforms where neither of these will be able to produce a connection? -- Kristian
Re: Features of the JUnit test execution harness
Myrna van Lunteren wrote: On 1/31/06, *Kristian Waagan* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Differences in output should be irrelevant. Although not what you mentioned above, the issue of (execution) control is very relevant. The logic for running the tests multiple times, each time with a different setup/environment must be located somewhere. I think Andreas' proposal of introducing a separate JUnit test type (see http://www.nabble.com/running-JUnit-tests-t887682.html#a2300670) makes sense, as it gives us more freedom w.r.t. handling of JUnit tests. Yes, that proposal made sense to me. I personally like the approach of having a class for various/different configurations. Although that could get out of hand. Does this 'throw away' the work that Rick is doing on DERBY-874? I think the work currently done on DERBY-874 was mainly to improve the DerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw that away. However I do propose to change DerbyJUnitTest to move out everything about configuration into a separate class. Following Andreas' approach we'd still be able to run the individual tests separately, yes? Yes - definetly. Andreas
Re: Features of the JUnit test execution harness
On 2/2/06, Andreas Korneliussen [EMAIL PROTECTED] wrote: I think the work currently done on DERBY-874 was mainly to improve theDerbyJUnitTest's JavaDoc, and to log exceptions. So I would not throw that away.However I do propose to change DerbyJUnitTest to move out everythingabout configuration into a separate class. cool. thx for the reply. I now noticed that the wiki says all suggestions are to be put on the list, so here I go rather than plopping them directly on the wiki: I think the following could qualify as 'more details' to the jvm, framework, version specific logic: 1. jvm-specific: 1.1. not all parameters are consistent for all jvms. Think here of jit settings / configurations, memory settings. For j2ME testing, that jvm doesn't come with a DriverManager implementation, so already from the start you know you have to go with DataSources. 1.2. Different versions of a vendor's jvm also may have slightly different implementations resulting in slightly different behavior - e.g. of the order of rows, for instance, or rounding of certain numeric values. 1.3. Some behavior will be only supported by later versions... 2.version specific. This really falls back to the discussion on...(can't find right now, raman's working on it, I think)... re mixed version testing. I think the conclusion was that the harness needs a way to cope with results from newer servers and clients - if they differ from results with same versions as the harness. 3. framework specific The tests needs to be able to cope with the following 3.1. different client drivers (e.g. DerbyClient, IBM Universal JDBC Driver) 3.2. server may need to be started by harness, or not 3.3. server may be on the localhost, or be running on a remote machine. certain individual tests may not be able to run in with this mechanism... 3.4 should be able to have the harness start the server in a differrent jvm. 4. one thing the current harness has no way of doing is to cope with different OSs. For instance, sometimes there are slight differences in behaviour of the same jvm version on different OSs. Like slightly different error messages (although this may be irrelevant if we're not gathering comparing output). I think the following details would be useful (in addition to the above and item 1 on the wiki): - there must be a way to skip individual tests without causing an error but with an informational message for certain configurations. eg. absence of optional jars (specifically thinking of db2jcc.jar), unsupported functionality with older jvms..., or when there is a problem that's being worked on, or that's been referred to some other organization ( e.g. in the case of jvm bugs, OS bugs...). - some way to compare runtimestatistics. Currently this is done by comparing the output, I have a hard time thinking of another mechanism. Ok, that'll do for now...:-) Myrna
Re: Features of the JUnit test execution harness
On 1/31/06, Kristian Waagan [EMAIL PROTECTED] wrote: Differences in output should be irrelevant. Although not what youmentioned above, the issue of (execution) control is very relevant. The logic for running the tests multiple times, each time with a differentsetup/environment must be located somewhere. I think Andreas' proposalof introducing a separate JUnit test type (see http://www.nabble.com/running-JUnit-tests-t887682.html#a2300670) makessense, as it gives us more freedom w.r.t. handling of JUnit tests. Yes, that proposal made sense to me. Ipersonallylike the approach of having a class for various/different configurations. Although that could get out of hand. Does this 'throw away' the work that Rick is doing on DERBY-874? Following Andreas' approach we'd still be able to run the individual tests separately, yes? Thanks for the answer Myrna. And good work on the readme! credit due: many people have reviewed, and improved it... snipRight now I feel that it is very hard to start writing a new JUnitharness, since the old harness seem to contain a lot of functionality and magic. I wonder if it would be best to just startconverting/writing some tests and create a framework along the way. Yes, a new harness is hard, and yes, the old harness hasbeen made to adapt to many different needs...and we should take those into account. (I'll add some thoughts on that wiki soon). But the old harness has grownunwieldy and hard to maintain and is slow.To take what we learnedandcraft it ontosomething new will be better, though daunting. Some ideas of Dan's are in this thread from about a year ago: http://mail-archives.apache.org/mod_mbox/db-derby-dev/200503.mbox/[EMAIL PROTECTED] The above approach would probably result in several major rewritings ofthe new framework and the existing JUnit tests, but it might be better than thinking out everything at once (that's hard!). Perhaps I willrewrite an old test I have been working a little on, and put it out forreview and comments, unless someone else beats me to it ;) --Kristian Personally, I do betterwhen I have code to prod and modify through iterations... I'd be interested to review. Myrna
Re: Features of the JUnit test execution harness
On 1/24/06, Kristian Waagan [EMAIL PROTECTED] wrote: Kristian Waagan wrote:4) Should it be possible to run single JUnit test/suite from the command line 2 yeahs snipped, 3rd: (Dan:) I think it's essential that a single test can be run. I don't think it's essential that it is from test's main method. The harness produces the useful functionality of setting up the correct environment for the test, not sure if that could be duplicated in evey main method for every test.Good point Dan.Citing Dan: ...harness produces the useful functionality of setting upthe correct environment for the test, Can somebody please explain to me what is actually done by the harness?I did have a quick look at the harness code, but I would need to studyit a bit more to understand what it actually does. Stating what the harness does when it comes to setting up the environment for tests wouldbe good for the general understanding of the test execution process. --- I've had this messages 'starred' in gmail for a bit, sorry for replying only now. There's no quick way to describe what the 'functiontest' harness does - I've tried to describe what I understand in the past (see the java/testing/README.htm)... Maybe if you have specific questions I'll be able to answer better. The main thing I think that the functiontest harness does is set up all sorts of fun properties for the tests to use, sothe same test can run in a number of different ways - in a suite, by itself, with different encryption algorithms, with encoding, locales, frameworks, with java.sql.DriverManager or instead with DataSource; to run in sequence or separately, start a new database or not...Depending on these properties, more or less 'magic' is done - for example, network server gets started or not, files needed for the test get copied around... Properties in a suite override properties set for individual tests lower level. Also, it is able to figure out what jvm is used and add specific (non-standard)properties as needed or applicable. A big part of it is to control, and eliminate uninteresting differences in the output, but that would be irrelevant with Junit, yes? Does this help at all? But maybe that wasn't what Dan meant? Myrna
Re: Features of the JUnit test execution harness
Silly question(s): with the Junit testsij is not used? if not, and we're going to convert all tests to junit, should there be a new test to test the tool ij specifically (at some point) ? Myrna
Re: Features of the JUnit test execution harness
Hi Myrna, I don't think we've discussed yet how or when we would convert existing tests to JUnit. I agree that ij needs to continue to be tested. Regards, -Rick Myrna van Lunteren wrote: Silly question(s): with the Junit tests ij is not used? if not, and we're going to convert all tests to junit, should there be a new test to test the tool ij specifically (at some point) ? Myrna
Re: Features of the JUnit test execution harness
Myrna van Lunteren wrote: On 1/24/06, *Kristian Waagan* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Kristian Waagan wrote: 4) Should it be possible to run single JUnit test/suite from the command line 2 yeahs snipped, 3rd: (Dan:) I think it's essential that a single test can be run. I don't think it's essential that it is from test's main method. The harness produces the useful functionality of setting up the correct environment for the test, not sure if that could be duplicated in evey main method for every test. Good point Dan. Citing Dan: ...harness produces the useful functionality of setting up the correct environment for the test, Can somebody please explain to me what is actually done by the harness? I did have a quick look at the harness code, but I would need to study it a bit more to understand what it actually does. Stating what the harness does when it comes to setting up the environment for tests would be good for the general understanding of the test execution process. --- I've had this messages 'starred' in gmail for a bit, sorry for replying only now. There's no quick way to describe what the 'functiontest' harness does - I've tried to describe what I understand in the past (see the java/testing/README.htm)... Maybe if you have specific questions I'll be able to answer better. The main thing I think that the functiontest harness does is set up all sorts of fun properties for the tests to use, so the same test can run in a number of different ways - in a suite, by itself, with different encryption algorithms, with encoding, locales, frameworks, with java.sql.DriverManager or instead with DataSource; to run in sequence or separately, start a new database or not...Depending on these properties, more or less 'magic' is done - for example, network server gets started or not, files needed for the test get copied around... Properties in a suite override properties set for individual tests lower level. Also, it is able to figure out what jvm is used and add specific (non-standard) properties as needed or applicable. A big part of it is to control, and eliminate uninteresting differences in the output, but that would be irrelevant with Junit, yes? Differences in output should be irrelevant. Although not what you mentioned above, the issue of (execution) control is very relevant. The logic for running the tests multiple times, each time with a different setup/environment must be located somewhere. I think Andreas' proposal of introducing a separate JUnit test type (see http://www.nabble.com/running-JUnit-tests-t887682.html#a2300670) makes sense, as it gives us more freedom w.r.t. handling of JUnit tests. Does this help at all? But maybe that wasn't what Dan meant? Myrna Thanks for the answer Myrna. And good work on the readme! I just skimmed through it now, but I will take the time to read it thoroughly soon. Right now I feel that it is very hard to start writing a new JUnit harness, since the old harness seem to contain a lot of functionality and magic. I wonder if it would be best to just start converting/writing some tests and create a framework along the way. The above approach would probably result in several major rewritings of the new framework and the existing JUnit tests, but it might be better than thinking out everything at once (that's hard!). Perhaps I will rewrite an old test I have been working a little on, and put it out for review and comments, unless someone else beats me to it ;) -- Kristian
Re: Features of the JUnit test execution harness
Rick Hillegas wrote:
Kristian Waagan wrote:
2) Should we support setting up a shared test fixture for a set of
tests?
This is a common issue with JUnit, and there are mechanisms to handle
it. For instance, we could let tests that require this to wrap itself
in a TestSetup instance (as described in the JUNit FAQ). Letting
DerbyJUnitTest extend TestSetup is another solution, but it might
have unwanted side-effects.
Hi Kristian,
I'm afraid I don't understand the issues here. I suspect your JUnit
experience is much deeper than mine. What is the advantage to having
DerbyJUnitTest extend TestSetup vs TestCase? It's hard to tell from the
FAQ. TestCase provides setup() and teardown() brackets like TestSetup.
In addition, TestCase provides the assertion machinery. What unwanted
side-effects do you have in mind?
Thanks,
-Rick
Hello Rick,
Your concern is valid. Having our Derby JUnit superclass
(DerbyJUnitTest) extend TestSetup is not a good idea.
The proper way to use it is as described in the FAQ, or to create a
separate TestSetup subclass. Just to make sure we are on the same track,
the reason for using TestSetup is to provide the test with a one-time
setUp() and a one-time tearDown() method. If your JUnit class has 3 test
methods, this sequence would be run (ordering of the test methods may
not be fixed):
TestSetup.setUp()
TestCase.setUp
TestCase.testMethodA()
TestCase.tearDown()
TestCase.setUp
TestCase.testMethodB()
TestCase.tearDown()
TestCase.setUp
TestCase.testMethodC()
TestCase.tearDown()
TestSetup.tearDown()
My concern is that many of our tests do not want to do a complete setUp
and tearDown for each test, as this can cause the runtime for the test
to be too long. By using the one-time methods, we can create a common
setup for all the tests in the suite.
To illustrate the two ways to use TestSetup, consider the following
suite() methods:
From the JUnit FAQ, the following example demonstrate how to use
TestSetup without creating a separate class.
public static Test suite() {
TestSuite suite = new TestSuite();
suite.addTest(SomeTest.suite());
suite.addTest(AnotherTest.suite());
TestSetup wrapper = new TestSetup(suite) {
protected void setUp() {
oneTimeSetUp();
}
protected void tearDown() {
oneTimeTearDown();
}
};
return wrapper;
}
public static void oneTimeSetUp() {
// one-time initialization code
}
public static void oneTimeTearDown() {
// one-time cleanup code
}
The other way.
public static Test suite() {
// Add all test methods in the class.
TestSuite suite = new TestSuite(MyJUnitTestClass.class);
// Create a new TestSetup subclass for our test.
// This would have a setUp() and a tearDown() method, and possibly
// several constructors and/or methods to control the setup to be
// created.
return new MyJUnitTestClassSetup(suite);
}
For the example above, there are lots of variants. We can return a suite
that runs all tests in the test with X different setups, and
additionally run a subset of the tests with a special setup. To do this,
we must either create several TestSetup subclasses, or parameterize the
TestSetup subclass.
Did this make any more sense?
--
Kristian
Re: Features of the JUnit test execution harness
I think there's a lot of value in having a suite-level setup mechanism,
I totally agree that doing setup for each test can be overkill and I am
often finding ways to work around it.
David
Kristian Waagan wrote:
Rick Hillegas wrote:
Kristian Waagan wrote:
2) Should we support setting up a shared test fixture for a set of
tests?
This is a common issue with JUnit, and there are mechanisms to
handle it. For instance, we could let tests that require this to
wrap itself in a TestSetup instance (as described in the JUNit FAQ).
Letting DerbyJUnitTest extend TestSetup is another solution, but it
might have unwanted side-effects.
Hi Kristian,
I'm afraid I don't understand the issues here. I suspect your JUnit
experience is much deeper than mine. What is the advantage to having
DerbyJUnitTest extend TestSetup vs TestCase? It's hard to tell from
the FAQ. TestCase provides setup() and teardown() brackets like
TestSetup. In addition, TestCase provides the assertion machinery.
What unwanted side-effects do you have in mind?
Thanks,
-Rick
Hello Rick,
Your concern is valid. Having our Derby JUnit superclass
(DerbyJUnitTest) extend TestSetup is not a good idea.
The proper way to use it is as described in the FAQ, or to create a
separate TestSetup subclass. Just to make sure we are on the same track,
the reason for using TestSetup is to provide the test with a one-time
setUp() and a one-time tearDown() method. If your JUnit class has 3 test
methods, this sequence would be run (ordering of the test methods may
not be fixed):
TestSetup.setUp()
TestCase.setUp
TestCase.testMethodA()
TestCase.tearDown()
TestCase.setUp
TestCase.testMethodB()
TestCase.tearDown()
TestCase.setUp
TestCase.testMethodC()
TestCase.tearDown()
TestSetup.tearDown()
My concern is that many of our tests do not want to do a complete setUp
and tearDown for each test, as this can cause the runtime for the test
to be too long. By using the one-time methods, we can create a common
setup for all the tests in the suite.
To illustrate the two ways to use TestSetup, consider the following
suite() methods:
From the JUnit FAQ, the following example demonstrate how to use
TestSetup without creating a separate class.
public static Test suite() {
TestSuite suite = new TestSuite();
suite.addTest(SomeTest.suite());
suite.addTest(AnotherTest.suite());
TestSetup wrapper = new TestSetup(suite) {
protected void setUp() {
oneTimeSetUp();
}
protected void tearDown() {
oneTimeTearDown();
}
};
return wrapper;
}
public static void oneTimeSetUp() {
// one-time initialization code
}
public static void oneTimeTearDown() {
// one-time cleanup code
}
The other way.
public static Test suite() {
// Add all test methods in the class.
TestSuite suite = new TestSuite(MyJUnitTestClass.class);
// Create a new TestSetup subclass for our test.
// This would have a setUp() and a tearDown() method, and possibly
// several constructors and/or methods to control the setup to be
// created.
return new MyJUnitTestClassSetup(suite);
}
For the example above, there are lots of variants. We can return a suite
that runs all tests in the test with X different setups, and
additionally run a subset of the tests with a special setup. To do this,
we must either create several TestSetup subclasses, or parameterize the
TestSetup subclass.
Did this make any more sense?
--
Kristian
begin:vcard
fn:David W Van Couvering
n:Van Couvering;David W
org:Sun Microsystems, Inc.;Database Technology Group
email;internet:[EMAIL PROTECTED]
title:Senior Staff Software Engineer
tel;work:510-550-6819
tel;cell:510-684-7281
x-mozilla-html:TRUE
version:2.1
end:vcard
Re: Features of the JUnit test execution harness
Thanks, Kristian. This is indeed helpful. My takeaway is that our JUnit
primer should recommend something like this as the pattern if developers
need to move the setup()/teardown() brackets outside a cluster of tests.
Regards,
-Rick
Kristian Waagan wrote:
Rick Hillegas wrote:
Kristian Waagan wrote:
2) Should we support setting up a shared test fixture for a set of
tests?
This is a common issue with JUnit, and there are mechanisms to
handle it. For instance, we could let tests that require this to
wrap itself in a TestSetup instance (as described in the JUNit
FAQ). Letting DerbyJUnitTest extend TestSetup is another solution,
but it might have unwanted side-effects.
Hi Kristian,
I'm afraid I don't understand the issues here. I suspect your JUnit
experience is much deeper than mine. What is the advantage to having
DerbyJUnitTest extend TestSetup vs TestCase? It's hard to tell from
the FAQ. TestCase provides setup() and teardown() brackets like
TestSetup. In addition, TestCase provides the assertion machinery.
What unwanted side-effects do you have in mind?
Thanks,
-Rick
Hello Rick,
Your concern is valid. Having our Derby JUnit superclass
(DerbyJUnitTest) extend TestSetup is not a good idea.
The proper way to use it is as described in the FAQ, or to create a
separate TestSetup subclass. Just to make sure we are on the same
track, the reason for using TestSetup is to provide the test with a
one-time setUp() and a one-time tearDown() method. If your JUnit class
has 3 test methods, this sequence would be run (ordering of the test
methods may not be fixed):
TestSetup.setUp()
TestCase.setUp
TestCase.testMethodA()
TestCase.tearDown()
TestCase.setUp
TestCase.testMethodB()
TestCase.tearDown()
TestCase.setUp
TestCase.testMethodC()
TestCase.tearDown()
TestSetup.tearDown()
My concern is that many of our tests do not want to do a complete
setUp and tearDown for each test, as this can cause the runtime for
the test to be too long. By using the one-time methods, we can create
a common setup for all the tests in the suite.
To illustrate the two ways to use TestSetup, consider the following
suite() methods:
From the JUnit FAQ, the following example demonstrate how to use
TestSetup without creating a separate class.
public static Test suite() {
TestSuite suite = new TestSuite();
suite.addTest(SomeTest.suite());
suite.addTest(AnotherTest.suite());
TestSetup wrapper = new TestSetup(suite) {
protected void setUp() {
oneTimeSetUp();
}
protected void tearDown() {
oneTimeTearDown();
}
};
return wrapper;
}
public static void oneTimeSetUp() {
// one-time initialization code
}
public static void oneTimeTearDown() {
// one-time cleanup code
}
The other way.
public static Test suite() {
// Add all test methods in the class.
TestSuite suite = new TestSuite(MyJUnitTestClass.class);
// Create a new TestSetup subclass for our test.
// This would have a setUp() and a tearDown() method, and possibly
// several constructors and/or methods to control the setup to be
// created.
return new MyJUnitTestClassSetup(suite);
}
For the example above, there are lots of variants. We can return a
suite that runs all tests in the test with X different setups, and
additionally run a subset of the tests with a special setup. To do
this, we must either create several TestSetup subclasses, or
parameterize the TestSetup subclass.
Did this make any more sense?
--
Kristian
Re: Features of the JUnit test execution harness
Daniel John Debrunner wrote (2006-01-24 20:23:27): The other issue is that we have a great opportunity to start out with JUnit tests that follow a consistent pattern and provide a great example for others to follow. +1 -- Bernt Marius Johnsen, Database Technology Group, Staff Engineer, Technical Lead Derby/Java DB Sun Microsystems, Trondheim, Norway pgpVWaBm3AABa.pgp Description: PGP signature
Re: Features of the JUnit test execution harness
I have to agree we're missing a well-defined pattern and associated guidebook on how to do this right. And I agree we should get this done before too many more JUnit tests are written. David Daniel John Debrunner wrote: Daniel John Debrunner wrote: David W. Van Couvering wrote: I agree with you that is disconcerting, but can't JUnit tests be written that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest cleaned up to everyone's satisfaction? Depends, at some point you put the potential burden of fixing all those new tests on the person doing the cleanup, that should have been part of the original submission. The other issue is that we have a great opportunity to start out with JUnit tests that follow a consistent pattern and provide a great example for others to follow. If we add a number of tests now that haphazardly use the methods in DerbyJUnitTest (because they are not well commented) then we have a set of tests like the current ones. No pattern, no obvious starting point leading to people inventing their own ways to get connections, run ddl, handle exceptions etc. Dan. begin:vcard fn:David W Van Couvering n:Van Couvering;David W org:Sun Microsystems, Inc.;Database Technology Group email;internet:[EMAIL PROTECTED] title:Senior Staff Software Engineer tel;work:510-550-6819 tel;cell:510-684-7281 x-mozilla-html:TRUE version:2.1 end:vcard
Re: Features of the JUnit test execution harness
I apologize for the radio silence on this topic. I have been on the road and unable to address these concerns. Today I will put some effort into beefing up the comments in DerbyJUnitTest and handling the swallowed exceptions. I agree that we need a primer for writing JUnit tests. I can mock up such a document and we can collectively polish it. I would like to start out with something small. I believe that other people are working on a larger JUnit framework and I would like their help in building up this document. Thanks for your patience, -Rick David W. Van Couvering wrote: I agree with you that is disconcerting, but can't JUnit tests be written that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest cleaned up to everyone's satisfaction? BTW, if you don't like the rampant catching and ignoring, is anything preventing you from going in and fixing it? :) I suppose we need to hear from Rick about his feelings around this before we go and reverse his changes... David Daniel John Debrunner wrote: I would like my concerns to be cleared up before too many tests use the current Junit facility. The rampant catching and ignoring of SQLExceptions is not good. Dan.
Re: Features of the JUnit test execution harness
Daniel John Debrunner wrote: Daniel John Debrunner wrote: David W. Van Couvering wrote: I agree with you that is disconcerting, but can't JUnit tests be written that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest cleaned up to everyone's satisfaction? Depends, at some point you put the potential burden of fixing all those new tests on the person doing the cleanup, that should have been part of the original submission. The other issue is that we have a great opportunity to start out with JUnit tests that follow a consistent pattern and provide a great example for others to follow. If we add a number of tests now that haphazardly use the methods in DerbyJUnitTest (because they are not well commented) then we have a set of tests like the current ones. No pattern, no obvious starting point leading to people inventing their own ways to get connections, run ddl, handle exceptions etc. One thing that would be great is if there was a well defined way to have tests handle jvm specific, framework specific or version specific logic, since there will have to be some replacement for the many properties in the test harness and the individual test checks for framework that we have now. I suggested something like this for the client/server compatibility testing, but did not consider JUnit. It would be good to have something laid out for JUnit testing so that tests are checking whether functionality is suppported and the details of whether that is because of the framework, jvm or whatever are not spread around all the tests. http://www.nabble.com/Server-and-client-compatibility-test-for-10.2-and-10.1-t913020.html#a2499854 Kathey
Re: Features of the JUnit test execution harness
One thing that would be great is if there was a well defined way to have tests handle jvm specific, framework specific or version specific logic, since there will have to be some replacement for the many properties in the test harness and the individual test checks for framework that we have now. I would also like to know how JUnit tests will map the current test harness properties. My concern is specifically how to run JUnit tests with CDC/Foundation/jsr169. I found the following test failing in this environment: derbyall/derbyall.fail:junitTests/lang/LangSuite.java The reason is the test uses DerbyJUnitTest.getConnection method which uses Driver class which is not available in jsr169. For tests which use the current test harness, ij.startJBMS() method is used to get connections which handles getting connection on different platforms. Can we have a similar central method to get connections in the JUnit harness? I think it would be good to get JUnit tests running in this environment before we move more tests. Thanks, Deepa
Re: Features of the JUnit test execution harness
Hi Deepa, Thanks for bringing this up. I agree that being able to run JUnit tests on the small device configuration would be a good idea. Regards, -Rick Deepa Remesh wrote: One thing that would be great is if there was a well defined way to have tests handle jvm specific, framework specific or version specific logic, since there will have to be some replacement for the many properties in the test harness and the individual test checks for framework that we have now. I would also like to know how JUnit tests will map the current test harness properties. My concern is specifically how to run JUnit tests with CDC/Foundation/jsr169. I found the following test failing in this environment: derbyall/derbyall.fail:junitTests/lang/LangSuite.java The reason is the test uses DerbyJUnitTest.getConnection method which uses Driver class which is not available in jsr169. For tests which use the current test harness, ij.startJBMS() method is used to get connections which handles getting connection on different platforms. Can we have a similar central method to get connections in the JUnit harness? I think it would be good to get JUnit tests running in this environment before we move more tests. Thanks, Deepa
Re: Features of the JUnit test execution harness
David W. Van Couvering wrote: Kristian Waagan wrote: 4) Should it be possible to run single JUnit test/suite from the command line by simply invoking the main method? Personally, I think this should be possible. This require us to remove the current use of main for running a test in the old test harness, which I believe is doable with a few changes (in the old harness). I think this is essential for debugging and incremental testing. +1 on this idea. I think it's essential that a single test can be run. I don't think it's essential that it is from test's main method. The harness produces the useful functionality of setting up the correct environment for the test, not sure if that could be duplicated in evey main method for every test. Dan.
Re: Features of the JUnit test execution harness
Daniel John Debrunner wrote: David W. Van Couvering wrote: Kristian Waagan wrote: 4) Should it be possible to run single JUnit test/suite from the command line by simply invoking the main method? Personally, I think this should be possible. This require us to remove the current use of main for running a test in the old test harness, which I believe is doable with a few changes (in the old harness). I think this is essential for debugging and incremental testing. +1 on this idea. I think it's essential that a single test can be run. I don't think it's essential that it is from test's main method. The harness produces the useful functionality of setting up the correct environment for the test, not sure if that could be duplicated in evey main method for every test. Good point Dan. Citing Dan: ...harness produces the useful functionality of setting up the correct environment for the test, Can somebody please explain to me what is actually done by the harness? I did have a quick look at the harness code, but I would need to study it a bit more to understand what it actually does. Stating what the harness does when it comes to setting up the environment for tests would be good for the general understanding of the test execution process. -- Kristian Dan.
Re: Features of the JUnit test execution harness
David W. Van Couvering wrote: Hi, Kristian, thanks for your questions. My one overriding thought is we should take this in incremental steps -- do all of these questions need to be answered before we can rewrite a single old canon-based test to a JUnit test? Can some of these questions be deferred? Yes, they can be deferred. I did not intend to say that we should try to fix/implement all issues/features at in one step. My main point of concern, is that it is too hard to write JUnit tests now, because there is so little information available. Just have a look at the number of JUnit tests that have been added to the repository - it sure ain't many! The conversion process, which I understand is fully based on it's my itch initiatives, is also moving along very, very slowly. Kristian Waagan wrote: Hello, I have wanted to rewrite a old canon based test to a JUnit test for a while, but each time I get started I always run into issues that we do not have guidelines for yet. So, I have a few questions I would like some feedback on. I know there are a lot of issues to consider, far more then the one I mention here, but I hope to get a feeling of the general direction we want to move in. 1) Should we support specific ordering of tests (within a single class)? This feature is wanted if a number of tests operate on shared data and each test method can be seen as a step in the test. If the order is mixed up, the intermediate results will not be as expected and all the tests may fail. In general, assuming anything about the execution order of tests is strongly disencouraged. However, since we are testing a database system, we might want to allow for this. This feature can influence the both the size of the test methods and the execution time for the test/suite. I thought if you name your tests correctly (e.g. test_010, test_020, etc.) then they are run in that order. Or is that not the case because all method names on a class are hashed? This is not the case. Tests are not ordered by method names. I do not know the implementation details about ordering when using reflection. I do know that when adding one test at a time in suite(), they are stored in a Vector and thus ordered (and executed in order). But please note that this is not a documented feature of JUnit. Assuming that's not the case, it seems to me that if you have specific ordering requirements, you could use multiple test classes, create a suite of those classes, and use the TestSetup mechanism you describe below to do any setup work prior to running the suite. Might work. In my opinion, the best thing is to write each test to be totally independent, but I am looking for ways to handle tests that are dependent on each other because I suspect that need will arise. 2) Should we support setting up a shared test fixture for a set of tests? This is a common issue with JUnit, and there are mechanisms to handle it. For instance, we could let tests that require this to wrap itself in a TestSetup instance (as described in the JUNit FAQ). Letting DerbyJUnitTest extend TestSetup is another solution, but it might have unwanted side-effects. I think there is a lot of value to having something like this (see above for an example). 3) Many tests just want a database to work on. Other tests want to create their own database due to specific requirements. How do we make this easy for the developer? I think if you have a TestSetup mechanism, then a suite of tests can depend on the fact that a database has been created. Creating a database is pretty darn easy already; are there particular things that are repetetive or error-prone? No. As long as you have the driver name and the connection string (including the correct protocol and database path/name), it is very easy. This information must be passed on to the test setup class when suite() is run, and with the current harness implementation (with respect to JUnit tests) this information is not available then. There are easy workarounds, but they are not written down anywhere... This is actually a matter of question 6. 4) Should it be possible to run single JUnit test/suite from the command line by simply invoking the main method? Personally, I think this should be possible. This require us to remove the current use of main for running a test in the old test harness, which I believe is doable with a few changes (in the old harness). I think this is essential for debugging and incremental testing. +1 on this idea. 5) Is the ultimate goal to end up with a pure JUnit based test harness? If this is the goal, we should design for this and accept some extra work for plugging JUnit tests into the old test harness. I can see us getting there over time, but this is a Very Big Effort. Are there particular things we have to decide upon now in order for this to be possible in the future? I don't know of any, but someone
Re: Features of the JUnit test execution harness
On 1/24/06, Kristian Waagan [EMAIL PROTECTED] wrote: David W. Van Couvering wrote: Hi, Kristian, thanks for your questions.My one overriding thought is we should take this in incremental steps -- do all of these questions need to be answered before we can rewrite a single old canon-based test to a JUnit test?Can some of these questions be deferred?Yes, they can be deferred. I did not intend to say that we should try tofix/implement all issues/features at in one step.My main point of concern, is that it is too hard to write JUnit tests now, because there is so little information available. Just have a lookat the number of JUnit tests that have been added to the repository - itsure ain't many! The conversion process, which I understand is fully based on it's my itch initiatives, is also moving along very, very slowly. +1 - I have now written some derby jUnit tests myself and I agree that it may not be that obvious from the beginning (different paradigm from canon-based tests) - I used jUnit before so it did help. I'm currently scratching a few itches but I'd be glad to post some instructions unless some itch-scratching idle volunteer want to take into that heroic task ;-) - jUnit aims at making unit testing easier and more effective therefore we just need to make the Derby jUnit adoption easier and more effective ;-)
Re: Features of the JUnit test execution harness
Francois Orsini wrote: On 1/24/06, *Kristian Waagan* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: David W. Van Couvering wrote: Hi, Kristian, thanks for your questions. My one overriding thought is we should take this in incremental steps -- do all of these questions need to be answered before we can rewrite a single old canon-based test to a JUnit test? Can some of these questions be deferred? Yes, they can be deferred. I did not intend to say that we should try to fix/implement all issues/features at in one step. My main point of concern, is that it is too hard to write JUnit tests now, because there is so little information available. Just have a look at the number of JUnit tests that have been added to the repository - it sure ain't many! The conversion process, which I understand is fully based on it's my itch initiatives, is also moving along very, very slowly. +1 - I have now written some derby jUnit tests myself and I agree that it may not be that obvious from the beginning (different paradigm from canon-based tests) - I used jUnit before so it did help. I'm currently scratching a few itches but I'd be glad to post some instructions unless some itch-scratching idle volunteer want to take into that heroic task ;-) - jUnit aims at making unit testing easier and more effective therefore we just need to make the Derby jUnit adoption easier and more effective ;-) I would like my concerns to be cleared up before too many tests use the current Junit facility. The rampant catching and ignoring of SQLExceptions is not good. Dan.
Re: Features of the JUnit test execution harness
I agree with you that is disconcerting, but can't JUnit tests be written that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest cleaned up to everyone's satisfaction? BTW, if you don't like the rampant catching and ignoring, is anything preventing you from going in and fixing it? :) I suppose we need to hear from Rick about his feelings around this before we go and reverse his changes... David Daniel John Debrunner wrote: I would like my concerns to be cleared up before too many tests use the current Junit facility. The rampant catching and ignoring of SQLExceptions is not good. Dan. begin:vcard fn:David W Van Couvering n:Van Couvering;David W org:Sun Microsystems, Inc.;Database Technology Group email;internet:[EMAIL PROTECTED] title:Senior Staff Software Engineer tel;work:510-550-6819 tel;cell:510-684-7281 x-mozilla-html:TRUE version:2.1 end:vcard
Re: Features of the JUnit test execution harness
David W. Van Couvering wrote: I agree with you that is disconcerting, but can't JUnit tests be written that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest cleaned up to everyone's satisfaction? Depends, at some point you put the potential burden of fixing all those new tests on the person doing the cleanup, that should have been part of the original submission. BTW, if you don't like the rampant catching and ignoring, is anything preventing you from going in and fixing it? :) Nope, apart from time. There's also nothing stopping me from veto'ing the original submission and reverting it until it handles exceptions in a way that does not have the potential to hide bugs. Guess which is less effort for me. :-) I suppose we need to hear from Rick about his feelings around this before we go and reverse his changes... Actually that doesn't matter, since they are not 'his changes', it's now code at Apache Derby. Dan.
Re: Features of the JUnit test execution harness
Daniel John Debrunner wrote: David W. Van Couvering wrote: I agree with you that is disconcerting, but can't JUnit tests be written that extend DerbyJUnitTest in parallel with getting DerbyJUnitTest cleaned up to everyone's satisfaction? Depends, at some point you put the potential burden of fixing all those new tests on the person doing the cleanup, that should have been part of the original submission. The other issue is that we have a great opportunity to start out with JUnit tests that follow a consistent pattern and provide a great example for others to follow. If we add a number of tests now that haphazardly use the methods in DerbyJUnitTest (because they are not well commented) then we have a set of tests like the current ones. No pattern, no obvious starting point leading to people inventing their own ways to get connections, run ddl, handle exceptions etc. Dan.
