Thanks, Olav, for your excellent investigation. This is quite tangled
and I don't think I fully understand the problem. I think you have
stumbled on an important clue: under derbyall the nist tests run inside
the same VM as the test harness. Why is that? Can't we run them in
another VM and get the same behavior we do when we run the nist suite
outside derbyall?
Here are my reactions to your proposals:
1) According to Lance Andersen, driver autoloading is a required JDBC4
feature. Section 10.2.1 of the JDBC4 spec (Loading a driver that
implements java.sql.Driver) says "The DriverManager.getConnection method
has been enhanced to support the Java Standard Edition Service Provider
mechanism. JDBC 4.0 Drivers must include the file
META-INF/services/java.sql.Driver." The "must" means "mandatory". So I
think that (1) is not an option.
2) I'm afraid I don't understand option (2). Is the embedded driver
silently failing to classload? If the engine is failing to boot, do you
know why?
3) I'm afraid I don't understand (3) either.
4) I think Dan has already discussed this point.
5) This one gets my vote until we can fully describe a bug in Derby
behavior. As I said, I'm confused about what that bug might be.
Thanks,
-Rick
Olav Sandstaa wrote:
So how should we fix this problem? Some more or less random ideas:
1. Disable autoloading of the embedded driver? [comment: are we
required by the JDBC 4 spec to support autoloading of the embedded
driver?]
2. Improve the code so that if the loading of the embedded driver is
unable to boot the database it does either (a) fail to load the
driver? (give error) or (b) don't register itself internally as
loaded? (silently) [comment: this is avoiding the seen problem to
occur]
3. Improve the code for loading the embedded driver so that it does no
longer assume that if the driver is loaded before it also have a
database. This would include explicitly checking of the database
being started, and if not boot the database [comment: this is to
handle the problem after it has occurred]
4. Split the loading of the driver and booting of the embedded Derby
database? During loading of the driver, only the driver is loaded.
When the first connection is attempted, the database is booted.
5. "Define" this as a test error. Fix derbyall to run successfully and
hope that no real application will experience this problem
[comment: I do think this problem can show up in real applications
too].
I appreciate any feedback on both the cause of the problem (as
reported in my previous email) and any suggestions for how to best fix
this problem.
Regards,
Olav
Olav Sandstaa wrote:
I think I have found what causes the Nist tests to fail when running
derbyall with jdk16. The Nist tests started to fail after DERBY-930
(Add support for autoloading of Derby client drivers) was check in.
Earlier the embedded driver was loaded into the jvm when the first of
the tests in the nist suite started and requested the Embedded driver
to be loaded explicitly (note that the Nist suite is the only suite
that run with useprocess=false). Due to autoloading the embedded
driver may now be loaded at an earlier time in the test than before,
and that happens in certain situations (you run the test from jars,
you have the DB2 driver in path, the nist test is run with
useprocess=false). What happens is that during startup of derbyall the
following code in RunList.shouldSkipTest is executed:
if (framework.equals("DerbyNet"))
{
// skip if the derbynet.jar is not in the Classpath
try {
Class.forName("org.apache.derby.drda.NetworkServerControl");
} catch (ClassNotFoundException cnfe) {
driverNotFound = true;
result = true;
}
// skip if the IBM Universal JDBC Driver is not in the Classpath
// note that that driver loads some javax.naming.* classes
which may not
// be present at runtime, and thus we need to catch a possible
error too
try {
Class.forName("com.ibm.db2.jcc.DB2Driver");
} catch (ClassNotFoundException cnfe) {
driverNotFound = true;
result = true;
} catch (NoClassDefFoundError err) {
driverNotFound = true;
result = true;
}
}
}
When the Class.forName("com.ibm.db2.jcc.DB2Driver") is executed (and
the DB2 driver is actually in the class path) it seems to call some
methods that happens to be defined by any JDBC driver (see also the
comment in the code). The class loader seems to touch derby.jar to try
to find these methods, and this makes the embedded driver to
"automagically" be loaded. And loading the embedded driver also
"automagically" attempts to boot an embedded Derby. Unfortunately (or
fortunately?) at this time the Derby properties has not been set for
defining an embedded Derby and the booting of Derby fails (silently).
As a result the VM has the embedded driver loaded (at least kind of),
but no embedded database. At this point it is no problem for derbyall
since the first tests to run has useprocess=true.
About an hour or two later when the first Nist test starts it also
requests the embedded driver to be loaded into the same VM. In the
code for loading the driver and booting the database the following
check is done (see JDBCBoot.boot()):
if (org.apache.derby.jdbc.InternalDriver.activeDriver() == null)
{
// request that the InternalDriver (JDBC) service and the
// authentication service be started.
//
addProperty("derby.service.jdbc",
"org.apache.derby.jdbc.InternalDriver");
addProperty("derby.service.authentication",
AuthenticationService.MODULE);
Monitor.startMonitor(bootProperties, logging);
The test for checking if the driver has been loaded returns a pointer
to the embedded driver loaded during starting of derbyall. Based on
this no Derby database is created (since it based on having the
embedded driver expect that it also must have an embedded
database). So the Nist tests are run with the embedded driver but no
embedded database. No surprise that this make the tests fail
eventually (with an security exception seen by the test).
Regards,
Olav
