[jira] Commented: (DERBY-1338) Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: DRDAProtocolExceptionInfo when run from classes dir
[ http://issues.apache.org/jira/browse/DERBY-1338?page=comments#action_12416840 ] Bryan Pendleton commented on DERBY-1338: Hi Dag, I looked at your patch. Unless anybody else has a better suggestion, I think we should go ahead and use your patch, since it clearly improves the behavior in your environment. I think that we should put a more complete comment in the code itself, though, so that people aren't required to visit JIRA unless they really need the detailed background. What do you think about the following comment -- is it accurate? // Work around a classloader bug involving interrupt handling during // class loading. If the first request to load the DRDAProtocolException // class occurs during shutdown, the loading of the class may be aborted // when the Network Server calls Thread.interrupt() on the DRDAConnThread. // By including a static reference to the DRDAProtocolException class here, // we ensure that it is loaded as soon as the DRDAConnThread class is loaded, // and therefore we know we won't be trying to load the class during shutdown. // See DERBY-1338 for more background, including pointers to the apparent // classloader bug in the JVM. If this comment seems reasonable to you, and if the community doesn't come up with any better suggestions, I can commit this change later this week. > Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: > DRDAProtocolExceptionInfo when run from classes dir > - > > Key: DERBY-1338 > URL: http://issues.apache.org/jira/browse/DERBY-1338 > Project: Derby > Type: Bug > Components: Network Server > Versions: 10.2.0.0 > Environment: Sun JDK 1.4.2, Sun JDK 1.5 on Solaris > Reporter: Dag H. Wanvik > Assignee: Dag H. Wanvik > Priority: Minor > Fix For: 10.2.0.0 > Attachments: derby-1338.diff, derby-1338.stat > > When run from Sun JDK 1.4.2 and Sun JDK 1.5, these two tests fail when run > from the > classes directory. They work, though, when run from jars. > Running NSinSameVM in my sandbox: > bash-3.00$ java -Dframework=DerbyNetClient > org.apache.derbyTesting.functionTests.harness.RunTest > derbynet/NSinSameJVM.java > *** Start: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:05 *** > Initialize for framework: DerbyNetClient > startServer = false. Bypass server startup > 6 add > > java.lang.NoClassDefFoundError: > > org/apache/derby/impl/drda/DRDAProtocolExceptionInfo > Test Failed. > *** End: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:27 *** > I get similar behavior for DerbyNetNewServer. I ran this on a Solaris > 10/i86x box. Davis has seen it too, see > initial discusson on this thread: > http://www.nabble.com/forum/ViewPost.jtp?post=4477600&framed=y -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: interested in working on DERBY-713
Hiya Rick,
Thanks for you helpful reply! I spent some time over the weekend
snooping around (mostly) InListOperatorNode and OptimizerImpl, but
never found out _where_ the table scan was chosen over an index scan
for finding the IN matches...I really wanted to find it before begging
for someone to point out the obvious :-) Specifically, I got all
tangled up around the dynamically generated classes which implement
the chosen query plan. Also, I didn't ever find any QueryPlan-like
class, but I guess this what the dynamic class(es) implement.
What did I want to do once I got there? Pseudo-code says it best:
// ISTR that the bound IN values are ordered and unique. If not,
// this should be done...
if( column is indexed ){
if( index is unique ){
// this should be a guaranteed win. We know we'll only get
// one match per IN value
do index scan
}
else if( table size > SOME_MAGIC_SIZE ){
// SOME_MAGIC_SIZE would represent the size of (the) table
// at which using the index would be reasonably known to
// be more expensive than scanning the table.
do index scan
}
else {
do table scan
}
}
else {
do table scan
}
Is that a workable approach (excepting the SOME_MAGIC_SIZE bit)?
I'll probably have some time later this week to immerse myself in locating
the magic location to apply this logic. A _little_ pointer would be
helpful: maybe a brief overview of how a query plan is determined -- I
suspect his is somewhere around where trulyTheBestTableAccess (or whatever
it's really called :-)) is defined. I'd rather have to bang my head
against this wall a bit, to get a better understanding of things, instead
of being pointed immediately to the solution -- my own version of smelling
the roses, I guess ;-)
Cheers!
Brent
[2006-06-19 11:30] Rick Hillegas said:
| Hi Brent,
|
| Sounds like you're off to a good start. From the initial bug report, it
| looks like there's a good idea about which heuristic is being
| mis-applied here. Once you've studied the optimizer papers, I recommend
| that you post some high-level candidate solutions. Try to avoid
| optimizer jargon and concentrate on simple descriptions:
|
| o What query plan would you rather see?
| o What heuristic would the optimizer apply that would lead it to your
| preferred plan?
| o How would the optimizer decide to apply the new heuristic rather than
| the old one?
|
| I think you'll get some good feedback if your post contains the phrase
| "Attention, optimizer experts." I think that the optimizer enthusiasts
| on the list will give you good feedback:
|
| o Maybe they can think of a better query plan or heuristic.
| o Maybe they can see some awkward corner cases in your heuristic.
| o They can advise you on whether your heuristic will short-circuit other
| optimizer choices.
| o They can advise you on whether your heuristic will cause an explosion
| in the time that the optimizer takes.
|
| Thanks for wanting to scratch this itch!
|
| Regards,
| -Rick
|
| Brent Verner wrote:
|
| >Hi,
| >
| >I've recently found need for an embedded java db and only Derby seems
| >even close to handling the task, however the broken query planning for IN
| >clauses makes it unusable :-(. I've decided to eschew an embedded db
| >in favor of PG for now, but I'd really like to be able to use Derby in the
| >near(ish) future for deployment.
| >
| >I'd like to try to fix the query planning around the IN clause. I'm
| >reviewing the internals papers right now, but I'd appreciate any input
| >that might point me in the right direction :-).
| >
| >cheers!
| > Brent
| >
| >
| >
Re: Driver autoloading and engine booting
Rick Hillegas wrote: This topic was raised in an earlier email thread (http://comments.gmane.org/gmane.comp.apache.db.derby.devel/20899) but it does not appear that we agreed on a solution. I would like to re-open this topic and hopefully we can converge on how we want to handle this issue. Olav has analyzed a problematic interaction between JDBC4-driver-autoloading and Derby-embedded-engine-booting. In addition to the above email thread, the problem is also discussed in DERBY-1399. Here's a brief sketch of the issue: 1) Remove the JDBC4-driver-autoloading. This removes a useful ease-of-development feature and makes us fail to be JDBC4-compliant. 2) Don't boot the engine when registering the embedded driver. Instead, boot the engine the first time that someone requests an embedded Connection. This approach involves a lot of testing. Thanks for explaing the problem clearly Rick. I believe it is not uncommon for the application to have a code that connects to different database from different vendors and only make connection to one of them at a time as needed in different parts of their code at runtime. I think booting derby (Actually it is just the monitor service modules that gets booted not underlying storage ..etc.) when user is using a different vendor database does not sound right. In my openion, this problem should be fixed if possible. Thanks -suresh
[jira] Updated: (DERBY-1338) Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: DRDAProtocolExceptionInfo when run from classes dir
[ http://issues.apache.org/jira/browse/DERBY-1338?page=all ] Dag H. Wanvik updated DERBY-1338: - Component: Network Server (was: Regression Test Failure) Environment: Sun JDK 1.4.2, Sun JDK 1.5 on Solaris was: Sun JDK 1.4.2, Sun JDK 1.5 > Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: > DRDAProtocolExceptionInfo when run from classes dir > - > > Key: DERBY-1338 > URL: http://issues.apache.org/jira/browse/DERBY-1338 > Project: Derby > Type: Bug > Components: Network Server > Versions: 10.2.0.0 > Environment: Sun JDK 1.4.2, Sun JDK 1.5 on Solaris > Reporter: Dag H. Wanvik > Assignee: Dag H. Wanvik > Priority: Minor > Fix For: 10.2.0.0 > Attachments: derby-1338.diff, derby-1338.stat > > When run from Sun JDK 1.4.2 and Sun JDK 1.5, these two tests fail when run > from the > classes directory. They work, though, when run from jars. > Running NSinSameVM in my sandbox: > bash-3.00$ java -Dframework=DerbyNetClient > org.apache.derbyTesting.functionTests.harness.RunTest > derbynet/NSinSameJVM.java > *** Start: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:05 *** > Initialize for framework: DerbyNetClient > startServer = false. Bypass server startup > 6 add > > java.lang.NoClassDefFoundError: > > org/apache/derby/impl/drda/DRDAProtocolExceptionInfo > Test Failed. > *** End: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:27 *** > I get similar behavior for DerbyNetNewServer. I ran this on a Solaris > 10/i86x box. Davis has seen it too, see > initial discusson on this thread: > http://www.nabble.com/forum/ViewPost.jtp?post=4477600&framed=y -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1338) Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: DRDAProtocolExceptionInfo when run from classes dir
[ http://issues.apache.org/jira/browse/DERBY-1338?page=all ] Dag H. Wanvik updated DERBY-1338: - Derby Info: [Patch Available] > Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: > DRDAProtocolExceptionInfo when run from classes dir > - > > Key: DERBY-1338 > URL: http://issues.apache.org/jira/browse/DERBY-1338 > Project: Derby > Type: Bug > Components: Regression Test Failure > Versions: 10.2.0.0 > Environment: Sun JDK 1.4.2, Sun JDK 1.5 > Reporter: Dag H. Wanvik > Assignee: Dag H. Wanvik > Priority: Minor > Fix For: 10.2.0.0 > Attachments: derby-1338.diff, derby-1338.stat > > When run from Sun JDK 1.4.2 and Sun JDK 1.5, these two tests fail when run > from the > classes directory. They work, though, when run from jars. > Running NSinSameVM in my sandbox: > bash-3.00$ java -Dframework=DerbyNetClient > org.apache.derbyTesting.functionTests.harness.RunTest > derbynet/NSinSameJVM.java > *** Start: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:05 *** > Initialize for framework: DerbyNetClient > startServer = false. Bypass server startup > 6 add > > java.lang.NoClassDefFoundError: > > org/apache/derby/impl/drda/DRDAProtocolExceptionInfo > Test Failed. > *** End: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:27 *** > I get similar behavior for DerbyNetNewServer. I ran this on a Solaris > 10/i86x box. Davis has seen it too, see > initial discusson on this thread: > http://www.nabble.com/forum/ViewPost.jtp?post=4477600&framed=y -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1108) The test jdbcapi/setTransactionIsolation.java fails with ibm jvm1.5
[ http://issues.apache.org/jira/browse/DERBY-1108?page=comments#action_12416825 ] Kathey Marsden commented on DERBY-1108: --- Thanks Army for the comment. I think I understand escape analysis better now and how it relatest to this issue and I think 1) The JVM is ok. It should garbage collect and finalize rs and all its innards because they can never really be used again. 2) Derby is ok. setTransactionIsolation() does not need to consider a ResultSet that can never be used again . 3) The test is not ok. To test what it wants to test, that we can't change the isolation in the middle of retrieving results on a held cursor, it should have code to use rs after setTransactionIsolation, e.g. add an rs.next() after setTransactionIsolation() Does that sound right? > The test jdbcapi/setTransactionIsolation.java fails with ibm jvm1.5 > --- > > Key: DERBY-1108 > URL: http://issues.apache.org/jira/browse/DERBY-1108 > Project: Derby > Type: Bug > Components: Regression Test Failure > Versions: 10.2.0.0 > Environment: java version "1.5.0" > Java(TM) 2 Runtime Environment, Standard Edition (build pwi32dev-20051104) > IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 > j9vmwi3223-2005110 > 3 (JIT enabled) > J9VM - 20051027_03723_lHdSMR > JIT - 20051027_1437_r8 > GC - 20051020_AA) > JCL - 20051102 > Reporter: Manjula Kutty > Attachments: test.java > > The test jdbcapi/setTransactionIsolation.java fails with ibmjvm15. I think > the cause of this failure is , the jvm is not throwing an exception while > trying to change the transaction isolation when there is holdable cursor on > a resultset. Other jvms like sun jdk1.5, ibm142 throws the expected > exception. while ibm15 does allow to change the transaction isolation when > there is a hold cursor on a result set. > Already reported this issue with the ibmjvm people and they are looking in to > it -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1108) The test jdbcapi/setTransactionIsolation.java fails with ibm jvm1.5
[
http://issues.apache.org/jira/browse/DERBY-1108?page=comments#action_12416823 ]
A B commented on DERBY-1108:
I'm not very well acquainted with this code, but based on the example of
"escape analysis" posted by the JVM team, I think I understand why this test is
failing with IBM 1.5.
If we look at the test program again, we can see that it very closely mimics
the example of "escape analysis" offered by the JVM team. In particular:
System.out.println("done creating table and inserting data.");
conn.setTransactionIsolation(Connection.TRANSACTION_READ_COMMITTED);
ResultSet rs = stmt.executeQuery("select * from T1");
==> Under Java 5.0 rs is eligible for GC here...
System.out.println("current isolation level: " +
getIsoLevelName(conn.getTransactionIsolation()));
conn.setTransactionIsolation(java.sql.Connection.TRANSACTION_SERIALIZABLE);
System.out.println("current isolation level: " +
getIsoLevelName(conn.getTransactionIsolation()));
==> Under previous versions, rs is eligible for GC here...
So with ibm15, rs becomes eligible for GC _prior_ to the call to
setTransactionIsolation because it (rs) is no longer referenced after it is
created. Then the call to gc() in verifyAllHeldResultSetsAreClosed() does the
escape analysis, sees that rs is now eligible for GC and so collects it. As
part of that collection process the JVM calls "finalize()", which takes us to
EmbedResultSet.finalize(), and there we see this:
/**
JDBC states that a ResultSet is closed when garbage collected.
We simply mark the activation as unused. Some later use
of the connection will clean everything up.
@exception Throwable Allows any exception to be thrown during
finalize
*/
protected void finalize() throws Throwable {
super.finalize();
if (finalizeActivation != null) {
finalizeActivation.markUnused();
}
}
Thus finalizeActiviation is now marked as "unused", which means we end up
ignoring the activation in the verifyAllHeldResultSetsAreClosed() method:
if (!a.isInUse())
{
continue;
}
And therefore we return "true" from the method and no error is thrown.
To prove this, I added a very simple (albeit meaningless) reference to "rs"
after the call to setTransactionIsolation:
System.out.println("done creating table and inserting data.");
conn.setTransactionIsolation(Connection.TRANSACTION_READ_COMMITTED);
ResultSet rs = stmt.executeQuery("select * from T1");
System.out.println("current isolation level: " +
getIsoLevelName(conn.getTransactionIsolation()));
conn.setTransactionIsolation(java.sql.Connection.TRANSACTION_SERIALIZABLE);
System.out.println("current isolation level: " +
getIsoLevelName(conn.getTransactionIsolation()));
System.out.println(rs); // <== Added by Army
With this simple line, the escape analysis will *not* mark rs eligible for GC
prior to setTransactionIsolation, which means we won't call "finalize()", which
means the activation will remain "in use", and therefore we'll get the
exception.
All of that said, I added System.outs to the test program to see what the
object id of "rs" is before the call to setTransactionIsolation, and I also
added System.outs to see what the object id of the "rs" is inside
verifyAllHeldResultSetsAreClosed(). Not surprisingly, the objects are NOT the
same. In the test program we see:
[EMAIL PROTECTED]
but in verifyAllHeldResultSetsAreClosed() we see:
[EMAIL PROTECTED]
So as far as I can tell, the test program no longer references the
EmbedResultSet, so it's marked as "eligible for GC" and is garbage collected by
escape analysis; however, that EmbedResultSet was in turn referencing another
object--BulkTableScanResultSet--that *is* still referenced (by the activation),
and that's the result set that should have caused the error--but since we
"bailed" on the error checking because the activiation was no longer "in use",
we never checked the underlying result set and therefore we didn't throw the
exception.
That's my take on it after looking at it briefly. I still don't know for sure
who's "at fault" here (the JVM or Derby)--I'd have to look at it more to say
one way or the other. But that's my first take on the issue...
Comments/input/feedback are, as always, much appreciated...
Army
> The test jdbcapi/setTransactionIsolation.java fails with ibm jvm1.5
> ---
>
> Key: DERBY-1108
> URL: http://issues.apache.org/jira/browse/DERBY-1108
> Project: Derby
> Type: Bug
> Components: Regression Test
[jira] Updated: (DERBY-1338) Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: DRDAProtocolExceptionInfo when run from classes dir
[ http://issues.apache.org/jira/browse/DERBY-1338?page=all ] Dag H. Wanvik updated DERBY-1338: - Attachment: derby-1338.stat derby-1338.diff Here is a simple patch which preloads the class causing the problem. It makes the tests run correctly - derbyall ran without errors on the classes directory on my Solaris 10/x86 box). I don't have to time to work on getting rid of the thread interrupts right now; although I would prefer that solution. I can leave this issue open or we could file another one. > Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: > DRDAProtocolExceptionInfo when run from classes dir > - > > Key: DERBY-1338 > URL: http://issues.apache.org/jira/browse/DERBY-1338 > Project: Derby > Type: Bug > Components: Regression Test Failure > Versions: 10.2.0.0 > Environment: Sun JDK 1.4.2, Sun JDK 1.5 > Reporter: Dag H. Wanvik > Assignee: Dag H. Wanvik > Priority: Minor > Fix For: 10.2.0.0 > Attachments: derby-1338.diff, derby-1338.stat > > When run from Sun JDK 1.4.2 and Sun JDK 1.5, these two tests fail when run > from the > classes directory. They work, though, when run from jars. > Running NSinSameVM in my sandbox: > bash-3.00$ java -Dframework=DerbyNetClient > org.apache.derbyTesting.functionTests.harness.RunTest > derbynet/NSinSameJVM.java > *** Start: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:05 *** > Initialize for framework: DerbyNetClient > startServer = false. Bypass server startup > 6 add > > java.lang.NoClassDefFoundError: > > org/apache/derby/impl/drda/DRDAProtocolExceptionInfo > Test Failed. > *** End: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:27 *** > I get similar behavior for DerbyNetNewServer. I ran this on a Solaris > 10/i86x box. Davis has seen it too, see > initial discusson on this thread: > http://www.nabble.com/forum/ViewPost.jtp?post=4477600&framed=y -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1338) Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: DRDAProtocolExceptionInfo when run from classes dir
[ http://issues.apache.org/jira/browse/DERBY-1338?page=comments#action_12416817 ] Dag H. Wanvik commented on DERBY-1338: -- From: Bryan Pendleton, Date: Tue, 23 May 2006 16:17:59 -0700 > The shutdown of the server *has* interrupted the thread, in > NetworkServerStarter#blockingStart. So, is this the reason we get this > exception? This is *very* interesting -- thank you for doing this great research! This sort of thing is exactly why I've been so scared to add more calls to interrupt() as part of DERBY-1219 and DERBY-1326 -- interrupt() is a very scary call to make. The good news is that this does mean that this is purely a shutdown problem, as the only place that we interrupt the DRDAConnThread instances right now is at the very end of the blockingStart() method. But it certainly means we'll have to be even more careful about whatever fix we eventually make for DERBY-1326, as we definitely don't want to casually add any more interrupt() calls, given the possibility of horrible side effects like this. > As I mentioned before, we can make sure the classes in question > (DRDAProtocolExceptionInfo and DRDAProtocolException) are already > loaded (by creating a dummy object) before the thread is interrupted, > thus sidestepping the issue. Alternatively, we could remove the interrupt() call in Network Server shutdown. Since we are shutting down the server anyway, I don't know how crucial it is to clean up those background threads. We have called close() on them already, so if they try to run again, they will quickly detect that they have been closed and exit themselves. And, so long as they have been marked as "daemon" threads, their mere presence won't prevent the server from shutting down. But I'm also fine with pre-loading the classes if that seems to help. These sorts of startup/shutdown bugs always seem very hard to me: trying to shut down a complex piece of server software while all sorts of complicated processing may be still underway has always been a dicey business. thanks, bryan > Client tests DerbyNetNewServer and NSinSameVM fail with NoClassDefFoundError: > DRDAProtocolExceptionInfo when run from classes dir > - > > Key: DERBY-1338 > URL: http://issues.apache.org/jira/browse/DERBY-1338 > Project: Derby > Type: Bug > Components: Regression Test Failure > Versions: 10.2.0.0 > Environment: Sun JDK 1.4.2, Sun JDK 1.5 > Reporter: Dag H. Wanvik > Assignee: Dag H. Wanvik > Priority: Minor > Fix For: 10.2.0.0 > > When run from Sun JDK 1.4.2 and Sun JDK 1.5, these two tests fail when run > from the > classes directory. They work, though, when run from jars. > Running NSinSameVM in my sandbox: > bash-3.00$ java -Dframework=DerbyNetClient > org.apache.derbyTesting.functionTests.harness.RunTest > derbynet/NSinSameJVM.java > *** Start: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:05 *** > Initialize for framework: DerbyNetClient > startServer = false. Bypass server startup > 6 add > > java.lang.NoClassDefFoundError: > > org/apache/derby/impl/drda/DRDAProtocolExceptionInfo > Test Failed. > *** End: NSinSameJVM jdk1.4.2_05 DerbyNetClient 2006-05-22 14:49:27 *** > I get similar behavior for DerbyNetNewServer. I ran this on a Solaris > 10/i86x box. Davis has seen it too, see > initial discusson on this thread: > http://www.nabble.com/forum/ViewPost.jtp?post=4477600&framed=y -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Driver autoloading and engine booting
Jean T. Anderson wrote:
> Rick Hillegas wrote:
>>Jean T. Anderson wrote:
>>>Rick Hillegas wrote:
> ...
>>>Does the "precedence for properties" documentation need to be updated to
>>>mention jdbc 4/jdk 1.6 behavior?
>>>
>>>http://db.apache.org/derby/docs/dev/tuning/ctunsetprop23308.html
>
>>I don't see that the precedence of properties has changed. What has
>>changed is when we read properties during the lifetime of a vm.
>>
>>Driver-autoloading does require many other edits to the user guides.
>>They are summarized in DERBY-1271.
>
> ok, good, thanks for the clarification.
I take the "ok" back based on Kathey's post to derby-user [1].
The actual precedence for a static variable will be whatever was visible
when another driver caused derby to be autoloaded, possibly before the
application set that property.
Kathey's derby.system.home example is probably the most likely confusion
to occur.
Let's say my application connects to MySQL to migrate a database to
Derby. When I connect to that MySQL database, the embedded derby driver
gets loaded and the embedded engine gets booted. Later, my application
calls System.setProperty("derby.system.home",
"/home/jta/DerbyDatabases") and creates the new derby database. But it
doesn't get created where I expected -- the derby.system.home I set is
ignored because the derby engine was already booted.
-jean
[1]
http://mail-archives.apache.org/mod_mbox/db-derby-user/200606.mbox/[EMAIL
PROTECTED]
[jira] Commented: (DERBY-578) Grouped select from temporary table raises null pointer exception in byte code generator
[ http://issues.apache.org/jira/browse/DERBY-578?page=comments#action_12416814 ] Manish Khettry commented on DERBY-578: -- Thanks for taking time to review the fix. The xaHelper is not related to this but when I was debugging using ij, this bit of code would always throw a NPE in the debugger. You are right about 2. Earlier, we would set the indexName field in DistinctScanResult to the conglomerate name (cd.getName()) used to scan the table. If the conglomerate was the base table itself then this was just plain wrong. The change, for this patch, passes null if no index is being used. This in turn ended up causing all the diffs to the master files. I really wish we had assert tests; all the test output wants to check in this case is that a distinct scan is performed and not have all the other cruft output by runtime stats. The change to FromBaseTable is similar to the code in BaseJoinStrategy:fillInScanArgs2. I was tempted to try and have all the differfent code paths (DistinctScan, specialMaxScan) share the same code when it comes to filling in the scan args but I took the easy way out for this bug for now. > Grouped select from temporary table raises null pointer exception in byte > code generator > > > Key: DERBY-578 > URL: http://issues.apache.org/jira/browse/DERBY-578 > Project: Derby > Type: Bug > Components: SQL, Services > Reporter: Rick Hillegas > Assignee: Manish Khettry > Attachments: 578.diff.new.txt, 578.diff.txt, predicatePushdown.out.patch > > The last statement in the following script: > autocommit off; > declare global temporary table session.ztemp > ( orderID varchar( 50 ) ) > not logged; > select orderID from session.ztemp group by orderID; > raises the following stack trace in the trunk: > java.lang.NullPointerException > at java.util.Hashtable.get(Hashtable.java:333) > at > org.apache.derby.iapi.services.classfile.ClassHolder.findMatchingEntry(ClassHolder.java:656) > at > org.apache.derby.iapi.services.classfile.ClassHolder.addUtf8Entry(ClassHolder.java:482) > at > org.apache.derby.iapi.services.classfile.ClassHolder.addString(ClassHolder.java:506) > at > org.apache.derby.iapi.services.classfile.ClassHolder.addConstant(ClassHolder.java:302) > at > org.apache.derby.impl.services.bytecode.BCMethod.push(BCMethod.java:452) > at > org.apache.derby.impl.sql.compile.FromBaseTable.generateDistinctScan(FromBaseTable.java:3216) > at > org.apache.derby.impl.sql.compile.FromBaseTable.generateResultSet(FromBaseTable.java:3032) > at > org.apache.derby.impl.sql.compile.FromBaseTable.generate(FromBaseTable.java:2978) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(ProjectRestrictNode.java:1196) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(ProjectRestrictNode.java:1145) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(ProjectRestrictNode.java:1196) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(ProjectRestrictNode.java:1145) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(ProjectRestrictNode.java:1196) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(ProjectRestrictNode.java:1145) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(ProjectRestrictNode.java:1196) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(ProjectRestrictNode.java:1145) > at > org.apache.derby.impl.sql.compile.ScrollInsensitiveResultSetNode.generate(ScrollInsensitiveResultSetNode.java:109) > at > org.apache.derby.impl.sql.compile.ReadCursorNode.generate(ReadCursorNode.java:118) > at > org.apache.derby.impl.sql.compile.CursorNode.generate(CursorNode.java:546) > at > org.apache.derby.impl.sql.compile.StatementNode.generate(StatementNode.java:232) > at > org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:433) > at > org.apache.derby.impl.sql.GenericStatement.prepare(GenericStatement.java:107) > at > org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:704) > at > org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:533) > at > org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:480) > at org.apache.derby.impl.tools.ij.ij.executeImmediate(ij.java:299) > at org.apache.derby.impl.tools.ij.utilMain.doCatch(utilMain.java:433) > at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:310) > at org.apache.derby.impl.tools.ij.Main.go(Main.java:203) > at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:169) >
[jira] Commented: (DERBY-1428) Generating derby properties on the fly can lead to non-deterministic engine startup behavior
[ http://issues.apache.org/jira/browse/DERBY-1428?page=comments#action_12416810 ] Rick Hillegas commented on DERBY-1428: -- Kathey adds the following: I think DERBY-1428 and DERBY-149 are not really related to generating derby.properties on the fly they are related to when Derby boots and when derby.system.home is recognized. derby.system.home *has* to be set with a system property. Anyone who sets derby.system.home or static derby properties programatically with System.setProperty() may be affected if they do not have control over their calling context. Another driver might cause derby to be autoloaded before these properties are set. Perhaps these issues would be better worded as: "Derby can have non-deterministic behaviour when derby.system.home or other static properties are set with System.setProperty()" More common reported symptoms will likely be after upgrading to Derby 10.2 and JDK 1.6 users will report the following systems because derby.system.home is not picked up properly. - I get XJ004.C=Database ''...'' not found.(Derby is looking in the wrong place for the db) - There is nothing in my database. (create=true was on the url and so app created a new db in the wrong place) - My database was created in the wrong place. - Properties set in derby.properties are not being picked up. Some more info: http://www.nabble.com/Re%3A--jira--Commented%3A-%28DERBY-930%29-Add-support-for-autoloading-of-Derby-client-drivers-p4909085.html > Generating derby properties on the fly can lead to non-deterministic engine > startup behavior > > > Key: DERBY-1428 > URL: http://issues.apache.org/jira/browse/DERBY-1428 > Project: Derby > Type: Bug > Components: Store > Versions: 10.0.2.0, 10.0.2.1, 10.0.2.2, 10.1.1.0, 10.2.0.0, 10.1.2.1, > 10.1.3.0, 10.1.2.2, 10.1.2.3, 10.3.0.0, 10.1.2.4, 10.1.2.5, 10.1.4.0, 10.1.3.1 > Reporter: Rick Hillegas > > A Heisenbug can arise if more than one embedded Derby application runs in the > same VM and at least one of them generates derby properties on the fly. > Here's the problem scenario: > o The customer runs two embedded Derby apps in the same VM: EmbeddedApp and > OtherApp. > o EmbeddedApp generates derby properties on the fly before connecting to a > database and triggering engine startup. > o Whether the engine picks up those generated properties depends on whether > EmbeddedApp or OtherApp runs first. > Here are two workarounds for the problem: > 1) Don't generate derby properties on the fly. Instead, specify them on the > VM startup command. Or specify them in a $DERBY_HOME/derby.properties file > which is generated before the VM comes up and which does not change during > the lifetime of the VM. > 2) If you can't do (1), then modify the VM startup script so that the > self-configuring EmbeddedApp runs first. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: How to best handle variations in output from ij in sql tests?
Hi Olav,
Thanks for digging into this. The differences in the embedded output are
caused by JDBC4's introduction of SQLException subclasses. These now
replace Derby's hand-rolled subclass of SQLException. The old Derby
subclass overrode the getMessage() method. That's why you used to see
"SQL Exception:..." and now you see
"java.sql.SQLSyntaxErrorException:..." I suppose that we could get
close to the old message text if we introduced a Derby subclass for each
of the JDBC4 exception subclasses and then override the getMessage() in
each of these. This seems like a lot of code bloat without much payback.
I think that handling these discrepancies with a sed script is fine.
I don't understand why the trailing SQLSTATE info has been stripped off
the network messages. Perhaps network serialization of chained
exceptions needs a closer look.
Regards,
-Rick
Olav Sandstaa wrote:
I am working on reducing the number of tests failing when running with
JDK 1.6. Some of the tests are now failing due to jdk16 specific
master files that have not been updated after introduction of textual
changes. To solve some of these failing tests I try eliminate the need
for jdk16 specific master files.
To avoid creating to more sed rules for filtering some of the
differences I wonder if some of the changes better had been fixed by
making sure that the output is more consistent between client and
embedded driver and between JDK 1.5 and JDK 1.6.
I would like to get feedback on whether the following example should
best be solved by fixing the difference between the output from ij or
by continue to have multiple master files. This example is from the
ieptests.sql which now fails with JDK 1.6 with the client driver:
SQL that makes the test fail:
=
ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE ('inventory', 'ORDERTABLE' ,
'extinout/order.dat', null, null, null) ;
Java 1.5 Embedded output:
=
ERROR 38000: The exception 'SQL Exception: Schema 'inventory' does not
exist' was thrown while evaluating an expression.
ERROR 42Y07: Schema 'inventory' does not exist
Java 1.6 Embedded output:
=
ERROR 38000: The exception 'java.sql.SQLSyntaxErrorException: Schema
'inventory' does not exist' was thrown while evaluating an expression.
ERROR 42Y07: Schema 'inventory' does not exist
Java 1.5 Client output:
===
ERROR 38000: The exception 'SQL Exception: Schema 'inventory' does not
exist' was thrown while evaluating an expression. SQLSTATE: 42Y07:
Schema 'inventory' does not exist
Java 1.6 Client output:
===
ERROR 38000: The exception 'java.sql.SQLSyntaxErrorException: Schema
'inventory' does not exist' was thrown while evaluating an expression.
The main difference is:
-embedded: with JDK 1.5 the output writes "SQL Exception" while on
JDK 1.6 it is "java.sql.SQLSyntaxErrorException" (more exact, but
less read-friendly?). This difference is handled by seding the
output.
-client: with JDK 1.5 the SQL state is included in the output, while
with JDK 1.6 the SQL state is not included. Is this a bug?
-embedded vs server: embedded writes the output split over two lines,
while the client has everything on one line. Could this be changed?
In addition to the detailed question about the specific output making
this test fail, I wonder if we should have some general rules that we
should attempt the output (error/warnings/responses) from ij be as
similar as possible for (a) between client and server and (b) between
different versions of the JVM?
Any feedback is appreciated!
Thanks,
Olav
[jira] Commented: (DERBY-578) Grouped select from temporary table raises null pointer exception in byte code generator
[ http://issues.apache.org/jira/browse/DERBY-578?page=comments#action_12416803 ] Rick Hillegas commented on DERBY-578: - Thanks, Manish. As you said, the fix looks pretty simple. I'm running regression tests now. I have a couple questions: 1) I don't understand the change to xaHelper. How is that related to this bugfix? 2) Can you explain the test diffs? On the surface, it looks like a number of index scans have been replaced with table scans. However, it's hard to trust the original claim that an index was being used because there doesn't seem to be a usable index. Please help me muddle through these diffs. -Rick > Grouped select from temporary table raises null pointer exception in byte > code generator > > > Key: DERBY-578 > URL: http://issues.apache.org/jira/browse/DERBY-578 > Project: Derby > Type: Bug > Components: SQL, Services > Reporter: Rick Hillegas > Assignee: Manish Khettry > Attachments: 578.diff.new.txt, 578.diff.txt, predicatePushdown.out.patch > > The last statement in the following script: > autocommit off; > declare global temporary table session.ztemp > ( orderID varchar( 50 ) ) > not logged; > select orderID from session.ztemp group by orderID; > raises the following stack trace in the trunk: > java.lang.NullPointerException > at java.util.Hashtable.get(Hashtable.java:333) > at > org.apache.derby.iapi.services.classfile.ClassHolder.findMatchingEntry(ClassHolder.java:656) > at > org.apache.derby.iapi.services.classfile.ClassHolder.addUtf8Entry(ClassHolder.java:482) > at > org.apache.derby.iapi.services.classfile.ClassHolder.addString(ClassHolder.java:506) > at > org.apache.derby.iapi.services.classfile.ClassHolder.addConstant(ClassHolder.java:302) > at > org.apache.derby.impl.services.bytecode.BCMethod.push(BCMethod.java:452) > at > org.apache.derby.impl.sql.compile.FromBaseTable.generateDistinctScan(FromBaseTable.java:3216) > at > org.apache.derby.impl.sql.compile.FromBaseTable.generateResultSet(FromBaseTable.java:3032) > at > org.apache.derby.impl.sql.compile.FromBaseTable.generate(FromBaseTable.java:2978) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(ProjectRestrictNode.java:1196) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(ProjectRestrictNode.java:1145) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(ProjectRestrictNode.java:1196) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(ProjectRestrictNode.java:1145) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(ProjectRestrictNode.java:1196) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(ProjectRestrictNode.java:1145) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generateMinion(ProjectRestrictNode.java:1196) > at > org.apache.derby.impl.sql.compile.ProjectRestrictNode.generate(ProjectRestrictNode.java:1145) > at > org.apache.derby.impl.sql.compile.ScrollInsensitiveResultSetNode.generate(ScrollInsensitiveResultSetNode.java:109) > at > org.apache.derby.impl.sql.compile.ReadCursorNode.generate(ReadCursorNode.java:118) > at > org.apache.derby.impl.sql.compile.CursorNode.generate(CursorNode.java:546) > at > org.apache.derby.impl.sql.compile.StatementNode.generate(StatementNode.java:232) > at > org.apache.derby.impl.sql.GenericStatement.prepMinion(GenericStatement.java:433) > at > org.apache.derby.impl.sql.GenericStatement.prepare(GenericStatement.java:107) > at > org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(GenericLanguageConnectionContext.java:704) > at > org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:533) > at > org.apache.derby.impl.jdbc.EmbedStatement.execute(EmbedStatement.java:480) > at org.apache.derby.impl.tools.ij.ij.executeImmediate(ij.java:299) > at org.apache.derby.impl.tools.ij.utilMain.doCatch(utilMain.java:433) > at org.apache.derby.impl.tools.ij.utilMain.go(utilMain.java:310) > at org.apache.derby.impl.tools.ij.Main.go(Main.java:203) > at org.apache.derby.impl.tools.ij.Main.mainCore(Main.java:169) > at org.apache.derby.impl.tools.ij.Main14.main(Main14.java:55) > at org.apache.derby.tools.ij.main(ij.java:60) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Driver autoloading and engine booting
Rick Hillegas wrote: > Jean T. Anderson wrote: >> Rick Hillegas wrote: ... >> >> Does the "precedence for properties" documentation need to be updated to >> mention jdbc 4/jdk 1.6 behavior? >> >> http://db.apache.org/derby/docs/dev/tuning/ctunsetprop23308.html > I don't see that the precedence of properties has changed. What has > changed is when we read properties during the lifetime of a vm. > > Driver-autoloading does require many other edits to the user guides. > They are summarized in DERBY-1271. ok, good, thanks for the clarification. -jean
How to best handle variations in output from ij in sql tests?
I am working on reducing the number of tests failing when running with
JDK 1.6. Some of the tests are now failing due to jdk16 specific
master files that have not been updated after introduction of textual
changes. To solve some of these failing tests I try eliminate the need
for jdk16 specific master files.
To avoid creating to more sed rules for filtering some of the
differences I wonder if some of the changes better had been fixed by
making sure that the output is more consistent between client and
embedded driver and between JDK 1.5 and JDK 1.6.
I would like to get feedback on whether the following example should
best be solved by fixing the difference between the output from ij or
by continue to have multiple master files. This example is from the
ieptests.sql which now fails with JDK 1.6 with the client driver:
SQL that makes the test fail:
=
ij> call SYSCS_UTIL.SYSCS_EXPORT_TABLE ('inventory', 'ORDERTABLE' ,
'extinout/order.dat', null, null, null) ;
Java 1.5 Embedded output:
=
ERROR 38000: The exception 'SQL Exception: Schema 'inventory' does not
exist' was thrown while evaluating an expression.
ERROR 42Y07: Schema 'inventory' does not exist
Java 1.6 Embedded output:
=
ERROR 38000: The exception 'java.sql.SQLSyntaxErrorException: Schema
'inventory' does not exist' was thrown while evaluating an expression.
ERROR 42Y07: Schema 'inventory' does not exist
Java 1.5 Client output:
===
ERROR 38000: The exception 'SQL Exception: Schema 'inventory' does not
exist' was thrown while evaluating an expression. SQLSTATE: 42Y07:
Schema 'inventory' does not exist
Java 1.6 Client output:
===
ERROR 38000: The exception 'java.sql.SQLSyntaxErrorException: Schema
'inventory' does not exist' was thrown while evaluating an expression.
The main difference is:
-embedded: with JDK 1.5 the output writes "SQL Exception" while on
JDK 1.6 it is "java.sql.SQLSyntaxErrorException" (more exact, but
less read-friendly?). This difference is handled by seding the
output.
-client: with JDK 1.5 the SQL state is included in the output, while
with JDK 1.6 the SQL state is not included. Is this a bug?
-embedded vs server: embedded writes the output split over two lines,
while the client has everything on one line. Could this be changed?
In addition to the detailed question about the specific output making
this test fail, I wonder if we should have some general rules that we
should attempt the output (error/warnings/responses) from ij be as
similar as possible for (a) between client and server and (b) between
different versions of the JVM?
Any feedback is appreciated!
Thanks,
Olav
Re: Driver autoloading and engine booting
Kathey Marsden wrote:
Rick Hillegas wrote:
I'm afraid I don't understand why we would want to revert our tests
to the old behavior. It seems to me that our tests are practicing
stunts which we strongly discourage: changing Derby properties on the
fly. In general, there will always be Heisenbugs for applications
which do this and we should discourage this practice in the strongest
possible terms. What is the benefit of reverting the test behavior?
I think the scope of the problem is beyond generating derby.properties
on the fly.
I think the greater risk is anyone setting system properties to affect
derby behaviour. The biggest trouble maker I am guessing would be
derby.system.home.
The new requirement (I think) is that you set derby.system.home before
any driver is autoloaded and would have the symptoms described in this
mail.
Is that description correct?
http://www.nabble.com/Re%3A--jira--Commented%3A-%28DERBY-930%29-Add-support-for-autoloading-of-Derby-client-drivers-p4909085.html
I find it not uncommon to see code like this:
System.setProperty("derby.system.home", MY_DERBY_DIR);
Class.forName("org.apache.derby.jdbc.EmbeddedDriver").newInstance();
You are right, that there are multiple ways to get to the bug. One way
is to regenerate derby.properties itself in the directory indicated by
derby.system.home. Another way to trigger the behavior is to reset the
value of derby.system.home so that it points to a directory with a
different derby.properties. I defer to your experience as to which of
these is the more travelled route.
If I understand the problem correctly, this will no longer reliably
work if any other code in the same JVM (or is it classloader
context?) makes a database connection to any database.
Is that correct?
Right. It does not reliably work today if the VM runs more than one
embedded derby application, one of which is self-configuring. That's bug
DERBY-1428. JDBC4 introduces a new scenario for the bug, as you
describe: the bug can turn up if jdk1.6 runs a self-configuring Derby
application together with another application, which connects to some
database. That's DERBY-1429.
The point of seeing if we can keep the tests the same is to see if we
can avoid user impact. We should olnly change the tests if and when
we decide the behavior change is ok. If later we add better tests
that more accurately reflect actual user scemarios like the one above
instead, that would be great, but the bottom line is just like the
tests, USERS DO STRANGE THINGS and expect things to still work. If
they upgrade and things suddenly don't work, they want the old
behaviour back and we can't give it to them because new users depend
on the new behaviour. We trap ourselves into a very ugly corner.and
this is a very bad and messy thing. It is better to keep things
working if we can. Every time we change a test to adapt to some new
usage requirement it should be a huge red flag that someone is going
to hit the problem.
Kathey
Thanks for the explanation. I'm looking forward to your analysis and to
more feedback from the user community so that we can decide what to do here.
Re: Driver autoloading and engine booting
Jean T. Anderson wrote: Rick Hillegas wrote: Kathey Marsden wrote: David Van Couvering wrote: - Log a bug - Work on fixing it by booting on first connection rather than when the driver is loaded. It would be good to get input from the user community as well. One possible approach would be to log the bug, mark it as a regression, existing application impact, and release note needed. and have a formal release note. Then send a note to derby-user with a pointer to the bug and Ricks summary to see if any users feel there will be impact. Unfortunately, my gut instinct is that this is the kind of issue that lays dormant and then hits someone hard on upgrade or when integrated with another product.It might broadside users unaware, like it did the nist tests and DerbyNetAutostart tests. It will be really helpful to have the bug and the release note to present to users to help them understand impact. Then we can close DERBY-930 back up too until it is decided whether this "Heisenbug" is a show stopper or not. I have logged two new bugs: DERBY-1428 describes the existing, pre-JDBC4 shape of the problem. DERBY-1429 describes the extra exposure introduced by JDBC4. I have added a release note to DERBY-930, which summarizes the problem and its workarounds. In addition, I have asked the user community to let us know how broad the extra impact is. Does the "precedence for properties" documentation need to be updated to mention jdbc 4/jdk 1.6 behavior? http://db.apache.org/derby/docs/dev/tuning/ctunsetprop23308.html -jean Hi Jean, I don't see that the precedence of properties has changed. What has changed is when we read properties during the lifetime of a vm. Driver-autoloading does require many other edits to the user guides. They are summarized in DERBY-1271. Regards, -Rick
Re: Driver autoloading and engine booting
Rick Hillegas wrote:
I'm afraid I don't understand why we would want to revert our tests to
the old behavior. It seems to me that our tests are practicing stunts
which we strongly discourage: changing Derby properties on the fly. In
general, there will always be Heisenbugs for applications which do
this and we should discourage this practice in the strongest possible
terms. What is the benefit of reverting the test behavior?
I think the scope of the problem is beyond generating derby.properties
on the fly.
I think the greater risk is anyone setting system properties to affect
derby behaviour. The biggest trouble maker I am guessing would be
derby.system.home.
The new requirement (I think) is that you set derby.system.home before
any driver is autoloaded and would have the symptoms described in this mail.
Is that description correct?
http://www.nabble.com/Re%3A--jira--Commented%3A-%28DERBY-930%29-Add-support-for-autoloading-of-Derby-client-drivers-p4909085.html
I find it not uncommon to see code like this:
System.setProperty("derby.system.home", MY_DERBY_DIR);
Class.forName("org.apache.derby.jdbc.EmbeddedDriver").newInstance();
If I understand the problem correctly, this will no longer reliably work
if any other code in the same JVM (or is it classloader context?)
makes a database connection to any database.
Is that correct?
The point of seeing if we can keep the tests the same is to see if we
can avoid user impact. We should olnly change the tests if and when
we decide the behavior change is ok. If later we add better tests
that more accurately reflect actual user scemarios like the one above
instead, that would be great, but the bottom line is just like the
tests, USERS DO STRANGE THINGS and expect things to still work. If
they upgrade and things suddenly don't work, they want the old behaviour
back and we can't give it to them because new users depend on the new
behaviour. We trap ourselves into a very ugly corner.and this is a very
bad and messy thing. It is better to keep things working if we can.
Every time we change a test to adapt to some new usage requirement it
should be a huge red flag that someone is going to hit the problem.
Kathey
[jira] Assigned: (DERBY-1241) logmirror.ctrl is getting accessed outside the privileged block when the checkpoint instant is invalid on log factory boot method.
[ http://issues.apache.org/jira/browse/DERBY-1241?page=all ] Suresh Thalamati reassigned DERBY-1241: --- Assign To: Suresh Thalamati > logmirror.ctrl is getting accessed outside the privileged block when the > checkpoint instant is invalid on log factory boot method. > > > Key: DERBY-1241 > URL: http://issues.apache.org/jira/browse/DERBY-1241 > Project: Derby > Type: Bug > Components: Store > Reporter: Suresh Thalamati > Assignee: Suresh Thalamati > Priority: Minor > > This problem was reported on the derby-dev list by Olav Sandstaa , filing > jira entry for it. > Olav Sandstaa wrote: > > Rick Hillegas <[EMAIL PROTECTED]> wrote: > > > > java.sql.SQLException: Java exception: 'access denied > > (java.io.FilePermission > > /export/home/tmp/derbyjdbc4/DerbyNetClient/TestConnectionMethods/wombat/log/logmirror.ctrl > > read): java.security.AccessControlException'. > > at > > java.security.AccessControlContext.checkPermission(AccessControlContext.java:321) > > at > > java.security.AccessController.checkPermission(AccessController.java:546) > > at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) > > at java.lang.SecurityManager.checkRead(SecurityManager.java:871) > > at java.io.File.exists(File.java:731) > > at > > org.apache.derby.impl.store.raw.log.LogToFile.boot(LogToFile.java:2940) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1996) > > at > > org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:290) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:542) > > at > > org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:418) > > at > > org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(BaseDataFileFactory.java:1762) > > at > > org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(BaseDataFileFactory.java:1218) > > at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:250) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1996) > > at > > org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:290) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:542) > > at > > org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:418) > > at > > org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:987) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1996) > > at > > org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:290) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:542) > > at > > org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:418) > > at > > org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:738) > > at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:178) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1996) > > at > > org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:290) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1831) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1697) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1577) > > at > > org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:990) > > at > > org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:541) > > at > > org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:1586) > > at > > org.apache.derby.impl.jdbc.EmbedConnection.(EmbedConnection.java:216) > > at > > org.apache.derby.impl.jdbc.EmbedConnection30.(EmbedConnection30.java:72) > > at > > org.apache.derby.impl.jdbc.EmbedConnection40.(EmbedConnection40.java:48) > > at > > org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:62) > > at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:199) > > at org.apache.derby.impl.drda.Database.makeConnection(Database.java:231) > > at > > org.apache.derby.impl.drda.DRDAConnThread.getConnFromDatabaseName(DRDAConnThread.java:1147) > > at > > org.apache.derby.impl.drda.DRDAConnThread.verifyUserIdPassword(DRDAConnThread.java:1125) > > at > > org.apache.derby.impl.drda.DRDAConnThread.parseSECCHK(DRDAConnThread.
Re: interested in working on DERBY-713
Hi Brent, Sounds like you're off to a good start. From the initial bug report, it looks like there's a good idea about which heuristic is being mis-applied here. Once you've studied the optimizer papers, I recommend that you post some high-level candidate solutions. Try to avoid optimizer jargon and concentrate on simple descriptions: o What query plan would you rather see? o What heuristic would the optimizer apply that would lead it to your preferred plan? o How would the optimizer decide to apply the new heuristic rather than the old one? I think you'll get some good feedback if your post contains the phrase "Attention, optimizer experts." I think that the optimizer enthusiasts on the list will give you good feedback: o Maybe they can think of a better query plan or heuristic. o Maybe they can see some awkward corner cases in your heuristic. o They can advise you on whether your heuristic will short-circuit other optimizer choices. o They can advise you on whether your heuristic will cause an explosion in the time that the optimizer takes. Thanks for wanting to scratch this itch! Regards, -Rick Brent Verner wrote: Hi, I've recently found need for an embedded java db and only Derby seems even close to handling the task, however the broken query planning for IN clauses makes it unusable :-(. I've decided to eschew an embedded db in favor of PG for now, but I'd really like to be able to use Derby in the near(ish) future for deployment. I'd like to try to fix the query planning around the IN clause. I'm reviewing the internals papers right now, but I'd appreciate any input that might point me in the right direction :-). cheers! Brent
Re: Driver autoloading and engine booting
Rick Hillegas wrote: > Kathey Marsden wrote: > >> David Van Couvering wrote: >> >>> >>> - Log a bug >>> >>> - Work on fixing it by booting on first connection rather than when >>> the driver is loaded. >>> >> It would be good to get input from the user community as well. One >> possible approach would be to log the bug, mark it as a regression, >> existing application impact, and release note needed. and have a >> formal release note. Then send a note to derby-user with a pointer to >> the bug and Ricks summary to see if any users feel there will be >> impact. Unfortunately, my gut instinct is that this is the kind of >> issue that lays dormant and then hits someone hard on upgrade or when >> integrated with another product.It might broadside users >> unaware, like it did the nist tests and DerbyNetAutostart tests. >> It will be really helpful to have the bug and the release note to >> present to users to help them understand impact. >> Then we can close DERBY-930 back up too until it is decided whether >> this "Heisenbug" is a show stopper or not. > > I have logged two new bugs: DERBY-1428 describes the existing, pre-JDBC4 > shape of the problem. DERBY-1429 describes the extra exposure introduced > by JDBC4. I have added a release note to DERBY-930, which summarizes the > problem and its workarounds. In addition, I have asked the user > community to let us know how broad the extra impact is. Does the "precedence for properties" documentation need to be updated to mention jdbc 4/jdk 1.6 behavior? http://db.apache.org/derby/docs/dev/tuning/ctunsetprop23308.html -jean
Re: Driver autoloading and engine booting
Kathey Marsden wrote: David Van Couvering wrote: - Log a bug - Work on fixing it by booting on first connection rather than when the driver is loaded. It would be good to get input from the user community as well. One possible approach would be to log the bug, mark it as a regression, existing application impact, and release note needed. and have a formal release note. Then send a note to derby-user with a pointer to the bug and Ricks summary to see if any users feel there will be impact. Unfortunately, my gut instinct is that this is the kind of issue that lays dormant and then hits someone hard on upgrade or when integrated with another product.It might broadside users unaware, like it did the nist tests and DerbyNetAutostart tests. It will be really helpful to have the bug and the release note to present to users to help them understand impact. Then we can close DERBY-930 back up too until it is decided whether this "Heisenbug" is a show stopper or not. I have logged two new bugs: DERBY-1428 describes the existing, pre-JDBC4 shape of the problem. DERBY-1429 describes the extra exposure introduced by JDBC4. I have added a release note to DERBY-930, which summarizes the problem and its workarounds. In addition, I have asked the user community to let us know how broad the extra impact is. -1 on removing the autoloading feature, unless people have evidence that this is going to cause real problems for our users. I'll analyze some usage scenarios that I am aware of and see what the impact might be. The most important thing is that this be a deliberate choice and that we not rush new functionality and introduce a regression where there is a reasonable technical alternative, where we can have autoloading and avoid the regression. Thanks. Your analysis will be very helpful. I also have a couple of questions: 1) How much work is it and what are the risks to moving the boot to the first connection ? 2) What are the implications to derby.drdra.startNetworkServer with the DERBY-930 change. Olav mentioned that even if the boot was moved, DerbyNetAutostart would still need to be changed. 3) If derby booted at the first connection or the first instantiation of the driver (whichever came first) , would that solve the derby.drda.startNetworkServer problem. Hi Olav, could you address these questions? If we could get DerbyNetAutoStart running without changes and nist put back the way it was, I think we would be ok. The fact that these tests require changes after DERBY-930 is a good indicator that users will be impacted. I'm afraid I don't understand why we would want to revert our tests to the old behavior. It seems to me that our tests are practicing stunts which we strongly discourage: changing Derby properties on the fly. In general, there will always be Heisenbugs for applications which do this and we should discourage this practice in the strongest possible terms. What is the benefit of reverting the test behavior? Thanks, -Rick Kathey Kathey
Re: [VOTE] 10.1.3.0 release
On 6/19/06, Bryan Pendleton <[EMAIL PROTECTED]> wrote: I'm still getting the Forbidden error on http://people.apache.org/~fuzzylogic/10.1.3.0/db-derby-10.1.3.0-bin.zip Fixed (again). Thanks for spotting that. andrew
Re: [jira] Commented: (DERBY-930) Add support for autoloading of Derby client drivers
Hi Bryan, Thanks for the quick response. I have revised the release note. Hopefully, it now describes the VM-specific issues better. Regards, -Rick Bryan Pendleton wrote: Rick Hillegas (JIRA) wrote: Adding a release note for this issue: Hi Rick, thanks for writing the release note. It wasn't clear to me from reading the release note whether this issue only affects JDK 1.6 environments, or whether it can arise under older VMs as well. If it only affects JDK 1.6 environments, perhaps we could change the release note to read: If an embedded Derby application runs in a JDK 1.6/JDBC4 environment, and generates its own Derby properties on the fly, and... thanks, bryan
[jira] Commented: (DERBY-930) Add support for autoloading of Derby client drivers
[ http://issues.apache.org/jira/browse/DERBY-930?page=comments#action_12416781 ] Rick Hillegas commented on DERBY-930: - Here is a second rev of the release note. This attempts to clarify the extra exposure introduced by jdk1.6. PROBLEM If an embedded Derby application generates its own Derby properties on the fly, and that Derby application runs in the same 1.6 VM with other JDBC applications, then you are not guaranteed that the Derby engine will pick up the custom properties when it boots. SYMPTOM Derby startup behavior will be non-deterministic. Sometimes the engine will pick up the custom properties and sometimes it won't. CAUSE JDBC4 introduced driver-autoloading. This causes all JDBC drivers visible to the 1.6 VM to register themselves the first time some application requests a Connection. When the embedded Derby driver registers itself, it also boots the Derby engine and the engine picks up whatever Derby properties are currently visible. An embedded Derby application may want to configure the engine properties before asking for a Connection. That embedded Derby application will not get a chance to configure engine properties if some other JDBC application in the same 1.6 VM runs first and requests a Connection. Two related bugs describe this issue in greater detail: DERBY-1428 and DERBY-1429. DERBY-1428 affects existing customer installations running old versions of Derby on the 1.3, 1.4, and 1.5 VMs. DERBY-1429 describes the extra exposure introduced with the 1.6 VM. SOLUTION There is no general solution to the problem. If two self-configuring embedded Derby applications run in the same VM, then only one of them can win. WORKAROUND The following workarounds may be useful: 1) Don't configure Derby properties inside your applications. Instead, specify Derby properties either on the VM startup line or in a $DERBY_HOME/derby.properties which remains constant for the VM's lifetime. 2) Don't run on the 1.6 VM. This eliminates the problem provided that you are not running more than one embedded Derby application in the same VM. 3) If (1) and (2) are not possible, then make the self-configuring Derby application run first. > Add support for autoloading of Derby client drivers > --- > > Key: DERBY-930 > URL: http://issues.apache.org/jira/browse/DERBY-930 > Project: Derby > Type: New Feature > Components: Build tools, JDBC > Reporter: Rick Hillegas > Assignee: Rick Hillegas > Fix For: 10.2.0.0 > Attachments: ClassloadingTest.jar, bug930.diff, bug930_problem.diff, > bug930_problem2.diff, bug930_rev2.diff, bug930_rev3.diff > > Write Derby's driver names into the correct spot in derby.jar and > derbyclient.jar so that the 1.6 vm autoloads Derby drivers. Section 10.2.1 of > the JDBC4 spec describes the details. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1429) Additional vulnerability to non-deterministic startup behavior when applications generate derby properties on the fly
[ http://issues.apache.org/jira/browse/DERBY-1429?page=comments#action_12416778 ] Rick Hillegas commented on DERBY-1429: -- We could separate out the booting of the engine from the registration of the embedded driver. This would eliminate this bug in the case that there is only one embedded Derby application in the VM. However, we can't solve the general problem of two self-configuring Derby applications running in the same VM. Only one of them can win. The user guide needs to strongly recommend that application designers avoid generating Derby properties on the fly. > Additional vulnerability to non-deterministic startup behavior when > applications generate derby properties on the fly > - > > Key: DERBY-1429 > URL: http://issues.apache.org/jira/browse/DERBY-1429 > Project: Derby > Type: Bug > Components: Store > Versions: 10.2.0.0 > Reporter: Rick Hillegas > > JDBC4 driver-autoloading, introduced by DERBY-930, increases the exposure to > non-deterministic startup behavior described in DERBY-1428. With the > introduction of driver-autloading, DERBY-1428 can be triggered if OtherApp is > any application which uses a JDBC driver. That is, OtherApp could use a Derby > client driver, the DB2JCC driver, the Oracle client driver, etc.. The extra > exposure arises because, with driver auto-loading, all JDBC drivers are > registered and the Derby engine boots the first time some application asks > for a Connection. > The issues are summarized in an email thread > http://mail-archives.apache.org/mod_mbox/db-derby-dev/200606.mbox/browser and > bug report DERBY-1399. > Workarounds are similar to those for DERBY-1428: > 1) Determine the derby properties BEFORE the VM starts. > 2) If that is not possible, then force the self-configuring embedded > application to run first. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] 10.1.3.0 release
Andrew McIntyre wrote: > There was a permissions problem with this particular file on the > server. It appears one of my sftp clients deposits files on the remote > server with bad permissions. > > I've corrected the permissions. You should be able to download the > files now. Please let me know if you still can't. Hi Andrew, I'm still getting the Forbidden error on http://people.apache.org/~fuzzylogic/10.1.3.0/db-derby-10.1.3.0-bin.zip thanks, bryan
Re: [jira] Commented: (DERBY-930) Add support for autoloading of Derby client drivers
Rick Hillegas (JIRA) wrote: Adding a release note for this issue: Hi Rick, thanks for writing the release note. It wasn't clear to me from reading the release note whether this issue only affects JDK 1.6 environments, or whether it can arise under older VMs as well. If it only affects JDK 1.6 environments, perhaps we could change the release note to read: If an embedded Derby application runs in a JDK 1.6/JDBC4 environment, and generates its own Derby properties on the fly, and... thanks, bryan
[jira] Updated: (DERBY-930) Add support for autoloading of Derby client drivers
[ http://issues.apache.org/jira/browse/DERBY-930?page=all ] Rick Hillegas updated DERBY-930: Derby Info: [Existing Application Impact, Regression, Release Note Needed] (was: [Existing Application Impact, Release Note Needed]) > Add support for autoloading of Derby client drivers > --- > > Key: DERBY-930 > URL: http://issues.apache.org/jira/browse/DERBY-930 > Project: Derby > Type: New Feature > Components: Build tools, JDBC > Reporter: Rick Hillegas > Assignee: Rick Hillegas > Fix For: 10.2.0.0 > Attachments: ClassloadingTest.jar, bug930.diff, bug930_problem.diff, > bug930_problem2.diff, bug930_rev2.diff, bug930_rev3.diff > > Write Derby's driver names into the correct spot in derby.jar and > derbyclient.jar so that the 1.6 vm autoloads Derby drivers. Section 10.2.1 of > the JDBC4 spec describes the details. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-930) Add support for autoloading of Derby client drivers
[ http://issues.apache.org/jira/browse/DERBY-930?page=comments#action_12416775 ] Rick Hillegas commented on DERBY-930: - Adding a release note for this issue: PROBLEM If an embedded Derby application generates its own Derby properties on the fly, and that Derby application runs in the same VM with other JDBC applications, then you are not guaranteed that the Derby engine will pick up the custom properties when it boots. SYMPTOM Derby startup behavior will be non-deterministic. Sometimes the engine will pick up the custom properties and sometimes it won't. CAUSE JDBC4 introduced driver-autoloading. This causes all JDBC drivers visible to the VM to register themselves the first time some application requests a Connection. When the embedded Derby driver registers itself, it also boots the Derby engine and the engine picks up whatever Derby properties are currently visible. An embedded Derby application may want to configure the engine properties before asking for a Connection. That embedded Derby application will not get a chance to configure engine properties if some other JDBC application in the same VM runs first and requests a Connection. Two related bugs describe this issue in greater detail: DERBY-1428 and DERBY-1429. SOLUTION There is no general solution to the problem. If two self-configuring embedded Derby applications run in the same VM, then only one of them can win. WORKAROUND The following workarounds may be useful: 1) Don't configure Derby properties inside your applications. Instead, specify Derby properties either on the VM startup line or in a $DERBY_HOME/derby.properties which remains constant for the VM's lifetime. 2) If (1) is not possible, then make the self-configuring Derby application run first. > Add support for autoloading of Derby client drivers > --- > > Key: DERBY-930 > URL: http://issues.apache.org/jira/browse/DERBY-930 > Project: Derby > Type: New Feature > Components: Build tools, JDBC > Reporter: Rick Hillegas > Assignee: Rick Hillegas > Fix For: 10.2.0.0 > Attachments: ClassloadingTest.jar, bug930.diff, bug930_problem.diff, > bug930_problem2.diff, bug930_rev2.diff, bug930_rev3.diff > > Write Derby's driver names into the correct spot in derby.jar and > derbyclient.jar so that the 1.6 vm autoloads Derby drivers. Section 10.2.1 of > the JDBC4 spec describes the details. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1428) Generating derby properties on the fly can lead to non-deterministic engine startup behavior
[ http://issues.apache.org/jira/browse/DERBY-1428?page=all ] Rick Hillegas updated DERBY-1428: - Description: A Heisenbug can arise if more than one embedded Derby application runs in the same VM and at least one of them generates derby properties on the fly. Here's the problem scenario: o The customer runs two embedded Derby apps in the same VM: EmbeddedApp and OtherApp. o EmbeddedApp generates derby properties on the fly before connecting to a database and triggering engine startup. o Whether the engine picks up those generated properties depends on whether EmbeddedApp or OtherApp runs first. Here are two workarounds for the problem: 1) Don't generate derby properties on the fly. Instead, specify them on the VM startup command. Or specify them in a $DERBY_HOME/derby.properties file which is generated before the VM comes up and which does not change during the lifetime of the VM. 2) If you can't do (1), then modify the VM startup script so that the self-configuring EmbeddedApp runs first. was: A Heisenbug can arise if more than one embedded Derby application runs in the same VM and at least one of them generates derby properties on the fly. Here's the problem scenario: o The customer runs two embedded Derby apps in the same VM: EmbeddedApp and OtherApp. o EmbeddedApp generates derby properties on the fly before connecting to a database and triggering engine startup. o Whether the engine picks up those generated properties depends on whether EmbeddedApp or OtherApp runs first. Here are two workarounds for the problem: 1) Don't generate derby properties on the fly. Instead, specify them on the VM startup command. Or specify them in a $DERBY_HOME/derby.properties file which is generated before the VM comes up and which does not change during the lifetime of the VM. 2) If you can't do (1), then modify the VM startup script so that it forces the applications to run in a deterministic order. In workaround (2) be explicit about the order in which the applications have to run. > Generating derby properties on the fly can lead to non-deterministic engine > startup behavior > > > Key: DERBY-1428 > URL: http://issues.apache.org/jira/browse/DERBY-1428 > Project: Derby > Type: Bug > Components: Store > Versions: 10.0.2.0, 10.0.2.1, 10.0.2.2, 10.1.1.0, 10.2.0.0, 10.1.2.1, > 10.1.3.0, 10.1.2.2, 10.1.2.3, 10.3.0.0, 10.1.2.4, 10.1.2.5, 10.1.4.0, 10.1.3.1 > Reporter: Rick Hillegas > > A Heisenbug can arise if more than one embedded Derby application runs in the > same VM and at least one of them generates derby properties on the fly. > Here's the problem scenario: > o The customer runs two embedded Derby apps in the same VM: EmbeddedApp and > OtherApp. > o EmbeddedApp generates derby properties on the fly before connecting to a > database and triggering engine startup. > o Whether the engine picks up those generated properties depends on whether > EmbeddedApp or OtherApp runs first. > Here are two workarounds for the problem: > 1) Don't generate derby properties on the fly. Instead, specify them on the > VM startup command. Or specify them in a $DERBY_HOME/derby.properties file > which is generated before the VM comes up and which does not change during > the lifetime of the VM. > 2) If you can't do (1), then modify the VM startup script so that the > self-configuring EmbeddedApp runs first. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (DERBY-1429) Additional vulnerability to non-deterministic startup behavior when applications generate derby properties on the fly
Additional vulnerability to non-deterministic startup behavior when applications generate derby properties on the fly - Key: DERBY-1429 URL: http://issues.apache.org/jira/browse/DERBY-1429 Project: Derby Type: Bug Components: Store Versions: 10.2.0.0 Reporter: Rick Hillegas JDBC4 driver-autoloading, introduced by DERBY-930, increases the exposure to non-deterministic startup behavior described in DERBY-1428. With the introduction of driver-autloading, DERBY-1428 can be triggered if OtherApp is any application which uses a JDBC driver. That is, OtherApp could use a Derby client driver, the DB2JCC driver, the Oracle client driver, etc.. The extra exposure arises because, with driver auto-loading, all JDBC drivers are registered and the Derby engine boots the first time some application asks for a Connection. The issues are summarized in an email thread http://mail-archives.apache.org/mod_mbox/db-derby-dev/200606.mbox/browser and bug report DERBY-1399. Workarounds are similar to those for DERBY-1428: 1) Determine the derby properties BEFORE the VM starts. 2) If that is not possible, then force the self-configuring embedded application to run first. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1374) compatibility test fails with 'PROTOCOL Data Stream Syntax Error'
[ http://issues.apache.org/jira/browse/DERBY-1374?page=comments#action_12416770 ] Andreas Korneliussen commented on DERBY-1374: - I will run tests on the patch and commit it. > compatibility test fails with 'PROTOCOL Data Stream Syntax Error' > - > > Key: DERBY-1374 > URL: http://issues.apache.org/jira/browse/DERBY-1374 > Project: Derby > Type: Bug > Components: Regression Test Failure > Versions: 10.2.0.0 > Environment: OS: ALL > JVM: server: JVM-1.3, client: JVM-1.3 > Derby: server: 10.1.2.0, client: Trunk > Reporter: Ole Solberg > Assignee: Fernanda Pizzorno > Priority: Critical > Attachments: derby-1374.diff, derby-1374.stat > > compatibility/serverVM-1.3_server10.1.2.0_clientVM-1.3_clientTrunk: > VM Version = 1.3 > Could not find com.ibm.db2.jcc.DB2Driver > Could not find org.apache.derby.jdbc.EmbeddedDriver > Driver org.apache.derby.jdbc.ClientDriver Version = 10.2 > java.sql.SQLException: A network protocol error was encountered and the > connection has been terminated: A PROTOCOL Data Stream Syntax Error was > detected. Reason: 0x8,544. > at > org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source) > at org.apache.derby.client.am.SqlException.getSQLException(Unknown > Source) > at org.apache.derby.jdbc.ClientDriver.connect(Unknown Source) > at java.sql.DriverManager.getConnection(DriverManager.java:512) > at java.sql.DriverManager.getConnection(DriverManager.java:141) > at > org.apache.derbyTesting.functionTests.util.DerbyJUnitTest.getConnection(Unknown > Source) > at > org.apache.derbyTesting.functionTests.util.DerbyJUnitTest.getConnection(Unknown > Source) > at > org.apache.derbyTesting.functionTests.tests.junitTests.compatibility.CompatibilitySuite.findServer(Unknown > Source) > at > org.apache.derbyTesting.functionTests.tests.junitTests.compatibility.CompatibilitySuite.main(Unknown > Source) > Exception in thread "main" java.lang.Exception: Error lookup up server info: > A network protocol error was encountered and the connection has been > terminated: A PROTOCOL Data Stream Syntax Error was detected. Reason: > 0x8,544. > at > org.apache.derbyTesting.functionTests.tests.junitTests.compatibility.CompatibilitySuite.findServer(Unknown > Source) > at > org.apache.derbyTesting.functionTests.tests.junitTests.compatibility.CompatibilitySuite.main(Unknown > Source) > The compatibility test aborts on failure, so lots of combinations were not > tested. > The following combinations succeeded: > compatibility/embedded_VM-1.3 > compatibility/embedded_VM-1.4 > compatibility/embedded_VM-1.5 > compatibility/serverVM-1.3_serverTrunk_clientVM-1.3_clientTrunk > compatibility/serverVM-1.3_serverTrunk_clientVM-1.3_clientDerby10.1.2.0 > compatibility/serverVM-1.3_serverTrunk_clientVM-1.3_clientDerby10.1.1.0 > compatibility/serverVM-1.3_serverTrunk_clientVM-1.3_clientDB2JCC > compatibility/serverVM-1.3_serverTrunk_clientVM-1.4_clientTrunk > compatibility/serverVM-1.3_serverTrunk_clientVM-1.4_clientDerby10.1.2.0 > compatibility/serverVM-1.3_serverTrunk_clientVM-1.4_clientDerby10.1.1.0 > compatibility/serverVM-1.3_serverTrunk_clientVM-1.4_clientDB2JCC > compatibility/serverVM-1.3_serverTrunk_clientVM-1.5_clientTrunk > compatibility/serverVM-1.3_serverTrunk_clientVM-1.5_clientDerby10.1.2.0 > compatibility/serverVM-1.3_serverTrunk_clientVM-1.5_clientDerby10.1.1.0 > compatibility/serverVM-1.3_serverTrunk_clientVM-1.5_clientDB2JCC -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-1384) Increase default BLOB/CLOB length to maximum supported (2G?)
[ http://issues.apache.org/jira/browse/DERBY-1384?page=comments#action_12416763 ] Bernt M. Johnsen commented on DERBY-1384: - Docs Committed revision 415329. > Increase default BLOB/CLOB length to maximum supported (2G?) > > > Key: DERBY-1384 > URL: http://issues.apache.org/jira/browse/DERBY-1384 > Project: Derby > Type: Improvement > Components: SQL > Reporter: Bernt M. Johnsen > Assignee: Bernt M. Johnsen > Priority: Minor > Fix For: 10.2.0.0 > Attachments: derby-1384-code.diff, derby-1384-code.stat, > derby-1384-docs.diff, derby-1384-docs.stat > > Default BLOB/CLOB length should be the maximum length supported by Derby (2G?) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (DERBY-1384) Increase default BLOB/CLOB length to maximum supported (2G?)
[ http://issues.apache.org/jira/browse/DERBY-1384?page=all ] Bernt M. Johnsen resolved DERBY-1384: - Resolution: Fixed Committed revision 415328. > Increase default BLOB/CLOB length to maximum supported (2G?) > > > Key: DERBY-1384 > URL: http://issues.apache.org/jira/browse/DERBY-1384 > Project: Derby > Type: Improvement > Components: SQL > Reporter: Bernt M. Johnsen > Assignee: Bernt M. Johnsen > Priority: Minor > Fix For: 10.2.0.0 > Attachments: derby-1384-code.diff, derby-1384-code.stat, > derby-1384-docs.diff, derby-1384-docs.stat > > Default BLOB/CLOB length should be the maximum length supported by Derby (2G?) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (DERBY-1428) Generating derby properties on the fly can lead to non-deterministic engine startup behavior
Generating derby properties on the fly can lead to non-deterministic engine startup behavior Key: DERBY-1428 URL: http://issues.apache.org/jira/browse/DERBY-1428 Project: Derby Type: Bug Components: Store Versions: 10.0.2.0, 10.0.2.1, 10.0.2.2, 10.1.1.0, 10.2.0.0, 10.1.2.1, 10.1.3.0, 10.1.2.2, 10.1.2.3, 10.3.0.0, 10.1.2.4, 10.1.2.5, 10.1.4.0, 10.1.3.1 Reporter: Rick Hillegas A Heisenbug can arise if more than one embedded Derby application runs in the same VM and at least one of them generates derby properties on the fly. Here's the problem scenario: o The customer runs two embedded Derby apps in the same VM: EmbeddedApp and OtherApp. o EmbeddedApp generates derby properties on the fly before connecting to a database and triggering engine startup. o Whether the engine picks up those generated properties depends on whether EmbeddedApp or OtherApp runs first. Here are two workarounds for the problem: 1) Don't generate derby properties on the fly. Instead, specify them on the VM startup command. Or specify them in a $DERBY_HOME/derby.properties file which is generated before the VM comes up and which does not change during the lifetime of the VM. 2) If you can't do (1), then modify the VM startup script so that it forces the applications to run in a deterministic order. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (DERBY-1341) LOB setBytes method(s) are currently no supported, but part of the Java 1.4 JDBC interface
[ http://issues.apache.org/jira/browse/DERBY-1341?page=all ] Anurag Shekhar updated DERBY-1341: -- Attachment: derby-1341.diff This diff is only for review of aprocah I am taking to resolve this issue. I have introduced 3 new class in this patch. LOBStreamControl, LOBOutputStream and LOBInputStream. LOBStreamControl acts as wrpper on top of bytes stored in memory and the temporary file on the file system. The read and write methods exposed by LOBStreamControl are used by EmbeddedBlob and LOBStream. when the total data size is less than 4k its stored in array once either the setBytes on blob or write on Stream incheses the size more than 4k the data is written into a temporrary file and subsequent operations are performed on the file. If call to truncate reduces the size of the file below 4k the data is again moved back to memory. Chanes made in EmbeddedBlob Changed EmbeddedBlob to pass all the calls which were priviously acessing the bytes array to LOBStreamControl. > LOB setBytes method(s) are currently no supported, but part of the Java 1.4 > JDBC interface > -- > > Key: DERBY-1341 > URL: http://issues.apache.org/jira/browse/DERBY-1341 > Project: Derby > Type: Bug > Components: JDBC > Versions: 10.0.2.0, 10.0.2.1, 10.0.2.2, 10.1.1.0, 10.2.0.0, 10.1.2.0, > 10.1.1.1, 10.1.1.2, 10.1.2.1, 10.1.3.0, 10.1.2.2, 10.1.2.3, 10.3.0.0, > 10.1.2.4, 10.1.2.5 > Environment: Windows 2000 > Reporter: Keith McFarlane > Assignee: Anurag Shekhar > Attachments: derby-1341.diff > > JDBC LOB . getBtypes methods are not implemented in any Derby version to > date: there is a "place-holder" method that throws a SQLException reporting > that the methods are not implemented. > It would be excellent to have any efficient Derby implementation of the > getBytes LOB methods that provide "random-access" to the binary // character > content of database large objects. The specific context is implementing a > Lucene Directory interface that stores indexing data (index files) and other > binary data in a local encrypted Derby instance. > A work around is to write an encrypted RandomAccessFile implementation as a > file-sdystem buffer, perhaps writing to the database on closure. An efficient > Derby implementation of LOB . getBytes would avoid this an make for a clean > design. I can think of several reasons why random-access to LOBs would be > valuable in a "hostile" client environment. > -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (DERBY-1427) sysinfo does not write Java SE and JDBC version when running under JDK 1.6
sysinfo does not write Java SE and JDBC version when running under JDK 1.6 -- Key: DERBY-1427 URL: http://issues.apache.org/jira/browse/DERBY-1427 Project: Derby Type: Bug Components: Tools Versions: 10.2.0.0 Environment: Sun JDK 1.6 Reporter: Olav Sandstaa Priority: Minor When running sysinfo with jdk 1.6 information about which Java SE and JDBC versions that is supported is missing in the output. The following is written: - Derby Information JRE - JDBC: ?-? The complete output from sysinfo: -- Java Information -- Java Version:1.6.0-beta2 Java Vendor: Sun Microsystems Inc. Java home: /usr/local/java/jdk1.6.0_b86/jre Java classpath: jars/sane/derby.jar OS name: SunOS OS architecture: x86 OS version: 5.10 Java user name: os136802 Java user home: /home/os136802 Java user dir: /home/os136802/derby/develop/jdk16/trunk java.specification.name: Java Platform API Specification java.specification.version: 1.6 - Derby Information JRE - JDBC: ?-? [/home/os136802/derby/develop/jdk16/trunk/jars/sane/derby.jar] 10.2.0.4 alpha - (415258M) -- - Locale Information - Current Locale : [English/United States [en_US]] Found support for locale: [de_DE] version: 10.2.0.4 alpha - (415258M) Found support for locale: [es] version: 10.2.0.4 alpha - (415258M) Found support for locale: [fr] version: 10.2.0.4 alpha - (415258M) Found support for locale: [it] version: 10.2.0.4 alpha - (415258M) Found support for locale: [ja_JP] version: 10.2.0.4 alpha - (415258M) Found support for locale: [ko_KR] version: 10.2.0.4 alpha - (415258M) Found support for locale: [pt_BR] version: 10.2.0.4 alpha - (415258M) Found support for locale: [zh_CN] version: 10.2.0.4 alpha - (415258M) Found support for locale: [zh_TW] version: 10.2.0.4 alpha - (415258M) -- -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Regression Test Failure! - Derby 415160 - Sun DBTG
[Auto-generated mail] *Derby* 415160/2006-06-18 19:46:38 CEST *derbyall* Failed TestsOK Skip Duration Platform --- *Jvm: 1.6* 15715700 0 106.20% SunOS-5.10_i86pc-i386 Details in http://www.multinet.no/~solberg/public/Apache/DerbyJDK16Jvm1.6/Limited/testSummary-415160.html Attempted failure analysis in http://www.multinet.no/~solberg/public/Apache/Failures/DerbyJDK16Jvm1.6/415160.html *Jvm: 1.5* 4675671 2 110.49% CYGWIN_NT-5.1_i686-unknown 3675672 2 115.57% CYGWIN_NT-5.2_i686-unknown 0675675 2 110.56% Linux-2.6.14-1.1644_FC4_i686-i686 0675675 2 105.30% Linux-2.6.9-34.ELsmp_x86_64-x86_64 0675675 2 176.69% SunOS-5.10_i86pc-i386 0675675 2 136.79% SunOS-5.10_sun4u-sparc NA NA NANA SunOS-5.11_i86pc-i386 0675675 2 113.45% SunOS-5.9_sun4u-sparc Details in http://www.multinet.no/~solberg/public/Apache/Derby/Limited/testSummary-415160.html Attempted failure analysis in http://www.multinet.no/~solberg/public/Apache/Failures/Derby/415160.html *Jvm: 1.4* 3669666 4 104.59% CYGWIN_NT-5.1_i686-unknown 0669669 4 111.38% Linux-2.6.14-1.1644_FC4_i686-i686 1669668 4 101.63% Linux-2.6.9-34.ELsmp_x86_64-x86_64 0669669 4 211.08% SunOS-5.10_i86pc-i386 Details in http://www.multinet.no/~solberg/public/Apache/DerbyJvm1.4/Limited/testSummary-415160.html Attempted failure analysis in http://www.multinet.no/~solberg/public/Apache/Failures/DerbyJvm1.4/415160.html Changes in http://www.multinet.no/~solberg/public/Apache/Derby/UpdateInfo/415160.txt ( All results in http://www.multinet.no/~solberg/public/Apache/index.html )
[jira] Updated: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
[ http://issues.apache.org/jira/browse/DERBY-1156?page=all ] Suresh Thalamati updated DERBY-1156: Attachment: reencrypt_3.diff DERBY -1156 (partial) This patch adds some code required to support reconfigure(rencryption) of an already existing encrypted database with a new password(secret key) or an external user specified encryption key. Two new attributes "newBootPassword" and "newEncryptionkey" are introduced to support this functionality. -- modified the code to support two have instance of two cipher factories to exist. So that the existing data can decrypted with the old encryption key using one cipher factory and rewrite the data with new encryption keys using another cipher factory. -- reenryption of the database with new keys is similar to encrypting an already existing database. All the container data is read through the page cache and rewritten using the new encryption keys. -- Added test case to test the re-encryption of an encrypted database. TESTS : derbyall test suite passed on Windows XP/JDK142 It would be great if some can review this patch. svn stat: M java\engine\org\apache\derby\impl\services\jce\JCECipherFactory.java A java\engine\org\apache\derby\impl\services\jce\JCECipherFactoryBuilder.java M java\engine\org\apache\derby\impl\store\raw\RawStore.java A java\engine\org\apache\derby\iapi\services\crypto\CipherFactoryBuilder.java M java\engine\org\apache\derby\iapi\reference\Attribute.java M java\engine\org\apache\derby\iapi\reference\Module.java M java\engine\org\apache\derby\modules.properties M java\testing\org\apache\derbyTesting\unitTests\crypto\T_Cipher.java M java\testing\org\apache\derbyTesting\functionTests\tests\store\encryptDatabaseTest1.sql M java\testing\org\apache\derbyTesting\functionTests\master\encryptDatabaseTest1.out > allow the encrypting of an existing unencrypted db and allow the > re-encrypting of an existing encrypted db > -- > > Key: DERBY-1156 > URL: http://issues.apache.org/jira/browse/DERBY-1156 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.2.3 > Reporter: Mike Matrigali > Assignee: Suresh Thalamati > Priority: Minor > Fix For: 10.2.0.0 > Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, > reencrypt_3.diff > > encrypted database to be re-encrypted with a new password. > Here are some ideas for an initial implementation. > The easiest way to do this is to make sure we have exclusive access to the > data and that no log is required in the new copy of the db. I want to avoid > the log as it also is encrypted. Here is my VERY high level plan: > 1) Force exclusive access by putting all the work in the low level store, >offline boot method. We will do redo recovery as usual, but at the end >there will be an entry point to do the copy/encrypt operation. > copy/encrypt process: > 0) The request to encrypt/re-encrypt the db will be handled with a new set >of url flags passed into store at boot time. The new flags will provide >the same inputs as the current encrypt flags. So at high level the >request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags". >TODO - provide exact new flag syntax. > 1) Open a transaction do all logged work to do the encryption. All logging >will be done with existing encryption. > 2) Copy and encrypt every db file in the database. The target files will >be in the data directory. There will be a new suffix to track the new >files, similar to the current process used for handling drop table in >a transaction consistent manner without logging the entire table to the > log. >Entire encrypted destination file is guaranteed synced to disk before >transaction commits. I don't think this part needs to be logged. >Files will be read from the cache using existing mechanism and written >directly into new encrypted files (new encrypted data does not end up in >the cache). > 3) Switch encrypted files for old files. Do this under a new log operation >so the process can be correctly rolled back if the encrypt db operation >transaction fails. Rollback will do file at a time switches, no reading >of encrypted data is necessary. > 4) log a "change encryption of db" log record, but do not update >system.properties with the change. > 5) commit transaction. > 6) update system.properties and sync changes. > 7) TODO - need someway to handle crash between steps 5 and 6. > 6) checkpoint all data, at this point guaranteed that there is no outstanding >transaction, so after checkpoint is done there is no need for the log. > ISSUES: > o there probably should be something that catches a request to encrypt to > whatever db w
