[jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
[ http://issues.apache.org/jira/browse/DERBY-666?page=comments#action_12356460 ] Francois Orsini commented on DERBY-666: --- The new J2SE 5.0 has some new API to dump individual or all threads' stracktrace running in a JVM - There is a new notion of StackTraceElement object which represent a stack frame and can be output'ed as a String So you can get all frames of a particular thread's stack dump as well as for all threads in the JVM. http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html#getStackTrace() http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html#getAllStackTraces() http://java.sun.com/j2se/1.5.0/docs/api/java/lang/StackTraceElement.html Thread(s) stack dumps can also be performed on the command line using 'jstack' (1.5) utility to dump all the JVM's thread stack traces given a JVM pid. http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html fyi. > Enhance derby.locks.deadlockTrace to print stack traces for all threads > involved in a deadlock > -- > > Key: DERBY-666 > URL: http://issues.apache.org/jira/browse/DERBY-666 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.1.0 > Reporter: Bryan Pendleton > Priority: Minor > > I was reading http://www.linux-mag.com/content/view/2134/ (good article, > btw!), and it says: > > The next two properties are needed to diagnose concurrency (locking and > > deadlock) problems. > > > > *derby.locks.monitor=true logs all deadlocks that occur in the system. > > *derby.locks.deadlockTrace=true log a stack trace of all threads > > involved in lock-related rollbacks. > It seems, that, in my environment, the deadlockTrace property does not log a > stack trace of *all* threads involved in the deadlock. > Instead, it only logs a stack trace of the *victim* thread involved in the > deadlock. > I think it would be very useful if the derby.locks.deadlockTrace setting > could in fact log a stack trace of all involved threads. > In a posting to derby-dev, Mike Matrigali noted that an earlier > implementation of a similar feature had to be removed because it was too > expensive in both time and space, but he suggested that there might be > several possible ways to implement this in an acceptably efficient manner: > > A long time ago there use to be room in each lock to point at a > > stack trace for each lock, but that was removed to optimize the size > > of the lock data structure which can have many objects outstanding. > > And creating and storing the stack for every lock was incredibly slow > > and just was not very useful for any very active application. I think > > I was the only one who ever used it. > > > > The plan was sometime to add a per user data structure which could be > > filled in when it was about to wait on a lock, which would give most of > > what is interesting in a deadlock. > > > > The current deadlockTrace is meant to dump the lock table out to derby.log > > when a deadlock is encountered. > > > > I agree getting a dump of all stack traces would be very useful, and > > with the later jvm debug interfaces may now be possible - in earlier > > JVM's there weren't any java interfaces to do so. Does anyone have > > the code to donate to dump all thread stacks to a buffer? > Mike also suggested a manual technique as a workaround; it would be useful to > put this into the documentation somewhere, perhaps on the page which > documents derby.locks.deadlockTrace? Here's Mike's suggestion: > > What I do if I can reproduce easily is set try to catch the wait by > > hand and then depending on the environment either send the magic > > signal or hit ctrl-break in the server window which will send the JVM > > specific thread dumps to derby.log. > The magic signal, btw, is 'kill -QUIT', at least with Sun JVMs in my > experience. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
[
http://issues.apache.org/jira/browse/DERBY-666?page=comments#action_12356519 ]
Mike Matrigali commented on DERBY-666:
--
Here are some pointers for anyone interested in picking up this project. It
seems like a good 1st project for someone interested in doing derby development:
1) use the interfaces as described by Francois's comments above.
2) The code must be within a securiity block, I believe Dan will be providing
some more info on this area soon.
3) See posting by Dan on list under subject "Re: is getAllStackTraces()
something we are allowed to call from the server given the recent
SecurityManager changes?"
4) I would sugest coding a try/catch block around the security exception and
not fail the operation if we get a security exception.
In this case add the extra information to the lock error message if you
can, otherwise let the system continue. You might
print a message saying stack traces not available due to not enough
permission granted by security manager.
5) The code wants to go in
opensource/java/engine/org/apache/derby/impl/services/locks/Deadlock.java, in
the buildException() routine.
Lars posted some code, it would need to get the traces into a string buffer:
The following ought to do it:
public static void dumpTraces(PrintStream out) {
Map threadtraces =
Thread.getAllStackTraces();
for (Map.Entry threadtrace
: threadtraces.entrySet()) {
out.println(threadtrace.getKey().getName());
StackTraceElement[] traces = threadtrace.getValue();
for (StackTraceElement trace : traces) {
out.println(trace);
}
}
}
The locktable could possibly remember which thread corresponds to each
transaction and just dump those.
Mapping threads to transactions is hard as it is possible for the thread of a
transaction to change with every
jdbc interaction if a connection pool is involved. The interesting stack
traces are the ones that are waiting and those transactions could note their
thread id before waiting.
> Enhance derby.locks.deadlockTrace to print stack traces for all threads
> involved in a deadlock
> --
>
> Key: DERBY-666
> URL: http://issues.apache.org/jira/browse/DERBY-666
> Project: Derby
> Type: Improvement
> Components: Store
> Versions: 10.1.1.0
> Reporter: Bryan Pendleton
> Priority: Minor
>
> I was reading http://www.linux-mag.com/content/view/2134/ (good article,
> btw!), and it says:
> > The next two properties are needed to diagnose concurrency (locking and
> > deadlock) problems.
> >
> > *derby.locks.monitor=true logs all deadlocks that occur in the system.
> > *derby.locks.deadlockTrace=true log a stack trace of all threads
> > involved in lock-related rollbacks.
> It seems, that, in my environment, the deadlockTrace property does not log a
> stack trace of *all* threads involved in the deadlock.
> Instead, it only logs a stack trace of the *victim* thread involved in the
> deadlock.
> I think it would be very useful if the derby.locks.deadlockTrace setting
> could in fact log a stack trace of all involved threads.
> In a posting to derby-dev, Mike Matrigali noted that an earlier
> implementation of a similar feature had to be removed because it was too
> expensive in both time and space, but he suggested that there might be
> several possible ways to implement this in an acceptably efficient manner:
> > A long time ago there use to be room in each lock to point at a
> > stack trace for each lock, but that was removed to optimize the size
> > of the lock data structure which can have many objects outstanding.
> > And creating and storing the stack for every lock was incredibly slow
> > and just was not very useful for any very active application. I think
> > I was the only one who ever used it.
> >
> > The plan was sometime to add a per user data structure which could be
> > filled in when it was about to wait on a lock, which would give most of
> > what is interesting in a deadlock.
> >
> > The current deadlockTrace is meant to dump the lock table out to derby.log
> > when a deadlock is encountered.
> >
> > I agree getting a dump of all stack traces would be very useful, and
> > with the later jvm debug interfaces may now be possible - in earlier
> > JVM's there weren't any java interfaces to do so. Does anyone have
> > the code to donate to dump all thread stacks to a buffer?
> Mike also suggested a manual technique as a workaround; it would be useful to
> put this into the documentation somewhere, perhaps on the page which
> documents derby.locks.deadlockTrace? Here's Mike's suggestion:
> > What I do if I can reproduce easily is set try to catch the wait by
> > hand and then depending on the environment either send the magic
[jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
[ http://issues.apache.org/jira/browse/DERBY-666?page=comments#action_12356791 ] Bryan Pendleton commented on DERBY-666: --- Thanks Mike -- I'm thinking about giving this a try. How do I write some JDK 1.5 dependent code such that Derby can still build and run successfully in JDK 1.3 and 1.4 environments? Do I have to use Reflection to do this? Is there some better technique used elsewhere in Derby? Also, does adding JDK 1.5 dependent code into Derby mean that I'd need to adjust the build and test instructions and Ant scripts so that developers could build and test Derby with JDK 1.5? > Enhance derby.locks.deadlockTrace to print stack traces for all threads > involved in a deadlock > -- > > Key: DERBY-666 > URL: http://issues.apache.org/jira/browse/DERBY-666 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.1.0 > Reporter: Bryan Pendleton > Priority: Minor > > I was reading http://www.linux-mag.com/content/view/2134/ (good article, > btw!), and it says: > > The next two properties are needed to diagnose concurrency (locking and > > deadlock) problems. > > > > *derby.locks.monitor=true logs all deadlocks that occur in the system. > > *derby.locks.deadlockTrace=true log a stack trace of all threads > > involved in lock-related rollbacks. > It seems, that, in my environment, the deadlockTrace property does not log a > stack trace of *all* threads involved in the deadlock. > Instead, it only logs a stack trace of the *victim* thread involved in the > deadlock. > I think it would be very useful if the derby.locks.deadlockTrace setting > could in fact log a stack trace of all involved threads. > In a posting to derby-dev, Mike Matrigali noted that an earlier > implementation of a similar feature had to be removed because it was too > expensive in both time and space, but he suggested that there might be > several possible ways to implement this in an acceptably efficient manner: > > A long time ago there use to be room in each lock to point at a > > stack trace for each lock, but that was removed to optimize the size > > of the lock data structure which can have many objects outstanding. > > And creating and storing the stack for every lock was incredibly slow > > and just was not very useful for any very active application. I think > > I was the only one who ever used it. > > > > The plan was sometime to add a per user data structure which could be > > filled in when it was about to wait on a lock, which would give most of > > what is interesting in a deadlock. > > > > The current deadlockTrace is meant to dump the lock table out to derby.log > > when a deadlock is encountered. > > > > I agree getting a dump of all stack traces would be very useful, and > > with the later jvm debug interfaces may now be possible - in earlier > > JVM's there weren't any java interfaces to do so. Does anyone have > > the code to donate to dump all thread stacks to a buffer? > Mike also suggested a manual technique as a workaround; it would be useful to > put this into the documentation somewhere, perhaps on the page which > documents derby.locks.deadlockTrace? Here's Mike's suggestion: > > What I do if I can reproduce easily is set try to catch the wait by > > hand and then depending on the environment either send the magic > > signal or hit ctrl-break in the server window which will send the JVM > > specific thread dumps to derby.log. > The magic signal, btw, is 'kill -QUIT', at least with Sun JVMs in my > experience. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
[ http://issues.apache.org/jira/browse/DERBY-666?page=comments#action_12357750 ] Bryan Pendleton commented on DERBY-666: --- Argh. I hate getting old. Anyway, after following about 50 dead-end ratholes involving tweaking the build scripts, I finally took Mike's advice and just hand-compiled the one file with the suggested code to call Thread.getAllStackTraces, and built a system that I could play with. The good news is that this worked, and I was able to run this code, and sure enough it produces the stack traces! The bad news is that, in doing the experiment, I realized that in my environment this feature is of no use, for I am running in a Network Server environment, and so the result is that I get the stack traces of the *server*, and what I want are the stack traces of my various *client* threads. Duh. Should have been obvious, but, as I say, I'm getting old. I'm going to try writing a simple JDBC embedded app to verify that this does actually work correctly in the embedded case, and that it gets useful stack traces in that case, and then I'll at least have done *something* useful, even if I now realize that I can't use this feature for the case that I really cared about (my production system, which runs in NetworkServer mode). Thanks to everybody for their help so far. :) > Enhance derby.locks.deadlockTrace to print stack traces for all threads > involved in a deadlock > -- > > Key: DERBY-666 > URL: http://issues.apache.org/jira/browse/DERBY-666 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.1.0 > Reporter: Bryan Pendleton > Priority: Minor > > I was reading http://www.linux-mag.com/content/view/2134/ (good article, > btw!), and it says: > > The next two properties are needed to diagnose concurrency (locking and > > deadlock) problems. > > > > *derby.locks.monitor=true logs all deadlocks that occur in the system. > > *derby.locks.deadlockTrace=true log a stack trace of all threads > > involved in lock-related rollbacks. > It seems, that, in my environment, the deadlockTrace property does not log a > stack trace of *all* threads involved in the deadlock. > Instead, it only logs a stack trace of the *victim* thread involved in the > deadlock. > I think it would be very useful if the derby.locks.deadlockTrace setting > could in fact log a stack trace of all involved threads. > In a posting to derby-dev, Mike Matrigali noted that an earlier > implementation of a similar feature had to be removed because it was too > expensive in both time and space, but he suggested that there might be > several possible ways to implement this in an acceptably efficient manner: > > A long time ago there use to be room in each lock to point at a > > stack trace for each lock, but that was removed to optimize the size > > of the lock data structure which can have many objects outstanding. > > And creating and storing the stack for every lock was incredibly slow > > and just was not very useful for any very active application. I think > > I was the only one who ever used it. > > > > The plan was sometime to add a per user data structure which could be > > filled in when it was about to wait on a lock, which would give most of > > what is interesting in a deadlock. > > > > The current deadlockTrace is meant to dump the lock table out to derby.log > > when a deadlock is encountered. > > > > I agree getting a dump of all stack traces would be very useful, and > > with the later jvm debug interfaces may now be possible - in earlier > > JVM's there weren't any java interfaces to do so. Does anyone have > > the code to donate to dump all thread stacks to a buffer? > Mike also suggested a manual technique as a workaround; it would be useful to > put this into the documentation somewhere, perhaps on the page which > documents derby.locks.deadlockTrace? Here's Mike's suggestion: > > What I do if I can reproduce easily is set try to catch the wait by > > hand and then depending on the environment either send the magic > > signal or hit ctrl-break in the server window which will send the JVM > > specific thread dumps to derby.log. > The magic signal, btw, is 'kill -QUIT', at least with Sun JVMs in my > experience. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
[ http://issues.apache.org/jira/browse/DERBY-666?page=comments#action_12357760 ] Bryan Pendleton commented on DERBY-666: --- Hmmm.. I have no trouble writing a program which causes a deadlock to be detected when running in a Client/Server configuration, but when I run in an Embedded configuration I simply cannot cause a deadlock to be detected. I always get a lock timeout instead. I get a lock timeout in Embedded mode even when I set derby.locks.waitTimeout to -1. I'll attach the program I'm using to try to force a deadlock in Embedded mode; maybe somebody will spot my bug. > Enhance derby.locks.deadlockTrace to print stack traces for all threads > involved in a deadlock > -- > > Key: DERBY-666 > URL: http://issues.apache.org/jira/browse/DERBY-666 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.1.0 > Reporter: Bryan Pendleton > Priority: Minor > > I was reading http://www.linux-mag.com/content/view/2134/ (good article, > btw!), and it says: > > The next two properties are needed to diagnose concurrency (locking and > > deadlock) problems. > > > > *derby.locks.monitor=true logs all deadlocks that occur in the system. > > *derby.locks.deadlockTrace=true log a stack trace of all threads > > involved in lock-related rollbacks. > It seems, that, in my environment, the deadlockTrace property does not log a > stack trace of *all* threads involved in the deadlock. > Instead, it only logs a stack trace of the *victim* thread involved in the > deadlock. > I think it would be very useful if the derby.locks.deadlockTrace setting > could in fact log a stack trace of all involved threads. > In a posting to derby-dev, Mike Matrigali noted that an earlier > implementation of a similar feature had to be removed because it was too > expensive in both time and space, but he suggested that there might be > several possible ways to implement this in an acceptably efficient manner: > > A long time ago there use to be room in each lock to point at a > > stack trace for each lock, but that was removed to optimize the size > > of the lock data structure which can have many objects outstanding. > > And creating and storing the stack for every lock was incredibly slow > > and just was not very useful for any very active application. I think > > I was the only one who ever used it. > > > > The plan was sometime to add a per user data structure which could be > > filled in when it was about to wait on a lock, which would give most of > > what is interesting in a deadlock. > > > > The current deadlockTrace is meant to dump the lock table out to derby.log > > when a deadlock is encountered. > > > > I agree getting a dump of all stack traces would be very useful, and > > with the later jvm debug interfaces may now be possible - in earlier > > JVM's there weren't any java interfaces to do so. Does anyone have > > the code to donate to dump all thread stacks to a buffer? > Mike also suggested a manual technique as a workaround; it would be useful to > put this into the documentation somewhere, perhaps on the page which > documents derby.locks.deadlockTrace? Here's Mike's suggestion: > > What I do if I can reproduce easily is set try to catch the wait by > > hand and then depending on the environment either send the magic > > signal or hit ctrl-break in the server window which will send the JVM > > specific thread dumps to derby.log. > The magic signal, btw, is 'kill -QUIT', at least with Sun JVMs in my > experience. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
[ http://issues.apache.org/jira/browse/DERBY-666?page=comments#action_12357762 ] Bryan Pendleton commented on DERBY-666: --- The strange thing is, when I run the program in Embedded mode, and I get the lock timeout, the program *is* sensitive to the value of derby.locks.deadlockTimeout! If I set derby.locks.deadlockTimeout to 5, the program gives the "A lock could not be obtained within the time requested." error after 5 seconds, rather than after the standard 20 seconds. So it seems that the Embedded configuration is detecting the problem as a deadlock, fundamentally; it's just reporting it back out to my program as a lock timeout rather than as a deadlock. I think I'd better stop now. I've succeeded in getting myself thoroughly confused :) > Enhance derby.locks.deadlockTrace to print stack traces for all threads > involved in a deadlock > -- > > Key: DERBY-666 > URL: http://issues.apache.org/jira/browse/DERBY-666 > Project: Derby > Type: Improvement > Components: Store > Versions: 10.1.1.0 > Reporter: Bryan Pendleton > Priority: Minor > Attachments: repro.java > > I was reading http://www.linux-mag.com/content/view/2134/ (good article, > btw!), and it says: > > The next two properties are needed to diagnose concurrency (locking and > > deadlock) problems. > > > > *derby.locks.monitor=true logs all deadlocks that occur in the system. > > *derby.locks.deadlockTrace=true log a stack trace of all threads > > involved in lock-related rollbacks. > It seems, that, in my environment, the deadlockTrace property does not log a > stack trace of *all* threads involved in the deadlock. > Instead, it only logs a stack trace of the *victim* thread involved in the > deadlock. > I think it would be very useful if the derby.locks.deadlockTrace setting > could in fact log a stack trace of all involved threads. > In a posting to derby-dev, Mike Matrigali noted that an earlier > implementation of a similar feature had to be removed because it was too > expensive in both time and space, but he suggested that there might be > several possible ways to implement this in an acceptably efficient manner: > > A long time ago there use to be room in each lock to point at a > > stack trace for each lock, but that was removed to optimize the size > > of the lock data structure which can have many objects outstanding. > > And creating and storing the stack for every lock was incredibly slow > > and just was not very useful for any very active application. I think > > I was the only one who ever used it. > > > > The plan was sometime to add a per user data structure which could be > > filled in when it was about to wait on a lock, which would give most of > > what is interesting in a deadlock. > > > > The current deadlockTrace is meant to dump the lock table out to derby.log > > when a deadlock is encountered. > > > > I agree getting a dump of all stack traces would be very useful, and > > with the later jvm debug interfaces may now be possible - in earlier > > JVM's there weren't any java interfaces to do so. Does anyone have > > the code to donate to dump all thread stacks to a buffer? > Mike also suggested a manual technique as a workaround; it would be useful to > put this into the documentation somewhere, perhaps on the page which > documents derby.locks.deadlockTrace? Here's Mike's suggestion: > > What I do if I can reproduce easily is set try to catch the wait by > > hand and then depending on the environment either send the magic > > signal or hit ctrl-break in the server window which will send the JVM > > specific thread dumps to derby.log. > The magic signal, btw, is 'kill -QUIT', at least with Sun JVMs in my > experience. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
[ https://issues.apache.org/jira/browse/DERBY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510913 ] Kurt Huwig commented on DERBY-666: -- Bryan, I am using embedded Derby and have this issue. Could you post what you did, please? > Enhance derby.locks.deadlockTrace to print stack traces for all threads > involved in a deadlock > -- > > Key: DERBY-666 > URL: https://issues.apache.org/jira/browse/DERBY-666 > Project: Derby > Issue Type: Improvement > Components: Store >Affects Versions: 10.1.1.0 >Reporter: Bryan Pendleton >Priority: Minor > Attachments: repro.java > > > I was reading http://www.linux-mag.com/content/view/2134/ (good article, > btw!), and it says: > > The next two properties are needed to diagnose concurrency (locking and > > deadlock) problems. > > > > *derby.locks.monitor=true logs all deadlocks that occur in the system. > > *derby.locks.deadlockTrace=true log a stack trace of all threads > > involved in lock-related rollbacks. > It seems, that, in my environment, the deadlockTrace property does not log a > stack trace of *all* threads involved in the deadlock. > Instead, it only logs a stack trace of the *victim* thread involved in the > deadlock. > I think it would be very useful if the derby.locks.deadlockTrace setting > could in fact log a stack trace of all involved threads. > In a posting to derby-dev, Mike Matrigali noted that an earlier > implementation of a similar feature had to be removed because it was too > expensive in both time and space, but he suggested that there might be > several possible ways to implement this in an acceptably efficient manner: > > A long time ago there use to be room in each lock to point at a > > stack trace for each lock, but that was removed to optimize the size > > of the lock data structure which can have many objects outstanding. > > And creating and storing the stack for every lock was incredibly slow > > and just was not very useful for any very active application. I think > > I was the only one who ever used it. > > > > The plan was sometime to add a per user data structure which could be > > filled in when it was about to wait on a lock, which would give most of > > what is interesting in a deadlock. > > > > The current deadlockTrace is meant to dump the lock table out to derby.log > > when a deadlock is encountered. > > > > I agree getting a dump of all stack traces would be very useful, and > > with the later jvm debug interfaces may now be possible - in earlier > > JVM's there weren't any java interfaces to do so. Does anyone have > > the code to donate to dump all thread stacks to a buffer? > Mike also suggested a manual technique as a workaround; it would be useful to > put this into the documentation somewhere, perhaps on the page which > documents derby.locks.deadlockTrace? Here's Mike's suggestion: > > What I do if I can reproduce easily is set try to catch the wait by > > hand and then depending on the environment either send the magic > > signal or hit ctrl-break in the server window which will send the JVM > > specific thread dumps to derby.log. > The magic signal, btw, is 'kill -QUIT', at least with Sun JVMs in my > experience. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
[ https://issues.apache.org/jira/browse/DERBY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12510914 ] Bryan Pendleton commented on DERBY-666: --- Kurt, I'm sorry, I no longer have the experimental changes that I was working with in 2005. As I recall, I pretty much took the code from Mike's 11-Nov-2005 comment above, and pasted it into Deadlock.java, as he suggested, and it worked just fine, so I should think you'd be able to build a custom version of the Derby code for yourself with such a change without too much work. Give it a try, and if you run into trouble, I'm sure the community will be glad to help. > Enhance derby.locks.deadlockTrace to print stack traces for all threads > involved in a deadlock > -- > > Key: DERBY-666 > URL: https://issues.apache.org/jira/browse/DERBY-666 > Project: Derby > Issue Type: Improvement > Components: Store >Affects Versions: 10.1.1.0 >Reporter: Bryan Pendleton >Priority: Minor > Attachments: repro.java > > > I was reading http://www.linux-mag.com/content/view/2134/ (good article, > btw!), and it says: > > The next two properties are needed to diagnose concurrency (locking and > > deadlock) problems. > > > > *derby.locks.monitor=true logs all deadlocks that occur in the system. > > *derby.locks.deadlockTrace=true log a stack trace of all threads > > involved in lock-related rollbacks. > It seems, that, in my environment, the deadlockTrace property does not log a > stack trace of *all* threads involved in the deadlock. > Instead, it only logs a stack trace of the *victim* thread involved in the > deadlock. > I think it would be very useful if the derby.locks.deadlockTrace setting > could in fact log a stack trace of all involved threads. > In a posting to derby-dev, Mike Matrigali noted that an earlier > implementation of a similar feature had to be removed because it was too > expensive in both time and space, but he suggested that there might be > several possible ways to implement this in an acceptably efficient manner: > > A long time ago there use to be room in each lock to point at a > > stack trace for each lock, but that was removed to optimize the size > > of the lock data structure which can have many objects outstanding. > > And creating and storing the stack for every lock was incredibly slow > > and just was not very useful for any very active application. I think > > I was the only one who ever used it. > > > > The plan was sometime to add a per user data structure which could be > > filled in when it was about to wait on a lock, which would give most of > > what is interesting in a deadlock. > > > > The current deadlockTrace is meant to dump the lock table out to derby.log > > when a deadlock is encountered. > > > > I agree getting a dump of all stack traces would be very useful, and > > with the later jvm debug interfaces may now be possible - in earlier > > JVM's there weren't any java interfaces to do so. Does anyone have > > the code to donate to dump all thread stacks to a buffer? > Mike also suggested a manual technique as a workaround; it would be useful to > put this into the documentation somewhere, perhaps on the page which > documents derby.locks.deadlockTrace? Here's Mike's suggestion: > > What I do if I can reproduce easily is set try to catch the wait by > > hand and then depending on the environment either send the magic > > signal or hit ctrl-break in the server window which will send the JVM > > specific thread dumps to derby.log. > The magic signal, btw, is 'kill -QUIT', at least with Sun JVMs in my > experience. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
sorry for the delay, been out of town. Some others may need to weigh in on the build issues, I believe this would be the first jdk 1.5 dependent feature. I'll give a quick overview on how the code has done this in the past for jdk1.4 specific features. If it were me I would just first implement it directly and get that working without worrying about module stuff. Then address the module stuff - hopefully with some help from the list. Derby has addressed jdk specific features at the "module" level, implementing a different module to be loaded at boot time dependent on the jdk that is running. This is going to pretty heavy weight for the change you are considering - but I don't know a different way. An example of how this is done for a jdk 1.4 feature is the use of file locking by the io system to prevent 2 jvm's from accessing the same database at the same time. In this case a jdk1.4 specific derby module was created: opensource/java/engine/org/apache/derby/impl/io/DirStorageFactory4.java This factory extends the standard factory, and the monitor is in charge of making sure it is loaded rather than DirStorageFactory.java, if running in a jvm of 1.4 or higher. There is some hard coded handling of this factory loading in: opensource/java/engine/org/apache/derby/impl/services/monitor/BaseMonitor.java - look for DirStorageFactory This code actually surprised me a bit and I am going to post a separate thread on how the right way to do this. I was expecting some entries in the modules.properties following the scheme used by the jdbc30 stuff and - see file and search for J4: opensource/java/engine/org/apache/derby This factory then just implements the code which insure the correct java modules are loaded if running in jdk1.4 or not. In it's case it overrides the producer methods to make sure DirRandomAccessFile4.java and DirFile4.java are used. Also take a look in that directory's build.xml file it has some magic for building those specific files. So in your case I believe you will need a separate locking module. The standard locking module is: opensource/java/engine/org/apache/derby/impl/services/locks/SinglePool.java So you will need at least something like: opensource/java/engine/org/apache/derby/impl/services/locks/SinglePool5.java opensource/java/engine/org/apache/derby/impl/services/locks/DeadLock5.java I haven't looked at the locking code so you may need more, you need enough infrastructure to make sure the producer methods from singlepool produce the right deadlock. Let me know if you need some help here and I can look at the code. I am a little confused by the monitor stuff, I am going to post a question about that and see if I can get some help from some more knowledgeable. Bryan Pendleton (JIRA) wrote: > [ > http://issues.apache.org/jira/browse/DERBY-666?page=comments#action_12356791 > ] > > Bryan Pendleton commented on DERBY-666: > --- > > Thanks Mike -- I'm thinking about giving this a try. > > How do I write some JDK 1.5 dependent code such that Derby can still build > and run successfully in JDK 1.3 and 1.4 environments? Do I have to use > Reflection to do this? Is there some better technique used elsewhere in Derby? > > Also, does adding JDK 1.5 dependent code into Derby mean that I'd need to > adjust the build and test instructions and Ant scripts so that developers > could build and test Derby with JDK 1.5? > > >>Enhance derby.locks.deadlockTrace to print stack traces for all threads >>involved in a deadlock >>-- >> >> Key: DERBY-666 >> URL: http://issues.apache.org/jira/browse/DERBY-666 >> Project: Derby >>Type: Improvement >> Components: Store >>Versions: 10.1.1.0 >>Reporter: Bryan Pendleton >>Priority: Minor > > >>I was reading http://www.linux-mag.com/content/view/2134/ (good article, >>btw!), and it says: >> >>> The next two properties are needed to diagnose concurrency (locking and >>> deadlock) problems. >>> >>> *derby.locks.monitor=true logs all deadlocks that occur in the system. >>> *derby.locks.deadlockTrace=true log a stack trace of all threads >>> involved in lock-related rollbacks. >> >>It seems, that, in my environment, the deadlockTrace property does not log a >>stack trace of *all* threads involved in the deadlock. >>Instead, it only logs a stack trace of the *victim* thread involved in the >>deadlock. >>I think it would be very useful if the derby.locks.deadlockTrace setting >>could in fact log a stack trace of all involved threads. >>In a posting to derby-dev, Mike Matrigali noted that an earlier >>implementation of a similar feature had to be removed because it was too >>expensive in both time and space, but he suggested that there might be >>several possible ways to implement this in an acceptably efficient manner: >> >>>A long time ago there use t
Re: [jira] Commented: (DERBY-666) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock
I tried your repro on a build from the trunk and got a deadlock message from the embedded case, when I ran it on a clean database. I reran after that and inconsistently got sometimes deadlock and sometimes timeout - note each run will add another uncommitted row to each of the tables in the db and thus change the timing. As you say it seems like the amount of time waited is the deadlock wait time, not the lock timeout time in all cases. This seems like a bug to me. I know there are some "time" estimates used by the lock manager to avoid expensive time request calls in java for every lock request. Maybe this estimate is going wrong for some reason. Maybe Dan may have some advice here? /mikem ps. I guess any bug numbered 666 is destined to cause problems :-) Bryan Pendleton (JIRA) wrote: [ http://issues.apache.org/jira/browse/DERBY-666?page=comments#action_12357762 ] Bryan Pendleton commented on DERBY-666: --- The strange thing is, when I run the program in Embedded mode, and I get the lock timeout, the program *is* sensitive to the value of derby.locks.deadlockTimeout! If I set derby.locks.deadlockTimeout to 5, the program gives the "A lock could not be obtained within the time requested." error after 5 seconds, rather than after the standard 20 seconds. So it seems that the Embedded configuration is detecting the problem as a deadlock, fundamentally; it's just reporting it back out to my program as a lock timeout rather than as a deadlock. I think I'd better stop now. I've succeeded in getting myself thoroughly confused :) Enhance derby.locks.deadlockTrace to print stack traces for all threads involved in a deadlock -- Key: DERBY-666 URL: http://issues.apache.org/jira/browse/DERBY-666 Project: Derby Type: Improvement Components: Store Versions: 10.1.1.0 Reporter: Bryan Pendleton Priority: Minor Attachments: repro.java I was reading http://www.linux-mag.com/content/view/2134/ (good article, btw!), and it says: The next two properties are needed to diagnose concurrency (locking and deadlock) problems. *derby.locks.monitor=true logs all deadlocks that occur in the system. *derby.locks.deadlockTrace=true log a stack trace of all threads involved in lock-related rollbacks. It seems, that, in my environment, the deadlockTrace property does not log a stack trace of *all* threads involved in the deadlock. Instead, it only logs a stack trace of the *victim* thread involved in the deadlock. I think it would be very useful if the derby.locks.deadlockTrace setting could in fact log a stack trace of all involved threads. In a posting to derby-dev, Mike Matrigali noted that an earlier implementation of a similar feature had to be removed because it was too expensive in both time and space, but he suggested that there might be several possible ways to implement this in an acceptably efficient manner: A long time ago there use to be room in each lock to point at a stack trace for each lock, but that was removed to optimize the size of the lock data structure which can have many objects outstanding. And creating and storing the stack for every lock was incredibly slow and just was not very useful for any very active application. I think I was the only one who ever used it. The plan was sometime to add a per user data structure which could be filled in when it was about to wait on a lock, which would give most of what is interesting in a deadlock. The current deadlockTrace is meant to dump the lock table out to derby.log when a deadlock is encountered. I agree getting a dump of all stack traces would be very useful, and with the later jvm debug interfaces may now be possible - in earlier JVM's there weren't any java interfaces to do so. Does anyone have the code to donate to dump all thread stacks to a buffer? Mike also suggested a manual technique as a workaround; it would be useful to put this into the documentation somewhere, perhaps on the page which documents derby.locks.deadlockTrace? Here's Mike's suggestion: What I do if I can reproduce easily is set try to catch the wait by hand and then depending on the environment either send the magic signal or hit ctrl-break in the server window which will send the JVM specific thread dumps to derby.log. The magic signal, btw, is 'kill -QUIT', at least with Sun JVMs in my experience.
