RE: DeadlockException
Deadlocks are not fun. A deadlock is a thread related issue. Specifically when a group of threads are exclusively holding resources, then waiting on other threads for their resources, in a circle.. hence they will wait forever. Here is an example: Say thread A requires resource 1 and 2 to run. Thread B requires resource 1, 2 and 3 to run. What happens when thread A starts, get's a hold of resource 1, then thread B starts, gets a hold of resource 2 and 3, then sits waiting for thread A to let go of resource 1. Similarly, thread A will now wait for thread B to finish with resource 2. Thus they are waiting on one another in a 'deadlock' situation because neither will give up the resouce that they just acquired, but both need to run to completion. There are ways around this, none are pretty. 1. You can have a 'supervisor' thread that arbitrarily kills one of the waiting threads when it detects a deadlock (ugly, but there are many algorithms in computer science used to determine exactly which thread to kill). 2. You can modify your source so that resources are not exclusive. Meaning, multiple threads can work with them at once (ugly). 3. You can modify your threads so that they acquire all the resources they need in a single atomic statement, i.e. a synchronized statement that only one thread can go through at once, which will delegate resources which allow all threads to run to completion. This assumes the threads will then give up the resources later. Debugging of such things are very tricky. I'd start with putting trace statements in all of the resource allocating methods. Hope this helps. Matt MacGillivray Technical Developer Delfour Corporation (905) 415-9779 x2021 [EMAIL PROTECTED] -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Adam Cassar Sent: Tuesday, November 14, 2000 6:55 AM To: Orion-Interest Subject: DeadlockException Does anyone know what the following error means? 11/14/00 10:51 PM Error in bean com.netregistry.aucom.accounting.BillingAccount: Error in ejbPassivate com.evermind.server.DeadlockException: Deadlock detected: thread 11 is waiting for resource entity 1003 held by thread 11 Any ideas on how to track it down?
RE: DeadlockException
Hi all, Say thread A requires resource 1 and 2 to run. Thread B requires resource 1, 2 and 3 to run. What happens when thread A starts, get's a hold of resource 1, then thread B starts, gets a hold of resource 2 and 3, then sits waiting for thread A to let go of resource 1. Similarly, thread A will now wait for thread B to finish with resource 2. Thus they are waiting on one another in a 'deadlock' situation because neither will give up the resouce that they just acquired, but both need to run to completion. Did you notice the change you made to get a deadlock ? You changed the order of locking ! If resources are always locked in the same order, deadlocks cannot occur. That's the only rule ! But it's not always easy to implement ! Try to get a thread dump, then look for waiting threads, and monitors... Tracking deadlocks is not easy... Good luck, Christophe
RE: DeadlockException
This is a problem I've faced before... It Orion, protecting your data consistency (I LOVE THIS SERVER!) You'll see, the spirit of transaction processing is the management of resources. There's something known as transaction isolation, I figure that in your app it is set to 'SERIALIZABLE' Whenever there's stress on the server(a lotta HTTP connections targeting dinamic pages), this kind of problem should arise... To avoid, try to keep your jsp code-free, that is, perform in EJB as many work as you can, defining very granular methods( I use Stateless Session Beans for this), then call these methods from jsp... Another workaround, at the cost of less data consistency, is to change the isolation level to 'COMMITED'. If you stress test your app further, you'll discover that orion starts to choke, eating up DB connections and not releasing them (I use CMP), refer to bug #161 in bugzilla for a complete description My 2c, JP -Original Message- From: Adam Cassar [mailto:[EMAIL PROTECTED]] Sent: Martes, 14 de Noviembre de 2000 8:55 To: Orion-Interest Subject: DeadlockException Does anyone know what the following error means? 11/14/00 10:51 PM Error in bean com.netregistry.aucom.accounting.BillingAccount: Error in ejbPassivate com.evermind.server.DeadlockException: Deadlock detected: thread 11 is waiting for resource entity 1003 held by thread 11 Any ideas on how to track it down?
Re: DeadlockException
Hello, AC Does anyone know what the following error means? AC 11/14/00 10:51 PM Error in bean com.netregistry.aucom.accounting.BillingAccount: AC Error in ejbPassivate AC com.evermind.server.DeadlockException: Deadlock detected: thread 11 is waiting for AC resource entity 1003 held by thread 11 AC Any ideas on how to track it down? Just ten minutes before I read this message I encountered the same error in my application. (1.4.4) com.evermind.server.rmi.OrionRemoteException: Transaction was rolled back: com.evermind.server.DeadlockException: thread 6 is waiting for resource entity K-5 held by thread 5 in transaction [Transaction 4d:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:da] who is waiting for resource entity E-6 held by thread 6 in transaction [ Transaction 4a:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:93]; nested exception is: I am not quite sure what that means exactly. Can anybody help us? Thanks in advance, /Thomas -- | /Thomas Hertz | [y] hybris GmbH | Software Engineering | | [EMAIL PROTECTED] | Schwere-Reiter-Strasse 35 | tel +49 89 306697- 0 | | www.hybris.de | Haus 16, D-80797 Muenchen | fax +49 89 306697-99 | | pgp fingerprint: D070 5D86 BE2D C3AF E2CC D2D8 C29A 7F68 7407 629E |
Re: DeadlockException
Ah, nothing quite like two threads in a deadly embrace! At least the server detects the deadlock and prints some useful debug for deadlock situations. Try generating a VM dump using Windows: Ctrl Scroll Lock *nix : kill -QUIT server process id In general, you're looking for a thread in the MW state that has already locked a monitor on which some other MW thread is trying to synchronize. In your case, thread 5 and thread 6 are deadlocked, both waiting to synchronize on a monitor held by the other. The stack trace information may help you determine where the deadlock is occurring. Mike --- Thomas Hertz [EMAIL PROTECTED] wrote: Hello, AC Does anyone know what the following error means? AC 11/14/00 10:51 PM Error in bean com.netregistry.aucom.accounting.BillingAccount: AC Error in ejbPassivate AC com.evermind.server.DeadlockException: Deadlock detected: thread 11 is waiting for AC resource entity 1003 held by thread 11 AC Any ideas on how to track it down? Just ten minutes before I read this message I encountered the same error in my application. (1.4.4) com.evermind.server.rmi.OrionRemoteException: Transaction was rolled back: com.evermind.server.DeadlockException: thread 6 is waiting for resource entity K-5 held by thread 5 in transaction [Transaction 4d:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:da] who is waiting for resource entity E-6 held by thread 6 in transaction [ Transaction 4a:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:93]; nested exception is: I am not quite sure what that means exactly. Can anybody help us? Thanks in advance, /Thomas -- | /Thomas Hertz | [y] hybris GmbH | Software Engineering | | [EMAIL PROTECTED] | Schwere-Reiter-Strasse 35 | tel +49 89 306697- 0 | | www.hybris.de | Haus 16, D-80797 Muenchen | fax +49 89 306697-99 | | pgp fingerprint: D070 5D86 BE2D C3AF E2CC D2D8 C29A 7F68 7407 629E | __ Do You Yahoo!? Yahoo! Calendar - Get organized for the holidays! http://calendar.yahoo.com/
RE: DeadlockException
I suggest also reading: Principles of Transaction Processing Operating Systems: Design and Implementation Links to the books in Amazon.com attached You'll find here some other solutions to aviod deadlocks, such as resource locking graphs, semaphore algorithms, etc However, deadlocks don't ocurr in Orion (AFAIK), but they're trapped and some/all of the offending transactions fail... Like it's supposed to! This is the exception that orion throws, that is, a possible deadlock is detected, and the system prevents it by rollbacking a transaction; However, taking a closer look to the error, I notice it's thrown in ejbPassivate, thus, you must be acquiring a reference to a bean within a Stateful Session Bean... IMHO, a possible design error... Try to avoid Stateful SB whenever possible; they're long lived and almost never get destroyed, thus, locks acquired within actually will block sucesive calls on the same entities indefinetly My 2c (that's 4c so far), JP -Original Message- From: Juan Lorandi (Chile) [mailto:[EMAIL PROTECTED]] Sent: Martes, 14 de Noviembre de 2000 13:21 To: Orion-Interest Subject: RE: DeadlockException This is a problem I've faced before... It Orion, protecting your data consistency (I LOVE THIS SERVER!) You'll see, the spirit of transaction processing is the management of resources. There's something known as transaction isolation, I figure that in your app it is set to 'SERIALIZABLE' Whenever there's stress on the server(a lotta HTTP connections targeting dinamic pages), this kind of problem should arise... To avoid, try to keep your jsp code-free, that is, perform in EJB as many work as you can, defining very granular methods( I use Stateless Session Beans for this), then call these methods from jsp... Another workaround, at the cost of less data consistency, is to change the isolation level to 'COMMITED'. If you stress test your app further, you'll discover that orion starts to choke, eating up DB connections and not releasing them (I use CMP), refer to bug #161 in bugzilla for a complete description My 2c, JP -Original Message- From: Adam Cassar [mailto:[EMAIL PROTECTED]] Sent: Martes, 14 de Noviembre de 2000 8:55 To: Orion-Interest Subject: DeadlockException Does anyone know what the following error means? 11/14/00 10:51 PM Error in bean com.netregistry.aucom.accounting.BillingAccount: Error in ejbPassivate com.evermind.server.DeadlockException: Deadlock detected: thread 11 is waiting for resource entity 1003 held by thread 11 Any ideas on how to track it down? Principles of Transaction Processing (Morgan Kaufman Series in Data Management Systems).url Operating Systems- Design And Implementation.url
RE: DeadlockException
Does Orion do more than check a timeout value for deadlock detection? Some servers just assume a deadlock if a method call takes too long. That's the simplest type of deadlock detection. Anyway, I use CtrlPause/Break to get a stack dump on Windows (i.e, I don't think CtrlScrLk will work, will it?). The Ctrl\ combo also works on both Windows and UNIX. A real stack dump from the JVM will tell you more about what K-5 and E-6 are, and show you a call stack for the two threads that are deadlocked. So did the Orion server print out that deadlock info, or was that somehow percolated from the JVM? That's interesting. You're extremely lucky if the JVM detects a deadlock for you, which it sometimes does. Having spent many hours wading through stack dumps from problem apps and buggy JVMs, I think if you have the exact deadlock info like you do below (thread A is locking resource x and thread B is locking resource y, etc.) you're pretty darn fortunate. Lots of times that deadlocks occur, JVMs don't detect it. Disable JIT when before you try to catch this in a stack dump, or else you won't be able to see the line numbers in the stack traces. Scott Stirling -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Mike Clark Sent: Tuesday, November 14, 2000 2:23 PM To: Orion-Interest Subject: Re: DeadlockException Ah, nothing quite like two threads in a deadly embrace! At least the server detects the deadlock and prints some useful debug for deadlock situations. Try generating a VM dump using Windows: Ctrl Scroll Lock *nix : kill -QUIT server process id In general, you're looking for a thread in the MW state that has already locked a monitor on which some other MW thread is trying to synchronize. In your case, thread 5 and thread 6 are deadlocked, both waiting to synchronize on a monitor held by the other. The stack trace information may help you determine where the deadlock is occurring. Mike --- Thomas Hertz [EMAIL PROTECTED] wrote: Hello, AC Does anyone know what the following error means? AC 11/14/00 10:51 PM Error in bean com.netregistry.aucom.accounting.BillingAccount: AC Error in ejbPassivate AC com.evermind.server.DeadlockException: Deadlock detected: thread 11 is waiting for AC resource entity 1003 held by thread 11 AC Any ideas on how to track it down? Just ten minutes before I read this message I encountered the same error in my application. (1.4.4) com.evermind.server.rmi.OrionRemoteException: Transaction was rolled back: com.evermind.server.DeadlockException: thread 6 is waiting for resource entity K-5 held by thread 5 in transaction [Transaction 4d:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:da] who is waiting for resource entity E-6 held by thread 6 in transaction [ Transaction 4a:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:93]; nested exception is: I am not quite sure what that means exactly. Can anybody help us? Thanks in advance, /Thomas
Re: DeadlockException
For generating JVM dump on Windows, ctrl break and ctrl scroll lock work equally well. Mike Scott Stirling wrote: Does Orion do more than check a timeout value for deadlock detection? Some servers just assume a deadlock if a method call takes too long. That's the simplest type of deadlock detection. Anyway, I use CtrlPause/Break to get a stack dump on Windows (i.e, I don't think CtrlScrLk will work, will it?). The Ctrl\ combo also works on both Windows and UNIX. A real stack dump from the JVM will tell you more about what K-5 and E-6 are, and show you a call stack for the two threads that are deadlocked. So did the Orion server print out that deadlock info, or was that somehow percolated from the JVM? That's interesting. You're extremely lucky if the JVM detects a deadlock for you, which it sometimes does. Having spent many hours wading through stack dumps from problem apps and buggy JVMs, I think if you have the exact deadlock info like you do below (thread A is locking resource x and thread B is locking resource y, etc.) you're pretty darn fortunate. Lots of times that deadlocks occur, JVMs don't detect it. Disable JIT when before you try to catch this in a stack dump, or else you won't be able to see the line numbers in the stack traces. Scott Stirling -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Mike Clark Sent: Tuesday, November 14, 2000 2:23 PM To: Orion-Interest Subject: Re: DeadlockException Ah, nothing quite like two threads in a deadly embrace! At least the server detects the deadlock and prints some useful debug for deadlock situations. Try generating a VM dump using Windows: Ctrl Scroll Lock *nix : kill -QUIT server process id In general, you're looking for a thread in the MW state that has already locked a monitor on which some other MW thread is trying to synchronize. In your case, thread 5 and thread 6 are deadlocked, both waiting to synchronize on a monitor held by the other. The stack trace information may help you determine where the deadlock is occurring. Mike --- Thomas Hertz [EMAIL PROTECTED] wrote: Hello, AC Does anyone know what the following error means? AC 11/14/00 10:51 PM Error in bean com.netregistry.aucom.accounting.BillingAccount: AC Error in ejbPassivate AC com.evermind.server.DeadlockException: Deadlock detected: thread 11 is waiting for AC resource entity 1003 held by thread 11 AC Any ideas on how to track it down? Just ten minutes before I read this message I encountered the same error in my application. (1.4.4) com.evermind.server.rmi.OrionRemoteException: Transaction was rolled back: com.evermind.server.DeadlockException: thread 6 is waiting for resource entity K-5 held by thread 5 in transaction [Transaction 4d:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:da] who is waiting for resource entity E-6 held by thread 6 in transaction [ Transaction 4a:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:93]; nested exception is: I am not quite sure what that means exactly. Can anybody help us? Thanks in advance, /Thomas -- Mike Clark Clarkware Consulting Enterprise Java Architecture, Design, Development http://www.clarkware.com
Re: DeadlockException
I recall having run into similar problems previously. I think Orion merely checks a timeout value and then proclaims a deadlock. I can look this up, but there is a place in Orion's deployment descriptor where one can specify the timeout value. However, when I was solving such problems, that value was being ignored. Vidur Scott Stirling wrote: Does Orion do more than check a timeout value for deadlock detection? Some servers just assume a deadlock if a method call takes too long. That's the simplest type of deadlock detection. Anyway, I use CtrlPause/Break to get a stack dump on Windows (i.e, I don't think CtrlScrLk will work, will it?). The Ctrl\ combo also works on both Windows and UNIX. A real stack dump from the JVM will tell you more about what K-5 and E-6 are, and show you a call stack for the two threads that are deadlocked. So did the Orion server print out that deadlock info, or was that somehow percolated from the JVM? That's interesting. You're extremely lucky if the JVM detects a deadlock for you, which it sometimes does. Having spent many hours wading through stack dumps from problem apps and buggy JVMs, I think if you have the exact deadlock info like you do below (thread A is locking resource x and thread B is locking resource y, etc.) you're pretty darn fortunate. Lots of times that deadlocks occur, JVMs don't detect it. Disable JIT when before you try to catch this in a stack dump, or else you won't be able to see the line numbers in the stack traces. Scott Stirling -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Mike Clark Sent: Tuesday, November 14, 2000 2:23 PM To: Orion-Interest Subject: Re: DeadlockException Ah, nothing quite like two threads in a deadly embrace! At least the server detects the deadlock and prints some useful debug for deadlock situations. Try generating a VM dump using Windows: Ctrl Scroll Lock *nix : kill -QUIT server process id In general, you're looking for a thread in the MW state that has already locked a monitor on which some other MW thread is trying to synchronize. In your case, thread 5 and thread 6 are deadlocked, both waiting to synchronize on a monitor held by the other. The stack trace information may help you determine where the deadlock is occurring. Mike --- Thomas Hertz [EMAIL PROTECTED] wrote: Hello, AC Does anyone know what the following error means? AC 11/14/00 10:51 PM Error in bean com.netregistry.aucom.accounting.BillingAccount: AC Error in ejbPassivate AC com.evermind.server.DeadlockException: Deadlock detected: thread 11 is waiting for AC resource entity 1003 held by thread 11 AC Any ideas on how to track it down? Just ten minutes before I read this message I encountered the same error in my application. (1.4.4) com.evermind.server.rmi.OrionRemoteException: Transaction was rolled back: com.evermind.server.DeadlockException: thread 6 is waiting for resource entity K-5 held by thread 5 in transaction [Transaction 4d:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:da] who is waiting for resource entity E-6 held by thread 6 in transaction [ Transaction 4a:1:0:0:0:0:0:0:0:e2:0:e2:d4:67:89:93]; nested exception is: I am not quite sure what that means exactly. Can anybody help us? Thanks in advance, /Thomas
