Re: Solrj possible deadlock
: Well, in the same processes I am using a jdbc connection to get all the : relative paths to the documents I want to index, then I parse the documents : to plain text using tones of open source libraries like POI, PFDBox : etc.(which might account for java2d) then I add them to the index and commit : every 2000 documents. Since nothing in your threaddumps refers to solrj or solr (either as the current method, or in the call stack suggesting that it's code called by Solr(J)), there's really no indication that the problem is even remotely solr related. i suspect that if you commented out all of hte code where you use SolrJ so that you still did all of the parsing and then just wrote the resulting data to /dev/null you would probably still see this behavior -- perhaps one of the other libraries you are using has some semantics you aren't obeying (ie: a parser that must be used single threaded, or an object that must be closed so that it can reset some static state) that is acusing this problem only after some time has elapsed (or on particular permutations of data) -Hoss
Re: Solrj possible deadlock
Well, in the same processes I am using a jdbc connection to get all the relative paths to the documents I want to index, then I parse the documents to plain text using tones of open source libraries like POI, PFDBox etc.(which might account for java2d) then I add them to the index and commit every 2000 documents. I write a db row for each row I index so I can resume where I left off after a crash or exception. My app will happily index for hours then after it hangs, the resumed indexing run will only last a few additional minutes! The thread dumps look the same. Cheers. ryantxu wrote: > > do you have anything custom going on? > > The fact that the lock is in java2d seems suspicious... > > > On Sep 23, 2009, at 7:01 PM, pof wrote: > >> >> I had the same problem again yesterday except the process halted >> after about >> 20mins this time. >> >> >> pof wrote: >>> >>> Hello, I was running a batch index the other day using the Solrj >>> EmbeddedSolrServer when the process abruptly froze in it's tracks >>> after >>> running for about 4-5 hours and indexing ~400K documents. There >>> were no >>> document locks so it would seem likely that there was some kind of >>> thread >>> deadlock. I was hoping someone might be able to tell me some >>> information >>> about the following thread dump taken at the time: >>> >>> Full thread dump OpenJDK Client VM (1.6.0-b09 mixed mode): >>> >>> "DestroyJavaVM" prio=10 tid=0x9322a800 nid=0xcef waiting on condition >>> [0x..0x0018a044] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Java2D Disposer" daemon prio=10 tid=0x0a28cc00 nid=0xf1c in >>> Object.wait() >>> [0x0311d000..0x0311def4] >>> java.lang.Thread.State: WAITING (on object monitor) >>>at java.lang.Object.wait(Native Method) >>>- waiting on <0x97a96840> (a java.lang.ref.ReferenceQueue >>> $Lock) >>>at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: >>> 133) >>>- locked <0x97a96840> (a java.lang.ref.ReferenceQueue$Lock) >>>at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: >>> 149) >>>at sun.java2d.Disposer.run(Disposer.java:143) >>>at java.lang.Thread.run(Thread.java:636) >>> >>> "pool-1-thread-1" prio=10 tid=0x93a26c00 nid=0xcf7 waiting on >>> condition >>> [0x08a6a000..0x08a6b074] >>> java.lang.Thread.State: WAITING (parking) >>>at sun.misc.Unsafe.park(Native Method) >>>- parking to wait for <0x967acfd0> (a >>> java.util.concurrent.locks.AbstractQueuedSynchronizer >>> $ConditionObject) >>>at >>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >>>at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer >>> $ConditionObject.await(AbstractQueuedSynchronizer.java:1978) >>>at >>> java >>> .util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java: >>> 386) >>>at >>> java >>> .util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java: >>> 1043) >>>at >>> java >>> .util >>> .concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: >>> 1103) >>>at >>> java.util.concurrent.ThreadPoolExecutor >>> $Worker.run(ThreadPoolExecutor.java:603) >>>at java.lang.Thread.run(Thread.java:636) >>> >>> "Low Memory Detector" daemon prio=10 tid=0x93a00c00 nid=0xcf5 >>> runnable >>> [0x..0x] >>> java.lang.Thread.State: RUNNABLE >>> >>> "CompilerThread0" daemon prio=10 tid=0x09fe9800 nid=0xcf4 waiting on >>> condition [0x..0x096a7af4] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Signal Dispatcher" daemon prio=10 tid=0x09fe8800 nid=0xcf3 waiting >>> on >>> condition [0x..0x] >>> java.lang.Thread.State: RUNNABLE >>> >>> "Finalizer" daemon prio=10 tid=0x09fd7000 nid=0xcf2 in Object.wait() >>> [0x005ca000..0x005caef4] >>> java.lang.Thread.State: WAITING (on object monitor) >>>at java.lang.Object.wait(Native Method) >>>- waiting on <0x966e6d40> (a java.lang.ref.ReferenceQueue >>> $Lock) >>>at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: >>> 133) >>>- locked <0x966e6d40> (a java.lang.ref.ReferenceQueue$Lock) >>>at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: >>> 149) >>>at java.lang.ref.Finalizer >>> $FinalizerThread.run(Finalizer.java:177) >>> >>> "Reference Handler" daemon prio=10 tid=0x09fd2c00 nid=0xcf1 in >>> Object.wait() [0x00579000..0x00579d74] >>> java.lang.Thread.State: WAITING (on object monitor) >>>at java.lang.Object.wait(Native Method) >>>- waiting on <0x966e6dc8> (a java.lang.ref.Reference$Lock) >>>at java.lang.Object.wait(Object.java:502) >>>at >>> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) >>>- locked <0x966e6dc8> (a java.lang.ref.Reference$Lock) >>> >>> "VM Thread" prio=10 tid=0x09fcf800 nid=0xcf0 runnable >>> >>> "VM Periodic Task Thread" prio=10 tid=0x93a02400 nid=0xcf6 waiting on >>> condition >>> >>> JNI
Re: Solrj possible deadlock
do you have anything custom going on? The fact that the lock is in java2d seems suspicious... On Sep 23, 2009, at 7:01 PM, pof wrote: I had the same problem again yesterday except the process halted after about 20mins this time. pof wrote: Hello, I was running a batch index the other day using the Solrj EmbeddedSolrServer when the process abruptly froze in it's tracks after running for about 4-5 hours and indexing ~400K documents. There were no document locks so it would seem likely that there was some kind of thread deadlock. I was hoping someone might be able to tell me some information about the following thread dump taken at the time: Full thread dump OpenJDK Client VM (1.6.0-b09 mixed mode): "DestroyJavaVM" prio=10 tid=0x9322a800 nid=0xcef waiting on condition [0x..0x0018a044] java.lang.Thread.State: RUNNABLE "Java2D Disposer" daemon prio=10 tid=0x0a28cc00 nid=0xf1c in Object.wait() [0x0311d000..0x0311def4] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x97a96840> (a java.lang.ref.ReferenceQueue $Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: 133) - locked <0x97a96840> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: 149) at sun.java2d.Disposer.run(Disposer.java:143) at java.lang.Thread.run(Thread.java:636) "pool-1-thread-1" prio=10 tid=0x93a26c00 nid=0xcf7 waiting on condition [0x08a6a000..0x08a6b074] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x967acfd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer $ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer $ConditionObject.await(AbstractQueuedSynchronizer.java:1978) at java .util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java: 386) at java .util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java: 1043) at java .util .concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1103) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) "Low Memory Detector" daemon prio=10 tid=0x93a00c00 nid=0xcf5 runnable [0x..0x] java.lang.Thread.State: RUNNABLE "CompilerThread0" daemon prio=10 tid=0x09fe9800 nid=0xcf4 waiting on condition [0x..0x096a7af4] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x09fe8800 nid=0xcf3 waiting on condition [0x..0x] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x09fd7000 nid=0xcf2 in Object.wait() [0x005ca000..0x005caef4] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x966e6d40> (a java.lang.ref.ReferenceQueue $Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: 133) - locked <0x966e6d40> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: 149) at java.lang.ref.Finalizer $FinalizerThread.run(Finalizer.java:177) "Reference Handler" daemon prio=10 tid=0x09fd2c00 nid=0xcf1 in Object.wait() [0x00579000..0x00579d74] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x966e6dc8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x966e6dc8> (a java.lang.ref.Reference$Lock) "VM Thread" prio=10 tid=0x09fcf800 nid=0xcf0 runnable "VM Periodic Task Thread" prio=10 tid=0x93a02400 nid=0xcf6 waiting on condition JNI global references: 1072 Heap def new generation total 36288K, used 23695K [0x93f1, 0x9667, 0x9667) eden space 32256K, 73% used [0x93f1, 0x95633f60, 0x95e9) from space 4032K, 0% used [0x95e9, 0x95e9, 0x9628) to space 4032K, 0% used [0x9628, 0x9628, 0x9667) tenured generation total 483968K, used 72129K [0x9667, 0xb3f1, 0xb3f1) the space 483968K, 14% used [0x9667, 0x9ace04b8, 0x9ace0600, 0xb3f1) compacting perm gen total 23040K, used 22983K [0xb3f1, 0xb559, 0xb7f1) the space 23040K, 99% used [0xb3f1, 0xb5581ff8, 0xb5582000, 0xb559) No shared spaces configured. Cheers. Brett. -- View this message in context: http://www.nabble.com/Solrj-possible-deadlock-tp25530146p25531321.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj possible deadlock
I had the same problem again yesterday except the process halted after about 20mins this time. pof wrote: > > Hello, I was running a batch index the other day using the Solrj > EmbeddedSolrServer when the process abruptly froze in it's tracks after > running for about 4-5 hours and indexing ~400K documents. There were no > document locks so it would seem likely that there was some kind of thread > deadlock. I was hoping someone might be able to tell me some information > about the following thread dump taken at the time: > > Full thread dump OpenJDK Client VM (1.6.0-b09 mixed mode): > > "DestroyJavaVM" prio=10 tid=0x9322a800 nid=0xcef waiting on condition > [0x..0x0018a044] >java.lang.Thread.State: RUNNABLE > > "Java2D Disposer" daemon prio=10 tid=0x0a28cc00 nid=0xf1c in Object.wait() > [0x0311d000..0x0311def4] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x97a96840> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) > - locked <0x97a96840> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) > at sun.java2d.Disposer.run(Disposer.java:143) > at java.lang.Thread.run(Thread.java:636) > > "pool-1-thread-1" prio=10 tid=0x93a26c00 nid=0xcf7 waiting on condition > [0x08a6a000..0x08a6b074] >java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x967acfd0> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1978) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > > "Low Memory Detector" daemon prio=10 tid=0x93a00c00 nid=0xcf5 runnable > [0x..0x] >java.lang.Thread.State: RUNNABLE > > "CompilerThread0" daemon prio=10 tid=0x09fe9800 nid=0xcf4 waiting on > condition [0x..0x096a7af4] >java.lang.Thread.State: RUNNABLE > > "Signal Dispatcher" daemon prio=10 tid=0x09fe8800 nid=0xcf3 waiting on > condition [0x..0x] >java.lang.Thread.State: RUNNABLE > > "Finalizer" daemon prio=10 tid=0x09fd7000 nid=0xcf2 in Object.wait() > [0x005ca000..0x005caef4] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x966e6d40> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) > - locked <0x966e6d40> (a java.lang.ref.ReferenceQueue$Lock) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) > > "Reference Handler" daemon prio=10 tid=0x09fd2c00 nid=0xcf1 in > Object.wait() [0x00579000..0x00579d74] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x966e6dc8> (a java.lang.ref.Reference$Lock) > at java.lang.Object.wait(Object.java:502) > at > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) > - locked <0x966e6dc8> (a java.lang.ref.Reference$Lock) > > "VM Thread" prio=10 tid=0x09fcf800 nid=0xcf0 runnable > > "VM Periodic Task Thread" prio=10 tid=0x93a02400 nid=0xcf6 waiting on > condition > > JNI global references: 1072 > > Heap > def new generation total 36288K, used 23695K [0x93f1, 0x9667, > 0x9667) > eden space 32256K, 73% used [0x93f1, 0x95633f60, 0x95e9) > from space 4032K, 0% used [0x95e9, 0x95e9, 0x9628) > to space 4032K, 0% used [0x9628, 0x9628, 0x9667) > tenured generation total 483968K, used 72129K [0x9667, 0xb3f1, > 0xb3f1) >the space 483968K, 14% used [0x9667, 0x9ace04b8, 0x9ace0600, > 0xb3f1) > compacting perm gen total 23040K, used 22983K [0xb3f1, 0xb559, > 0xb7f1) >the space 23040K, 99% used [0xb3f1, 0xb5581ff8, 0xb5582000, > 0xb559) > No shared spaces configured. > > Cheers. Brett. > -- View this message in context: http://www.nabble.com/Solrj-possible-deadlock-tp25530146p25531321.html Sent from the Solr - User mailing list archive at Nabble.com.
