Re: Solr stops responding

Fuad Efendi Tue, 15 Jul 2008 09:18:10 -0700

Following lines are strange, looks like SOLR deals with OOM andrethrows own exception (so that in some cases JVM simply hangs insteadof exit):

  Apr 4, 2008 1:20:53 PM org.apache.solr.common.SolrException log
  SEVERE: java.lang.OutOfMemoryError: Java heap space

This is full Thread Dump after OOM, made in April with Tomcat 6.Deadlock at Tomcat? Looks like some queries succeed, but I was forcedKILL -9.

=================


SEVERE: Error allocating socket processor
java.lang.OutOfMemoryError: Java heap space
Apr 4, 2008 1:57:36 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Exception in thread "catalina-exec-4" java.lang.OutOfMemoryError: Javaheap space

Apr 4, 2008 1:58:18 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 1:59:01 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 1:59:01 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 1:59:01 PM org.apache.tomcat.util.net.AprEndpoint$Acceptor run
SEVERE: Socket accept failed
java.lang.OutOfMemoryError: Java heap space
Apr 4, 2008 1:59:39 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 1:59:53 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=webcam&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.excaliberpc.com"&hl=true 018

Apr 4, 2008 2:00:51 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=pepe+jeans&qt=dismax&version=2.2&facet.field=country&facet.field=host&hl=true 038544

Apr 4, 2008 2:02:11 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 2:02:11 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 2:02:11 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 2:02:11 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=10&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=category:"core"&qt=standard&version=2.2&facet.field=country&facet.field=host&hl=true 079439

Apr 4, 2008 2:02:21 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=robot&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.clickonit.com"&hl=true 017

Apr 4, 2008 2:02:35 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=Cognac&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.designersimports.com"&hl=true 019

Apr 4, 2008 2:03:12 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=prada&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.theprincessescloset.com"&hl=true 01

Apr 4, 2008 2:04:55 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space

Apr 4, 2008 2:04:55 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&sort=price+desc&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=velodyne+DD+15&qt=dismax&version=2.2&facet.field=country&facet.field=host&hl=true 053

Apr 4, 2008 2:05:21 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=velodyne+DD+15&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.hometheaterstore.com"&hl=true 03

Apr 4, 2008 2:06:06 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&sort=id+asc&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=sex&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.moviesunlimited.com"&fq=category:"video"&hl=true 039

Apr 4, 2008 2:06:24 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=10&start=0&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=id:[*+TO+*]&qt=standard&version=2.2&facet.field=country&facet.field=host&hl=true 0859

Apr 4, 2008 2:07:03 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=0&sort=host+asc&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=sex&qt=dismax&version=2.2&facet.field=country&facet.field=host&fq=host:"www.moviesunlimited.com"&fq=category:"video"&hl=true 011

Apr 4, 2008 2:07:42 PM org.apache.solr.core.SolrCore execute

INFO: /selectwt=xml&facet.limit=100&rows=100&start=100&facet=true&facet.mincount=1&fl=id,item_name,category,price,price_txt,url,host,country&q=www.cheapbats.com&qt=dismax&version=2.2&facet.field=country&facet.field=host&hl=true 038441

Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_14-b03 mixed mode):

"catalina-exec-52" daemon prio=1 tid=0x00002aaaaffff5d0 nid=0x732bwaiting on condition [0x0000000041b6b000..0x0000000041b6bda0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

atjava.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)atjava.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)atjava.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:674)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-51" daemon prio=1 tid=0x00002aaaab18c460 nid=0x729fwaiting on condition [0x0000000043888000..0x0000000043888d20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-50" daemon prio=1 tid=0x00002aaaad17d800 nid=0x5dcdwaiting on condition [0x0000000044999000..0x0000000044999ca0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-49" daemon prio=1 tid=0x00002aaaad17cbc0 nid=0x5dccwaiting on condition [0x0000000044898000..0x0000000044898c20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-48" daemon prio=1 tid=0x00002aaaab121af0 nid=0x5dcbwaiting on condition [0x0000000044797000..0x0000000044797ba0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-47" daemon prio=1 tid=0x00002aaaac63e040 nid=0x5dcawaiting on condition [0x0000000044696000..0x0000000044696b20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-46" daemon prio=1 tid=0x00002aaaac63c1b0 nid=0x5dc9waiting on condition [0x0000000044595000..0x0000000044595ea0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-45" daemon prio=1 tid=0x00002aaaab11fd60 nid=0x5dc8waiting on condition [0x0000000044494000..0x0000000044494e20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-44" daemon prio=1 tid=0x00002aaaac63f390 nid=0x5dc7waiting on condition [0x0000000044393000..0x0000000044393da0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-43" daemon prio=1 tid=0x00002aaaac63d300 nid=0x5dc6waiting on condition [0x0000000044292000..0x0000000044292d20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-42" daemon prio=1 tid=0x00002aaaac63c6c0 nid=0x5dc5waiting on condition [0x0000000044191000..0x0000000044191ca0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-41" daemon prio=1 tid=0x00002aaaac63b2d0 nid=0x5dc4waiting on condition [0x0000000044090000..0x0000000044090c20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-40" daemon prio=1 tid=0x00002aaaac63a690 nid=0x5dc3waiting on condition [0x0000000043f8f000..0x0000000043f8fba0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-39" daemon prio=1 tid=0x00002aaaac639a10 nid=0x5dc2waiting on condition [0x0000000043e8e000..0x0000000043e8eb20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-38" daemon prio=1 tid=0x00002aaaab857210 nid=0x5dc1waiting on condition [0x0000000043d8d000..0x0000000043d8dea0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-37" daemon prio=1 tid=0x00002aaaab8565d0 nid=0x5dc0waiting on condition [0x0000000043c8c000..0x0000000043c8ce20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-36" daemon prio=1 tid=0x00002aaaab16ca10 nid=0x5dbfwaiting on condition [0x0000000043b8b000..0x0000000043b8bda0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-35" daemon prio=1 tid=0x00002aaaab168d30 nid=0x5dbewaiting on condition [0x0000000043a8a000..0x0000000043a8ad20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-34" daemon prio=1 tid=0x00002aaaab1680f0 nid=0x5dbdwaiting on condition [0x0000000043989000..0x0000000043989ca0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-32" daemon prio=1 tid=0x00002aaaab166870 nid=0x5dbbwaiting on condition [0x0000000043787000..0x0000000043787ba0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-31" daemon prio=1 tid=0x00002aaaab165c30 nid=0x5dbawaiting on condition [0x0000000043686000..0x0000000043686b20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-30" daemon prio=1 tid=0x00002aaaaab00850 nid=0x5db9waiting on condition [0x0000000043585000..0x0000000043585ea0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-29" daemon prio=1 tid=0x00002aaaaaaffc50 nid=0x5db8waiting on condition [0x0000000043484000..0x0000000043484e20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-28" daemon prio=1 tid=0x00002aaaaaaff010 nid=0x5db7waiting on condition [0x0000000043383000..0x0000000043383da0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-27" daemon prio=1 tid=0x00002aaaaaafe3d0 nid=0x5db6waiting on condition [0x0000000043282000..0x0000000043282d20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-26" daemon prio=1 tid=0x00002aaaabf401e0 nid=0x5db5waiting on condition [0x0000000043181000..0x0000000043181ca0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-25" daemon prio=1 tid=0x00002aaaabf3eca0 nid=0x5db4waiting on condition [0x0000000043080000..0x0000000043080c20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-24" daemon prio=1 tid=0x00002aaaab8533c0 nid=0x5db3waiting on condition [0x0000000042f7f000..0x0000000042f7fba0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-23" daemon prio=1 tid=0x00002aaaaf2b18a0 nid=0x5db2waiting on condition [0x0000000042e7e000..0x0000000042e7eb20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-22" daemon prio=1 tid=0x00002aaaaf2b0d90 nid=0x5db1waiting on condition [0x0000000042d7d000..0x0000000042d7dea0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-21" daemon prio=1 tid=0x00002aaaab557410 nid=0x5db0waiting on condition [0x0000000042c7c000..0x0000000042c7ce20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-20" daemon prio=1 tid=0x00002aaaaaafcf40 nid=0x5dafwaiting on condition [0x0000000042b7b000..0x0000000042b7bda0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-19" daemon prio=1 tid=0x00002aaaaaafbfe0 nid=0x5daewaiting on condition [0x0000000042a7a000..0x0000000042a7ad20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-18" daemon prio=1 tid=0x00002aaaae3d6c10 nid=0x5dadwaiting on condition [0x0000000042979000..0x0000000042979ca0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-17" daemon prio=1 tid=0x00002aaaae3d5fd0 nid=0x5dacwaiting on condition [0x0000000042878000..0x0000000042878c20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-16" daemon prio=1 tid=0x00002aaaae3d53c0 nid=0x5dabwaiting on condition [0x0000000042777000..0x0000000042777ba0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-15" daemon prio=1 tid=0x00002aaaabea4070 nid=0x5daawaiting on condition [0x0000000042676000..0x0000000042676b20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-14" daemon prio=1 tid=0x00002aaaaf4bfab0 nid=0x5da9waiting on condition [0x0000000042575000..0x0000000042575ea0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-13" daemon prio=1 tid=0x00002aaaae3d7bc0 nid=0x5da8waiting on condition [0x0000000042474000..0x0000000042474e20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-12" daemon prio=1 tid=0x00002aaaadbe4130 nid=0x5da7waiting on condition [0x0000000042373000..0x0000000042373da0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-11" daemon prio=1 tid=0x00002aaaae3902f0 nid=0x5da6waiting on condition [0x0000000042272000..0x0000000042272d20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-10" daemon prio=1 tid=0x00002aaaae3910a0 nid=0x5da5waiting on condition [0x0000000042171000..0x0000000042171ca0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-9" daemon prio=1 tid=0x00002aaaabecf790 nid=0x5da4waiting on condition [0x0000000042070000..0x0000000042070c20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-8" daemon prio=1 tid=0x00002aaaabece030 nid=0x5da3waiting on condition [0x0000000041f6f000..0x0000000041f6fba0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-7" daemon prio=1 tid=0x00002aaaab556c30 nid=0x5da1waiting on condition [0x0000000041e6e000..0x0000000041e6eb20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-6" daemon prio=1 tid=0x00002aaaaf2adfd0 nid=0x5da0waiting on condition [0x0000000041d6d000..0x0000000041d6dea0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-5" daemon prio=1 tid=0x00002aaaae38bb70 nid=0x5d9fwaiting on condition [0x0000000041c6c000..0x0000000041c6ce20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-3" daemon prio=1 tid=0x00002aaaab596170 nid=0x5d9dwaiting on condition [0x0000000041a6a000..0x0000000041a6ad20]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-2" daemon prio=1 tid=0x00002aaaabf35340 nid=0x5d9cwaiting on condition [0x0000000041969000..0x0000000041969ca0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"ajp-18009-Acceptor-0" daemon prio=1 tid=0x00002aaaafb29d90 nid=0x5d9brunnable [0x0000000041868000..0x0000000041868c20]

        at org.apache.tomcat.jni.Socket.accept(Native Method)
        at 
org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1110)
        at java.lang.Thread.run(Thread.java:595)

"ajp-18009-CometPoller-0" daemon prio=1 tid=0x00002aaaafb4d150nid=0x5d9a in Object.wait() [0x0000000041767000..0x0000000041767ba0]

        at java.lang.Object.wait(Native Method)

- waiting on <0x00002b95478569d0> (aorg.apache.tomcat.util.net.AprEndpoint$Poller)

        at java.lang.Object.wait(Object.java:474)
        at 
org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1267)

- locked <0x00002b95478569d0> (aorg.apache.tomcat.util.net.AprEndpoint$Poller)

        at java.lang.Thread.run(Thread.java:595)

"ajp-18009-Poller-0" daemon prio=1 tid=0x00002aaaafb4cc40 nid=0x5d99in Object.wait() [0x0000000041666000..0x0000000041666b20]

        at java.lang.Object.wait(Native Method)

- waiting on <0x00002b9547856c30> (aorg.apache.tomcat.util.net.AprEndpoint$Poller)

        at java.lang.Object.wait(Object.java:474)
        at 
org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1267)

- locked <0x00002b9547856c30> (aorg.apache.tomcat.util.net.AprEndpoint$Poller)

        at java.lang.Thread.run(Thread.java:595)

"catalina-exec-1" daemon prio=1 tid=0x00002aaaae38c5d0 nid=0x5d98waiting on condition [0x0000000041565000..0x0000000041565ea0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"http-18080-Acceptor-0" daemon prio=1 tid=0x00002aaaafb010b0nid=0x5d97 runnable [0x0000000041464000..0x0000000041464e20]

        at org.apache.tomcat.jni.Socket.accept(Native Method)
        at 
org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1110)
        at java.lang.Thread.run(Thread.java:595)

"http-18080-Sendfile-0" daemon prio=1 tid=0x00002aaaafb50b20nid=0x5d96 in Object.wait() [0x0000000041363000..0x0000000041363da0]

        at java.lang.Object.wait(Native Method)

- waiting on <0x00002b954d8bca60> (aorg.apache.tomcat.util.net.AprEndpoint$Sendfile)

        at java.lang.Object.wait(Object.java:474)
        at 
org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:1725)

- locked <0x00002b954d8bca60> (aorg.apache.tomcat.util.net.AprEndpoint$Sendfile)

        at java.lang.Thread.run(Thread.java:595)

"http-18080-CometPoller-0" daemon prio=1 tid=0x00002aaaafb5e680nid=0x5d95 in Object.wait() [0x0000000041262000..0x0000000041262d20]

        at java.lang.Object.wait(Native Method)

- waiting on <0x00002b954d8bcb78> (aorg.apache.tomcat.util.net.AprEndpoint$Poller)

        at java.lang.Object.wait(Object.java:474)
        at 
org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1267)

- locked <0x00002b954d8bcb78> (aorg.apache.tomcat.util.net.AprEndpoint$Poller)

        at java.lang.Thread.run(Thread.java:595)

"pool-1-thread-1" prio=1 tid=0x00002aaaafb03a40 nid=0x5d92 waiting oncondition [0x0000000040f5f000..0x0000000040f5fba0]

        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)

        at java.lang.Thread.run(Thread.java:595)

"Low Memory Detector" daemon prio=1 tid=0x00002aaaaaba0a50 nid=0x5d90runnable [0x0000000000000000..0x0000000000000000]

"CompilerThread1" daemon prio=1 tid=0x00002aaaaab9ef60 nid=0x5d8fwaiting on condition [0x0000000000000000..0x0000000040c5b740]

"CompilerThread0" daemon prio=1 tid=0x00002aaaaab9d9b0 nid=0x5d8ewaiting on condition [0x0000000000000000..0x0000000040b5a6f0]

"AdapterThread" daemon prio=1 tid=0x00002aaaaab9c390 nid=0x5d8dwaiting on condition [0x0000000000000000..0x0000000000000000]

"Signal Dispatcher" daemon prio=1 tid=0x00002aaaaab9b030 nid=0x5d8crunnable [0x0000000000000000..0x0000000000000000]

"Finalizer" daemon prio=1 tid=0x00002aaaaab86840 nid=0x5d8b inObject.wait() [0x0000000040858000..0x0000000040858c20]

        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002b954d994708> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
        - locked <0x00002b954d994708> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=1 tid=0x00002aaaaab85d60 nid=0x5d8a inObject.wait() [0x0000000040757000..0x0000000040757ba0]

        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002b95477625d8> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:474)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x00002b95477625d8> (a java.lang.ref.Reference$Lock)

"main" prio=1 tid=0x0000000040116020 nid=0x5d83 runnable[0x00007fff70172000..0x00007fff70172d00]

        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x00002b954d994920> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at 
org.apache.catalina.core.StandardServer.await(StandardServer.java:389)
        at org.apache.catalina.startup.Catalina.await(Catalina.java:642)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:602)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:413)

"VM Thread" prio=1 tid=0x00002aaaaab81870 nid=0x5d89 runnable

"GC task thread#0 (ParallelGC)" prio=1 tid=0x0000000040132b60nid=0x5d85 runnable

"GC task thread#1 (ParallelGC)" prio=1 tid=0x00000000401339e0nid=0x5d86 runnable

"GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040134860nid=0x5d87 runnable

"GC task thread#3 (ParallelGC)" prio=1 tid=0x00000000401356e0nid=0x5d88 runnable

"VM Periodic Task Thread" prio=1 tid=0x0000000040129ab0 nid=0x5d91waiting on condition













==============
http://www.linkedin.com/in/liferay


Quoting Noble Paul ??????? ?????? <[EMAIL PROTECTED]>:

Can we collect more information. It would be nice to know what the
threads are doing when it hangs.
If you are using *nix  issue kill -3 <pid>
it would print out the stacktrace of all the threads in the VM . That
may tell us what is the state of each thread which could help us
suggest something


On Tue, Jul 15, 2008 at 8:59 PM, Fuad Efendi <[EMAIL PROTECTED]> wrote:

I constantly have the same problem; sometimes I have OutOfMemoryError in
logs, sometimes
not. Not-predictable. I minimized all caches, it still happens even with
8192M. CPU usage
is 375%-400% (two double-core Opterons), SUN Java 5. Moved to BEA JRockit 5
yesterday,
looks 30 times faster (25% CPU load with 4096M RAM); no any problem yet,
let's see...

Strange: Tomcat simply hangs instead of exit(...)

There are some posts related to OutOfMemoryError in solr-user list.


==============
http://www.linkedin.com/in/liferay

Quoting Doug Steigerwald <[EMAIL PROTECTED]>:

Since we pushed Solr out to production a few weeks ago, we've seen a
few issues with Solr not responding to requests (searches or admin
pages).  There doesn't seem to be any reason for it from what we can
tell.  We haven't seen it in QA or development.

We're running Solr with basically the example Solr setup with Jetty
(6.1.3).  We package our Solr install by using 'ant example' and
replacing configs/etc.  Whenever Solr stops responding, there are no
messages in the logs, nothing.  Requests just time out.

We have also only seen this on our slaves.  The master doesn't seem to
be hitting this issue.  All the boxes are the same, version of java is
the same, etc.

We don't have a stack trace and no JMX set up.  Once we see this issue,
our support folks just stop and start Solr on that machine.

Has anyone else run into anything like this with Solr?

Thanks.
Doug




--
--Noble Paul

Re: Solr stops responding

Reply via email to