LRUCache causing locked threads
I've searched through the forums and seen a few similar problems to this, but nothing that seemed to help much. We're running Solr 1.3 on Tomcat 6.0.16 and Java 6. We've been having performance problems with our search, causing long query times under normal traffic. We've taken a thread dump and have seen many threads locked or waiting for LRUCache (see below). Our cache values are as follows: "http-8983-99" daemon prio=10 tid=0x002beb3f5800 nid=0x2fb9 waiting for monitor entry [0x47ea5000..0x47ea6c30] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.solr.search.LRUCache.get(LRUCache.java:130) - waiting to lock <0x002a9fb94be8> (a org.apache.solr.search.LRUCache$1) at org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:605) at org.apache.solr.search.SolrIndexSearcher.numDocs(SolrIndexSearcher.java:1556) at org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) "http-8983-83" daemon prio=10 tid=0x002bead1a000 nid=0x2f76 waiting for monitor entry [0x46e95000..0x46e96c30] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.solr.search.LRUCache.get(LRUCache.java:130) - locked <0x002a9fb94be8> (a org.apache.solr.search.LRUCache$1) at org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:605) at org.apache.solr.search.SolrIndexSearcher.numDocs(SolrIndexSearcher.java:1556) at org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) at org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) at org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) at org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) Has anyone else experienced this or does anyone have an idea of why this might be happening? -- View this message in context: http://www.nabble.com/LRUCache-causing-locked-threads-tp24040421p24040421.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: LRUCache causing locked threads
Solr 1.4 has a cache implementation that's lockless for gets, and faster for gets. There's a new faceting implementation as well. -Yonik http://www.lucidimagination.com On Mon, Jun 15, 2009 at 2:39 PM, CameronL wrote: > > I've searched through the forums and seen a few similar problems to this, but > nothing that seemed to help much. > > We're running Solr 1.3 on Tomcat 6.0.16 and Java 6. We've been having > performance problems with our search, causing long query times under normal > traffic. We've taken a thread dump and have seen many threads locked or > waiting for LRUCache (see below). Our cache values are as follows: > > autowarmCount="1"/> > autowarmCount="5000"/> > autowarmCount="0"/> > > > "http-8983-99" daemon prio=10 tid=0x002beb3f5800 nid=0x2fb9 waiting for > monitor entry [0x47ea5000..0x47ea6c30] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.solr.search.LRUCache.get(LRUCache.java:130) > - waiting to lock <0x002a9fb94be8> (a > org.apache.solr.search.LRUCache$1) > at > org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:605) > at > org.apache.solr.search.SolrIndexSearcher.numDocs(SolrIndexSearcher.java:1556) > at > org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) > at > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) > at > org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) > at > org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) > at > org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) > "http-8983-83" daemon prio=10 tid=0x002bead1a000 nid=0x2f76 waiting for > monitor entry [0x46e95000..0x46e96c30] > java.lang.Thread.State: BLOCKED (on object monitor) > at org.apache.solr.search.LRUCache.get(LRUCache.java:130) > - locked <0x002a9fb94be8> (a org.apache.solr.search.LRUCache$1) > at > org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:605) > at > org.apache.solr.search.SolrIndexSearcher.numDocs(SolrIndexSearcher.java:1556) > at > org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) > at > org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) > at > org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) > at > org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) > at > org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) > > Has anyone else experienced this or does anyone have an idea of why this > might be happening? > -- > View this message in context: > http://www.nabble.com/LRUCache-causing-locked-threads-tp24040421p24040421.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
Re: LRUCache causing locked threads
Unfortunately upgrading to 1.4 isn't an option for us at the moment. Since we're stuck using 1.3, is there anything in particular we need to do to prevent these threads from locking (through configuration or something) or is this sort of expected/unavoidable using 1.3? Yonik Seeley-2 wrote: > > Solr 1.4 has a cache implementation that's lockless for gets, and > faster for gets. There's a new faceting implementation as well. > > -Yonik > http://www.lucidimagination.com > > On Mon, Jun 15, 2009 at 2:39 PM, CameronL > wrote: >> >> I've searched through the forums and seen a few similar problems to this, >> but >> nothing that seemed to help much. >> >> We're running Solr 1.3 on Tomcat 6.0.16 and Java 6. We've been having >> performance problems with our search, causing long query times under >> normal >> traffic. We've taken a thread dump and have seen many threads locked or >> waiting for LRUCache (see below). Our cache values are as follows: >> >> > autowarmCount="1"/> >> > autowarmCount="5000"/> >> > autowarmCount="0"/> >> >> >> "http-8983-99" daemon prio=10 tid=0x002beb3f5800 nid=0x2fb9 waiting >> for >> monitor entry [0x47ea5000..0x47ea6c30] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at org.apache.solr.search.LRUCache.get(LRUCache.java:130) >> - waiting to lock <0x002a9fb94be8> (a >> org.apache.solr.search.LRUCache$1) >> at >> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:605) >> at >> org.apache.solr.search.SolrIndexSearcher.numDocs(SolrIndexSearcher.java:1556) >> at >> org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) >> at >> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) >> at >> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) >> at >> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) >> at >> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) >> at >> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) >> "http-8983-83" daemon prio=10 tid=0x002bead1a000 nid=0x2f76 waiting >> for >> monitor entry [0x46e95000..0x46e96c30] >> java.lang.Thread.State: BLOCKED (on object monitor) >> at org.apache.solr.search.LRUCache.get(LRUCache.java:130) >> - locked <0x002a9fb94be8> (a >> org.apache.solr.search.LRUCache$1) >> at >> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:605) >> at >> org.apache.solr.search.SolrIndexSearcher.numDocs(SolrIndexSearcher.java:1556) >> at >> org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) >> at >> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) >> at >> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) >> at >> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) >> at >> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) >> at >> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) >> at >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) >> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) >> at >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) >> at >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) >> >> Has anyone else experienced this or does anyone have an idea of why this >> might be happening? >> -- >> View this message in context: >> http://www.nabble.com/LRUCache-causing-locked-threads-tp24040421p24040421.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://www.nabble.com/LRUCache-causing-locked-threads-tp24040421p24040772.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: LRUCache causing locked threads
On Mon, Jun 15, 2009 at 2:58 PM, CameronL wrote: > Unfortunately upgrading to 1.4 isn't an option for us at the moment. > > Since we're stuck using 1.3, is there anything in particular we need to do > to prevent these threads from locking (through configuration or something) Not really. > or is this sort of expected/unavoidable using 1.3? Throughput will be less when faceting with LRUCache, but not a lot less under reasonable loads. Just because you're seeing threads blocked on LRUCache doesn't mean it would perform well if it were lockless. How many CPU cores do you have on your box, and how many requests typically execute at the same time? What's your CPU utilization under load? Does a single faceting request return in acceptable time when no other requests are running? -Yonik http://www.lucidimagination.com > > Yonik Seeley-2 wrote: >> >> Solr 1.4 has a cache implementation that's lockless for gets, and >> faster for gets. There's a new faceting implementation as well. >> >> -Yonik >> http://www.lucidimagination.com >> >> On Mon, Jun 15, 2009 at 2:39 PM, CameronL >> wrote: >>> >>> I've searched through the forums and seen a few similar problems to this, >>> but >>> nothing that seemed to help much. >>> >>> We're running Solr 1.3 on Tomcat 6.0.16 and Java 6. We've been having >>> performance problems with our search, causing long query times under >>> normal >>> traffic. We've taken a thread dump and have seen many threads locked or >>> waiting for LRUCache (see below). Our cache values are as follows: >>> >>> >> autowarmCount="1"/> >>> >> autowarmCount="5000"/> >>> >> autowarmCount="0"/> >>> >>> >>> "http-8983-99" daemon prio=10 tid=0x002beb3f5800 nid=0x2fb9 waiting >>> for >>> monitor entry [0x47ea5000..0x47ea6c30] >>> java.lang.Thread.State: BLOCKED (on object monitor) >>> at org.apache.solr.search.LRUCache.get(LRUCache.java:130) >>> - waiting to lock <0x002a9fb94be8> (a >>> org.apache.solr.search.LRUCache$1) >>> at >>> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:605) >>> at >>> org.apache.solr.search.SolrIndexSearcher.numDocs(SolrIndexSearcher.java:1556) >>> at >>> org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) >>> at >>> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) >>> at >>> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) >>> at >>> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) >>> at >>> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) >>> at >>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) >>> at >>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) >>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) >>> at >>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) >>> at >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) >>> "http-8983-83" daemon prio=10 tid=0x002bead1a000 nid=0x2f76 waiting >>> for >>> monitor entry [0x46e95000..0x46e96c30] >>> java.lang.Thread.State: BLOCKED (on object monitor) >>> at org.apache.solr.search.LRUCache.get(LRUCache.java:130) >>> - locked <0x002a9fb94be8> (a >>> org.apache.solr.search.LRUCache$1) >>> at >>> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:605) >>> at >>> org.apache.solr.search.SolrIndexSearcher.numDocs(SolrIndexSearcher.java:1556) >>> at >>> org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) >>> at >>> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) >>> at >>> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) >>> at >>> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) >>> at >>> org.apache.solr.handler.component
Re: LRUCache causing locked threads
>>>> at >>>> org.apache.solr.request.SimpleFacets.getFacetTermEnumCounts(SimpleFacets.java:377) >>>> at >>>> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:156) >>>> at >>>> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:182) >>>> at >>>> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:96) >>>> at >>>> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:70) >>>> at >>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:169) >>>> at >>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) >>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) >>>> at >>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) >>>> at >>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) >>>> >>>> Has anyone else experienced this or does anyone have an idea of why this >>>> might be happening? >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/LRUCache-causing-locked-threads-tp24040421p24040421.html >>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>> >>>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/LRUCache-causing-locked-threads-tp24040421p24040772.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > -- - Noble Paul | Principal Engineer| AOL | http://aol.com
