Hello folks!
I was under the impression that this sync bottleneck was fixed in
recent versions of Solr/Lucene, but we're seeing it with 1.4-dev right
now. When we load test a server with >100 threads (using jmeter), we
see several threads all blocked at the same spot:
"http-8080-exec-505" - Thread t...@594
java.lang.Thread.State: BLOCKED on
org.apache.lucene.index.segmentrea...@2b6f5d18 owned by: http-8080-
exec-434
at org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
737)
at
org
.apache
.lucene.index.MultiSegmentReader.isDeleted(MultiSegmentReader.java:266)
at org.apache.solr.search.function.FunctionQuery
$AllScorer.next(FunctionQuery.java:118)
at org.apache.solr.search.function.FunctionQuery
$AllScorer.skipTo(FunctionQuery.java:137)
at
org
.apache
.lucene
.search.BooleanScorer2$SingleMatchScorer.skipTo(BooleanScorer2.java:170)
at
org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:76)
at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:
357)
at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:
320)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:
136)
at org.apache.lucene.search.Searcher.search(Searcher.java:126)
at org.apache.lucene.search.Searcher.search(Searcher.java:105)
at
org
.apache
.solr
.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:
1231)
at
org
.apache
.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:917)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:
338)
at
org
.apache
.solr.handler.component.QueryComponent.process(QueryComponent.java:164)
at
org
.apache
.solr
.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:
171)
at
org
.apache
.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org
.apache
.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at
org
.apache
.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at
org
.apache
.catalina
.core
.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
235)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org
.apache
.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
233)
at
org
.apache
.catalina.core.StandardContextValve.invoke(StandardContextValve.java:
175)
at
org
.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
128)
at
org
.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
102)
at
org
.apache
.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
286)
at
org
.apache
.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:879)
at org.apache.coyote.http11.Http11NioProtocol
$Http11ConnectionHandler.process(Http11NioProtocol.java:719)
at org.apache.tomcat.util.net.NioEndpoint
$SocketProcessor.run(NioEndpoint.java:2080)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.reentrantlock$nonfairs...@4d54c7be
I checked the Lucene SVN and it looks like that's still appearing to
be a bottleneck.
http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/index/SegmentReader.java?view=markup
Any tips?
Thanks for your time!
Matthew Runo
Software Engineer, Zappos.com
mr...@zappos.com - 702-943-7833
