I see a ReadOnlySegmentReader now - we're on an optimized index now
which gets around the isDeleted() check.
(solr4, optimized)
searcherName : searc...@260f8e27 main
caching : true
numDocs : 139583
maxDoc : 139583
readerImpl : ReadOnlySegmentReader
readerDir : org.apache.lucene.store.NIOFSDirectory@/opt/solr-data/zeta-
main/index
indexVersion : 1233423823917
openedAt : Thu Feb 26 13:29:25 PST 2009
registeredAt : Thu Feb 26 13:29:42 PST 2009
warmupTime : 16910
(solr1, non optimized)
searcherName : searc...@36be11a1 main
caching : true
numDocs : 139561
maxDoc : 139591
readerImpl : ReadOnlyMultiSegmentReader
readerDir : org.apache.lucene.store.NIOFSDirectory@/opt/solr-data/zeta-
main/index
indexVersion : 1233423823924
openedAt : Thu Feb 26 13:48:16 PST 2009
registeredAt : Thu Feb 26 13:49:11 PST 2009
warmupTime : 54785
I did a thread dump against the optimized server just now, but didn't
find anything blocked to check which reader was actually in use this
time.
Thanks for your time!
Matthew Runo
Software Engineer, Zappos.com
mr...@zappos.com - 702-943-7833
On Feb 26, 2009, at 1:39 PM, Yonik Seeley wrote:
That's interesting.
We should be using read-only readers, which should not synchronize on
the deleted docs check. But as your stack trace shows, you're using
SegmentReader and MultiSegmentReader.
Right now, if I look at the admin/statistics page at the searcher, it
shows the following for the reader:
reader:SolrIndexReader
{this=42f352,r=readonlymultisegmentrea...@42f352,segments=6}
Hopefully the fact that it's a ReadOnlyMultiSegmentReader means that
it contains ReadOnlySegmentReader instances, which don't synchronize
on isDeleted.
What do you see?
-Yonik
On Thu, Feb 26, 2009 at 4:09 PM, Matthew Runo <mr...@zappos.com>
wrote:
Hello folks!
I was under the impression that this sync bottleneck was fixed in
recent
versions of Solr/Lucene, but we're seeing it with 1.4-dev right
now. When we
load test a server with >100 threads (using jmeter), we see several
threads
all blocked at the same spot:
"http-8080-exec-505" - Thread t...@594
java.lang.Thread.State: BLOCKED on
org.apache.lucene.index.segmentrea...@2b6f5d18 owned by: http-8080-
exec-434
at
org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:
737)
at
org
.apache
.lucene.index.MultiSegmentReader.isDeleted(MultiSegmentReader.java:
266)
at
org.apache.solr.search.function.FunctionQuery
$AllScorer.next(FunctionQuery.java:118)
at
org.apache.solr.search.function.FunctionQuery
$AllScorer.skipTo(FunctionQuery.java:137)
at
org
.apache
.lucene
.search.BooleanScorer2$SingleMatchScorer.skipTo(BooleanScorer2.java:
170)
at
org.apache.lucene.search.ReqOptSumScorer.score(ReqOptSumScorer.java:
76)
at
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:
357)
at
org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:
320)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:136)
at org.apache.lucene.search.Searcher.search(Searcher.java:126)
at org.apache.lucene.search.Searcher.search(Searcher.java:105)
at
org
.apache
.solr
.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearcher.java:
1231)
at
org
.apache
.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:
917)
at
org
.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:
338)
at
org
.apache
.solr.handler.component.QueryComponent.process(QueryComponent.java:
164)
at
org
.apache
.solr
.handler
.component.SearchHandler.handleRequestBody(SearchHandler.java:171)
at
org
.apache
.solr
.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:
131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org
.apache
.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at
org
.apache
.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
232)
at
org
.apache
.catalina
.core
.ApplicationFilterChain
.internalDoFilter(ApplicationFilterChain.java:235)
at
org
.apache
.catalina
.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
206)
at
org
.apache
.catalina
.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org
.apache
.catalina
.core.StandardContextValve.invoke(StandardContextValve.java:175)
at
org
.apache
.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org
.apache
.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org
.apache
.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
109)
at
org
.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
286)
at
org
.apache
.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:
879)
at
org.apache.coyote.http11.Http11NioProtocol
$Http11ConnectionHandler.process(Http11NioProtocol.java:719)
at
org.apache.tomcat.util.net.NioEndpoint
$SocketProcessor.run(NioEndpoint.java:2080)
at
java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:885)
at
java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- locked
java.util.concurrent.locks.reentrantlock$nonfairs...@4d54c7be
I checked the Lucene SVN and it looks like that's still appearing
to be a
bottleneck.
http://svn.apache.org/viewvc/lucene/java/trunk/src/java/org/apache/lucene/index/SegmentReader.java?view=markup
Any tips?
Thanks for your time!
Matthew Runo
Software Engineer, Zappos.com
mr...@zappos.com - 702-943-7833
--
-Yonik
http://www.lucidimagination.com
