[
https://issues.apache.org/jira/browse/SOLR-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14490307#comment-14490307
]
Timothy Potter commented on SOLR-6820:
--------------------------------------
Thanks for the nice description of this Yonik! I think this thread dump shows
the problem nicely:
{code}
"recoveryExecutor-7-thread-1" prio=10 tid=0x00007f0fe821e800 nid=0xbafc
runnable [0x00007f1003c30000]
java.lang.Thread.State: RUNNABLE
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsEnum.nextDoc(Lucene41PostingsReader.java:440)
at
org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:111)
at
org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:157)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:351)
at
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
at
org.apache.lucene.index.BufferedUpdatesStream.applyQueryDeletes(BufferedUpdatesStream.java:545)
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:284)
- locked <0x00000005d9e96768> (a
org.apache.lucene.index.BufferedUpdatesStream)
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3238)
- locked <0x00000005d9a4f2a8> (a org.apache.solr.update.SolrIndexWriter)
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3229)
- locked <0x00000005d9a4f2a8> (a org.apache.solr.update.SolrIndexWriter)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:384)
- locked <0x00000005d9a4f2a8> (a org.apache.solr.update.SolrIndexWriter)
- locked <0x00000005dc1943a8> (a java.lang.Object)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:289)
at
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:274)
at
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1465)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:424)
- locked <0x00000005dd011ab8> (a org.apache.solr.update.UpdateLog)
at
org.apache.solr.update.DirectUpdateHandler2.addAndDelete(DirectUpdateHandler2.java:449)
- locked <0x00000005dc1943d8> (a java.lang.Object)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:216)
at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:160)
at
org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:69)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:51)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:928)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1082)
- locked <0x00000005c2b560d0> (a org.apache.solr.update.VersionBucket)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:695)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
at
org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1343)
at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1217)
...
{code}
and
{code}
"recoveryExecutor-7-thread-2" prio=10 tid=0x00007ec5b4003000 nid=0xc131 waiting
for monitor entry [0x00007f1003aab000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:999)
- waiting to lock <0x00000005c2b560d0> (a
org.apache.solr.update.VersionBucket)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:695)
at
org.apache.solr.update.processor.LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:104)
at
org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1343)
at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1217)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
...
{code}
> The sync on the VersionInfo bucket in DistributedUpdateProcesser#addDocument
> appears to be a large bottleneck when using replication.
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-6820
> URL: https://issues.apache.org/jira/browse/SOLR-6820
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
> Reporter: Mark Miller
> Assignee: Timothy Potter
> Attachments: SOLR-6820.patch, threads.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
