Hi All,
We're running a fairly non-standard solr configuration. We ingest into named
shards in master cores and then replicate out to slaves running solr cloud. So
in effect we are using solrcloud only to manage the config files and more
importantly to look after the cluster state. Our corpus and search workload,
is such that this makes sense to reduce the need to query every shard for each
search since the majority of queries contain values that allow is to target
search towards the shards holding the appropriate documents, also this isolates
the searching slaves from the costs of indexing (we index fairly infrequently,
but in fairly large volumes). I'm happy to expand on this if anyone's is
interested or take suggestions as to how to we might better be doing things.
We've been running 4.6.0 for the past 3 years or so, but have recently upgraded
to 5.5.2 - we'll likely be upgrading to 6.3.0 shortly. However we hit a
problem when running 5.5.2, which we also replicated in 6.2.1 and 6.3.0. When
a partial replication starts this will usually block all subsequent requests to
solr, whilst replication continues in the background. Whilst in this blocked
state we took thread dumps using VisualVM; we see this when running 6.3.0:
"explicit-fetchindex-cmd" - Thread t@71
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
......
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1463)
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1429)
at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:855)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:434)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:397)
at
org.apache.solr.handler.ReplicationHandler.lambda$handleRequestBody$0(ReplicationHandler.java:279)
at
org.apache.solr.handler.ReplicationHandler$$Lambda$82/776974667.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- locked <4c18799d> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
- locked <64a00f> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
and
"qtp1873653341-61" - Thread t@61
java.lang.Thread.State: TIMED_WAITING
at sun.misc.Unsafe.park(Native Method)
- waiting to lock <64a00f> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) owned by
"explicit-fetchindex-cmd" t@71
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.tryLock(ReentrantReadWriteLock.java:871)
at
org.apache.solr.update.DefaultSolrCoreState.lock(DefaultSolrCoreState.java:159)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:104)
at
org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1781)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1931)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1677)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1577)
.....
The cause of the problem seems to be that in IndexFetcher.fetchLatestIndex,
when the running as solrcloud, the searcher is shut down prior to cleaning up
the existing segment files and downloading the new ones.
6.3.0 - Lines(407-409)
if
(solrCore.getCoreDescriptor().getCoreContainer().isZooKeeperAware()) {
solrCore.closeSearcher();
}
Subsequently solrCore.getUpdateHandler().newIndexWriter(true); takes a write
lock on the indexwriter, which is not released until the openIndexWriter call
after the new files have been copied. So because openNewSearcher needs to take
a read lock on the index writer, and it can't take that whilst the write lock
is in place, all subsequent requests are blocked.
To test this we queued up a load of search requests, then manually triggered
replication, reasoning that a new searcher might be created before the write
lock is taken. On a test instance manually triggering replication would almost
always result in all subsequent requests being blocked, but when we queued up
search requests and ran these whilst triggering replication this never resulted
in the blocking behaviour we were seeing.
We then patched solr locally, to comment out the closeSearcher call, on the
basis that whilst we are running solrcloud, if the core is also running as a
slave there is no need to close the searcher. This seems to work fine;
replication works, nothing hangs.
This seems like a bug to me, but we could find no other reports of the problem.
So my questions are: Is it worth raising an issue in JIRA and working up a
proper patch? Or is our setup so unique there is little value to this? Or am
I missing something else?
Thanks,
Jeremy
________________________________
This message is confidential and may contain privileged information. You should
not disclose its contents to any other person. If you are not the intended
recipient, please notify the sender named above immediately. It is expressly
declared that this e-mail does not constitute nor form part of a contract or
unilateral obligation. Opinions, conclusions and other information in this
message that do not relate to the official business of findmypast shall be
understood as neither given nor endorsed by it.
________________________________
__________________________________________________________________________
This email has been checked for virus and other malicious content prior to
leaving our network.
__________________________________________________________________________
