All:
As part of SOLR-4196, I'm opening and closing cores at a furious rate.
My tests are running for 20-40 minutes then locking up quite
regularly. Of course the first place I'm looking is my recent code,
since it has a bunch of synchronized blocks.
The deadlock is definitely happening at a call from the new code to
close a Solr core, so to really look at this anyone will need to get
the patch I'll put up in a minute. The deadlock trace is below.
But without going that far, I question whether it's really anything to
do with new synchronizations I'm doing or whether it's just something
that's been lurking for a while and I'm flushing out of the woodwork.
One of the deadlocked threads may be called form my code, but as far
as I can tell none of the actual synchronization objects I'm using are
held. I have the full jstack output if anyone needs it...
Of course I'll continue looking, but at a glance I'm wondering if this
code has really ever been stressed this way before or whether these
have existed for a while. All synchronization should be approached
with fear and loathing IMO.....
One thread blocks at a synchronized method, but should this method
really be synchronized?
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:78)
(here's the method) @Override
public synchronized RefCounted<IndexWriter> getIndexWriter(SolrCore core)
throws IOException {
and a little later in the method there's:
synchronized (writerPauseLock) {
if (core == null) {
and the other thread blocks at:
at
org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:668),
(here's the method)
// IndexWriterCloser interface method - called from solrCoreState.decref(this)
@Override
public void closeWriter(IndexWriter writer) throws IOException {
boolean clearRequestInfo = false;
commitLock.lock(); **********locking here!
try {
SolrQueryRequest req = new LocalSolrQueryRequest(core, new
ModifiableSolrParams());
SolrQueryResponse rsp = new SolrQueryResponse();
if (SolrRequestInfo.getRequestInfo() == null) {
clearRequestInfo = true;
Java stack information for the threads listed above:
===================================================
"commitScheduler-42617-thread-1":
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:78)
- waiting to lock <78b4aa518> (a org.apache.solr.update.DefaultSolrCoreState)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1359)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:561)
- locked <7884ca730> (a java.lang.Object)
at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
*********
Other thread
"qtp1401888126-32":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <788d73208> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at
org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:668)
at
org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:64)
- locked <78b4aa518> (a org.apache.solr.update.DefaultSolrCoreState)
at
org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:272)
- locked <78b4aa518> (a org.apache.solr.update.DefaultSolrCoreState)
at org.apache.solr.core.SolrCore.decrefSolrCoreState(SolrCore.java:888)
- locked <78b4aa518> (a org.apache.solr.update.DefaultSolrCoreState)
at org.apache.solr.core.SolrCore.close(SolrCore.java:980)
at org.apache.solr.core.CoreMaps.putTransientCore(CoreContainer.java:1465)
at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:730)
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1137)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:190)
at
Thanks,
Erick
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
