[
https://issues.apache.org/jira/browse/SOLR-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972813#comment-14972813
]
Mark Miller commented on SOLR-6406:
-----------------------------------
A more recent set of stack traces:
{noformat}
[junit4] ERROR 0.00s | HdfsChaosMonkeyNothingIsSafeTest (suite) <<<
[junit4] > Throwable #1: java.lang.AssertionError: ERROR:
SolrIndexSearcher opens=39 closes=38
[junit4] > at
__randomizedtesting.SeedInfo.seed([71608A03B4692CB]:0)
[junit4] > at
org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:468)
[junit4] > at
org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
[junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE
scope at org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest:
[junit4] > 1) Thread[id=243, name=qtp487431535-243, state=WAITING,
group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
[junit4] > at java.lang.Object.wait(Native Method)
[junit4] > at java.lang.Object.wait(Object.java:502)
[junit4] > at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient.blockUntilFinished(ConcurrentUpdateSolrClient.java:404)
[junit4] > at
org.apache.solr.update.StreamingSolrClients.blockUntilFinished(StreamingSolrClients.java:103)
[junit4] > at
org.apache.solr.update.SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:231)
[junit4] > at
org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:89)
[junit4] > at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:778)
[junit4] > at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1622)
[junit4] > at
org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:183)
[junit4] > at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
[junit4] > at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:151)
[junit4] > at
org.apache.solr.core.SolrCore.execute(SolrCore.java:2079)
[junit4] > at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:667)
[junit4] > at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
[junit4] > at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
[junit4] > at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
[junit4] > at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] > at
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
[junit4] > at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] > at
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
[junit4] > at
org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
[junit4] > at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
[junit4] > at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
[junit4] > at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
[junit4] > at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
[junit4] > at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
[junit4] > at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] > at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
[junit4] > at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] > at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
[junit4] > at
org.eclipse.jetty.server.Server.handle(Server.java:499)
[junit4] > at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
[junit4] > at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
[junit4] > at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
[junit4] > at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[junit4] > at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > 2) Thread[id=266, name=searcherExecutor-30-thread-1,
state=WAITING, group=TGRP-HdfsChaosMonkeyNothingIsSafeTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
[junit4] > at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
[junit4] > at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > at
__randomizedtesting.SeedInfo.seed([71608A03B4692CB]:0)
{noformat}
It would seem that runners.wait() can race with a final runners.notifyAll() or
something.
> ConcurrentUpdateSolrServer hang in blockUntilFinished.
> ------------------------------------------------------
>
> Key: SOLR-6406
> URL: https://issues.apache.org/jira/browse/SOLR-6406
> Project: Solr
> Issue Type: Bug
> Reporter: Mark Miller
> Fix For: 5.0, Trunk
>
> Attachments: CPU Sampling.png
>
>
> Not sure what is causing this, but SOLR-6136 may have taken us a step back
> here. I see this problem occasionally pop up in ChaosMonkeyNothingIsSafeTest
> now - test fails because of a thread leak, thread leak is due to a
> ConcurrentUpdateSolrServer hang in blockUntilFinished. Only started popping
> up recently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
