Another odd thing is that I've seen it hang the same way, just very rarely, on linux. And this is with a timeout. Somehow the timeout is ignored. This just happened on fullmetal - it's stuck even though there is a low timeout for this http call:
[junit4:junit4] 1> Thread[RecoveryThread,5,TGRP-BasicDistributedZkTest] [junit4:junit4] 1> java.net.SocketInputStream.socketRead0(Native Method) [junit4:junit4] 1> java.net.SocketInputStream.read(SocketInputStream.java:150) [junit4:junit4] 1> java.net.SocketInputStream.read(SocketInputStream.java:121) [junit4:junit4] 1> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:149) [junit4:junit4] 1> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:111) [junit4:junit4] 1> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:264) [junit4:junit4] 1> org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98) [junit4:junit4] 1> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252) [junit4:junit4] 1> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:282) [junit4:junit4] 1> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247) [junit4:junit4] 1> org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:216) [junit4:junit4] 1> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298) [junit4:junit4] 1> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) [junit4:junit4] 1> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:647) [junit4:junit4] 1> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464) [junit4:junit4] 1> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) [junit4:junit4] 1> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) [junit4:junit4] 1> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) [junit4:junit4] 1> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:352) [junit4:junit4] 1> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) [junit4:junit4] 1> org.apache.solr.handler.SnapPuller.getLatestVersion(SnapPuller.java:248) [junit4:junit4] 1> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:308) [junit4:junit4] 1> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) [junit4:junit4] 1> org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153) [junit4:junit4] 1> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408) [junit4:junit4] 1> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223) On Dec 14, 2012, at 7:41 PM, Mark Miller <[email protected]> wrote: > There is nothing you can do that I have looked at in a reasonable amount of > time and effort. > > These are tests, and something like that would take a fair amount of mocking > / dependency injection. If someone starts tackling that type of thing, I'll > help, but it's not something I'm working on right now. And it would be better > for a specific test case, not this large integration test stuff. > > I'd rather also have full integration tests that don't mock and inject odd > test code. That's all I fully trust at the end of the day. The rest is just > nice and helpful. > > If you have a patch to propose, I'd be happy to comment on it though. > > - Mark > > > On Dec 14, 2012, at 6:50 PM, Uwe Schindler <[email protected]> wrote: > >> Why have hardcoded sleeps? Could it not be wait/notify? When node shutted >> down it sends notify() and the waiting node waits without limit?? >> >> ----- >> Uwe Schindler >> H.-H.-Meier-Allee 63, D-28213 Bremen >> http://www.thetaphi.de >> eMail: [email protected] >> >> >>> -----Original Message----- >>> From: Mark Miller [mailto:[email protected]] >>> Sent: Saturday, December 15, 2012 12:45 AM >>> To: [email protected] >>> Subject: Re: [JENKINS-MAVEN] Lucene-Solr-Maven-trunk #708: POMs out of >>> sync >>> >>> The problem is that the black hole problem with no query side timeout. >>> >>> I have to do all these waits that slow down tests to try avoiding ever >>> querying a >>> node that is down. So in many cases, I have to wait as long as the zkclient >>> timeout until the ephemeral node representing a SolrCore times out. That is >>> so that we avoid reading it and god forbid, make a request to it. That >>> request >>> will never come back home. >>> >>> The below fail is doing such a wait. It's waiting to see the node go away, >>> but I >>> guess not waiting long enough. >>> >>> I'd love to remove these's waits, speed up tests, and actually test what >>> happens when you query a node that is down. >>> >>> But FreeBDS + Blackhole thwarts me. >>> >>> - Mark >>> >>> >>> On Dec 14, 2012, at 6:38 PM, Uwe Schindler <[email protected]> wrote: >>> >>>> "SOLR-4189: Fix hanging threads on FreeBSD in JettySolrRunner by making >>> connector configureable through sysprop" is fixed. Also my cool >>> syspropertyset worked to pass down sysprop to maven. >>>> >>>> The failing test is a well-known one. >>>> >>>> ----- >>>> Uwe Schindler >>>> H.-H.-Meier-Allee 63, D-28213 Bremen >>>> http://www.thetaphi.de >>>> eMail: [email protected] >>>> >>>> >>>>> -----Original Message----- >>>>> From: Apache Jenkins Server [mailto:[email protected]] >>>>> Sent: Saturday, December 15, 2012 12:35 AM >>>>> To: [email protected] >>>>> Subject: [JENKINS-MAVEN] Lucene-Solr-Maven-trunk #708: POMs out of >>>>> sync >>>>> >>>>> Build: https://builds.apache.org/job/Lucene-Solr-Maven-trunk/708/ >>>>> >>>>> 1 tests failed. >>>>> FAILED: org.apache.solr.cloud.SyncSliceTest.testDistribSearch >>>>> >>>>> Error Message: >>>>> Shard still reported as live in zk - 0 jetty >>>>> >>>>> Stack Trace: >>>>> java.lang.AssertionError: Shard still reported as live in zk - 0 jetty >>>>> at >>>>> >>> __randomizedtesting.SeedInfo.seed([F48C972F3236E094:756A1937456980 >>>>> A8]:0) >>>>> at org.junit.Assert.fail(Assert.java:93) >>>>> at >>>>> org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitToSeeNotLive( >>>>> Abstrac >>>>> tFullDistribZkTestBase.java:1239) >>>>> at >>>>> org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitToSeeNotLive( >>>>> Abstrac >>>>> tFullDistribZkTestBase.java:1224) >>>>> at >>>>> org.apache.solr.cloud.SyncSliceTest.waitToSeeDownInClusterState(SyncS >>>>> liceT >>>>> est.java:259) >>>>> at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:223) >>>>> at >>>>> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseD >>>>> istribu >>>>> tedSearchTestCase.java:793) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at >>>>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. >>>>> jav >>>>> a:57) >>>>> at >>>>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces >>>>> so >>>>> rImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:616) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(Randomized >>>>> Runner.java:1559) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(Random >>>>> izedRunner.java:79) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Random >>>>> i >>>>> zedRunner.java:737) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Random >>>>> i >>>>> zedRunner.java:773) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Random >>>>> i >>>>> zedRunner.java:787) >>>>> at >>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$ >>>>> 1.e >>>>> valuate(SystemPropertiesRestoreRule.java:53) >>>>> at >>>>> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRu >>>>> leS >>>>> etupTeardownChained.java:50) >>>>> at >>>>> org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFi >>>>> eldCac >>>>> heSanity.java:51) >>>>> at >>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef >>>>> oreAf >>>>> terRule.java:46) >>>>> at >>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRul >>>>> e$1.e >>>>> valuate(SystemPropertiesInvariantRule.java:55) >>>>> at >>>>> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleT >>>>> hre >>>>> adAndTestName.java:49) >>>>> at >>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test >>>>> RuleIg >>>>> noreAfterMaxFailures.java:70) >>>>> at >>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai >>>>> lure.j >>>>> ava:48) >>>>> at >>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St >>>>> atem >>>>> entAdapter.java:36) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner. >>>>> ru >>>>> n(ThreadLeakControl.java:358) >>>>> at >>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTa >>>>> sk( >>>>> ThreadLeakControl.java:782) >>>>> at >>>>> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(Threa >>>>> dL >>>>> eakControl.java:442) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Ran >>>>> do >>>>> mizedRunner.java:746) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Random >>>>> i >>>>> zedRunner.java:648) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Random >>>>> i >>>>> zedRunner.java:682) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Random >>>>> i >>>>> zedRunner.java:693) >>>>> at >>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St >>>>> atem >>>>> entAdapter.java:36) >>>>> at >>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$ >>>>> 1.e >>>>> valuate(SystemPropertiesRestoreRule.java:53) >>>>> at >>>>> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBef >>>>> oreAf >>>>> terRule.java:46) >>>>> at >>>>> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStor >>>>> eClas >>>>> sName.java:42) >>>>> at >>>>> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRul >>>>> e$1.e >>>>> valuate(SystemPropertiesInvariantRule.java:55) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth >>>>> odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMeth >>>>> odsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) >>>>> at >>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St >>>>> atem >>>>> entAdapter.java:36) >>>>> at >>>>> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRule >>>>> Asser >>>>> tionsRequired.java:43) >>>>> at >>>>> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFai >>>>> lure.j >>>>> ava:48) >>>>> at >>>>> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(Test >>>>> RuleIg >>>>> noreAfterMaxFailures.java:70) >>>>> at >>>>> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIg >>>>> noreTe >>>>> stSuites.java:55) >>>>> at >>>>> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(St >>>>> atem >>>>> entAdapter.java:36) >>>>> at >>>>> >>> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner. >>>>> ru >>>>> n(ThreadLeakControl.java:358) >>>>> at java.lang.Thread.run(Thread.java:679) >>>>> >>>>> >>>>> >>>>> >>>>> Build Log: >>>>> [...truncated 21206 lines...] >>>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [email protected] For >>>> additional commands, e-mail: [email protected] >>>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] For additional >>> commands, e-mail: [email protected] >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
