Build: https://builds.apache.org/job/Lucene-Solr-tests-only-trunk-java7/2950/
1 tests failed. FAILED: org.apache.solr.cloud.FullSolrCloudTest.testDistribSearch Error Message: Shard still reported as live in zk Stack Trace: java.lang.AssertionError: Shard still reported as live in zk at __randomizedtesting.SeedInfo.seed([87EFC7C5D91A084C:60949DDAE456870]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.FullSolrCloudTest.brindDownShardIndexSomeDocsAndRecover(FullSolrCloudTest.java:715) at org.apache.solr.cloud.FullSolrCloudTest.doTest(FullSolrCloudTest.java:559) at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:679) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:818) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:891) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:825) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132) at com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:671) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:697) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:736) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) at org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53) at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132) at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551) Build Log: [...truncated 12216 lines...] [junit4:junit4] 2> at org.apache.solr.cloud.Overseer$CloudStateUpdater.run(Overseer.java:113) [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722) [junit4:junit4] 2> [junit4:junit4] 2> 96995 T56 oasc.Overseer$CloudStateUpdater.amILeader According to ZK I (id=88015724125356034-127.0.0.1:27285_solr-n_0000000000) am no longer a leader. [junit4:junit4] 2> 97341 T105 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:57073 [junit4:junit4] 2> 97688 T68 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:57073 [junit4:junit4] 2> 98026 T92 oaz.ClientCnxn$SendThread.startConnect Opening socket connection to server localhost/127.0.0.1:57073 [junit4:junit4] 2> 108141 T69 oaz.ClientCnxn$EventThread.run EventThread shut down [junit4:junit4] 2> 108141 T34 oaz.ZooKeeper.close Session: 0x138b1d5b3240003 closed [junit4:junit4] 2> 108143 T34 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null} [junit4:junit4] 2> 108196 T34 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 53538 [junit4:junit4] 2> 108196 T34 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 37233 [junit4:junit4] 2> 108196 T34 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=8849257 [junit4:junit4] 2> 108197 T34 oasc.RecoveryStrategy.close WARNING Stopping recovery for core collection1 zkNodeName=127.0.0.1:37233_solr_collection1 [junit4:junit4] 2> 108197 T34 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6390c7a [junit4:junit4] 2> 108203 T34 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request. [junit4:junit4] 2> 108203 T34 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=7,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=36,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0} [junit4:junit4] 2> 108206 T34 oasc.RecoveryStrategy.close WARNING Stopping recovery for core collection1 zkNodeName=127.0.0.1:37233_solr_collection1 [junit4:junit4] 2> 108307 T93 oaz.ClientCnxn$EventThread.run EventThread shut down [junit4:junit4] 2> 108307 T34 oaz.ZooKeeper.close Session: 0x138b1d5b3240005 closed [junit4:junit4] 2> 108309 T34 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null} [junit4:junit4] 2> 108361 T34 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 12825 [junit4:junit4] 2> 108361 T34 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1872108915 [junit4:junit4] 2> 108361 T34 oasc.RecoveryStrategy.close WARNING Stopping recovery for core collection1 zkNodeName=127.0.0.1:12825_solr_collection1 [junit4:junit4] 2> 108463 T34 oaz.ZooKeeper.close Session: 0x138b1d5b3240006 closed [junit4:junit4] 2> 108464 T34 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null} [junit4:junit4] 2> 108745 T34 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch [junit4:junit4] 1> /solr/collections/control_collection/leader_elect/control_shard/election/88015724125356034-127.0.0.1:27285_solr_collection1-n_0000000000 (0) [junit4:junit4] 1> /solr/collections/control_collection/leaders (1) [junit4:junit4] 1> /solr/collections/control_collection/leaders/control_shard (0) [junit4:junit4] 1> DATA: [junit4:junit4] 1> { [junit4:junit4] 1> "core":"collection1", [junit4:junit4] 1> "node_name":"127.0.0.1:27285_solr", [junit4:junit4] 1> "base_url":"http://127.0.0.1:27285/solr"} [junit4:junit4] 1> /solr/clusterstate.json (0) [junit4:junit4] 1> DATA: [junit4:junit4] 1> { [junit4:junit4] 1> "collection1":{ [junit4:junit4] 1> "shard1":{ [junit4:junit4] 1> "127.0.0.1:10248_solr_collection1":{ [junit4:junit4] 1> "shard":"shard1", [junit4:junit4] 1> "roles":null, [junit4:junit4] 1> "leader":"true", [junit4:junit4] 1> "state":"active", [junit4:junit4] 1> "core":"collection1", [junit4:junit4] 1> "collection":"collection1", [junit4:junit4] 1> "node_name":"127.0.0.1:10248_solr", [junit4:junit4] 1> "base_url":"http://127.0.0.1:10248/solr"}, [junit4:junit4] 1> "127.0.0.1:37233_solr_collection1":{ [junit4:junit4] 1> "shard":"shard1", [junit4:junit4] 1> "roles":null, [junit4:junit4] 1> "state":"active", [junit4:junit4] 1> "core":"collection1", [junit4:junit4] 1> "collection":"collection1", [junit4:junit4] 1> "node_name":"127.0.0.1:37233_solr", [junit4:junit4] 1> "base_url":"http://127.0.0.1:37233/solr"}}, [junit4:junit4] 1> "shard2":{ [junit4:junit4] 1> "127.0.0.1:53538_solr_collection1":{ [junit4:junit4] 1> "shard":"shard2", [junit4:junit4] 1> "leader":"true", [junit4:junit4] 1> "roles":null, [junit4:junit4] 1> "state":"active", [junit4:junit4] 1> "core":"collection1", [junit4:junit4] 1> "collection":"collection1", [junit4:junit4] 1> "node_name":"127.0.0.1:53538_solr", [junit4:junit4] 1> "base_url":"http://127.0.0.1:53538/solr"}, [junit4:junit4] 1> "127.0.0.1:12825_solr_collection1":{ [junit4:junit4] 1> "shard":"shard2", [junit4:junit4] 1> "roles":null, [junit4:junit4] 1> "state":"active", [junit4:junit4] 1> "core":"collection1", [junit4:junit4] 1> "collection":"collection1", [junit4:junit4] 1> "node_name":"127.0.0.1:12825_solr", [junit4:junit4] 1> "base_url":"http://127.0.0.1:12825/solr"}}}, [junit4:junit4] 1> "control_collection":{"control_shard":{"127.0.0.1:27285_solr_collection1":{ [junit4:junit4] 1> "shard":"control_shard", [junit4:junit4] 1> "roles":null, [junit4:junit4] 1> "leader":"true", [junit4:junit4] 1> "state":"active", [junit4:junit4] 1> "core":"collection1", [junit4:junit4] 1> "collection":"control_collection", [junit4:junit4] 1> "node_name":"127.0.0.1:27285_solr", [junit4:junit4] 1> "base_url":"http://127.0.0.1:27285/solr"}}}} [junit4:junit4] 1> /zookeeper (1) [junit4:junit4] 1> DATA: [junit4:junit4] 1> [junit4:junit4] 1> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=FullSolrCloudTest -Dtests.method=testDistribSearch -Dtests.seed=87EFC7C5D91A084C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh -Dtests.timezone=America/Goose_Bay -Dtests.file.encoding=US-ASCII [junit4:junit4] 1> [junit4:junit4] 2> [junit4:junit4] > (@AfterClass output) [junit4:junit4] 2> 108792 T34 oas.SolrTestCaseJ4.deleteCore ###deleteCore [junit4:junit4] 2> 109463 T106 oasc.SyncStrategy.syncToMe Could not tell a replica to recover org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:53538/solr/collection1 [junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:414) [junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:182) [junit4:junit4] 2> at org.apache.solr.cloud.SyncStrategy.syncToMe(SyncStrategy.java:237) [junit4:junit4] 2> at org.apache.solr.cloud.SyncStrategy.syncReplicas(SyncStrategy.java:119) [junit4:junit4] 2> at org.apache.solr.cloud.SyncStrategy.sync(SyncStrategy.java:82) [junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:158) [junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:151) [junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:96) [junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:57) [junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:125) [junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) [junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) [junit4:junit4] 2> Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 127.0.0.1:53538 timed out [junit4:junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:125) [junit4:junit4] 2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148) [junit4:junit4] 2> at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150) [junit4:junit4] 2> at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121) [junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575) [junit4:junit4] 2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425) [junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) [junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) [junit4:junit4] 2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) [junit4:junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:353) [junit4:junit4] 2> ... 11 more [junit4:junit4] 2> [junit4:junit4] 2> 109464 T106 oasc.RecoveryStrategy.close WARNING Stopping recovery for core collection1 zkNodeName=127.0.0.1:12825_solr_collection1 [junit4:junit4] 2> 109465 T106 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@6ce96831 [junit4:junit4] 2> 109471 T106 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request. [junit4:junit4] 2> 109478 T106 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=36,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0} [junit4:junit4] 2> 109479 T106 oasc.RecoveryStrategy.close WARNING Stopping recovery for core collection1 zkNodeName=127.0.0.1:12825_solr_collection1 [junit4:junit4] 2> 109480 T106 oasc.LeaderElector$1.process WARNING org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn- [junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) [junit4:junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) [junit4:junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:643) [junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:244) [junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:241) [junit4:junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:67) [junit4:junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:241) [junit4:junit4] 2> at org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:236) [junit4:junit4] 2> at org.apache.solr.cloud.ZkController.publish(ZkController.java:734) [junit4:junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:168) [junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:151) [junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:96) [junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:57) [junit4:junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:125) [junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526) [junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) [junit4:junit4] 2> [junit4:junit4] 2> 109481 T106 oascc.ZkStateReader$3.process Updating live nodes [junit4:junit4] 2> 109481 T106 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4:junit4] 2> 109481 T106 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42c3d37a name:ZooKeeperConnection Watcher:127.0.0.1:57073/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None [junit4:junit4] 2> 109481 T106 oaz.ClientCnxn$EventThread.run EventThread shut down [junit4:junit4] 2> NOTE: test params are: codec=Lucene40: {n_f1=PostingsFormat(name=MockRandom), foo_b=MockVariableIntBlock(baseBlockSize=50), foo_d=PostingsFormat(name=NestedPulsing), foo_f=MockVariableIntBlock(baseBlockSize=50), n_tl1=PostingsFormat(name=MockRandom), n_d1=PostingsFormat(name=Memory doPackFST= true), rnd_b=PostingsFormat(name=NestedPulsing), intDefault=PostingsFormat(name=Memory doPackFST= true), n_td1=PostingsFormat(name=MockRandom), id=PostingsFormat(name=NestedPulsing), timestamp=PostingsFormat(name=Memory doPackFST= true), range_facet_sl=MockVariableIntBlock(baseBlockSize=50), range_facet_si=PostingsFormat(name=MockRandom), oddField_s=MockVariableIntBlock(baseBlockSize=50), multiDefault=PostingsFormat(name=MockRandom), n_tf1=PostingsFormat(name=Memory doPackFST= true), n_dt1=PostingsFormat(name=MockRandom), n_ti1=PostingsFormat(name=NestedPulsing), range_facet_l=PostingsFormat(name=Memory doPackFST= true), text=MockVariableIntBlock(baseBlockSize=50), _version_=PostingsFormat(name=Memory doPackFST= true), SubjectTerms_mfacet=PostingsFormat(name=Memory doPackFST= true), a_t=PostingsFormat(name=Memory doPackFST= true), n_tdt1=PostingsFormat(name=Memory doPackFST= true), other_tl1=PostingsFormat(name=Memory doPackFST= true), n_l1=PostingsFormat(name=Memory doPackFST= true), a_si=PostingsFormat(name=MockRandom)}, sim=DefaultSimilarity, locale=zh, timezone=America/Goose_Bay [junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Oracle Corporation 1.7.0_04 (64-bit)/cpus=16,threads=1,free=54955664,total=153092096 [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestRangeQuery, CloudStateTest, TestIndexSearcher, FullSolrCloudTest] [junit4:junit4] 2> [junit4:junit4] Completed on J1 in 109.83s, 1 test, 1 failure <<< FAILURES! [...truncated 782 lines...] [...truncated 13149 lines...] [...truncated 13149 lines...] [...truncated 13149 lines...] [...truncated 13149 lines...] [...truncated 13136 lines...] BUILD FAILED /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/build.xml:29: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/solr/build.xml:144: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/lucene/module-build.xml:62: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/lucene/common-build.xml:1094: The following error occurred while executing this line: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-trunk-java7/checkout/lucene/common-build.xml:789: There were test failures: 297 suites, 1054 tests, 1 failure, 5 ignored Total time: 35 minutes 55 seconds Build step 'Execute shell' marked build as failure Archiving artifacts Recording test results Email was triggered for: Failure Sending email for trigger: Failure
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org