Build: https://builds.apache.org/job/Lucene-Solr-tests-only-4.x/312/
1 tests failed.
REGRESSION: org.apache.solr.cloud.FullSolrCloudTest.testDistribSearch
Error Message:
Shard still reported as live in zk
Stack Trace:
java.lang.AssertionError: Shard still reported as live in zk
at
__randomizedtesting.SeedInfo.seed([D6106D46F6D8368E:57F6E35E818756B2]:0)
at org.junit.Assert.fail(Assert.java:93)
at
org.apache.solr.cloud.FullSolrCloudTest.brindDownShardIndexSomeDocsAndRecover(FullSolrCloudTest.java:712)
at
org.apache.solr.cloud.FullSolrCloudTest.doTest(FullSolrCloudTest.java:556)
at
org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:679)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:818)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:877)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:891)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:825)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:671)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:697)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:736)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:747)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at
org.apache.lucene.util.TestRuleIcuHack$1.evaluate(TestRuleIcuHack.java:51)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at
org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
at
org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)
Build Log:
[...truncated 11536 lines...]
[junit4:junit4] 2> 85129 T296 oazs.SessionTrackerImpl.run SessionTrackerImpl
exited loop!
[junit4:junit4] 2> 94470 T350 oaz.ClientCnxn$SendThread.run Client session
timed out, have not heard from server in 12001ms for sessionid
0x138b0741cab0005, closing socket connection and attempting reconnect
[junit4:junit4] 2> 94475 T363 oaz.ClientCnxn$SendThread.run Client session
timed out, have not heard from server in 12008ms for sessionid
0x138b0741cab0006, closing socket connection and attempting reconnect
[junit4:junit4] 2> 94597 T327 oaz.ClientCnxn$EventThread.run EventThread shut
down
[junit4:junit4] 2> 94597 T293 oaz.ZooKeeper.close Session: 0x138b0741cab0003
closed
[junit4:junit4] 2> 94599 T293 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 94656 T293 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
25221
[junit4:junit4] 2> 94657 T293 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
23258
[junit4:junit4] 2> 94657 T293 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=1907293563
[junit4:junit4] 2> 94658 T293 oasc.RecoveryStrategy.close WARNING Stopping
recovery for core collection1 zkNodeName=127.0.0.1:23258_solr_collection1
[junit4:junit4] 2> 94658 T293 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@7ac9f71b
[junit4:junit4] 2> 94664 T293 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 94664 T293 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=7,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=36,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 94672 T293 oasc.RecoveryStrategy.close WARNING Stopping
recovery for core collection1 zkNodeName=127.0.0.1:23258_solr_collection1
[junit4:junit4] 2> 96032 T364 oasc.SyncStrategy.syncToMe Could not tell a
replica to recover org.apache.solr.client.solrj.SolrServerException:
IOException occured when talking to server at:
http://127.0.0.1:25221/solr/collection1
[junit4:junit4] 2> at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:414)
[junit4:junit4] 2> at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:182)
[junit4:junit4] 2> at
org.apache.solr.cloud.SyncStrategy.syncToMe(SyncStrategy.java:237)
[junit4:junit4] 2> at
org.apache.solr.cloud.SyncStrategy.syncReplicas(SyncStrategy.java:119)
[junit4:junit4] 2> at
org.apache.solr.cloud.SyncStrategy.sync(SyncStrategy.java:82)
[junit4:junit4] 2> at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:158)
[junit4:junit4] 2> at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:151)
[junit4:junit4] 2> at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:96)
[junit4:junit4] 2> at
org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:57)
[junit4:junit4] 2> at
org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:125)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4] 2> Caused by: org.apache.http.conn.ConnectTimeoutException:
Connect to 127.0.0.1:25221 timed out
[junit4:junit4] 2> at
org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:125)
[junit4:junit4] 2> at
org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:148)
[junit4:junit4] 2> at
org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:150)
[junit4:junit4] 2> at
org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:121)
[junit4:junit4] 2> at
org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:575)
[junit4:junit4] 2> at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:425)
[junit4:junit4] 2> at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
[junit4:junit4] 2> at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
[junit4:junit4] 2> at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
[junit4:junit4] 2> at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:353)
[junit4:junit4] 2> ... 11 more
[junit4:junit4] 2>
[junit4:junit4] 2> 96033 T364 oasc.RecoveryStrategy.close WARNING Stopping
recovery for core collection1 zkNodeName=127.0.0.1:31180_solr_collection1
[junit4:junit4] 2> 96147 T363 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server localhost/127.0.0.1:30687
[junit4:junit4] 2> 96305 T350 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server localhost/127.0.0.1:30687
[junit4:junit4] 2> 106577 T363 oaz.ClientCnxn$SendThread.run Client session
timed out, have not heard from server in 12001ms for sessionid
0x138b0741cab0006, closing socket connection and attempting reconnect
[junit4:junit4] 2> 106674 T351 oaz.ClientCnxn$EventThread.run EventThread
shut down
[junit4:junit4] 2> 106674 T293 oaz.ZooKeeper.close Session: 0x138b0741cab0005
closed
[junit4:junit4] 2> 106675 T293 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 106727 T293 oasc.ChaosMonkey.monkeyLog monkey: stop shard!
31180
[junit4:junit4] 2> 106727 T293 oasc.CoreContainer.shutdown Shutting down
CoreContainer instance=1682580949
[junit4:junit4] 2> 106727 T293 oasc.RecoveryStrategy.close WARNING Stopping
recovery for core collection1 zkNodeName=127.0.0.1:31180_solr_collection1
[junit4:junit4] 2> 107832 T363 oaz.ClientCnxn$SendThread.startConnect Opening
socket connection to server localhost/127.0.0.1:30687
[junit4:junit4] 2> 118780 T293 oaz.ZooKeeper.close Session: 0x138b0741cab0006
closed
[junit4:junit4] 2> 118781 T293 oejsh.ContextHandler.doStop stopped
o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4] 2> 118879 T293 oas.SolrTestCaseJ4.tearDown ###Ending
testDistribSearch
[junit4:junit4] 1>
/solr/collections/control_collection/leader_elect/control_shard/election/88014205477388290-127.0.0.1:22131_solr_collection1-n_0000000000
(0)
[junit4:junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4:junit4] 1>
/solr/collections/control_collection/leaders/control_shard (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:22131_solr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:22131/solr"}
[junit4:junit4] 1> /solr/clusterstate.json (0)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1> {
[junit4:junit4] 1> "collection1":{
[junit4:junit4] 1> "shard1":{
[junit4:junit4] 1> "127.0.0.1:53704_solr_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "leader":"true",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:53704_solr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:53704/solr"},
[junit4:junit4] 1> "127.0.0.1:23258_solr_collection1":{
[junit4:junit4] 1> "shard":"shard1",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:23258_solr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:23258/solr"}},
[junit4:junit4] 1> "shard2":{
[junit4:junit4] 1> "127.0.0.1:25221_solr_collection1":{
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "leader":"true",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:25221_solr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:25221/solr"},
[junit4:junit4] 1> "127.0.0.1:31180_solr_collection1":{
[junit4:junit4] 1> "shard":"shard2",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"collection1",
[junit4:junit4] 1> "node_name":"127.0.0.1:31180_solr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:31180/solr"}}},
[junit4:junit4] 1>
"control_collection":{"control_shard":{"127.0.0.1:22131_solr_collection1":{
[junit4:junit4] 1> "shard":"control_shard",
[junit4:junit4] 1> "roles":null,
[junit4:junit4] 1> "leader":"true",
[junit4:junit4] 1> "state":"active",
[junit4:junit4] 1> "core":"collection1",
[junit4:junit4] 1> "collection":"control_collection",
[junit4:junit4] 1> "node_name":"127.0.0.1:22131_solr",
[junit4:junit4] 1> "base_url":"http://127.0.0.1:22131/solr"}}}}
[junit4:junit4] 1> /zookeeper (1)
[junit4:junit4] 1> DATA:
[junit4:junit4] 1>
[junit4:junit4] 1>
[junit4:junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=FullSolrCloudTest -Dtests.method=testDistribSearch
-Dtests.seed=D6106D46F6D8368E -Dtests.multiplier=3 -Dtests.slow=true
-Dtests.locale=en_ZA -Dtests.timezone=America/Rankin_Inlet
-Dtests.file.encoding=UTF-8
[junit4:junit4] 1>
[junit4:junit4] 2>
[junit4:junit4] > (@AfterClass output)
[junit4:junit4] 2> 118934 T293 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 119781 T364 oasc.SolrCore.close [collection1] CLOSING
SolrCore org.apache.solr.core.SolrCore@4a23d87f
[junit4:junit4] 2> 119786 T364 oasc.SolrCore.closeSearcher [collection1]
Closing main searcher on request.
[junit4:junit4] 2> 119788 T364 oasu.DirectUpdateHandler2.close closing
DirectUpdateHandler2{commits=6,autocommits=0,soft
autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=36,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 119789 T364 oasc.RecoveryStrategy.close WARNING Stopping
recovery for core collection1 zkNodeName=127.0.0.1:31180_solr_collection1
[junit4:junit4] 2> 119789 T364 oasc.LeaderElector$1.process WARNING
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode =
Session expired for /overseer/queue/qn-
[junit4:junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
[junit4:junit4] 2> at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
[junit4:junit4] 2> at
org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:643)
[junit4:junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:244)
[junit4:junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:241)
[junit4:junit4] 2> at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:67)
[junit4:junit4] 2> at
org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:241)
[junit4:junit4] 2> at
org.apache.solr.cloud.DistributedQueue.offer(DistributedQueue.java:236)
[junit4:junit4] 2> at
org.apache.solr.cloud.ZkController.publish(ZkController.java:734)
[junit4:junit4] 2> at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:168)
[junit4:junit4] 2> at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:151)
[junit4:junit4] 2> at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:96)
[junit4:junit4] 2> at
org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:57)
[junit4:junit4] 2> at
org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:125)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
[junit4:junit4] 2> at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
[junit4:junit4] 2>
[junit4:junit4] 2> 119790 T364 oascc.ZkStateReader$3.process Updating live
nodes
[junit4:junit4] 2> 119790 T364 oascc.ZkStateReader$3.process WARNING
ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 119791 T364 oascc.ConnectionManager.process Watcher
org.apache.solr.common.cloud.ConnectionManager@3ff9f663
name:ZooKeeperConnection Watcher:127.0.0.1:30687/solr got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
[junit4:junit4] 2> 119791 T364 oaz.ClientCnxn$EventThread.run EventThread
shut down
[junit4:junit4] 2> NOTE: test params are: codec=Appending,
sim=DefaultSimilarity, locale=en_ZA, timezone=America/Rankin_Inlet
[junit4:junit4] 2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc.
1.6.0_32 (64-bit)/cpus=16,threads=1,free=46168640,total=170000384
[junit4:junit4] 2> NOTE: All tests run in this JVM:
[TestCollationKeyRangeQueries, TestSynonymMap, TestIBSimilarityFactory,
FieldMutatingUpdateProcessorTest, TestReplicationHandler, TestIndexSearcher,
CloudStateUpdateTest, SoftAutoCommitTest, FullSolrCloudTest]
[junit4:junit4] 2>
[junit4:junit4] Completed on J1 in 119.95s, 1 test, 1 failure <<< FAILURES!
[...truncated 778 lines...]
[...truncated 12465 lines...]
[...truncated 12465 lines...]
[...truncated 12465 lines...]
[...truncated 12465 lines...]
[...truncated 12450 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-4.x/checkout/build.xml:29:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-4.x/checkout/solr/build.xml:144:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-4.x/checkout/lucene/module-build.xml:62:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-4.x/checkout/lucene/common-build.xml:1094:
The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-tests-only-4.x/checkout/lucene/common-build.xml:789:
There were test failures: 305 suites, 1085 tests, 1 failure, 5 ignored
Total time: 28 minutes 12 seconds
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
Publishing Clover coverage report...
No Clover report will be published due to a Build Failure
Email was triggered for: Failure
Sending email for trigger: Failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
