Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1747/ Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC
1 tests failed. REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure Error Message: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed Stack Trace: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed at __randomizedtesting.SeedInfo.seed([2261C2321782AA10:26694DC105274531]:0) at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314) at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221) at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155) at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:457) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:454) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398) at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136) at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31) at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131) ... 46 more Build Log: [...truncated 10811 lines...] [junit4] Suite: org.apache.solr.cloud.OverseerTest [junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-2261C2321782AA10-001/init-core-data-001 [junit4] 2> 145014 T199 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false) [junit4] 2> 145015 T199 oas.SolrTestCaseJ4.initCore ####initCore [junit4] 2> 145015 T199 oas.SolrTestCaseJ4.initCore ####initCore end [junit4] 2> 145035 T199 oas.SolrTestCaseJ4.setUp ###Starting testOverseerFailure [junit4] 2> 145036 T199 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 145037 T200 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 145038 T199 oasc.ZkTestServer.run start zk server on port:49257 [junit4] 2> 145077 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 145082 T206 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71d1ff7f name:ZooKeeperConnection Watcher:127.0.0.1:49257 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 145083 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 145090 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 145094 T208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@112995a5 name:ZooKeeperConnection Watcher:127.0.0.1:49257 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 145094 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 145095 T199 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 145106 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 145110 T210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b77552 name:ZooKeeperConnection Watcher:127.0.0.1:49257/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 145111 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 145111 T199 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 145119 T199 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 145125 T199 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 145130 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 145161 T212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6156bf2d name:ZooKeeperConnection Watcher:127.0.0.1:49257/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 145161 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 145162 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 145180 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 145187 T199 oascc.SolrZkClient.makePath makePath: /live_nodes/node1 [junit4] 2> 145203 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 145209 T216 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f822e2 name:ZooKeeperConnection Watcher:127.0.0.1:49257/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 145210 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 145212 T199 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 145217 T199 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 145226 T199 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49257_solr [junit4] 2> 145226 T199 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 145232 T199 oasc.Overseer.start Overseer (id=92209745550311428-127.0.0.1:49257_solr-n_0000000000) starting [junit4] 2> 145236 T199 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 145244 T199 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 145252 T199 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 145260 T199 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 145268 T199 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 145275 T199 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 145289 T218 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 145292 T217 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 146301 T219 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 146307 T217 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 146309 T217 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"recovering", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 146309 T217 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1] [junit4] 2> 146309 T217 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 146316 T219 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 146318 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 146319 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 146801 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 146829 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 146846 T219 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 146847 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 146852 T217 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 146856 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 146856 T219 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 146868 T217 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 146869 T217 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 146877 T219 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 146983 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 146983 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 147303 T218 oasc.OverseerCollectionProcessor.run WARN Overseer cannot talk to ZK [junit4] 2> 148049 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 148071 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 148074 T221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@24f8d1e2 name:ZooKeeperConnection Watcher:127.0.0.1:49257/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 148074 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 148076 T217 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49257_solr [junit4] 2> 148078 T222 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 148085 T199 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49257_solr [junit4] 2> 148085 T199 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 148093 T199 oasc.Overseer.start Overseer (id=92209745550311429-127.0.0.1:49257_solr-n_0000000001) starting [junit4] 2> 148126 T224 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 148131 T223 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 148140 T223 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 148143 T223 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"recovering", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"core_node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 148143 T223 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 148151 T225 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 148152 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 148152 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 148211 T225 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 148212 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 148217 T223 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 148254 T213 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 148256 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 148257 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 148266 T223 oasc.SolrException.log ERROR Problem cleaning up collection in zk:collection1:org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /collections/collection1/leader_elect/shard1/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:125) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:184) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:181) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:181) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:665) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.clean(SolrZkClient.java:660) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.removeCore(Overseer.java:1094) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.processMessage(Overseer.java:359) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:242) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 148285 T225 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 148289 T223 oasc.Overseer$ClusterStateUpdater.setShardLeader ERROR Could not mark shard leader for non existing collection:collection1 [junit4] 2> 148297 T225 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 148409 T213 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 148411 T214 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 149278 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 149279 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 150292 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 150293 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 151301 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 151301 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 152315 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 152315 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 153329 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 153329 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 154341 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 154342 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 155362 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 155362 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 156371 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 156373 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 157383 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 157383 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 158396 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 158396 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 159408 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 159408 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 160424 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 160424 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 161433 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 161435 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 162445 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 162446 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 163459 T226 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 163460 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 163467 T226 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed [junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144) [junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163) [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125) [junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55) [junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1 [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:457) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:454) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398) [junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136) [junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31) [junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131) [junit4] 2> ... 10 more [junit4] 2> [junit4] 2> 163480 T199 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49257 49257 [junit4] 2> 163500 T200 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49257 49257 [junit4] 2> 163740 T199 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure [junit4] 2> 163741 T223 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49257_solr [junit4] 2> 163742 T227 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=2261C2321782AA10 -Dtests.slow=true -Dtests.locale=es_CR -Dtests.timezone=Europe/Rome -Dtests.file.encoding=UTF-8 [junit4] ERROR 18.8s | OverseerTest.testOverseerFailure <<< [junit4] > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed [junit4] > at __randomizedtesting.SeedInfo.seed([2261C2321782AA10:26694DC105274531]:0) [junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:144) [junit4] > at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163) [junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125) [junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314) [junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221) [junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155) [junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1 [junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:119) [junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) [junit4] > at org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:457) [junit4] > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:454) [junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:411) [junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:398) [junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:136) [junit4] > at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31) [junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:131) [junit4] > ... 46 more [junit4] 2> 163842 T199 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange [junit4] 2> 163843 T199 oasc.ZkTestServer.run STARTING ZK TEST SERVER [junit4] 1> client port:0.0.0.0/0.0.0.0:0 [junit4] 2> 163844 T228 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server [junit4] 2> 163944 T199 oasc.ZkTestServer.run start zk server on port:49264 [junit4] 2> 163946 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 163951 T234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77b1b790 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 163951 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 163954 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 163957 T236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f730eb1 name:ZooKeeperConnection Watcher:127.0.0.1:49264 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 163957 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 163965 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 163968 T238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39951022 name:ZooKeeperConnection Watcher:127.0.0.1:49264 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 163969 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 163969 T199 oascc.SolrZkClient.makePath makePath: /solr [junit4] 2> 163979 T199 oascc.SolrZkClient.makePath makePath: /live_nodes [junit4] 2> 163987 T239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 163990 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 163992 T241 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e891f79 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 163995 T239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 163995 T243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6c7179a0 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 163997 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 163998 T239 oascc.SolrZkClient.makePath makePath: /overseer_elect [junit4] 2> 163999 T199 oascc.SolrZkClient.makePath makePath: /clusterstate.json [junit4] 2> 164003 T239 oascc.SolrZkClient.makePath makePath: /overseer_elect/election [junit4] 2> 164006 T199 oascc.SolrZkClient.makePath makePath: /aliases.json [junit4] 2> 164011 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 164014 T239 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49264_solr [junit4] 2> 164015 T239 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 164021 T199 oascc.SolrZkClient.makePath makePath: /live_nodes/node1 [junit4] 2> 164024 T239 oasc.Overseer.start Overseer (id=92209746782846979-127.0.0.1:49264_solr-n_0000000000) starting [junit4] 2> 164030 T239 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 164033 T199 oascc.SolrZkClient.makePath makePath: /overseer/queue [junit4] 2> 164044 T239 oascc.SolrZkClient.makePath makePath: /overseer/queue-work [junit4] 2> 164053 T239 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure [junit4] 2> 164062 T239 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running [junit4] 2> 164093 T239 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed [junit4] 2> 164104 T239 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work [junit4] 2> 164120 T239 oasc.OverseerTest$OverseerRestarter.run Killing overseer. [junit4] 2> 164121 T246 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 164125 T239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 164129 T248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e1c8a36 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 164129 T239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 164129 T245 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=92209746782846979-127.0.0.1:49264_solr-n_0000000000) am no longer a leader. [junit4] 2> 164130 T245 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 164130 T245 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49264_solr [junit4] 2> 164130 T246 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92209746782846979-127.0.0.1:49264_solr-n_0000000000) am no longer a leader. [junit4] 2> 164131 T246 oasc.SolrException.log ERROR :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:99) [junit4] 2> at org.apache.solr.cloud.DistributedQueue.getTailId(DistributedQueue.java:483) [junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:241) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 164131 T246 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:251) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:248) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:248) [junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:412) [junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:256) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 164133 T249 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 164142 T239 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49264_solr [junit4] 2> 164142 T239 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 164149 T239 oasc.Overseer.start Overseer (id=92209746782846981-127.0.0.1:49264_solr-n_0000000001) starting [junit4] 2> 164181 T251 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 164186 T250 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 164193 T250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 164195 T250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 164195 T250 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1] [junit4] 2> 164195 T250 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 164203 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164204 T244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 164542 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 164568 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 164583 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164589 T250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 164599 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164599 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 164602 T250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state2", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 164602 T250 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 164612 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164619 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 164623 T254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2121bf06 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 164623 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 164627 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 164636 T199 oascc.SolrZkClient.makePath makePath: /live_nodes/node2 [junit4] 2> 164649 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164649 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 164652 T250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 164652 T250 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 164660 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164669 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 164769 T255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 164769 T244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 164875 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 164884 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164885 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164886 T257 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 164892 T250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 164894 T250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state2", [junit4] 2> "node_name":"node2", [junit4] 2> "core":"core4", [junit4] 2> "core_node_name":"node2", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node2/solr/"} [junit4] 2> 164895 T250 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1 [junit4] 2> 164896 T250 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1 [junit4] 2> 164904 T257 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165010 T244 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 165010 T255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 165386 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 165411 T244 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94) [junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55) [junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 165412 T243 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3df51218 rejected from java.util.concurrent.ThreadPoolExecutor@205cd1d[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 6] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) [junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:201) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) [junit4] 2> [junit4] 2> 165417 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 165418 T255 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 165433 T257 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165433 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165433 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165438 T250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 165447 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165554 T255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 165621 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 165625 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 165629 T260 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d18daf6 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 165630 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 165635 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 165658 T199 oascc.SolrZkClient.makePath makePath: /live_nodes/node1 [junit4] 2> 165684 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 165686 T252 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165693 T250 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 165695 T256 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165696 T257 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165697 T250 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 165698 T250 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 165704 T239 oasc.OverseerTest$OverseerRestarter.run Killing overseer. [junit4] 2> 165710 T239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 165716 T263 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f66492a name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 165717 T239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 165719 T250 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49264_solr [junit4] 2> 165724 T264 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 165732 T239 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49264_solr [junit4] 2> 165732 T239 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 165734 T261 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 165737 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 165742 T239 oasc.Overseer.start Overseer (id=92209746782846984-127.0.0.1:49264_solr-n_0000000002) starting [junit4] 2> 165765 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 165769 T266 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b71dc32 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 165770 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 165777 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 165787 T199 oascc.SolrZkClient.makePath makePath: /live_nodes/node2 [junit4] 2> 165793 T268 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 165815 T267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 165821 T267 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue. [junit4] 2> 165824 T267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 165824 T267 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 165828 T261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 165828 T269 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 165831 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 165865 T267 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 165874 T267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 165877 T267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 165878 T267 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 165887 T270 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165892 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 165894 T267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state2", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 165895 T267 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 165901 T270 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165911 T270 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 165914 T267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 165914 T267 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 165926 T270 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 166033 T269 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 166034 T261 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 166100 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 166107 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 166109 T271 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 166108 T270 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 166114 T272 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 166121 T267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 166123 T267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state2", [junit4] 2> "node_name":"node2", [junit4] 2> "core":"core4", [junit4] 2> "core_node_name":"node2", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node2/solr/"} [junit4] 2> 166123 T267 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 166134 T272 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 166146 T273 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94) [junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55) [junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 166149 T269 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 166164 T272 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 166176 T272 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 166284 T269 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 167646 T261 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94) [junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55) [junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 167649 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 167652 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 167657 T275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ce4ad3d name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 167658 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 167662 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 167672 T199 oascc.SolrZkClient.makePath makePath: /live_nodes/node1 [junit4] 2> 167687 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 167687 T271 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 167687 T272 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 167688 T270 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 167695 T267 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 167697 T267 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 167698 T267 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 167703 T270 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 167715 T276 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 167716 T270 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 167717 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 167718 T239 oasc.OverseerTest$OverseerRestarter.run Killing overseer. [junit4] 2> 167721 T267 oasc.Overseer$ClusterStateUpdater.run WARN Solr cannot talk to ZK, exiting Overseer main queue loop org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue/qn-0000000013 [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:604) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:261) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 167723 T267 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49264_solr [junit4] 2> 167724 T239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 167726 T279 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 167732 T278 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58109b76 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 167732 T239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 167741 T239 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49264_solr [junit4] 2> 167742 T239 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 167746 T239 oasc.Overseer.start Overseer (id=92209746782846987-127.0.0.1:49264_solr-n_0000000003) starting [junit4] 2> 167747 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 167750 T281 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d5fcd57 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 167751 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 167753 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 167760 T199 oascc.SolrZkClient.makePath makePath: /live_nodes/node2 [junit4] 2> 167780 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 167782 T284 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 167790 T282 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 167793 T282 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue. [junit4] 2> 167796 T282 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 167796 T282 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 167799 T283 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 167800 T276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 167806 T282 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue [junit4] 2> 167807 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 167813 T282 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 167815 T282 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state2", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 167816 T282 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 167823 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 167832 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 167835 T282 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 167835 T282 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 167842 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 167949 T283 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 167950 T276 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2) [junit4] 2> 168012 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 168020 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 168022 T286 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 168022 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 168023 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 168029 T282 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 168031 T282 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state2", [junit4] 2> "node_name":"node2", [junit4] 2> "core":"core4", [junit4] 2> "core_node_name":"node2", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node2/solr/"} [junit4] 2> 168031 T282 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 168038 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 168046 T287 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94) [junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55) [junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 168045 T275 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3e7199e2 rejected from java.util.concurrent.ThreadPoolExecutor@142de475[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 6] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) [junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:201) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) [junit4] 2> [junit4] 2> 168050 T283 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 168063 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 168074 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 168181 T283 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 169543 T276 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> 169546 T199 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 169549 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 169553 T289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51ef869d name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 169553 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 169558 T199 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... [junit4] 2> 169568 T199 oascc.SolrZkClient.makePath makePath: /live_nodes/node1 [junit4] 2> 169583 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 169583 T286 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 169583 T285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 169585 T291 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 169594 T282 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 169597 T282 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"state1", [junit4] 2> "node_name":"node1", [junit4] 2> "core":"core1", [junit4] 2> "core_node_name":"node1", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"1", [junit4] 2> "base_url":"http://node1/solr/"} [junit4] 2> 169598 T282 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered [junit4] 2> 169608 T239 oasc.OverseerTest$OverseerRestarter.run Killing overseer. [junit4] 2> 169620 T229 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1479846b1d2000b, likely client has closed socket [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 169620 T239 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 169628 T293 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@df4fdf3 name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 169634 T239 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] 2> 169635 T282 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49264_solr [junit4] 2> 169640 T295 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 169648 T239 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49264_solr [junit4] 2> 169649 T239 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader [junit4] 2> 169657 T239 oasc.Overseer.start Overseer (id=92209746782846990-127.0.0.1:49264_solr-n_0000000004) starting [junit4] 2> 169664 T294 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1 [junit4] 2> 169667 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election [junit4] 2> 169705 T298 oasc.OverseerCollectionProcessor.run Process current queue of collection creations [junit4] 2> 169707 T199 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper [junit4] 2> 169715 T299 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27a52e8f name:ZooKeeperConnection Watcher:127.0.0.1:49264/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None [junit4] 2> 169717 T199 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper [junit4] [...truncated too long message...] tate:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 277958 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election [junit4] 2> 277992 T712 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 277992 T714 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 277992 T713 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 277997 T706 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... [junit4] 2> 278000 T706 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={ [junit4] 2> "operation":"state", [junit4] 2> "state":"active", [junit4] 2> "node_name":"127.0.0.1", [junit4] 2> "core":"core6", [junit4] 2> "core_node_name":"node6", [junit4] 2> "collection":"collection1", [junit4] 2> "numShards":"3", [junit4] 2> "base_url":"http://127.0.0.1/solr/"} [junit4] 2> 278001 T706 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3 [junit4] 2> 278001 T706 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3 [junit4] 2> 278016 T713 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged [junit4] 2> 278128 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 278129 T710 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1) [junit4] 2> 278494 T199 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election [junit4] 2> 278536 T709 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2143f66e rejected from java.util.concurrent.ThreadPoolExecutor@74e7c0d1[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 7] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) [junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:201) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) [junit4] 2> [junit4] 2> 278536 T715 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:285) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:282) [junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94) [junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55) [junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3$1.run(SolrZkClient.java:204) [junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [junit4] 2> at java.lang.Thread.run(Thread.java:745) [junit4] 2> [junit4] 2> 278537 T709 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5df28026 rejected from java.util.concurrent.ThreadPoolExecutor@74e7c0d1[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 7] [junit4] 2> at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) [junit4] 2> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) [junit4] 2> at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:201) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522) [junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498) [junit4] 2> [junit4] 2> 280035 T711 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK [junit4] 2> 280042 T199 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49371 49371 [junit4] 2> 280140 T693 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:49371 49371 [junit4] 2> 280661 T199 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment [junit4] 2> 280662 T706 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:49371_solr [junit4] 2> 280667 T716 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) [junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) [junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:302) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) [junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:299) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85) [junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289) [junit4] 2> [junit4] 2> 283673 T199 oas.SolrTestCaseJ4.deleteCore ###deleteCore [junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-2261C2321782AA10-001 [junit4] 2> NOTE: test params are: codec=Lucene46, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=es_CR, timezone=Europe/Rome [junit4] 2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=2,threads=1,free=41175632,total=103051264 [junit4] 2> NOTE: All tests run in this JVM: [ConvertedLegacyTest, DistribCursorPagingTest, OverseerTest] [junit4] Completed in 138.77s, 9 tests, 1 error <<< FAILURES! [...truncated 1322 lines...] BUILD FAILED /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1298: The following error occurred while executing this line: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:922: There were test failures: 422 suites, 1756 tests, 1 error, 44 ignored (17 assumptions) Total time: 167 minutes 41 seconds Build step 'Invoke Ant' marked build as failure [description-setter] Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC Archiving artifacts Recording test results Email was triggered for: Failure - Any Sending email for trigger: Failure - Any
--------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org