I am running a zookeeper ensemble of 3 zookeeper instances and established a solrCloud to work with it (2 masters , 2 slaves) on one of the masters I keep noticing ZooKeeper related exceptions which I can't understand: And the other is java.lang.IllegalArgumentException: Path cannot be null (PathUtils.java:45)
Here are my logs (I set the log level to FINE on zookeeper package) Anyone can identify the issue? (I could not yet get any help from the solrCloud community) FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: -8,101 replyHeader:: -8,-1,0 request:: 30064776552,v{'/collections},v{},v{'/collections/ENPwl/shards/ENPWL1,'/collections/ENPwl/shards/ENPWL4,'/collections/ENPwl/shards/ENPWL2,'/collections,'/collections/ENPwl/shards/ENPWL3,'/collections/ENPwlMaster/shards/ENPWLMaster_3,'/collections/ENPwlMaster/shards/ENPWLMaster_4,'/live_nodes,'/collections/ENPwlMaster/shards/ENPWLMaster_1,'/collections/ENPwlMaster/shards/ENPWLMaster_2} response:: null Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: 540,8 replyHeader:: 540,-1,0 request:: '/collections,F response:: v{'ENPwl,'ENPwlMaster} Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error SEVERE: Error while calling watcher java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) at org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315) at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process INFO: Updating live nodes:org.apache.solr.common.cloud.solrzkcli...@55308275 Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Updating live nodes from ZooKeeper... Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: 541,8 replyHeader:: 541,-1,0 request:: '/live_nodes,F response:: v{'ob1078.nydc1.outbrain.com:8983_solr2,'ob1078.nydc1.outbrain.com:8983_solr1,'ob1061.nydc1.outbrain.com:8983_solr2,'ob1062.nydc1.outbrain.com:8983_solr1,'ob1062.nydc1.outbrain.com:8983_solr2,'ob1061.nydc1.outbrain.com:8983_solr1,'ob1077.nydc1.outbrain.com:8983_solr2,'ob1077.nydc1.outbrain.com:8983_solr1} Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error SEVERE: Error while calling watcher java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) at org.apache.solr.cloud.ZkController$2.process(ZkController.java:321) at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager process INFO: Watcher org.apache.solr.common.cloud.connectionmana...@339bb448 name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event WatchedEvent: Server state change. New state: Disconnected path:null type:None Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_1 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_3 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_2 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_4 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$5 process INFO: Detected a new or removed collection Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: 542,8 replyHeader:: 542,-1,0 request:: '/collections,F response:: v{'ENPwl,'ENPwlMaster} Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error SEVERE: Error while calling watcher java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) at org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315) at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process INFO: Updating live nodes:org.apache.solr.common.cloud.solrzkcli...@55308275 Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Updating live nodes from ZooKeeper... Aug 25, 2010 5:18:19 AM org.apache.log4j.Category debug FINE: Reading reply sessionid:0x12a97312613010b, packet:: clientPath:null serverPath:null finished:false header:: 543,8 replyHeader:: 543,-1,0 request:: '/live_nodes,F response:: v{'ob1078.nydc1.outbrain.com:8983_solr2,'ob1078.nydc1.outbrain.com:8983_solr1,'ob1061.nydc1.outbrain.com:8983_solr2,'ob1062.nydc1.outbrain.com:8983_solr1,'ob1062.nydc1.outbrain.com:8983_solr2,'ob1061.nydc1.outbrain.com:8983_solr1,'ob1077.nydc1.outbrain.com:8983_solr2,'ob1077.nydc1.outbrain.com:8983_solr1} Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error SEVERE: Error while calling watcher java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) at org.apache.solr.cloud.ZkController$2.process(ZkController.java:321) at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager process INFO: Watcher org.apache.solr.common.cloud.connectionmana...@339bb448 name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event WatchedEvent: Server state change. New state: SyncConnected path:null type:None Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_1 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_3 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info INFO: Attempting connection to server zook2/192.168.252.62:2181 Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info INFO: Priming connection to java.nio.channels.SocketChannel[connected local=/192.168.252.77:54892 remote=zook2/192.168.252.62:2181] Aug 25, 2010 5:18:19 AM org.apache.log4j.Category info INFO: Server connection successful Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error SEVERE: Error while calling watcher java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) at org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315) at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_2 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process INFO: Updating live nodes:org.apache.solr.common.cloud.solrzkcli...@29d03e78 Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Updating live nodes from ZooKeeper... Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error SEVERE: Error while calling watcher java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) at org.apache.solr.cloud.ZkController$2.process(ZkController.java:321) at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_4 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_1 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL3 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL2 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ConnectionManager process INFO: Watcher org.apache.solr.common.cloud.connectionmana...@40bb2bc3 name:ZooKeeperConnection Watcher:zook1:2181,zook2:2181,zook3:2181 got event WatchedEvent: Server state change. New state: SyncConnected path:null type:None Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_3 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$5 process INFO: Detected a new or removed collection Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error SEVERE: Error while calling watcher java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) at org.apache.solr.common.cloud.ZkStateReader$5.process(ZkStateReader.java:315) at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL1 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_2 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process INFO: Updating live nodes:org.apache.solr.common.cloud.solrzkcli...@29d03e78 Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Updating live nodes from ZooKeeper... Aug 25, 2010 5:18:19 AM org.apache.log4j.Category error SEVERE: Error while calling watcher java.lang.IllegalArgumentException: Path cannot be null at org.apache.zookeeper.common.PathUtils.validatePath(PathUtils.java:45) at org.apache.zookeeper.ZooKeeper.getChildren(zookeeper:ZooKeeper.java):1196) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:200) at org.apache.solr.cloud.ZkController$2.process(ZkController.java:321) at org.apache.zookeeper.ClientCnxn$EventThread.run(zookeeper:ClientCnxn.java):425) Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_4 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWL4 in collection:ENPwl Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$4 process INFO: Detected a shard change under ShardId:ENPWLMaster_1 in collection:ENPwlMaster Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader$6 process INFO: Notified of CloudState change Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Cloud state update for ZooKeeper already scheduled Aug 25, 2010 5:18:19 AM org.apache.solr.cloud.ZkController$2 process INFO: Updating live nodes:org.apache.solr.common.cloud.solrzkcli...@29d03e78 Aug 25, 2010 5:18:19 AM org.apache.solr.common.cloud.ZkStateReader updateCloudState INFO: Updating live nodes from ZooKeeper... Aug 25, 2010 5:18:22 AM org.apache.log4j.Category debug FINE: Got ping response for sessionid:0x12a97312613010b after 0ms