Re: unstable cluster

2016-04-11 Thread Ted Yu
>From region server log: 2016-04-11 03:11:51,589 WARN org.apache.zookeeper.ClientCnxnSocket: Connected to an old server; r-o mode will be unavailable 2016-04-11 03:11:51,589 INFO org.apache.zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x52ee1452fec5ac has expired,

unstable cluster

2016-04-11 Thread Ted Tuttle
Hello - We've started experiencing regular failures of our HBase cluster. For the last week we've had nightly failures about 1hr after a heavy batch process starts. In the logs below we see the failure starting at 2016-04-11 03:11 in zookeeper, master and region server logs: zookeeper: