Sergey Shelukhin created HIVE-12167: ---------------------------------------
Summary: HBase metastore causes massive number of ZK exceptions in MiniTez tests Key: HIVE-12167 URL: https://issues.apache.org/jira/browse/HIVE-12167 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Daniel Dai I ran some random test (vectorization_10) with HBase metastore, and I see large number of exceptions in hive.log {noformat} $ grep -c "ConnectionLoss" hive.log 52 $ grep -c "Connection refused" hive.log 1014 {noformat} These log lines' count has increased by ~33% since merging llap branch, but it is still high before that (39/~700) for the same test). These lines are not present if I disable HBase metastore. The exceptions are: {noformat} 2015-10-13T17:51:06,232 WARN [Thread-359-SendThread(localhost:2181)]: zookeeper.ClientCnxn (ClientCnxn.java:run(1102)) - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_45] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:1.8.0_45] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) [zookeeper-3.4.6.jar:3.4.6-1569965] {noformat} that is retried for some seconds and then {noformat} 2015-10-13T17:51:22,867 WARN [Thread-359]: zookeeper.ZKUtil (ZKUtil.java:checkExists(544)) - hconnection-0x1da6ef180x0, quorum=localhost:2181, baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid) org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222) ~[hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:541) [hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65) [hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105) [hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:879) [hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:635) [hbase-client-1.1.1.jar:1.1.1] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_45] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_45] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_45] at java.lang.reflect.Constructor.newInstance(Constructor.java:422) [?:1.8.0_45] at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) [hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:420) [hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hbase.client.ConnectionManager.createConnectionInternal(ConnectionManager.java:329) [hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:144) [hbase-client-1.1.1.jar:1.1.1] at org.apache.hadoop.hive.metastore.hbase.VanillaHBaseConnection.connect(VanillaHBaseConnection.java:56) [hive-metastore-2.0.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.metastore.hbase.HBaseReadWrite.<init>(HBaseReadWrite.java:227) [hive-metastore-2.0.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.metastore.hbase.HBaseReadWrite.<init>(HBaseReadWrite.java:83) [hive-metastore-2.0.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.metastore.hbase.HBaseReadWrite$1.initialValue(HBaseReadWrite.java:157) [hive-metastore-2.0.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.metastore.hbase.HBaseReadWrite$1.initialValue(HBaseReadWrite.java:151) [hive-metastore-2.0.0-SNAPSHOT.jar:?] at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180) [?:1.8.0_45] at java.lang.ThreadLocal.get(ThreadLocal.java:170) [?:1.8.0_45] at org.apache.hadoop.hive.metastore.hbase.HBaseReadWrite.getInstance(HBaseReadWrite.java:205) [hive-metastore-2.0.0-SNAPSHOT.jar:?] at org.apache.hadoop.hive.metastore.hbase.StatsCache$Invalidator.run(StatsCache.java:309) [hive-metastore-2.0.0-SNAPSHOT.jar:?] {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)