BELUGA BEHR created HIVE-20844: ---------------------------------- Summary: Cache Instances of CacheManager in DummyTxnManager Key: HIVE-20844 URL: https://issues.apache.org/jira/browse/HIVE-20844 Project: Hive Issue Type: Improvement Components: HiveServer2, Locking Affects Versions: 3.1.0, 2.3.2, 4.0.0 Reporter: BELUGA BEHR
I noticed that the {{DummyTxnManager}} class instantiates quite a few instances of {{ZooKeeperHiveLockManager}}. The ZooKeeper LM creates a connection to ZK for each instance created. It also does some initialization steps that are almost always just noise and pressure on ZooKeeper because it has already been initialized and the steps are therefore NOOPs. {{ZooKeeperHiveLockManager}} should be a singleton class with one long-lived connection to the ZooKeeper service. Perhaps the {{HiveLockManager}} interface could have a {{isSingleton()}} method which indicates that the LM should only be instantiated once and cached for subsequent sessions. {code:java} 2018-05-14 22:45:30,574 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252389]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager 2018-05-14 22:51:27,865 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252671]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager 2018-05-14 22:51:37,552 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252686]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager 2018-05-14 22:51:49,046 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252736]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager 2018-05-14 22:51:50,664 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1252742]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager 2018-05-14 23:00:54,314 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1253479]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager 2018-05-14 23:17:26,867 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1254180]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager 2018-05-14 23:24:25,426 INFO org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager: [HiveServer2-Background-Pool: Thread-1255493]: Creating lock manager of type org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager {code} {code:java|title=DummyTxnManager.java} @Override public HiveLockManager getLockManager() throws LockException { if (lockMgr == null) { boolean supportConcurrency = conf.getBoolVar(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY); if (supportConcurrency) { String lockMgrName = conf.getVar(HiveConf.ConfVars.HIVE_LOCK_MANAGER); if ((lockMgrName == null) || (lockMgrName.isEmpty())) { throw new LockException(ErrorMsg.LOCKMGR_NOT_SPECIFIED.getMsg()); } try { // CACHE LM HERE LOG.info("Creating lock manager of type " + lockMgrName); lockMgr = (HiveLockManager)ReflectionUtils.newInstance( conf.getClassByName(lockMgrName), conf); lockManagerCtx = new HiveLockManagerCtx(conf); lockMgr.setContext(lockManagerCtx); } catch (Exception e) { ... {code} [https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockManager.java] {code:java|title=ZooKeeperHiveLockManager Initialization} try { curatorFramework = CuratorFrameworkSingleton.getInstance(conf); parent = conf.getVar(HiveConf.ConfVars.HIVE_ZOOKEEPER_NAMESPACE); try{ curatorFramework.create().withMode(CreateMode.PERSISTENT).forPath("/" + parent, new byte[0]); } catch (Exception e) { // ignore if the parent already exists if (!(e instanceof KeeperException) || ((KeeperException)e).code() != KeeperException.Code.NODEEXISTS) { LOG.warn("Unexpected ZK exception when creating parent node /" + parent, e); } } {code} https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java#L96-L106 -- This message was sent by Atlassian JIRA (v7.6.3#76005)