[ https://issues.apache.org/jira/browse/HIVE-20794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Bapat updated HIVE-20794: ---------------------------------- Attachment: HIVE-20794.07 Status: Patch Available (was: In Progress) Fixed findbug notice in the last run and updated PR. > Use Zookeeper for metastore service discovery > --------------------------------------------- > > Key: HIVE-20794 > URL: https://issues.apache.org/jira/browse/HIVE-20794 > Project: Hive > Issue Type: Improvement > Reporter: Ashutosh Bapat > Assignee: Ashutosh Bapat > Priority: Major > Labels: pull-request-available > Attachments: HIVE-20794.01, HIVE-20794.02, HIVE-20794.03, > HIVE-20794.03, HIVE-20794.04, HIVE-20794.05, HIVE-20794.06, HIVE-20794.07 > > > Right now, multiple metastore services can be specified in > hive.metastore.uris configuration, but that list is static and can not be > modified dynamically. Use Zookeeper for dynamic service discovery of > metastore. > h3. Improve ZooKeeperHiveHelper class (suggestions for name welcome) > The Zookeeper related code (for service discovery) accesses Zookeeper > parameters directly from HiveConf. The class is changed so that it could be > used for both HiveServer2 and Metastore server and works with both the > configurations. Following methods from HiveServer2 are now moved into > ZooKeeperHiveHelper. # startZookeeperClient # addServerInstanceToZooKeeper # > removeServerInstanceFromZooKeeper > h3. HiveMetaStore conf changes > # THRIFT_URIS (hive.metastore.uris) can also be used to specify ZooKeeper > quorum. When THRIFT_SERVICE_DISCOVERY_MODE > (hive.metastore.service.discovery.mode) is set to "zookeeper" the URIs are > used as ZooKeeper quorum. When it's set to be empty, the URIs are used to > locate the metastore directly. > # Here's list of Hiveserver2's parameters and their proposed metastore conf > counterparts. It looks odd that the Metastore related configurations do not > have their macros start with METASTORE, but start with THRIFT. I have just > followed naming convention used for other parameters. > ** HIVE_SERVER2_ZOOKEEPER_NAMESPACE - THRIFT_ZOOKEEPER_NAMESPACE > (hive.metastore.zookeeper.namespace) > ** HIVE_ZOOKEEPER_CLIENT_PORT - THRIFT_ZOOKEEPER_CLIENT_PORT > (hive.metastore.zookeeper.client.port) > ** HIVE_ZOOKEEPER_CONNECTION_TIMEOUT - THRIFT_ZOOKEEPER_CONNECTION_TIMEOUT - > (hive.metastore.zookeeper.connection.timeout) > ** HIVE_ZOOKEEPER_CONNECTION_MAX_RETRIES - > THRIFT_ZOOKEEPER_CONNECTION_MAX_RETRIES > (hive.metastore.zookeeper.connection.max.retries) > ** HIVE_ZOOKEEPER_CONNECTION_BASESLEEPTIME - > THRIFT_ZOOKEEPER_CONNECTION_BASESLEEPTIME > (hive.metastore.zookeeper.connection.basesleeptime) > # Additional configuration THRIFT_BIND_HOST is used to specify the host > address to bind Metastore service to. Right now Metastore binds to *, i.e all > addresses. Metastore doesn't then know which of those addresses it should add > to the ZooKeeper. THRIFT_BIND_HOST solves that problem. When this > configuration is specified the metastore server binds to that address and > also adds it to the ZooKeeper if dynamic service discovery mode is ZooKeeper. > Following Hive ZK configurations seem to be related to managing locks and > seem irrelevant for MS ZK. > # HIVE_ZOOKEEPER_SESSION_TIMEOUT > # HIVE_ZOOKEEPER_CLEAN_EXTRA_NODES > Since there is no configuration to be published, > HIVE_ZOOKEEPER_PUBLISH_CONFIGS does not have a THRIFT counterpart. > h3. HiveMetaStore class changes > # startMetaStore should also register the instance with Zookeeper, when > configured. > # When shutting a metastore server down it should deregister itself from > Zookeeper, when configured. > # These changes use the refactored code described above. > h3. HiveMetaStoreClient class changes > When service discovery mode is zookeeper, we fetch the metatstore URIs from > the specified ZooKeeper and treat those as if they were specified in > THRIFT_URIS i.e. use the existing mechanisms to choose a metastore server to > connect to and establish a connection. -- This message was sent by Atlassian JIRA (v7.6.3#76005)