Hi Charles & Team, Just want to quickly follow up on this. Is there any config tweak that I missed that would help in this situation?
Thanks, Ravi On Tue, Jul 14, 2020 at 1:47 PM ravi kanth <[email protected]> wrote: > Thanks again for the prompt response Charles. Following is the cluster > stack: > > 1. Drill - v1.17 > 2. Hadoop - v3.1.2 > 3. Hive - v3.1.2 > > Please let me know if you need any further information. > > Regards, > Ravi > > > On Tue, Jul 14, 2020 at 1:44 PM Charles Givre <[email protected]> wrote: > >> Hi Ravi, >> Gotcha! I just saw the error and immediately thought it was a host >> resolution error. Let me look into that. Can you share what version(s) of >> Drill, Hive and Hadoop you are using? >> THanks, >> -- C >> >> >> >> > On Jul 14, 2020, at 4:42 PM, ravi kanth <[email protected]> wrote: >> > >> > Hi Charles, >> > >> > Thanks for responding. >> > >> > *hadoop-qa* is the nameservice our HA enabled Hadoop cluster uses and >> it's >> > not technically a server to have an IP rather a logical name. So, to >> your >> > question, unfortunately, I cannot change that to an IP. Although, if I >> try >> > to change that, Hive mappings in the metastore database for Hadoop paths >> > are all mapped with Hadoop Nameservice and not IP. >> > >> > From Hadoop docs, >> > >> > - >> > >> > *dfs.nameservices* - the logical name for this new nameservice >> > >> > Choose a logical name for this nameservice, for example “mycluster”, >> and >> > use this logical name for the value of this config option. The name >> you >> > choose is arbitrary. It will be used both for configuration and as the >> > authority component of absolute HDFS paths in the cluster. >> > >> > *Note:* If you are also using HDFS Federation, this configuration >> > setting should also include the list of other nameservices, HA or >> > otherwise, as a comma-separated list. >> > >> > <property> >> > <name>dfs.nameservices</name> >> > <value>mycluster</value> >> > </property> >> > >> > >> > >> > Is this something Drill support or tested? >> > >> > Thanks, >> > Ravi >> > >> > >> > On Tue, Jul 14, 2020 at 1:24 PM Charles Givre <[email protected]> wrote: >> > >> >> Could you try with the IP address of hadoop-qa and see if that works? >> >> -- C >> >> >> >>> On Jul 14, 2020, at 4:22 PM, ravi kanth <[email protected]> wrote: >> >>> >> >>> Hi Team, >> >>> >> >>> I have a Drill cluster with 4 nodes. I wanted to make use of the Drill >> >>> cluster to query a Hive Metastore Service which is built on top of HA >> >>> enabled HDFS cluster. >> >>> >> >>> As the Hadoop cluster is HA enabled the paths for the Hive tables >> >>> partitions are identified by a nameservice instead of a URL. >> >>> >> >>> In our case, HDFS Nameservice is hadoop-qa and so, are the HDFS >> partition >> >>> paths in Hive like "hdfs://hadoop-qa/dwh/data/tables/base_table/" >> >>> >> >>> Following is the storage-plugins-override.conf: >> >>> >> >>> "storage":{ >> >>> hive: { >> >>> type: "hive", >> >>> configProps: { >> >>> "hive.metastore.uris": "thrift://<host_name>:9083", >> >>> "hive.metastore.warehouse.dir": >> >>> "hdfs://hadoop-qa:8020/user/hive/warehouse", >> >>> "fs.default.name": "hdfs://hadoop-qa:8020", >> >>> "hive.metastore.sasl.enabled": "false", >> >>> }, >> >>> enabled: true >> >>> } >> >>> } >> >>> >> >>> When I try to query Hive using Drill, it fails to understand the >> >>> nameservice *hadoop-qa* and throws the following exception: >> >>> >> >>> >> >>> Error: SYSTEM ERROR: UnknownHostException: hadoop-qa >> >>> >> >>> Can someone suggest if the configurations are wrong? I followed the >> ones >> >> in >> >>> the documentation. Any help will be greatly appreciated. >> >>> >> >>> *We are trying to evaluate Drill for our Production facing analytical >> use >> >>> cases and this is a blocker for us in order to proceed with Drill.* >> >>> >> >>> Thanks, >> >>> Ravi >> >> >> >> >> >>
