Attila Csaba Marosi created HIVE-21181:
------------------------------------------
Summary: Hive pre-upgrade tool not working with HDFS HA, tries
connecting to nameservice as it was a NameNode
Key: HIVE-21181
URL: https://issues.apache.org/jira/browse/HIVE-21181
Project: Hive
Issue Type: Bug
Components: Hive
Affects Versions: 1.2.1
Environment: Centos 7.4.1708
kernel 3.10.0-693.11.6.el7.x86_64
Ambari 2.6.2.2
HDP-2.6.5.0-292
Hive 1.2.1000
HDFS 2.7.3
Reporter: Attila Csaba Marosi
While preparing a production cluster HDP-2.6.5 -> HDP-3.1 upgrades, we've
noticed issues with the hive-pre-upgrade tool, when we tried running it, we got
the exception:
{{Found Acid table: default.hello_acid
2019-01-28 15:54:20,331 ERROR [main] acid.PreUpgradeTool
(PreUpgradeTool.java:main(152)) - PreUpgradeTool failed
java.lang.IllegalArgumentException: java.net.UnknownHostException: mytestcluster
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:439)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:321)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:696)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:636)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2796)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2830)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2812)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at
org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:417)
at
org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:384)
at
org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:374)
at
org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:235)
at
org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:149)
Caused by: java.net.UnknownHostException: mytestcluster
... 17 more}}
We tried running it on a kerberized test cluster built based on the same
blueprint like the production clusters, with HDP-2.6.5.0-292, Hive 1.2.1000,
HDFS 2.7.3, with HDFS HA and without Hive HA.
We enabled Hive ACID, created the same example ACID table as shown in
https://hortonworks.com/tutorial/using-hive-acid-transactions-to-insert-update-and-delete-data/
We followed the steps described at
https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/prepare_hive_for_upgrade.html
, kinit-ed, used the "-Djavax.security.auth.useSubjectCredsOnly=false"
parameter.
Without the ACID table there is no issue.
I'm attaching the hdfs-site.xml and core-site.xml.
Feel free to ping me directly on slack, if any additional detail is needed, we
can reproduce the issue on a lab cluster any time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
