Reza Safi created HIVE-23240:
--------------------------------
Summary: loadDynamicPartition complains about static partitions
even when they are provided in the description
Key: HIVE-23240
URL: https://issues.apache.org/jira/browse/HIVE-23240
Project: Hive
Issue Type: Bug
Reporter: Reza Safi
Hive is computing valid dynamic partitions here:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2853
However it later uses the specification provided by client here:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2879
(partSpec is exactly what client has provided and partSpec.keySet() contains
both static and dynamic partitions key)
As a result the makeSpecFromName here will expect both static and dynamic
partitions in requiredKeys:
https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java#L580
However since the curPath that is passed to the above method is just like
"somePath/dynamicPart=value" which miss the static partitions and a result the
method will ignore static partition keys then complains in log a warning that
the static partition keys are missing. Returning false to Hive.java, a log
warning that "dynamicPart=value" is an invalid partition will be issued,
despite the fact that the dynamic partition has been validated before:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L2880
This will cause a silent data corruption in some clients. As an example spark
will suffer from this when working with hive metastore in master branch.
It seems that if the goal was just to warn the client, there is no need to
ignore the valid dynamic partition.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
