[
https://issues.apache.org/jira/browse/SPARK-44058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732806#comment-17732806
]
Yuming Wang commented on SPARK-44058:
-------------------------------------
For Hive 0.13 and later, we use
https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala#L753-L768.
> Remove deprecated API usage in HiveShim.scala
> ---------------------------------------------
>
> Key: SPARK-44058
> URL: https://issues.apache.org/jira/browse/SPARK-44058
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 3.4.0
> Reporter: Aman Raj
> Priority: Major
>
> Spark's HiveShim.scala calls this particular method in Hive :
> createPartitionMethod.invoke(
> hive,
> table,
> spec,
> location,
> params, // partParams
> null, // inputFormat
> null, // outputFormat
> -1: JInteger, // numBuckets
> null, // cols
> null, // serializationLib
> null, // serdeParams
> null, // bucketCols
> null) // sortCols
> }
>
> We do not have any such implementation of createPartition in Hive. We only
> have this definition :
> public Partition createPartition(Table tbl, Map<String, String> partSpec)
> throws HiveException {
> try
> { org.apache.hadoop.hive.metastore.api.Partition part =
> Partition.createMetaPartitionObject(tbl, partSpec, null);
> AcidUtils.TableSnapshot tableSnapshot = AcidUtils.getTableSnapshot(conf,
> tbl); part.setWriteId(tableSnapshot != null ?
> tableSnapshot.getWriteId() : 0); return new Partition(tbl,
> getMSC().add_partition(part)); }
> catch (Exception e)
> { LOG.error(StringUtils.stringifyException(e)); throw new
> HiveException(e); }
> }
> *The 12 parameter implementation was removed in HIVE-5951*
>
> The issue is that this 12 parameter implementation of createPartition method
> was added in Hive-0.12 and then was removed in Hive-0.13. When Hive 0.12 was
> used in Spark, SPARK-15334 commit in Spark added this 12 parameters
> implementation. But after Hive migrated to newer APIs somehow this was not
> changed in Spark OSS and it looks to us like a Bug from the Spark end.
>
> We need to migrate to the newest implementation of Hive createPartition
> method otherwise this flow can break
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
