[ https://issues.apache.org/jira/browse/SPARK-23710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yuming Wang updated SPARK-23710: -------------------------------- Fix Version/s: 3.0.0 > Upgrade the built-in Hive to 2.3.5 for hadoop-3.2 > ------------------------------------------------- > > Key: SPARK-23710 > URL: https://issues.apache.org/jira/browse/SPARK-23710 > Project: Spark > Issue Type: Umbrella > Components: SQL > Affects Versions: 2.4.0 > Reporter: Yuming Wang > Assignee: Yuming Wang > Priority: Critical > Fix For: 3.0.0 > > > Spark fail to run on Hadoop 3.x, because Hive's ShimLoader considers Hadoop > 3.x to be an unknown Hadoop version. see SPARK-18673 and HIVE-16081 for more > details. So we need to upgrade the built-in Hive for Hadoop-3.x. This is an > umbrella JIRA to track this upgrade. > > *Upgrade Plan*: > # SPARK-27054 Remove the Calcite dependency. This can avoid some jar > conflicts. > # SPARK-23749 Replace built-in Hive API (isSub/toKryo) and remove > OrcProto.Type usage > # SPARK-27158, SPARK-27130 Update dev/* to support dynamic change profiles > when testing > # Fix ORC dependency conflict to makes it test passed on Hive 1.2.1 and > compile passed on Hive 2.3.4 > # Add an empty hive-thriftserverV2 module. then we could test all test cases > in next step > # Make Hadoop-3.1 with Hive 2.3.4 test passed > # Adapted hive-thriftserverV2 from hive-thriftserver with Hive 2.3.4's > [TCLIService.thrift|https://github.com/apache/hive/blob/rel/release-2.3.4/service-rpc/if/TCLIService.thrift] > > I have completed the [initial > work|https://github.com/apache/spark/pull/24044] and plan to finish this > upgrade step by step. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org