Dear Spark Developers and Users,


Hyukjin and I plan to upgrade the built-in Hive from 1.2.1-spark2
<https://github.com/JoshRosen/hive/tree/release-1.2.1-spark2> to 2.3.4
<https://github.com/apache/hive/releases/tag/rel%2Frelease-2.3.4> to solve
some critical issues, such as support Hadoop 3.x, solve some ORC and
Parquet issues. This is the list:

*Hive issues*:

[SPARK-26332 <https://issues.apache.org/jira/browse/SPARK-26332>][HIVE-10790]
Spark sql write orc table on viewFS throws exception

[SPARK-25193 <https://issues.apache.org/jira/browse/SPARK-25193>][HIVE-12505]
insert overwrite doesn't throw exception when drop old data fails

[SPARK-26437 <https://issues.apache.org/jira/browse/SPARK-26437>][HIVE-13083]
Decimal data becomes bigint to query, unable to query

[SPARK-25919 <https://issues.apache.org/jira/browse/SPARK-25919>][HIVE-11771]
Date value corrupts when tables are "ParquetHiveSerDe" formatted and target
table is Partitioned

[SPARK-12014 <https://issues.apache.org/jira/browse/SPARK-12014>][HIVE-11100]
Spark SQL query containing semicolon is broken in Beeline



*Spark issues*:

[SPARK-23534 <https://issues.apache.org/jira/browse/SPARK-23534>] Spark run
on Hadoop 3.0.0

[SPARK-20202 <https://issues.apache.org/jira/browse/SPARK-20202>] Remove
references to org.spark-project.hive

[SPARK-18673 <https://issues.apache.org/jira/browse/SPARK-18673>]
Dataframes doesn't work on Hadoop 3.x; Hive rejects Hadoop version

[SPARK-24766 <https://issues.apache.org/jira/browse/SPARK-24766>]
CreateHiveTableAsSelect and InsertIntoHiveDir won't generate decimal column
stats in parquet





Since the code for the *hive-thriftserver* module has changed too much for
this upgrade, I split it into two PRs for easy review.

The first PR <https://github.com/apache/spark/pull/23552> does not contain
the changes of hive-thriftserver. Please ignore the failed test in
hive-thriftserver.

The second PR <https://github.com/apache/spark/pull/23553> is complete
changes.



I have created a Spark distribution for Apache Hadoop 2.7, you might
download it via Google Drive
<https://drive.google.com/open?id=1cq2I8hUTs9F4JkFyvRfdOJ5BlxV0ujgt> or Baidu
Pan <https://pan.baidu.com/s/1b090Ctuyf1CDYS7c0puBqQ>.

Please help review and test. Thanks.

Reply via email to