GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/17232
[SPARK-18112] [SQL] Support reading data from Hive 2.1 metastore [WIP] ### What changes were proposed in this pull request? This PR is to support reading data from Hive 2.1 metastore. Need to update shim class because of the Hive API changes caused by the following two Hive JIRAs: - [HIVE-12730 MetadataUpdater: provide a mechanism to edit the basic statistics of a table (or a partition)](https://issues.apache.org/jira/browse/HIVE-12730) - [Hive-13341 Stats state is not captured correctly: differentiate load table and create table](https://issues.apache.org/jira/browse/HIVE-13341) There two new fields have been added in Hive. - `EnvironmentContext environmentContext`. So far, this is always set to `null`. This was introduced for supporting DDL `alter table s update statistics set ('numRows'='NaN')`. Using this DDL, users can specify the statistics. So far, our Spark SQL does not need it, because we use different table properties to store our generated statistics values. However, when Spark SQL issues ALTER TABLE DDL statements, Hive metastore always automatically invalidate the Hive-generated statistics. In the follow-up PR, we can fix it by explicitly adding a property to `environmentContext`. ```JAVA putToProperties(StatsSetupConst.STATS_GENERATED, StatsSetupConst.USER) ``` - `boolean hasFollowingStatsTask`. We always set it to `false`. TODO: more investigation about this ### How was this patch tested? Added test cases to VersionsSuite.scala You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark Hive21 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17232.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17232 ---- commit af81cee9f54abc13d7d07a12e4b499e49cd0dbcb Author: Xiao Li <gatorsm...@gmail.com> Date: 2017-03-10T03:35:38Z fix ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org