thesuperzapper opened a new pull request #638: [HUDI-12] Upgrade to Spark 2.4, 
Avro 1.8.2, Parquet 1.10.0... (WIP)
URL: https://github.com/apache/incubator-hudi/pull/638
 
 
   This is still a work in progress, and comments/suggestions are welcome. 
   
   **Implications of this PR:**
   * Spark 2.4+ will become a dependency for Hudi.
     * This is because spark-avro was only introduced in Spark 2.4.
     * This is necessary because spark-avro in Spark 2.4 has different schema 
conversion internals, (supporting Avro 1.8.2, and [logical 
types](https://avro.apache.org/docs/1.8.2/spec.html#Logical+Types))
     * This will allow us to move to the [Datasource V2 
API](https://databricks.com/session/apache-spark-data-source-v2), added in 
Spark 2.3.
   
   **This PR makes the following changes:**
   * Upgrades to Spark 2.4
   * Upgrades to Avro 1.8.2
   * Upgrades to Parquet 1.10.0
   * Upgrades the demo docker images
   * Cleans up the dependency poms
   * Replaces `com.databricks:spark-avro` with `org.apache.spark:spark-avro`
   * Replaces `com.twitter:parquet-avro` with `org.apache.parquet:parquet-avro`
   * Replaces `com.twitter:parquet-hadoop` with 
`org.apache.parquet:parquet-hadoop`
   * Makes `HoodieWrapperFileSystem.java` do what it says on the tin
   
   **Known Issues:**
   * There is something going on with non-trivial Hive-MR queries on 
merge-on-read tables. (Specifically ```select symbol, max(ts) from 
stock_ticks_mor group by symbol HAVING symbol = 'GOOG';``` from the demo)
     * _If you can reproduce this can you post the underlying stacktrace? As I 
dont have a good test environment for this._

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to