Thanks Fokko, I will definitely take a look at this.
Cheers Andrew From: "Driesprong, Fokko" <fo...@driesprong.frl> Date: Friday, August 24, 2018 at 2:39 AM To: "reubensaw...@hotmail.com" <reubensaw...@hotmail.com> Cc: "dev@spark.apache.org" <dev@spark.apache.org> Subject: Re: Spark data quality bug when reading parquet files from hive metastore Hi Andrew, This blog gives an idea how to schema is resolved: https://blog.godatadriven.com/multiformat-spark-partition There is some optimisation going on when reading Parquet using Spark. Hope this helps. Cheers, Fokko Op wo 22 aug. 2018 om 23:59 schreef t4 <reubensaw...@hotmail.com<mailto:reubensaw...@hotmail.com>>: https://issues.apache.org/jira/browse/SPARK-23576 ? -- Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org>
