Github user maropu commented on the issue: https://github.com/apache/spark/pull/16030 This is not a bug in `VectorizedParquetReader ` as @liancheng said, and the root cause is that wrongly inferred types are passed into the reader in `VectorizedParquetReader#initBatch` https://github.com/apache/spark/blob/master/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java#L176 yea, I know this functionality is helpful for skilful users, and, on the other hand, newbies easily could break query results via this interface as @brkyvz said. Therefore, in case of that we allow data schema to contain (part of) the partition columns, IMO it'd be better to alert that risk to users via `logWarning` or something. BTW this is the original fix of this bug (Sorry, but I wrongly overrode and remove this commit ): https://github.com/apache/spark/compare/master...maropu:SPARK-18108-2. The fix of this commit is to fill in correct data types when data schema contains (part of) the partition columns.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org