voonhous commented on code in PR #20136:
URL: https://github.com/apache/hudi/pull/20136#discussion_r4143164211
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieParquetReadSupport.scala:
##########
@@ -49,7 +49,11 @@ class HoodieParquetReadSupport(
} else {
readContext.getRequestedSchema
}
- val trimmedParquetSchema =
HoodieParquetReadSupport.trimParquetSchema(requestedParquetSchema,
context.getFileSchema)
+ // Same condition as Spark's own intersectParquetGroups: only the
row-based reader wants a
+ // requested schema restricted to what the file has; the vectorized reader
null-fills a missing
+ // column itself and matches columns by position.
+ val trimmedParquetSchema =
HoodieParquetReadSupport.trimParquetSchema(requestedParquetSchema,
+ context.getFileSchema, dropMissingTopLevelFields =
!enableVectorizedReader)
Review Comment:
Confirmed. With an update before the DDL, the parquet-log MOR leg fails on
the pushed arm with `INVALID_VARIANT_SHREDDING_SCHEMA` out of
`HoodieSparkParquetReader`. The top-level drop is now unconditional and the
flag is gone (cbdd8de95443). Both MOR legs write a log block before the DDL;
the avro-block leg read it correctly before the change as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
