cshuo commented on issue #20438: URL: https://github.com/apache/hudi/issues/20438#issuecomment-6435778627
Thanks for the profile. In Hudi 1.1.1, the sampled `HoodieFileGroupReaderBasedFileFormat.projectIter` is called by the [base-file read path](https://github.com/apache/hudi/blob/release-1.1.1/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala#L399-L417) when some partition columns are read from the file and others are supplied from partition values. For comparison, the ordinary non-bootstrap COW snapshot path in Hudi 0.14.1 without schema-on-read generally reads the partition columns either all from the file or all from partition values. It does not have this mixed-partition `projectIter` stage, although Spark may still perform its own row conversion. The 1.1.1 Hudi projection may be needed to reorder the partition columns. However, if `from == to` and the reader returns a `ColumnarBatchRow`, Hudi still materializes it as an `UnsafeRow`, even though Spark may subsequently perform another conversion. Could you share: - The ordered partition fields, key generator, ordering/precombine field, and partition read options. - If possible, the `from` and `to` schemas passed to `projectIter`, including field order and types, and the input row class. These would help determine whether the Hudi projection reorders columns in your case or is an identity projection. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
