voonhous commented on PR #19687: URL: https://github.com/apache/hudi/pull/19687#issuecomment-5413560990
Head `fab55f1948ca`, three commits on `8389e2646ad3`: - **Spark 4.0**: `VariantType` is an `AtomicType` on 4.0 and batch reads are disabled for variant only on 4.1+, so the default COW/MOR base read was vectorized through Spark's own `ParquetReadSupport` and never reached the guard (Spark failed it with the column-less `WRONG_NUM_FIELDS`). `Spark40ParquetReader.read` now rejects off the footer before either reader; a `PushVariantIntoScan` rewrite struct is rejected on shape. - **Hive guards**: legs for the 3-level `list`/`element` layout, map values and a realtime split with the file group reader disabled; native log names skip the `.parquet` gate. - **Bootstrap**: the footer-schema read also shifted every data column by one on partitioned tables whose external files omit the partition column (legacy merge path); pinned by the partitioned leg, and a non-nullable evolved column now fails naming it. - **Sort columns**: the error names the unorderable member (`'s.m'`, `'arr[]'`). - **Tests**: legacy `BaseFileOnlyRelation` guard pinned in `TestLegacyParquetReadPath`; nested row-writer leg reads through both parquet readers; shared guard fixtures; duplicate legs dropped. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
