voonhous commented on issue #20157: URL: https://github.com/apache/hudi/issues/20157#issuecomment-5912612582
History of this, and the cause is now confirmed. - #18334: Spark 4.0 reads the variant group by position as `[value, metadata]`, while Hudi writes `[metadata, value]`. - #17833 worked around it in `HoodieParquetReadSupport` by reordering every variant-shaped group of the requested schema, at any depth. - #18674 (in 1.2.0) moved the reorder into `Spark40HoodieParquetReadSupport` and gated it on the catalyst schema: only top-level fields typed `VariantType` are reordered, so that a user `struct<value: binary, metadata: binary>` is left alone. A top-level `struct<inner: variant>` is then skipped as a whole, which is this issue. Confirmed on master with Spark 4.0.2: with the reorder walking the catalyst schema through structs, the repro above reads correctly on COW and MOR, and the COW rewrite writes `metadata` and `value` into the right columns. Variants inside arrays and maps were not tried. Why it went unnoticed: the reorder unit tests cover a top-level variant only, the nested end-to-end tests are gated to Spark 4.1+, and master CI builds no Spark 4.0 profile. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
