voonhous opened a new pull request, #20332: URL: https://github.com/apache/hudi/pull/20332
**TLDR:** brings Hudi's read of shredded variant base files to parity with Spark 4. Each `typed_value` leaf is now read from its Parquet type, as Spark 4's own reader does, so leaves that came back wrong are rebuilt correctly and leaves Spark rejects fail when the file is opened. Stacked on #20326 (write-side parity); only the last commit is new. ### Describe the issue this Pull Request addresses Read side of #19442. `HoodieAvroParquetReader` rebuilt shredded variants from the footer's Avro conversion, which drops DECIMAL on INT32 / INT64, the unsigned flag, JSON and NANOS. So for files written by other engines: - DECIMAL on INT32 / INT64 came back as the unscaled integer (1.50 as 150), with no error. - Plain FLBA failed with a `ClassCastException`, and TIMESTAMP(MILLIS) with an NPE. - TIME and TIMESTAMP(NANOS) came back as integers, where Spark 4 refuses the file. Map of every case: https://github.com/apache/hudi/issues/19442#issuecomment-6165989504 ### Summary and Changelog - `HoodieAvroParquetReader` passes the file's Parquet schema to `HoodieVariantReconstruction`, which prepares one rebuilder per shredded variant column when the file is opened (records, both LIST encodings, map values). - `VariantShreddingProvider.rebuildVariantRecord` becomes `prepareRebuild(columnPath, parquetGroup, unshreddedSchema)`. - `Spark4VariantShreddingProvider` maps each leaf from its Parquet type the way Spark's `ParquetToSparkSchemaConverter` and `SparkShreddingUtils.buildVariantSchema` do, and decodes values per leaf. Dead code from the old path is removed. - Tests in `TestSpark4VariantShreddingProvider`, `TestHoodieVariantReconstructionRoundTrip` (real files through the reader) and `TestHoodieVariantReconstruction`. ### Impact Shredded files written by Hudi read as before. Files from other engines now read like Spark 4. A file Spark 4 refuses (TIME, TIMESTAMP(NANOS), FLOAT16, UUID, INTERVAL, MAP `typed_value`) now fails at open with an error naming the column, including such files written by Hudi before #20326 from a hand-written shredding schema. ### Risk Level medium <details><summary>Per-encoding behaviour (matches Spark 4.1.1)</summary> | `typed_value` encoding | Read as | |---|---| | BOOLEAN, FLOAT, DOUBLE, INT32, INT64, INT(8/16/32/64 signed), DATE, TIMESTAMP(MICROS) | unchanged | | STRING, ENUM, JSON on BINARY | string (JSON was binary) | | BINARY, BSON, plain FLBA | binary (plain FLBA threw) | | DECIMAL on INT32 / INT64 / BINARY / FLBA | decimal (INT32 / INT64 were unscaled integers) | | TIMESTAMP(MILLIS), UTC / local | timestamp / timestamp_ntz, scaled to micros (threw) | | INT(32,false) | long, unsigned (was negative above 2^31) | | INT(64,false) | decimal(20,0) (was a signed long) | | TIME, TIMESTAMP(NANOS), FLOAT16, UUID on FLBA(16), INTERVAL | rejected at open | | INT96 | rejected; not supported by Hudi's Avro reader (Spark reads it as a timestamp) | </details> <details><summary>Known differences from Spark</summary> - Spark applies datetime rebase to very old DATE / TIMESTAMP values; Hudi passes raw values through (Spark's CORRECTED mode). - Spark throws `AnalysisException`; Hudi throws `HoodieException`. - `spark.sql.legacy.parquet.nanosAsLong` is not honoured: NANOS is always rejected. </details> ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
