yihua opened a new pull request, #20387: URL: https://github.com/apache/hudi/pull/20387
### Describe the issue this Pull Request addresses closes #20386 The Avro-based parquet reader returns wrong values for decimal columns stored as INT32 or INT64, the encoding Spark, Iceberg, Trino, DuckDB and parquet-rs use for precision up to 18. parquet-avro only keeps the DECIMAL annotation on BINARY and FIXED_LEN_BYTE_ARRAY, so the column surfaces as a plain `long`, and Hudi then promotes the unscaled integer to the table's decimal as if it were the value: 17.00 reads as 1700.00 with no error. The Hive reader in hudi-hadoop-mr shares the problem since it derives its row schema the same way. ### Summary and Changelog Integer-backed decimals now read correctly through `HoodieAvroParquetReader` and the Hive reader, and a shape the reader cannot handle fails instead of returning wrong numbers. - `AvroSchemaConverterWithTimestampNTZ` maps an INT32/INT64 DECIMAL column to a fixed Avro decimal of the minimal width for its precision, the type Spark derives for the same column; `HoodieSchema.minBytesForDecimalPrecision` is the shared width helper. - `SchemaRepair.alignDecimalPhysicalTypes` re-types decimal leaves of the requested projection to the file's INT32/INT64 physical type so the projection matches the column chunks. - `HoodieAvroReadSupport` applies that alignment in `init`, supplies the Hudi-converted schema as the read schema when neither the caller nor the footer provides one, and wraps the root converter of the parent's materializer in `IntegerDecimalRecordConverter`, whose `IntegerDecimalConverter` turns plain and dictionary-encoded int/long values into the fixed or bytes decimal of the read schema, including inside nested records. Such columns inside arrays or maps throw `UnsupportedOperationException` rather than read wrong. The wrapper keeps the parent's converters for every other field, and the parent materializer is obtained from `AvroReadSupport` itself (through a super-call method handle on parquet 1.14+, whose record reader dispatches through `ParquetConfiguration`), so no parquet-avro constructor is named: hudi-hadoop-common compiles against parquet 1.12.3, 1.13.1, 1.15.2 and 1.17.0. - The hudi-hadoop-mr `HoodieAvroParquetReader` reads through `HoodieAvroReadSupport` instead of plain `AvroReadSupport`. - A table schema that recorded such a column as plain int or long, which is what schema inference produced before this change, keeps reading the unscaled integer it read before: `HoodieAvroUtils` and `HoodieArrayWritableSchemaUtils` rewrite a decimal to a plain int/long target that way. Tests: `TestHoodieAvroParquetReader` (both modules) read INT32/INT64 decimal files written with parquet-mr's example writer against fixed, bytes, file-derived and reordered reader schemas, with and without dictionary encoding, plus nulls, a nested record, a wider and a narrower scale, and the array rejection. Without the fix they fail with `Cannot encode decimal with precision 11 as max precision 9`, a `LongWritable` where a `HiveDecimalWritable` is expected, and `cannot support rewrite value for schema type ... decimal ... since the old schema type is: "int"`. The two `TestAvroSchemaConverter` cases that pinned the lossy int/long conversion now expect the decimal. ### Impact Reader-only. Base files with FIXED_LEN_BYTE_ARRAY decimals, which is everything Hudi writes today, take the same path as before. Two visible changes for files with integer-backed decimals: schema inference from such a file now yields a decimal instead of int/long, and a projection containing such a column is read with parquet-avro's regular record converter rather than its compat converter (strings arrive as `Utf8` instead of `String` on paths that default to compat, such as Hive). `ParquetUtils.readAvroRecords` still uses the stock parquet-avro reader and is left for a follow-up. ### Risk Level low. The new converter only engages when the requested projection contains an INT32/INT64 decimal column. The Hive reader additionally picks up `HoodieAvroReadSupport`'s legacy list and map handling; the hudi-hadoop-common and hudi-hadoop-mr suites were run against the change. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
