yihua opened a new pull request, #19512: URL: https://github.com/apache/hudi/pull/19512
### Describe the issue this Pull Request addresses `HoodieRowParquetWriteSupport`, the Spark row write support used by bulk insert, clustering, and file-group-reader-based MOR compaction, sizes decimal `FIXED_LEN_BYTE_ARRAY` columns from `Decimal.minBytesForPrecision()[precision]` and discards the declared Avro `fixed` size. A Spark `DecimalType` carries only precision and scale, so an Avro `fixed(N)` decimal whose declared size is wider than the precision-minimal width (for example a `fixed(10)` decimal(20,2), whose minimal width is 9) is written as `FIXED_LEN_BYTE_ARRAY(9)`. This diverges from the Avro write path (inserts), which preserves the declared 10. Mixing the two paths (insert then compaction, or upsert then clustering) leaves a table with mixed-width decimal columns under one logical schema. ### Summary and Changelog Add a helper `decimalFixedLen(HoodieSchema resolvedSchema, int precision)`: when the resolved schema is a `HoodieSchema.Decimal` backed by an Avro `fixed` type (`isFixed()`), use its `getFixedSize()`; otherwise fall back to `Decimal.minBytesForPrecision()[precision]`. It is applied at both decimal branches that already receive the resolved `HoodieSchema`: - the output Parquet type in the schema converter (`convertField`), so the column is declared `FIXED_LEN_BYTE_ARRAY(N)`, and - the value writer (`makeWriter`), so the written value bytes are padded to the same `N`. The padding buffer is now resolved once per column instead of once per record. The shared `decimalBuffer` is only sized for the precision-minimal maximum, so a declared `fixed(N)` wider than that gets its own per-column buffer, which also closes a latent overflow when honoring a wide fixed size. No code copied. ### Impact Behavior change on the Spark row write path: a decimal declared as an Avro `fixed(N)` is now written as `FIXED_LEN_BYTE_ARRAY(N)`, matching the table schema and the Avro write path, instead of the precision-minimal width. Existing narrower files still read correctly (Spark decodes decimals by precision). The scope is limited to decimals backed by an Avro `fixed` type; `bytes`-backed decimals and pure-Spark tables (no Avro fixed size) are unaffected. ### Risk Level low The change is confined to the decimal branch and only diverges from prior behavior when the resolved schema is a `fixed`-backed decimal. A new unit test asserts an Avro `fixed(10)` decimal(20,2) is declared `FIXED_LEN(10)` and a `bytes` decimal(20,2) stays `FIXED_LEN(9)`. A new functional MOR test builds a table whose Avro schema declares a `fixed(10)` decimal, runs compaction and row-writer clustering over two file groups, asserts the rewritten base files stay `FIXED_LEN_BYTE_ARRAY(10)`, and asserts the commit-metadata table schema is unchanged after both operations. Both fixed-width assertions fail (expected 10 but was 9) when the fix is reverted. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
