init-js opened a new pull request, #12288:
URL: https://github.com/apache/arrow-rs/pull/12288

   # Which issue does this PR close?
   
   - Closes #12287
   
   # Rationale for this change
   
   A `BYTE_ARRAY` `DECIMAL` stores its unscaled value as a big-endian two's 
complement integer. The spec does not limit its length: it says "the minimum 
number of bytes to store the unscaled value should be used" 
([LogicalTypes.md#L241-L242](https://github.com/apache/parquet-format/blob/ec90dd4661ff1c4d77f06b5b5232c5aedf68ffc5/LogicalTypes.md?plain=1#L241-L242)),
 not must. A writer can therefore sign-extend a value with redundant leading 
`0x00` or `0xFF` bytes, and the file is still valid.
   
   Reading a `DECIMAL(38, 0)` that holds `9` in 19 bytes panics with:
   
   ```
   Array too large, expected less than 16
   ```
   
   The same happens for `Decimal256` past 32 bytes.
   
   `StatisticsConverter` has the same assumption. `from_bytes_to_i32`, 
`from_bytes_to_i64`, `from_bytes_to_i128` and `from_bytes_to_i256` 
([statistics.rs#L41-L66](https://github.com/apache/arrow-rs/blob/4213ecfee2871df8cf46e44f6a50c4b5ebeaf677/parquet/src/arrow/arrow_reader/statistics.rs#L41-L66))
 sign-extend the min and max bytes to the width of the Arrow type, so 
`row_group_mins`, `row_group_maxes`, `data_page_mins` and `data_page_maxes` 
panic on such a file for every Arrow decimal type.
   
   # What changes are included in this PR?
   
   - `bit_util::try_sign_extend_be` sign-extends like `sign_extend_be`, but 
also accepts input longer than `N` bytes when the extra leading bytes only 
repeat the sign. It returns `None` when `b` is empty or the value does not fit 
in `N` bytes.
   - `ByteArrayReader` converts `Decimal128` and `Decimal256` through one 
helper that uses `try_sign_extend_be`. Values that fit read back unchanged. A 
value that does not fit returns `BYTE_ARRAY decimal value of {n} bytes is out 
of range for {data_type}` instead of panicking.
   - `StatisticsConverter`'s `from_bytes_to_*` functions use 
`try_sign_extend_be` and return `Option`. A min or max that does not fit the 
Arrow type becomes null, as an `INT64` statistic that does not fit a 
`Decimal32` already does 
([statistics.rs#L296-L299](https://github.com/apache/arrow-rs/blob/4213ecfee2871df8cf46e44f6a50c4b5ebeaf677/parquet/src/arrow/arrow_reader/statistics.rs#L296-L299),
 
[#L1165](https://github.com/apache/arrow-rs/blob/4213ecfee2871df8cf46e44f6a50c4b5ebeaf677/parquet/src/arrow/arrow_reader/statistics.rs#L1165)).
   
   The path for values of normal width is unchanged apart from one length 
comparison.
   
   The `FIXED_LEN_BYTE_ARRAY` reader does not hit this panic: the column width 
selects an Arrow type that holds it, and a narrower schema hint is rejected 
with an error.
   
   # Are these changes tested?
   
   Yes. New tests in `arrow::array_reader::byte_array::tests` write 
`BYTE_ARRAY` decimal columns and read them back:
   
   - sign-extended values for `Decimal128` (9 and -9 in 19 bytes, `i128::MIN` 
in 17 bytes) and `Decimal256` (5 in 34 bytes, `i256::MIN` in 40 bytes), with 
nulls;
   - values that do not fit `Decimal128` or `Decimal256`, which now return an 
error.
   
   A new test in `tests/arrow_reader/statistics.rs` writes sign-extended 
`BYTE_ARRAY` decimals with page statistics, and checks the row group and data 
page mins and maxes for `Decimal32`, `Decimal64`, `Decimal128` and 
`Decimal256`, including a max that only fits the wider types, and an empty min 
and max, which become null.
   
   All three fail on `main` and pass with this change. A unit test for 
`try_sign_extend_be` covers the boundary cases, where the dropped bytes look 
like sign bytes but do not match the sign of the kept bytes.
   
   # Are there any significant user-facing changes?
   
   No API change.
   
   # AI usage disclosure
   
   The fix started from a patch I wrote and use internally. An AI assistant 
(Claude) adapted it to current `main`, extended it to `StatisticsConverter`, 
drafted the tests and this description, and ran the tests against `main` and 
this branch. I reviewed the changes and chose the behaviour for values that do 
not fit: a read error in the reader, and a null statistic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to