init-js opened a new pull request, #12288: URL: https://github.com/apache/arrow-rs/pull/12288
# Which issue does this PR close? - Closes #12287 # Rationale for this change A `BYTE_ARRAY` `DECIMAL` stores its unscaled value as a big-endian two's complement integer. The spec does not limit its length: it says "the minimum number of bytes to store the unscaled value should be used" ([LogicalTypes.md#L241-L242](https://github.com/apache/parquet-format/blob/ec90dd4661ff1c4d77f06b5b5232c5aedf68ffc5/LogicalTypes.md?plain=1#L241-L242)), not must. A writer can therefore sign-extend a value with redundant leading `0x00` or `0xFF` bytes, and the file is still valid. Reading a `DECIMAL(38, 0)` that holds `9` in 19 bytes panics with: ``` Array too large, expected less than 16 ``` The same happens for `Decimal256` past 32 bytes. `StatisticsConverter` has the same assumption. `from_bytes_to_i32`, `from_bytes_to_i64`, `from_bytes_to_i128` and `from_bytes_to_i256` ([statistics.rs#L41-L66](https://github.com/apache/arrow-rs/blob/4213ecfee2871df8cf46e44f6a50c4b5ebeaf677/parquet/src/arrow/arrow_reader/statistics.rs#L41-L66)) sign-extend the min and max bytes to the width of the Arrow type, so `row_group_mins`, `row_group_maxes`, `data_page_mins` and `data_page_maxes` panic on such a file for every Arrow decimal type. # What changes are included in this PR? - `bit_util::try_sign_extend_be` sign-extends like `sign_extend_be`, but also accepts input longer than `N` bytes when the extra leading bytes only repeat the sign. It returns `None` when `b` is empty or the value does not fit in `N` bytes. - `ByteArrayReader` converts `Decimal128` and `Decimal256` through one helper that uses `try_sign_extend_be`. Values that fit read back unchanged. A value that does not fit returns `BYTE_ARRAY decimal value of {n} bytes is out of range for {data_type}` instead of panicking. - `StatisticsConverter`'s `from_bytes_to_*` functions use `try_sign_extend_be` and return `Option`. A min or max that does not fit the Arrow type becomes null, as an `INT64` statistic that does not fit a `Decimal32` already does ([statistics.rs#L296-L299](https://github.com/apache/arrow-rs/blob/4213ecfee2871df8cf46e44f6a50c4b5ebeaf677/parquet/src/arrow/arrow_reader/statistics.rs#L296-L299), [#L1165](https://github.com/apache/arrow-rs/blob/4213ecfee2871df8cf46e44f6a50c4b5ebeaf677/parquet/src/arrow/arrow_reader/statistics.rs#L1165)). The path for values of normal width is unchanged apart from one length comparison. The `FIXED_LEN_BYTE_ARRAY` reader does not hit this panic: the column width selects an Arrow type that holds it, and a narrower schema hint is rejected with an error. # Are these changes tested? Yes. New tests in `arrow::array_reader::byte_array::tests` write `BYTE_ARRAY` decimal columns and read them back: - sign-extended values for `Decimal128` (9 and -9 in 19 bytes, `i128::MIN` in 17 bytes) and `Decimal256` (5 in 34 bytes, `i256::MIN` in 40 bytes), with nulls; - values that do not fit `Decimal128` or `Decimal256`, which now return an error. A new test in `tests/arrow_reader/statistics.rs` writes sign-extended `BYTE_ARRAY` decimals with page statistics, and checks the row group and data page mins and maxes for `Decimal32`, `Decimal64`, `Decimal128` and `Decimal256`, including a max that only fits the wider types, and an empty min and max, which become null. All three fail on `main` and pass with this change. A unit test for `try_sign_extend_be` covers the boundary cases, where the dropped bytes look like sign bytes but do not match the sign of the kept bytes. # Are there any significant user-facing changes? No API change. # AI usage disclosure The fix started from a patch I wrote and use internally. An AI assistant (Claude) adapted it to current `main`, extended it to `StatisticsConverter`, drafted the tests and this description, and ran the tests against `main` and this branch. I reviewed the changes and chose the behaviour for values that do not fit: a read error in the reader, and a null statistic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
