init-js opened a new issue, #12287: URL: https://github.com/apache/arrow-rs/issues/12287
### Describe the bug Reading a `BYTE_ARRAY` `DECIMAL` column through the Arrow reader panics if a value is stored in more bytes than the Arrow decimal type holds, even when the value itself fits. For example, a `DECIMAL(38, 0)` holding `9` written in 19 bytes panics with: ``` thread '...' panicked at parquet/src/arrow/buffer/bit_util.rs:43:5: Array too large, expected less than 16 ``` The Parquet spec allows this: for `BYTE_ARRAY`, "the minimum number of bytes to store the unscaled value should be used" ([LogicalTypes.md](https://github.com/apache/parquet-format/blob/ec90dd4661ff1c4d77f06b5b5232c5aedf68ffc5/LogicalTypes.md?plain=1#L241-L242)), not must, so leading `0x00` or `0xFF` bytes that only repeat the sign are valid. `StatisticsConverter` has the same problem: `row_group_mins`, `row_group_maxes`, `data_page_mins` and `data_page_maxes` panic on such a column for `Decimal32`, `Decimal64`, `Decimal128` and `Decimal256` ([statistics.rs](https://github.com/apache/arrow-rs/blob/4213ecfee2871df8cf46e44f6a50c4b5ebeaf677/parquet/src/arrow/arrow_reader/statistics.rs#L41-L66)). They also panic on an empty (0-byte) min or max. ### To Reproduce ```rust use bytes::Bytes; use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder; use parquet::data_type::{ByteArray, ByteArrayType}; use parquet::file::writer::SerializedFileWriter; use parquet::schema::parser::parse_message_type; use std::sync::Arc; let schema = parse_message_type("message m { REQUIRED BYTE_ARRAY d (DECIMAL(38, 0)); }").unwrap(); let mut buffer = Vec::new(); let mut writer = SerializedFileWriter::new(&mut buffer, Arc::new(schema), Default::default()).unwrap(); let mut row_group = writer.next_row_group().unwrap(); let mut column = row_group.next_column().unwrap().unwrap(); // 9, sign-extended to 19 bytes let mut nine = vec![0u8; 18]; nine.push(9); column .typed::<ByteArrayType>() .write_batch(&[ByteArray::from(nine)], None, None) .unwrap(); column.close().unwrap(); row_group.close().unwrap(); writer.close().unwrap(); let reader = ParquetRecordBatchReaderBuilder::try_new(Bytes::from(buffer)) .unwrap() .build() .unwrap(); for batch in reader { println!("{:?}", batch.unwrap().column(0)); // panics } ``` ### Expected behavior - A value whose extra leading bytes only repeat its sign reads back unchanged (`9` above). - A value that does not fit the Arrow decimal type returns an error instead of panicking. - In `StatisticsConverter`, a min or max that does not fit the Arrow type, or is empty, is null (unknown) instead of panicking, as already happens for an `INT64` statistic that does not fit a `Decimal32`. ### Additional context I have a fix and will open a PR. ### Willingness to contribute I plan to contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
