init-js opened a new issue, #12287:
URL: https://github.com/apache/arrow-rs/issues/12287

   ### Describe the bug
   
   Reading a `BYTE_ARRAY` `DECIMAL` column through the Arrow reader panics if a 
value is stored in more bytes than the Arrow decimal type holds, even when the 
value itself fits. For example, a `DECIMAL(38, 0)` holding `9` written in 19 
bytes panics with:
   
   ```
   thread '...' panicked at parquet/src/arrow/buffer/bit_util.rs:43:5:
   Array too large, expected less than 16
   ```
   
   The Parquet spec allows this: for `BYTE_ARRAY`, "the minimum number of bytes 
to store the unscaled value should be used" 
([LogicalTypes.md](https://github.com/apache/parquet-format/blob/ec90dd4661ff1c4d77f06b5b5232c5aedf68ffc5/LogicalTypes.md?plain=1#L241-L242)),
 not must, so leading `0x00` or `0xFF` bytes that only repeat the sign are 
valid.
   
   `StatisticsConverter` has the same problem: `row_group_mins`, 
`row_group_maxes`, `data_page_mins` and `data_page_maxes` panic on such a 
column for `Decimal32`, `Decimal64`, `Decimal128` and `Decimal256` 
([statistics.rs](https://github.com/apache/arrow-rs/blob/4213ecfee2871df8cf46e44f6a50c4b5ebeaf677/parquet/src/arrow/arrow_reader/statistics.rs#L41-L66)).
 They also panic on an empty (0-byte) min or max.
   
   
   
   ### To Reproduce
   
   ```rust
   use bytes::Bytes;
   use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
   use parquet::data_type::{ByteArray, ByteArrayType};
   use parquet::file::writer::SerializedFileWriter;
   use parquet::schema::parser::parse_message_type;
   use std::sync::Arc;
   
   let schema = parse_message_type("message m { REQUIRED BYTE_ARRAY d 
(DECIMAL(38, 0)); }").unwrap();
   let mut buffer = Vec::new();
   let mut writer = SerializedFileWriter::new(&mut buffer, Arc::new(schema), 
Default::default()).unwrap();
   let mut row_group = writer.next_row_group().unwrap();
   let mut column = row_group.next_column().unwrap().unwrap();
   // 9, sign-extended to 19 bytes
   let mut nine = vec![0u8; 18];
   nine.push(9);
   column
       .typed::<ByteArrayType>()
       .write_batch(&[ByteArray::from(nine)], None, None)
       .unwrap();
   column.close().unwrap();
   row_group.close().unwrap();
   writer.close().unwrap();
   
   let reader = ParquetRecordBatchReaderBuilder::try_new(Bytes::from(buffer))
       .unwrap()
       .build()
       .unwrap();
   for batch in reader {
       println!("{:?}", batch.unwrap().column(0)); // panics
   }
   ```
   
   ### Expected behavior
   
   - A value whose extra leading bytes only repeat its sign reads back 
unchanged (`9` above).
   - A value that does not fit the Arrow decimal type returns an error instead 
of panicking.
   - In `StatisticsConverter`, a min or max that does not fit the Arrow type, 
or is empty, is null (unknown) instead of panicking, as already happens for an 
`INT64` statistic that does not fit a `Decimal32`.
   
   ### Additional context
   
   I have a fix and will open a PR.
   
   ### Willingness to contribute
   
   I plan to contribute a fix for this bug independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to