techgopal opened a new issue, #11691:
URL: https://github.com/apache/arrow-rs/issues/11691
### Describe the bug
Reading a nullable column whose definition levels contain a bit-packed run
that is longer than the remaining level data panics in `arrow-buffer` instead
of returning an error.
`PackedDecoder::next_rle_block`
(`parquet/src/arrow/record_reader/definition_levels.rs`) sets `packed_count`
from the run header, and `PackedDecoder::read` then calls
```rust
buffer.append_packed_range(offset..offset + to_read, self.data.as_ref());
```
without checking that `self.data` holds `ceil((offset + to_read) / 8)`
bytes. `BooleanBufferBuilder::append_packed_range` documents that it panics in
that case, and it does, from `BitChunks::new`:
```
assert!(ceil(end, 8) <= buffer.len(), "offset + len out of bounds");
```
This is different from #11516 / #11517 (overflow of the run length): here
the run length is small and valid, it just exceeds the bytes available in the
page.
### To Reproduce
With `parquet = "60.0.0"`, `arrow-array = "60.0.0"`, `arrow-schema =
"60.0.0"`, `bytes = "1"`:
```rust
use std::sync::Arc;
use arrow_array::{Int32Array, RecordBatch};
use arrow_schema::{DataType, Field, Schema};
use bytes::Bytes;
use parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder;
use parquet::arrow::ArrowWriter;
use parquet::file::properties::WriterProperties;
fn main() {
// A nullable INT32 column with every other value null: the data page
// (v1, uncompressed, plain) starts with the definition levels as
// [4-byte length = 127][bit-packed header 0x7F (63 groups = 504
levels)][63 bytes]...
let array: Int32Array = (0..1000).map(|i| (i % 2 ==
0).then_some(i)).collect();
let schema = Arc::new(Schema::new(vec![Field::new("c", DataType::Int32,
true)]));
let batch = RecordBatch::try_new(schema.clone(),
vec![Arc::new(array)]).unwrap();
let props =
WriterProperties::builder().set_dictionary_enabled(false).build();
let mut buf = Vec::new();
let mut writer = ArrowWriter::try_new(&mut buf, schema,
Some(props)).unwrap();
writer.write(&batch).unwrap();
writer.close().unwrap();
// Shrink the definition-level length prefix from 127 to 3 bytes; the
// first bit-packed run header still claims 504 levels.
let pattern = [0x7F, 0, 0, 0, 0x7F, 0x55];
let pos = buf.windows(pattern.len()).position(|w| w ==
pattern).expect("pattern");
buf[pos] = 3;
let reader = ParquetRecordBatchReaderBuilder::try_new(Bytes::from(buf))
.unwrap()
.build()
.unwrap();
for batch in reader {
println!("{:?}", batch.map(|b| b.num_rows()));
}
}
```
Output (debug and release):
```
thread 'main' panicked at
arrow-buffer-60.0.0/src/util/bit_chunk_iterator.rs:235:9:
offset + len out of bounds
```
Backtrace (abridged):
```
arrow_buffer::util::bit_chunk_iterator::BitChunks::new
arrow_buffer::util::bit_util::byte_aligned_bitwise_bin_op_helper
arrow_buffer::util::bit_util::apply_bitwise_binary_op
arrow_buffer::builder::boolean::BooleanBufferBuilder::append_packed_range
parquet::arrow::record_reader::definition_levels::PackedDecoder::read
<parquet::arrow::record_reader::definition_levels::DefinitionLevelBufferDecoder
as DefinitionLevelDecoder>::read_def_levels
parquet::column::reader::GenericColumnReader<R,D,V>::read_records_with_reservation
...
```
The same panic is reachable through `ParquetPushDecoder::try_decode`.
### Expected behavior
The reader returns a `ParquetError` (e.g. unexpected end of definition level
data) instead of panicking.
### Additional context
- Reproduces with `parquet` / `arrow-buffer` 60.0.0 and on `main` at
33d338ddf (`definition_levels.rs:446`).
- Found while fuzzing a Parquet reader built on arrow-rs. Related: #11516,
#9742.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
