This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new e30b544d13 Return errors for undersized struct validity buffers in IPC
reader (#11658)
e30b544d13 is described below
commit e30b544d13f78d5644c3d587a9d0f98a5dc6b311
Author: Aleksandar Zecevic <[email protected]>
AuthorDate: Sun Oct 18 01:46:37 2026 +0000
Return errors for undersized struct validity buffers in IPC reader (#11658)
# Which issue does this PR close?
- Relates to #7124 and #11044.
# Rationale for this change
#11044 made validated IPC reads return an error, instead of panicking,
when a validity buffer is too short. Its check is in
`ArrayDataBuilder::build`. The IPC reader does not build struct arrays
with `ArrayDataBuilder`: when a struct node has `null_count > 0`,
`create_struct_array` calls `BooleanBuffer::new(null_buffer, 0, len)`
directly. A struct validity buffer shorter than `ceil(len / 8)` bytes
therefore still panics with "buffer not large enough", even with
validation enabled.
# What changes are included in this PR?
- `create_struct_array` checks the validity buffer length before it
builds the `BooleanBuffer`, and returns the same error as #11044
(`null_bit_buffer size too small. got {} needed {}`).
- The check runs only when validation is enabled; the
`with_skip_validation` path is unchanged. `arrow-ipc` has no
`force_validate` feature, so unlike #11044 there is no condition on it.
- A regression test next to the #11044 one: it writes a `StructArray`
with a short validity buffer (built with `StructArray::new_unchecked`)
with `StreamWriter` and reads it back with `StreamReader`.
# Are these changes tested?
Yes. `test_stream_reader_rejects_short_struct_validity_buffer` panics
with "buffer not large enough" without the fix and passes with it.
# Are there any significant user-facing changes?
No.
---
arrow-ipc/src/reader.rs | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/arrow-ipc/src/reader.rs b/arrow-ipc/src/reader.rs
index f891b6dd3c..35ca545ac3 100644
--- a/arrow-ipc/src/reader.rs
+++ b/arrow-ipc/src/reader.rs
@@ -369,6 +369,17 @@ impl RecordBatchDecoder<'_> {
let skip_validation = self.skip_validation.get();
let nulls = if null_count > 0 {
+ if !skip_validation {
+ // Check before constructing the BooleanBuffer, which would
otherwise panic.
+ let needed_len = len.div_ceil(8);
+ if null_buffer.len() < needed_len {
+ return Err(ArrowError::InvalidArgumentError(format!(
+ "null_bit_buffer size too small. got {} needed {}",
+ null_buffer.len(),
+ needed_len
+ )));
+ }
+ }
let validity_buffer = BooleanBuffer::new(null_buffer, 0, len);
let null_buffer = if skip_validation {
// safety: flag can only be set via unsafe code
@@ -3618,6 +3629,30 @@ mod tests {
);
}
+ #[test]
+ fn test_stream_reader_rejects_short_struct_validity_buffer() {
+ // Serialize a StructArray with too few validity bits.
+ let fields = Fields::from(vec![Field::new("b", DataType::Int32,
false)]);
+ let child: ArrayRef = Arc::new(Int32Array::from_iter_values(0..8000));
+ let nulls = NullBuffer::from(&[true, false, true, false]);
+ let array: ArrayRef =
+ unsafe { Arc::new(StructArray::new_unchecked(fields, vec![child],
Some(nulls))) };
+ let batch = RecordBatch::try_from_iter([("a", array)]).unwrap();
+
+ let mut stream = Vec::new();
+ let mut writer =
+ crate::writer::StreamWriter::try_new(&mut stream,
&batch.schema()).unwrap();
+ writer.write(&batch).unwrap();
+ writer.finish().unwrap();
+
+ let mut reader = StreamReader::try_new(Cursor::new(stream),
None).unwrap();
+ let err = reader.next().unwrap().unwrap_err();
+ assert_eq!(
+ err.to_string(),
+ "Invalid argument error: null_bit_buffer size too small. got 1
needed 1000"
+ );
+ }
+
#[test]
fn test_invalid_struct_array_ipc_read_errors() {
let a_field = Field::new("a", DataType::Int32, false);
