pingzh opened a new issue, #11869: URL: https://github.com/apache/arrow-rs/issues/11869
### Describe the bug `BatchCoalescer` can combine individually valid batches into an array whose cumulative offsets exceed the array type's capacity. Its row target does not bound string/binary payload bytes or nested list/map child counts. A small reproduction uses two valid one-row `List<Null>` batches, each with `2^30` children. There are only **two top-level rows**, far below a target of 8,192, but concatenating their children requires the final i32 offset `2^31`. `finish_buffered_batch()` panics rather than emitting the rows in separate representable batches. `NullArray` stores a length without allocating a payload per child, so this reproduction crosses the real offset boundary without allocating gigabytes. Here the offset counts **child elements**, not bytes. ### To Reproduce Reproduced on Apache `main` at [`df4dc3ede59b`](https://github.com/apache/arrow-rs/commit/df4dc3ede59bd43b5f71847231212381d068bb72) (`60.0.0` development version), using a standalone program with path dependencies on that checkout. The equivalent pinned Git dependencies are: `Cargo.toml`: ```toml [package] name = "coalescer-offset-repro" version = "0.1.0" edition = "2024" [dependencies] arrow-array = { git = "https://github.com/apache/arrow-rs.git";, rev = "df4dc3ede59bd43b5f71847231212381d068bb72" } arrow-buffer = { git = "https://github.com/apache/arrow-rs.git";, rev = "df4dc3ede59bd43b5f71847231212381d068bb72" } arrow-schema = { git = "https://github.com/apache/arrow-rs.git";, rev = "df4dc3ede59bd43b5f71847231212381d068bb72" } arrow-select = { git = "https://github.com/apache/arrow-rs.git";, rev = "df4dc3ede59bd43b5f71847231212381d068bb72" } ``` `src/main.rs`: ```rust use std::sync::Arc; use arrow_array::{ArrayRef, ListArray, NullArray, RecordBatch}; use arrow_buffer::OffsetBuffer; use arrow_schema::{DataType, Field}; use arrow_select::coalesce::BatchCoalescer; fn main() -> Result<(), Box<dyn std::error::Error>> { // NullArray stores a length, without allocating one value per child. let child_count = 1usize << 30; let list = ListArray::new( Arc::new(Field::new("item", DataType::Null, true)), OffsetBuffer::new(vec![0i32, child_count as i32].into()), Arc::new(NullArray::new(child_count)), None, ); let batch = RecordBatch::try_from_iter(vec![("items", Arc::new(list) as ArrayRef)])?; let mut coalescer = BatchCoalescer::new(batch.schema(), 8192); coalescer.push_batch(batch.clone())?; coalescer.push_batch(batch)?; // Only two top-level rows, but 2^31 child elements when concatenated. coalescer.finish_buffered_batch()?; Ok(()) } ``` Run `cargo run`. The failure occurs in `finish_buffered_batch()`: ```text arrow-buffer/src/buffer/offset.rs:139:62: offset overflow: 2147483648 does not fit in i32 ``` ### Expected behavior The coalescer should preserve all input rows and their schema while producing smaller batches when another append would exceed an offset limit. For this example, two one-row batches are a valid result. Check cumulative offsets before modifying any column builder, flush the buffered rows, then continue with the incoming rows. The checks need to account for relevant nested children and sliced inputs, and keep all columns aligned. This does not require splitting a single logical value: each input row in the reproduction is already representable. The current API documents an exact row target. Supporting earlier flushes for offset capacity would require clarifying that contract or exposing an appropriate batching policy. An overflow error is preferable to a panic, but the desired batching behavior is to continue processing representable rows. ### Additional context The current path is [`BatchCoalescer::finish_buffered_batch`](https://github.com/apache/arrow-rs/blob/df4dc3ede59bd43b5f71847231212381d068bb72/arrow-select/src/coalesce.rs#L635) → [`GenericInProgressArray::finish`](https://github.com/apache/arrow-rs/blob/df4dc3ede59bd43b5f71847231212381d068bb72/arrow-select/src/coalesce/generic.rs#L94) → [`concat_lists`](https://github.com/apache/arrow-rs/blob/df4dc3ede59bd43b5f71847231212381d068bb72/arrow-select/src/concat.rs#L189), which creates combined offsets using `OffsetBuffer::from_lengths`. Related issues: - #8247 addresses reporting overflow when string concatenation exceeds i32 offsets; this request is about preventing an oversized coalesced output by emitting multiple batches. - #8850 discusses batching criteria including row and byte limits; this is a concrete offset-capacity failure with a small reproduction. The report and standalone reproducer were prepared with AI assistance and verified locally against the upstream revision above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
