Rich-T-kid opened a new issue, #11115: URL: https://github.com/apache/arrow-rs/issues/11115
**Is your feature request related to a problem or challenge?** Part of https://github.com/apache/arrow-rs/issues/7761 \`ListView<T>\` and \`LargeListView<T>\` go through \`GenericInProgressArray\` → \`concat_list_view\`. The ListView format stores per-row (offset, size) pairs rather than cumulative offsets, making the values buffer layout similar to BinaryView — offsets are absolute, so source value buffers can potentially be Arc-shared without copying. **Describe the solution you'd like** Analogous to \`InProgressByteViewArray\` for StringView/BinaryView, an \`InProgressListViewArray\` could: 1. **Dense buffer reuse**: If the source ListView's values buffer is dense (actual used bytes ≈ total buffer size), Arc-share the values buffer and only adjust the offsets 2. **Sparse detection**: If the buffer is sparse (many unused value-buffer gaps from prior filtering), fall back to copying values into a fresh buffer 3. **Incremental offset/size tracking**: accumulate (adjusted_offset, size) pairs as batches are pushed **\`set_source\` logic:** - Determine if source values buffer is dense: \`used_bytes / total_bytes > threshold\` - If dense: store a reference to the values buffer + record the global adjustment needed for offsets - If sparse: flag for value copy **\`copy_rows(offset, len)\` logic:** - Append outer nulls - For each row i in [offset, offset+len: - Read `src_offset = source.offsets()[i]` and `src_size = source.sizes()[i]` - If dense path: store `(src_offset + global_adjustment, src_size)` in the view metadata - If sparse path: copy values to a fresh buffer and store the new offset - Advance the global offset counter **Why this matches the ByteView opportunity:** - ListView is to List what BinaryView is to Binary — the view format allows direct buffer references - `concat_list_view` currently recalculates all offsets by adding cumulative `global_offset` per input array; the dense path avoids this for buffer-aligned data - Expected benefit: similar to `InProgressByteViewArray` for BinaryView (15-40% depending on sparsity) **Implementation reference:** See `arrow-select/src/coalesce/byte_view.rs` for the dense/sparse detection pattern and buffer GC logic. **Benchmarks** ``` cargo bench --bench coalesce_kernels --features test_utils -- list_view ``` EOF ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
