cetra3 opened a new issue, #11280: URL: https://github.com/apache/arrow-rs/issues/11280
### Describe the bug When investigating some shortcomings in https://github.com/apache/datafusion/pull/23565 I found that the `interleave` method doesn't reuse existing buffers, and clones them. While this is fine for memory/perf, since cloning a buffer is a refcount rather than a full data clone, the problem arises in things that don't know about this accounting: - The issue as is in DF, writing out IPC can *inflate* the output drastically & `gc()` can make things *much worse* - `BatchCoalescer` gc check can inflate values as well ### To Reproduce ```rust use arrow::array::{Array, AsArray, StringViewArray}; use arrow::compute::interleave; let array = StringViewArray::from_iter_values( (0..4).map(|i| format!("a string long enough to not be inlined {i}")), ); assert_eq!(array.data_buffers().len(), 1); let a = array.slice(0, 2); let b = array.slice(2, 2); let result = interleave(&[&a, &b], &[(0, 0), (1, 0), (0, 1), (1, 1)]).unwrap(); // fails: the single shared buffer is emitted twice assert_eq!(result.as_string_view().data_buffers().len(), 1); ``` The effect grows with the number of inputs sharing buffers. Interleaving all 8192 rows of a `StringViewArray` (5 data buffers, 638,976 bytes) back from `k` equal slices: | k | output buffers | `get_buffer_memory_size` | IPC stream bytes | |---|----------------|--------------------------|------------------| | 1 | 5 | 638,976 | 517,640 | | 2 | 6 | 770,048 | 648,776 | | 4 | 8 | 1,097,728 | 853,704 | | 8 | 12 | 1,654,784 | 1,288,072 | ### Expected behavior I'd expect `interleave` to reuse existing view buffers ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
