JoeryH opened a new issue, #12305:
URL: https://github.com/apache/arrow-rs/issues/12305

   **Describe the bug**
   
   `BatchCoalescer`'s StringView/BinaryView path (`coalesce/byte_view.rs`) 
allocates the blocks it copies strings into from a `BufferSource` whose size 
doubles on every new block (8 KiB → 1 MiB) and is never reset. 
`InProgressByteViewArray::finish()` resets the size counters but keeps 
`buffer_source`, so once a coalescer has emitted a few batches, **every later 
output batch gets at least one 1 MiB block**, no matter how few string bytes it 
holds.
   
   `get_array_memory_size()` reports buffer capacity, so a long-running 
coalescer reports far more memory than it stores, and the per-row figure grows 
the longer the coalescer runs. Most of the extra capacity is never touched, so 
RSS stays much lower than what is accounted.
   
   **To Reproduce**
   
   Arrow 60.1.0 (same code in 59.x):
   
   ```rust
   use std::sync::Arc;
   use arrow::array::{Array, RecordBatch, StringViewArray, UInt32Array};
   use arrow::compute::{take, BatchCoalescer};
   use arrow::datatypes::{DataType, Field, Schema};
   
   fn main() {
       // One large source array whose strings live in shared data buffers 
(like a
       // decoded parquet batch); 10% of the strings are longer than 12 bytes.
       let n = 1_000_000;
       let strings: Vec<String> = (0..n)
           .map(|i| if i % 10 == 0 { format!("long-string-{i:012}") } else { 
format!("s{}", i % 1000) })
           .collect();
       let source = StringViewArray::from_iter_values(strings.iter());
       let schema = Arc::new(Schema::new(vec![Field::new("s", 
DataType::Utf8View, false)]));
   
       // Like a hash repartitioner: push small takes of the source into one 
coalescer,
       // so each pushed array barely uses the buffers it points into and gets 
gc'd (copied).
       let mut coalescer = BatchCoalescer::new(schema.clone(), 8192);
       let mut out = 0;
       for start in (0..n).step_by(160) {
           let idx = UInt32Array::from_iter_values(start as u32..(start + 
160).min(n) as u32);
           let col = take(&source, &idx, None).unwrap();
           coalescer.push_batch(RecordBatch::try_new(schema.clone(), 
vec![col]).unwrap()).unwrap();
           while let Some(batch) = coalescer.next_completed_batch() {
               let a = 
batch.column(0).as_any().downcast_ref::<StringViewArray>().unwrap();
               let stored: usize = a.data_buffers().iter().map(|b| 
b.len()).sum();
               let capacity: usize = a.data_buffers().iter().map(|b| 
b.capacity()).sum();
               if out < 3 || out % 30 == 0 {
                   println!("batch {out:3}: stored {stored:>8} B  capacity 
{capacity:>8} B  get_array_memory_size {:>8} B",
                            a.get_array_memory_size());
               }
               out += 1;
           }
       }
   }
   ```
   
   Output:
   
   ```
   batch   0: stored    19680 B  capacity    24576 B  get_array_memory_size   
155760 B
   batch   1: stored    19656 B  capacity    32768 B  get_array_memory_size   
163952 B
   batch   2: stored    19656 B  capacity    65536 B  get_array_memory_size   
196720 B
   batch  30: stored    19680 B  capacity  1048576 B  get_array_memory_size  
1179760 B
   batch  60: stored    19680 B  capacity  1048576 B  get_array_memory_size  
1179760 B
   batch  90: stored    19680 B  capacity  1048576 B  get_array_memory_size  
1179760 B
   ```
   
   Identical batches, 7.5x larger after the coalescer warms up. Data buffer 
capacity is 53x what is stored.
   
   **Impact in DataFusion**
   
   `RepartitionExec` (hash) runs one coalescer per output partition and feeds 
it `take`-ed slices of scan batches, which is exactly this pattern. Partitions 
are fed concurrently, so they also end up with many partly filled 256 KiB–1 MiB 
blocks. With a hash join building on a repartitioned parquet scan (several 
Utf8View columns, 50 partitions), the build side's accounted bytes per row grew 
with input size even though the stored string bytes grew linearly:
   
   | input (same files, ×N) | stored string bytes | data buffer capacity | 
accounted bytes/row |
   |---|---|---|---|
   | ×1 | 22 MB | 81 MB | 336 |
   | ×2 | 45 MB | 430 MB | 637 |
   | ×4 | 90 MB | 1.4 GB | 954 |
   | ×8 | 180 MB | 3.6 GB | 1188 |
   
   On real queries the memory pool peak was ~3.5x process RSS, which leads to 
spills or `ResourcesExhausted` well before memory is actually short. Switching 
the same query to `Utf8` dropped the build from 15.6 GB to 1.6 GB.
   
   **Expected behavior**
   
   An output batch's buffer capacity should be in proportion to the bytes it 
holds, and should not depend on how many batches the coalescer has emitted 
before.
   
   **Possible fix**
   
   I patched arrow-select locally and measured the ×8 case above (stock 3.94 GB 
accounted):
   
   1. Reset `buffer_source` in `InProgressByteViewArray::finish()` 
(`self.buffer_source = BufferSource::new();`): **1.53 GB**.
   2. Also `shrink_to_fit()` a block in `finish_current()` when it is less than 
half full: **1.09 GB**.
   3. Also size blocks by need with a floor (`max(min_size, 64 KiB)`) instead 
of doubling: **0.76 GB**. Bytes per row stay flat (~230) as input grows, 
against ~100 for `Utf8`. The rest is the 16-byte views.
   
   (1) is a one-liner and removes the dependence on history. (2) costs one 
realloc per under-filled block. (3) means more, smaller blocks per batch. Happy 
to open a PR for whichever you prefer.
   
   **Additional context**
   
   Related but different: #7765, #11280, #11635, apache/datafusion#25712. 
#11770 touches the same file for performance and does not change `BufferSource`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to