jayzhan211 opened a new issue, #11635: URL: https://github.com/apache/arrow-rs/issues/11635
**Describe the bug** `BatchCoalescer::push_batch_with_filter` does not compact `StringViewArray`/`BinaryViewArray` data buffers on its sparse path, while `filter` + `push_batch` does. A selective filter therefore keeps every input data buffer alive for the few selected strings. When `selected_count <= filter_len / 16`, `push_batch_with_filtered_columns` calls `InProgressByteViewArray::copy_rows_by_filter_from`, which filters the views and then calls `append_views_and_update_buffer_index(filtered.views(), filtered.data_buffers(), false)`, reusing the source buffers. `push_batch` goes through `set_source`, which sets `need_gc` when the buffers are more than 2x what the views use and then copies the strings (`append_views_and_copy_strings`). **To Reproduce** 1600 strings of 39 bytes in one `StringViewArray`, filter selecting every 100th row, `push_batch_with_filter`, `finish_buffered_batch`: the output's data buffers have the same capacity as the input (114,688 bytes) for 16 strings. With `filter_record_batch` + `push_batch` the output buffers hold only the selected strings. In DataFusion this showed up as 2-3x higher memory on ClickBench queries with selective string filters (apache/datafusion#26436, which works around it by calling `filter_record_batch` + `push_batch` for view columns). **Expected behavior** `copy_rows_by_filter_from` applies the same `need_gc` decision as `copy_rows`: when the filtered views use less than half of the source buffers, copy the selected strings instead of reusing the buffers. **Additional context** arrow-select 60.0.0, same code on main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
