Rich-T-kid opened a new pull request, #11696: URL: https://github.com/apache/arrow-rs/pull/11696
Depends on #11174. This is intentionally a draft while that PR is under review. # Why `Utf8`, `LargeUtf8`, `Binary`, and `LargeBinary` still use the generic in-progress coalescer. Very sparse filter and indexed selections therefore materialize an intermediate array before copying it into the output batch. # What changed - Add `InProgressByteArray` for variable-width string and binary arrays. - Write very sparse filter and indexed selections directly into offset, value, and validity buffers. - Override only `copy_rows_by_filter_from` and `copy_rows_by_indices`; retain the trait defaults for the other optional methods. - Use the direct path only at selectivity at or below `1 / 1024`, retaining the existing materialized kernels above the benchmarked cutoff. - Preserve value bytes under null slots and validate offset growth before construction. # Results Criterion comparison against the clean #11174 head at 0.1% selectivity (`8192` input rows, 20 samples): | Path | Schema | Nulls | Before | After | Change | |---|---|---:|---:|---:|---:| | filter | single `Utf8` | 0% | 37.962 ms | 22.778 ms | -40.00% | | filter | single `Utf8` | 10% | 44.181 ms | 26.099 ms | -40.93% | | filter | primitive + `Utf8` | 0% | 23.766 ms | 11.489 ms | -51.66% | | filter | primitive + `Utf8` | 10% | 28.201 ms | 15.441 ms | -45.24% | | take | single `Utf8` | 0% | 18.035 ms | 9.346 ms | -48.18% | | take | single `Utf8` | 10% | 21.655 ms | 13.831 ms | -36.13% | | take | primitive + `Utf8` | 0% | 11.860 ms | 4.144 ms | -65.06% | | take | primitive + `Utf8` | 10% | 16.718 ms | 10.711 ms | -35.93% | The full UTF-8 matrix also covered 1%, 10%, and 80% selectivity. Those cases retain the existing materialized paths. # Test coverage - Direct filter and indexed copies compared with the existing `filter` and `take` kernels. - `Utf8`, `LargeUtf8`, `Binary`, and `LargeBinary` integration coverage. - Nulls, repeated and reordered indices, filter slices and indices, empty output, target-batch splitting, and non-empty bytes under null slots. - Offset and memory-size accounting. - `cargo test -p arrow-select`: 475 unit tests and 19 doctests passed. - `cargo clippy -p arrow-select --all-targets -- -D warnings` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
