mbutrovich opened a new issue, #379:
URL: https://github.com/apache/datafusion-spark/issues/379

   ### Is your feature request related to a problem or challenge?
   
   Several `datafusion-functions-nested` kernels count set bits with one call 
per list row. Each call builds a new bit chunk, and all but `array_compact` 
also slice a buffer or array per row, which clones its `Arc`:
   
   - `array_remove` with a scalar needle 
([remove.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/remove.rs#L575-L576))
   - `inner_product` 
([inner_product.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/inner_product.rs#L183-L185))
   - `cosine_distance` 
([cosine_distance.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/cosine_distance.rs#L177-L179))
   - `array_normalize` 
([array_normalize.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/array_normalize.rs#L151-L152))
   - `array_compact` for non-primitive values 
([array_compact.rs](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/array_compact.rs#L219-L220))
   
   `array_filter` already avoids this with a private 
[`SetBitCounter`](https://github.com/apache/datafusion/blob/c6c1cc31d2f433930c9b4f3ca9fea023846ac276/datafusion/functions-nested/src/utils.rs#L110-L147).
 apache/arrow-rs#12346 adds a `SetBitCounter` to `arrow-buffer`. It takes the 
bitmap instead of a word iterator, so callers can't pass an unpadded final 
word, and it returns correct counts when a position decreases instead of 
looping.
   
   @neilconway's apache/datafusion#26507 used a public DataFusion counter at 
these call sites, with benchmarks for each in its description. It was closed so 
the counter could move to arrow-rs.
   
   ### Describe the solution you'd like
   
   Once DataFusion depends on an arrow release with `SetBitCounter`, replace 
the private counter with it and use it at the call sites above. The commits of 
#26507 are a starting point: its benchmark commits 
([c3d5ca8](https://github.com/apache/datafusion/pull/26507/commits/c3d5ca84b6382b65b89a0efa698e008fa4acf610))
 and the call site changes 
([18b2c43](https://github.com/apache/datafusion/pull/26507/commits/18b2c43a99232651929b36a71f7ece3d036f441a),
 
[6225dfc](https://github.com/apache/datafusion/pull/26507/commits/6225dfc43b1d43c9c599ee994a611875a483f368)).
   
   The review of #26507 asked for these changes, which still apply:
   
   - Skip counting when the element validity has no nulls, because `nulls()` 
can return an all-valid buffer 
(https://github.com/apache/datafusion/pull/26507#discussion_r4318870152). Add 
an all-valid `NullBuffer` case to the `vector_math` benchmark, because its 
`no_nulls` cases pass `None` and never reach the counter 
(https://github.com/apache/datafusion/pull/26507#discussion_r4340399645).
   - In `inner_product` and `cosine_distance`, OR the two arguments' row 
bitmaps once before the loop instead of looking up both per row 
(https://github.com/apache/datafusion/pull/26507#discussion_r4318870156).
   - Name the new tests `test_sliced_capacity` like their siblings, and add 
multi-row `.slt` cases with a NULL element in only the first argument and in 
only the second 
(https://github.com/apache/datafusion/pull/26507#discussion_r4318870163).
   - Build the counter once above the branch in `prepare_list_filter` 
(https://github.com/apache/datafusion/pull/26507#pullrequestreview-5577773030).
   
   ### Describe alternatives you've considered
   
   Making the private counter public in `datafusion-common` would duplicate the 
arrow-rs type, and `datafusion-spark` would need it too.
   
   ### Additional context
   
   Routing `array_compact` through `prepare_list_filter` made it slower when 
NULL list rows still store elements: `parent_nulls/utf8/mostly` was 42% slower 
in #26507. A PR that includes that change has to measure and report it again.
   
   In the arrow-rs benchmark, rows of exactly 256 elements starting on a word 
boundary are 0.73x with the counter on x86 at the default target, and 1.10x 
with `-C target-cpu=x86-64-v3` (apache/arrow-rs#12347). The `vector_math` 
`width_256` cases cover that shape.
   
   ### Willingness to contribute
   
   I plan to contribute to this feature independently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to