mbutrovich opened a new pull request, #7691:
URL: https://github.com/apache/datafusion-comet/pull/7691

   ## Which issue does this PR close?
   
   - None. This PR depends on #7687, and the branch includes its commit.
   
   ## Rationale for this change
   
   Once a fixed-width column has a null, Comet's murmur3 and xxhash64 test the 
null bit row by row 
([`hash_array_primitive!`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/hash_funcs/utils.rs#L108-L112)),
 so the loop has a branch per row and doesn't vectorize. Shuffle hash 
partitioning hits this on nullable keys. In TPC-DS, the foreign keys in 
`store_sales` are 4.5% null, so nearly every batch has a null.
   
   ## What changes are included in this PR?
   
   Integer and float columns with nulls are hashed 64 rows at a time, adapting 
apache/datafusion#27559. A chunk with at least 16 valid rows hashes every row 
with the vectorized loop from #7687 and then restores the previous hashes of 
its null rows. A sparser chunk hashes only its valid rows. Small decimals, 
intervals, and booleans keep the per-row loop, because `i64::try_from` must not 
run on a null slot.
   
   ## Benchmark results
   
   These are local runs on an Apple M5 Max. They don't predict the Neoverse-N1 
and x86-64-v3 builds that Comet releases. Both sides used `-Ctarget-cpu=native` 
and thin LTO, and they alternated within one run. The aligned layout adds `-C 
llvm-args=-align-all-functions=7 -C llvm-args=-align-all-nofallthru-blocks=7` 
to both sides. The baseline is #7687.
   
   <img width="1600" height="2148" alt="hash-nulls-local" 
src="https://github.com/user-attachments/assets/32fd4e29-f351-4072-8820-0a2b6a2c23b3";
 />
   
   A dense chunk hashes its null slots too and then writes their previous 
hashes back, so its cost grows with the number of nulls up to the threshold. A 
chunk just above 16 valid rows pays the most for that. These runs don't include 
a null rate near that point (about 75%).
   
   Could a reviewer run the same comparison on x86 with 
`-Ctarget-cpu=x86-64-v3` and on Neoverse-N1 with `-Ctarget-cpu=neoverse-n1`, 
against #7687? On both sides:
   
   ```shell
   cd native
   RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo bench -p 
datafusion-comet-spark-expr --bench hash_nulls
   ```
   
   The `no_nulls` case doesn't reach the changed code, so it shows the noise 
floor.
   
   ## How are these changes tested?
   
   `hash_fixed_width_with_nulls_matches_per_row` in `hash_funcs/utils.rs` 
checks the chunk loop against hashing row by row, for lengths from 1 to 200, 
offsets 0 and 5, and validity patterns at and one below the threshold, with a 
null run longer than a chunk, and all null. 
`create_hashes_with_nulls_matches_per_row` checks int32, int64, and float64 
columns, including -0.0 and NaN, through murmur3 and xxhash64.
   
   # AI usage disclosure
   
   I wrote this PR with help from Claude but understand and support the core 
approach, implementation, and test coverage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to