mbutrovich opened a new pull request, #7687: URL: https://github.com/apache/datafusion-comet/pull/7687
## Which issue does this PR close? - None. ## Rationale for this change The loops that hash fixed-width columns without nulls in Comet's murmur3 and xxhash64 index both the values and the hash buffer per row ([`hash_array_primitive!`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/hash_funcs/utils.rs#L99-L105)). Inside `create_murmur3_hashes`, LLVM reloads the values pointer and checks bounds on every row, and reports `loop not vectorized: cannot identify array bounds`. In the 1.2.0 jar, `create_murmur3_hashes` has no vector multiply in either the amd64 or the aarch64 `libcomet.so`. Murmur3 is the hash for shuffle hash partitioning ([`multi_partition.rs`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/shuffle/src/partitioners/multi_partition.rs#L302)). ## What changes are included in this PR? The loops for arrays without nulls iterate the hash buffer and the values together, so the primitive, float, boolean, interval, and small decimal loops can vectorize. With this change, the Int32 and Int64 murmur3 loops vectorize in a thin-LTO `libcomet` built with `-Ctarget-cpu=x86-64-v3` and with `-Ctarget-cpu=neoverse-n1`. Each column's length is now checked against the hash buffer, because a zipped loop would skip rows instead of panicking. The loops for arrays with nulls are unchanged. ## How are these changes tested? The existing hash tests check the hash values. `test_hashes_buffer_length_mismatch_is_error` in `murmur3.rs` and `xxhash64.rs` checks the length check. Could a reviewer with an x86 machine benchmark this against `main`? I don't have x86 hardware, and the xxhash64 case is the open question: on x86-64-v3, LLVM vectorizes it by emulating 64-bit multiplies with `vpmuludq`, and only a benchmark shows whether that beats the scalar loop. On both sides: ```shell cd native RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo bench -p datafusion-comet-spark-expr \ --bench xxhash64 --bench murmur3 --bench hash -- \ 'fallback/custom_seed_i64$|compatible/primitive_i64$|^spark_xxhash64/8192/|^spark_murmur3_hash/8192/|^murmur3/int32$' ``` `compatible/primitive_i64` runs through datafusion-spark, which this PR doesn't change, so it shows the noise floor. A shuffle-heavy TPC-H or TPC-DS run with an x86-64-v3 release jar would show the effect on shuffle hash partitioning. # AI usage disclosure I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
