Rich-T-kid opened a new pull request, #12203: URL: https://github.com/apache/arrow-rs/pull/12203
# Which issue does this PR close? - Part of #11518. # Rationale for this change DataFusion owns the Arrow array hashing used by repartitioning, joins, grouping, and approximate distinct counts. Arrow and Parquet code cannot reuse it, so consumers that need row hashes or hashes for Bloom filters and sketches must keep separate implementations. This draft starts the shared layer in `arrow-arith` so we can settle the API, hash compatibility, and supported representations before moving DataFusion callers. # What changes are included in this PR? This adds hasher-generic `create_hashes` and `create_hashes_buffer` kernels for primitive, Boolean, string, binary, view, fixed-size binary, and dictionary arrays. The caller chooses the hash builder. The aligned-buffer API uses Arrow allocation, the dense primitive path works in fixed-size chunks, and dictionaries hash values once before scattering hashes by key. This is still WIP. Nested arrays, benchmarks against DataFusion, and the DataFusion migration remain. Bloom-filter batch APIs are already being developed in #11641, so this does not add a second Bloom implementation. Theta sketches should follow separately once the project decides whether compatibility with the Apache DataSketches format used by Iceberg is required. # Are these changes tested? Tests cover chunk boundaries and tails, sliced and nullable primitives, multi-column ordering, null dictionary keys and null dictionary values, inline and out-of-line views, float normalization, aligned output, and invalid or unsupported input. # Are there any significant user-facing changes? Yes. This adds new public hashing kernels under `arrow_arith::hash` and re-exports them from `arrow::compute`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
