mhk197 opened a new pull request, #12057: URL: https://github.com/apache/arrow-rs/pull/12057
# Which issue does this PR close? - Relates to #1851, which made the same change for `StringDictionaryBuilder` in #1861. # Rationale for this change `PrimitiveDictionaryBuilder` hashes with SipHash and does a `get` followed by an `insert` for each new value. `cast` to a primitive dictionary goes through this builder, so high-cardinality casts spend most of their time hashing. # What changes are included in this PR? - Hash with `ahash::RandomState`, which arrow-array already depends on and `GenericByteDictionaryBuilder` uses. - Look up each value once with the entry API. - Hash a value's bytes without the slice length. - Add a `primitive_dictionary_builder` benchmark. # Are these changes tested? Covered by the existing tests. Benchmark with Int32 keys, median of 3 interleaved runs on an Apple M5 Max: | distinct values / rows | UInt32 | UInt64 | Decimal128 | |---|---|---|---| | 20 / 1,000 | −63% | −68% | −70% | | 100 / 10,000 | −60% | −64% | −70% | | 10,000 / 100,000 | −62% | −69% | −69% | # Are there any significant user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
