geoffreyclaude opened a new pull request, #24181: URL: https://github.com/apache/datafusion/pull/24181
## Which issue does this PR close? - Part of #19241. - Stacked on [#24088](https://github.com/apache/datafusion/pull/24088) and [#24102](https://github.com/apache/datafusion/pull/24102), which follow [#23014](https://github.com/apache/datafusion/pull/23014). - Extracted from #19390. ## Rationale for this change Large constant integer `IN` lists perform a hash lookup for every input row. Reducing that lookup cost speeds up these filters. ## What changes are included in this PR? - Adds a private immutable set for `Int32`, `UInt32`, `Int64`, and `UInt64`. - Stores two values directly in each bucket and sends additional collisions to an overflow table, reusing the computed hash. - Replaces float wrapper types with `to_bits()` keys in the standard `HashSet`. - Reuses `build_in_list_result` for null and negation handling. ## Are these changes tested? Yes. Tests cover collisions, slices, nulls, dictionary arrays, and float bit equality. ## Are there any user-facing changes? No. This is an internal performance optimization only. ## Local benchmark snapshot Built and run in separate target directories after compilation completed, using the existing `in_list_strategy` benchmark: ```bash cargo bench --target-dir <target-dir> \ -p datafusion-physical-expr \ --bench in_list_strategy -- 'primitive/i(32|64)/large_list|f32/large_list' --noplot ``` Criterion defaults were used, comparing median point estimates. Filter construction is outside the timed loop. Lower is better. Compared baselines: exact combined heads of [#24088](https://github.com/apache/datafusion/pull/24088) and [#24102](https://github.com/apache/datafusion/pull/24102) -> this PR 10 of 11 cases improved, with a 35.3% geometric-mean time reduction. | Benchmark | Before | After | Change | |---|---:|---:|---:| | `f32 / 64 / miss` | 15.94 us | 14.89 us | -6.6% | | `f32 / 64 / 50% hit` | 41.03 us | 30.13 us | -26.6% | | `i32 / 64 / miss` | 15.81 us | 14.17 us | -10.3% | | `i32 / 64 / 50% hit` | 35.36 us | 15.62 us | -55.8% | | `i32 / 256 / miss` | 16.48 us | 13.62 us | -17.4% | | `i32 / 256 / 50% hit` | 33.51 us | 15.18 us | -54.7% | | `i64 / 32 / miss` | 14.57 us | 15.65 us | +7.5% | | `i64 / 32 / 50% hit` | 37.08 us | 13.11 us | -64.6% | | `i64 / 128 / miss` | 16.07 us | 15.22 us | -5.3% | | `i64 / 128 / 50% hit` | 35.14 us | 17.17 us | -51.1% | | `nullable i32 / 64 / 50% hit / 20% null` | 29.60 us | 13.68 us | -53.8% | The baseline and this PR were measured without competing Cargo or rustc processes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
