geoffreyclaude opened a new pull request, #24662: URL: https://github.com/apache/datafusion/pull/24662
## Which issue does this PR close? - Closes #24658. - Stacked on #24102. ## Rationale for this change Dictionary handling is currently repeated in each concrete `IN LIST` filter. That makes `datafusion-physical-expr` substantially larger to compile and link. ## What changes are included in this PR? Dictionary-encoded inputs use one small wrapper around the selected filter. The filter checks each dictionary value once, then maps those results through the dictionary keys. Plain inputs continue to call their selected filter directly. Nested dictionary list values are flattened before choosing a filter. This preserves SQL null behavior while still allowing the final value type to use its specialized filter. The PR also adds small-batch Criterion cases for both plain and dictionary inputs. ### Measurements Measured locally on an M1 Max against the exact #24102 head plus the benchmark-only commit. | Size measurement | Before | After | Change | |---|---:|---:|---:| | `datafusion-physical-expr` LLVM IR | 675,256 lines | 647,201 lines | -4.15% | | `StaticFilter::contains` LLVM IR | 33,502 lines | 5,343 lines | -84.05% | | Release benchmark Mach-O `__text` | 8,713,520 bytes | 8,500,880 bytes | -2.44% | Criterion median changes for batches of 1 / 8 / 64 / 8,192 rows (negative is faster): | Filter | Plain input | Dictionary input | |---|---:|---:| | Branchless, list 4 | -2.31% / -4.86% / -0.36% / -0.25% | -16.73% / -11.74% / -15.16% / -0.62% | | Hash set, list 64 | -7.52% / -7.98% / -1.42% / +0.22% | -14.50% / -21.18% / -16.38% / +0.14% | Criterion reports no statistically significant change in any of the four 8,192-row cases. ## Are these changes tested? Yes. Tests cover every dictionary key type, nested dictionaries, dictionary nulls, unreferenced null values, timestamp timezones, decimal metadata, and nested dictionary scalar inputs. - `cargo test -p datafusion-physical-expr --all-features` (1,694 passed, 1 ignored; 13 doctests passed) - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo fmt --all` ## Are there any user-facing changes? No. This keeps the existing `IN LIST` behavior while reducing generated code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
