mbutrovich commented on PR #27399: URL: https://github.com/apache/datafusion/pull/27399#issuecomment-6961888717
I ran property-based tests from [arrow-pbt](https://github.com/mbutrovich/arrow-pbt) (mbutrovich/arrow-pbt#5) against this PR at dbc4514bb, with arrow-rs `main` at d65329d3e7. Every join test passed seeds 1 to 6 at 1,000 cases each, including [one that compares `enable_migration_hash_join` on and off](https://github.com/mbutrovich/arrow-pbt/blob/2c4a66ab8d645443a5157e44936430cb632c54e4/arrow-pbt-datafusion/tests/joins.rs#L147-L158). That test runs the same `HashJoinExec` with the option on and off, and [checks](https://github.com/mbutrovich/arrow-pbt/blob/2c4a66ab8d645443a5157e44936430cb632c54e4/arrow-pbt-datafusion/src/join.rs#L490-L558) that both fail, or both return the same output types and the same rows in any order, with every output column valid under the Arrow format. Two more tests compare the hash join with a `NestedLoopJoinExec` whose filter is the residual filter and one `=` or `IS NOT DISTINCT FROM` per key, and the perfect hash join (`ArrayMap`) with the general hash map. The [cases](https://github.com/mbutrovich/arrow-pbt/blob/2c4a66ab8d645443a5157e44936430cb632c54e4/arrow-pbt-datafusion/src/join.rs#L82-L162) cover every join type, both `NullEquality` values, a residual filter in half of them, and `CollectLeft` and `Partitioned` with one partition per side. Keys are one or two columns of 24 types, including dictionaries, views, structs, lists, and floats whose values include NaN, -0.0, and 0.0. Both sides draw each key column from one small pool of values, and `batch_size` is 1 to 8, 16, 63 to 65, 1024, or 8192. In 31% of cases a probe row matches more build rows than `batch_size`, so its matches span chunks and reach [`last_probe_row_with_on_match`](https://github.com/apache/datafusion/blob/dbc4514bbd8710e384316282606d17a949989a50/datafusion/physical-plan/src/joins/hash_join/batched_join_processor.rs#L423-L435) for the joins that keep unmatched probe rows. That share depends only on the inputs. On `main` at e2dc8b048b, 4.2% of cases built an `Arr ayMap`. The comparison with the nested loop join also discarded 175 cases on three open arrow-rs bugs that the joins' non-key columns reach: apache/arrow-rs#11607, apache/arrow-rs#10925, and apache/arrow-rs#11835. In the `enable_migration_hash_join` test, a case where both paths fail on one of those bugs counts as agreement. To rerun it, check out the `datafusion-27399` branch of arrow-pbt, whose `Cargo.toml` points at this PR, and run: ```sh ARROW_PBT_SEEDS=1-6 ARROW_PBT_CASES=1000 cargo test -p arrow-pbt-datafusion --test joins ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
