mrhard9090 opened a new pull request, #26010: URL: https://github.com/apache/datafusion/pull/26010
## Which issue does this PR close? - Closes #26002. ## Rationale for this change `SortMergeJoinExec` only evaluated its join filter when the filter read at least one column. A filter that reads none (a bound parameter such as `$1 > 5`, or a volatile function such as `random() > 2`) was skipped, so every key match was kept. For `FULL JOIN` that returns matched pairs the filter rejects, where hash join null-pads both sides. ## What changes are included in this PR? - `freeze_streamed_matched` (`sort_merge_join/materializing_stream.rs`) now evaluates the filter whenever the join has one. The filter batch is built with `RecordBatchOptions::with_row_count(total_matched_rows)`, because a batch with no columns cannot infer its row count. - Two queries in `sort_merge_join.slt`: `FULL JOIN ... AND random() > 2` (every row null-padded) and `AND random() < 2` (all key matches kept). I did not touch `bitwise_stream.rs` (semi/anti/mark joins), which builds its filter batch in a different place. ## What is the testing strategy for this PR? `cargo test -p datafusion-sqllogictest --test sqllogictests -- sort_merge_join`: all 4 files pass. With the `materializing_stream.rs` change reverted, `sort_merge_join.slt` fails. `cargo test -p datafusion-physical-plan --lib sort_merge_join`: 247 passed. `cargo fmt` and `cargo clippy -p datafusion-physical-plan --all-targets -- -D warnings` pass. ## Are there any user-facing changes? `FULL JOIN` (and other join types run by sort-merge join) with a column-free `ON` condition now returns the same rows as hash join. No API change. AI assistance: written with Claude Code; I reproduced the bug, reviewed the diff and ran the tests above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
