Sunninsky opened a new pull request, #27356: URL: https://github.com/apache/datafusion/pull/27356
## Which issue does this PR close? - Part of #23805 (the 54.2.0 patch release tracking issue). - Backport of #26027 to `branch-54`. #26027 fixed #26001. ## Rationale for this change With a sort-merge join (`datafusion.optimizer.prefer_hash_join = false`), a `LEFT`, `RIGHT` or `FULL` join with an extra join filter fails with `Arrow error: Invalid argument error: Column '...' is declared as non-nullable but contains null values` when the NULL-padded side has `NOT NULL` columns referenced by the filter. Hash join returns the correct result for the same queries. This is a failed query, which makes it eligible for backport as a correctness fix. The four new sqllogictest queries fail on `branch-54` exactly as on `main`. #26027 fixed it on `main` (labelled `v56.0.0`); it is on `branch-56` but not on `branch-54`. It was requested for `branch-54` on #26868 (https://github.com/apache/datafusion/issues/26868#issuecomment-6813785483), "to prevent regressions on upgrade to 54". ## What changes are included in this PR? - A `git cherry-pick -x` of 8248a579695b92b7ca94709eedd63a71689513ce (#26027, by @CuteChuanChuan), with the conflicts in `materializing_stream.rs` resolved by hand. The sort-merge join on this branch predates the generator refactor (#23761, #23976), so the surrounding code differs. - `datafusion/physical-plan/src/joins/sort_merge_join/materializing_stream.rs`: `MaterializingSortMergeJoinStream::try_new` builds the filter schema once with the buffered-side fields marked nullable, and filter evaluation uses it. It is only built when the join needs deferred filtering. This branch has no cached `deferred_filtering` field and calls `needs_deferred_filtering(&self.filter, self.join_type)` where it is needed, so the filter schema is built from that call; there is no other change in the constructor. The filter result of a NULL-padded row does not affect the output, so this does not change results. - `datafusion/sqllogictest/test_files/sort_merge_join.slt`: the cases from #26027, unchanged (they applied cleanly). ## Are these changes tested? Yes. On this branch (Rust 1.95.0) I ran: - `cargo test -p datafusion-sqllogictest --test sqllogictests -- join`: all 14 matching files pass, including `sort_merge_join.slt` with the new cases. - With the change in `materializing_stream.rs` reverted and the new slt cases kept, the four new queries fail with the Arrow error above (`sort_merge_join.slt` lines 990, 997, 1004 and 1013). - `cargo test -p datafusion-physical-plan --lib`: 1439 passed. - `cargo fmt --all -- --check` and `cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings` pass. ## Are there any user-facing changes? No. Bug fix only; no public API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
