adriangb opened a new pull request, #25559: URL: https://github.com/apache/datafusion/pull/25559
## Which issue does this PR close? - Closes https://github.com/apache/datafusion/issues/25336. > [!IMPORTANT] > **Stacked on https://github.com/apache/datafusion/pull/25558.** GitHub cannot base a PR on another fork's branch, so the diff here shows both. **Review only the last commit** (`fix: apply the join filter when a null-aware hash join marks rows UNKNOWN`). Draft until #25558 lands; I will rebase then. ## Rationale for this change A correlated `NOT IN` whose correlation cannot become an equi-join key leaves a residual join filter. The null-aware hash join ignored that filter when deciding whether a NULL on the subquery side makes `NOT IN` UNKNOWN, so a NULL the filter excludes still poisoned every outer row: ```sql CREATE TABLE oc(id INT, g INT) AS VALUES (1,5),(2,5),(3,0),(4,NULL),(NULL,5),(NULL,0); CREATE TABLE ic(id INT) AS VALUES (1),(NULL); SELECT id, g FROM oc WHERE oc.id NOT IN (SELECT ic.id FROM ic WHERE oc.g > 0); ``` returns no rows; DuckDB and PostgreSQL return `3|0`, `4|NULL`, `NULL|0`. `oc.g > 0` holds only for `id` 1, 2 and the NULL-id row, so only those three see the subquery `{1, NULL}`; the rest see an empty subquery, and `NOT IN` over an empty set is TRUE. The plan was already right — `LeftAnti ... Filter: oc.g > Int32(0) null_aware` — so this is purely an execution fix. No optimizer change is involved. ## What changes are included in this PR? A NULL now makes `NOT IN` UNKNOWN only for the build rows whose correlation scope and residual filter keep that NULL, recorded per build row in a null-indices bitmap. Candidates come from a scope-map lookup when there are correlation keys and from a cross product otherwise, then pass the filter. Cost is proportional to the number of NULLs and is zero when the data has none; build rows already marked UNKNOWN are skipped. Null-aware `LeftAnti` also accepts more than one join key, which the equality-correlated shape needs. `RightAnti` still requires exactly one. ## What is the testing strategy for this PR? The coverage landed in #25558. This PR flips the expectations it fixes — the diff in `null_aware_anti_join.slt` and the Q05–Q07 canaries is the behaviour change. `datafusion/physical-plan/src/joins/hash_join/exec.rs` also gains unit tests for the filter-only anti and mark paths at several batch sizes. ## Are there any user-facing changes? Correlated `NOT IN` with a residual filter returns correct results. Some shapes that failed to plan now run. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
