viirya opened a new issue, #25691: URL: https://github.com/apache/datafusion/issues/25691
### Describe the bug Null-aware `LeftMark` hash joins inspect the physical null bitmap of dictionary-encoded join keys instead of their logical nulls. A `DictionaryArray` key can be physically valid while pointing to a `NULL` dictionary value, so the join misses that `NULL` and produces an incorrect `FALSE` mark where SQL three-valued logic requires `NULL`. The problem affects both sides of an uncorrelated null-aware mark join: * A logical `NULL` build key is treated as non-null when the final mark column is built. * A logical `NULL` probe key is not recorded in the shared probe summary, including when it is seen by a different probe partition from the partition that completes the join. ### To Reproduce At the `HashJoinExec` level, use dictionary keys whose key bitmap contains no nulls but whose dictionary values contain a null: * Build values `[1, NULL, 4]`, probe values `[1, 2, 3]`: the expected marks are `[TRUE, NULL, FALSE]`, but the build-side logical null is marked `FALSE`. * Build values `[1, 4]`, probe values containing `[1, NULL]`: the expected marks are `[TRUE, NULL]`, but the unmatched build row is marked `FALSE`. ### Expected behavior Null-aware `LeftMark` joins should use Arrow logical-null semantics for dictionary arrays on both build and probe sides, producing the same SQL results as non-dictionary arrays. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
