saadtajwar commented on code in PR #23957:
URL: https://github.com/apache/datafusion/pull/23957#discussion_r3739622020
##########
datafusion/physical-plan/src/joins/hash_join/stream.rs:
##########
@@ -742,40 +742,58 @@ impl HashJoinStream {
let timer = self.join_metrics.join_time.timer();
// Null-aware anti join semantics:
+
// For LeftAnti: output LEFT (build) rows where LEFT.key NOT IN
RIGHT.key
// 1. If RIGHT (probe) contains NULL in any batch, no LEFT rows should
be output
// 2. LEFT rows with NULL keys should not be output (handled in final
stage)
+
+ // For RightAnti: output RIGHT (probe) rows where RIGHT.key NOT IN
LEFT.key
+ // 1. If LEFT (build) contains NULL, no RIGHT rows should be output
+ // 2. RIGHT rows with NULL keys should not be output
+ // 3. If LEFT (build) is empty, all RIGHT rows should be output
if self.null_aware {
- // Mark that we've seen a probe batch with actual rows (probe side
is non-empty)
- // Only set this if batch has rows - empty batches don't count
- // Use shared atomic state so all partitions can see this global
information
- if state.batch.num_rows() > 0 {
- build_side
- .left_data
- .probe_side_non_empty
- .store(true, Ordering::Relaxed);
- }
+ match self.join_type {
+ JoinType::RightAnti => {
Review Comment:
Thanks, done! Added that condition to `can_swap_hash_join`
##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -2330,6 +2338,8 @@ async fn collect_left_input(
bounds = None;
}
+ let build_has_null = !left_values.is_empty() &&
left_values[0].null_count() > 0;
Review Comment:
Ah good catch - fixed to use those logical null masks!
##########
datafusion/physical-plan/src/joins/hash_join/stream.rs:
##########
@@ -894,14 +912,26 @@ impl HashJoinStream {
last_joined_right_idx.map_or(0, |v| v + 1)
};
- let (left_indices, right_indices) = adjust_indices_by_join_type(
+ let (left_indices, mut right_indices) = adjust_indices_by_join_type(
left_indices,
right_indices,
index_alignment_range_start..index_alignment_range_end,
self.join_type,
self.right_side_ordered,
)?;
+ // If null-aware RightAnti join, we don't want to emit NULL probe keys
Review Comment:
Ah I see, that makes sense, thanks! Pushed a change to address, please let
me know your thoughts!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
