viirya commented on code in PR #26647:
URL: https://github.com/apache/datafusion/pull/26647#discussion_r4366088339
##########
datafusion/physical-plan/src/joins/hash_join/exec.rs:
##########
@@ -11210,6 +10911,89 @@ mod tests {
Ok(Arc::new(swapped.builder().reset_state().build()?))
}
+ /// A correlated null-aware join whose build value keys are all NULL: under
+ /// `NullEqualsNothing` its full-key map is empty, but the UNKNOWN marks
+ /// still depend on each scope. `NULL NOT IN (5)` and `NULL NOT IN (7)` are
+ /// UNKNOWN; `NULL NOT IN (empty)`, in scope 3, is TRUE. The probe side
+ /// arrives in two batches.
+ #[apply(hash_join_exec_configs)]
+ #[tokio::test]
+ async fn test_null_aware_correlated_all_null_build_values(
+ batch_size: usize,
+ ) -> Result<()> {
+ fn table(id: Vec<Option<i32>>, g: Vec<Option<i32>>) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("id", DataType::Int32, true),
+ Field::new("g", DataType::Int32, true),
+ ]));
+ RecordBatch::try_new(
+ schema,
+ vec![
+ Arc::new(Int32Array::from(id)),
+ Arc::new(Int32Array::from(g)),
+ ],
+ )
+ .unwrap()
+ }
+ let build = table(vec![None, None, None], vec![Some(1), Some(2),
Some(3)]);
+ let probe = [
+ table(vec![Some(5)], vec![Some(1)]),
+ table(vec![Some(7)], vec![Some(2)]),
+ ];
+
+ let mut results = vec![];
+ for join_type in [JoinType::LeftAnti, JoinType::LeftMark] {
Review Comment:
Thanks. Putting `(9, 4)` in the same batch as `(7, 2)` is better than what I
suggested: it also checks that the empty-build path drops rows, not whole
batches.
##########
datafusion/physical-plan/src/joins/array_map.rs:
##########
@@ -245,6 +245,9 @@ impl ArrayMap {
self.data.capacity() * size_of::<u32>() + self.next.capacity() *
size_of::<u32>()
}
+ // Kept out of line: inlined into `HashJoinStream::process_probe_batch`,
+ // this loop measured ~4-6% slower on high-fanout probes.
+ #[inline(never)]
Review Comment:
`null_aware_join` is in as well: 11 of 12 queries are unchanged or faster.
Q08 is 2% slower (20.4 → 20.8 ms), but Q06, with the same NULL fraction and no
scope key, is 2% faster, and Q08's marking path is moved unchanged, so I read
it as code placement. I'm fine leaving the pins out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
