ewoodbury commented on code in PR #2848:
URL: 
https://github.com/apache/datafusion-ballista/pull/2848#discussion_r4582238614


##########
ballista/scheduler/src/state/aqe/optimizer_rule/distributed_exchange.rs:
##########
@@ -60,21 +61,18 @@ impl DistributedExchangeRule {
         config: &datafusion::config::ConfigOptions,
     ) -> datafusion::error::Result<Transformed<Arc<dyn ExecutionPlan>>> {
         // DataFusion's null-aware hash join coordinates visited rows and
-        // probe-side NULL state with in-process atomics. A CollectLeft join
-        // running once per probe partition therefore produces duplicate or
-        // incorrect output. This is the final plan-mutating rule, so enforce 
an
-        // explicit probe-side coalesce here after DataFusion's optimizers have
-        // finished. Add an exchange when the subtree has no existing boundary.
-        if let Some(hash_join) = execution_plan.downcast_ref::<HashJoinExec>()
-            && hash_join.null_aware
-            && *hash_join.partition_mode() == PartitionMode::CollectLeft
-            && hash_join
-                .right()
-                .downcast_ref::<CoalescePartitionsExec>()
-                .is_none()
+        // probe-side NULL state with in-process atomics, and a nested-loop 
join
+        // tracks matched build rows the same way. Such a join running once per
+        // probe partition therefore produces duplicate or incorrect output.
+        // This is the final plan-mutating rule, so enforce an explicit
+        // probe-side coalesce here after DataFusion's optimizers have 
finished.
+        // Add an exchange when the subtree has no existing boundary.
+        if probe_needs_one_task(&execution_plan)
+            && let [left, right] = execution_plan.children().as_slice()
+            && right.downcast_ref::<CoalescePartitionsExec>().is_none()
         {
-            let left = hash_join.left().clone();
-            let right = hash_join.right().clone();
+            let left = Arc::clone(left);
+            let right = Arc::clone(right);
             let right = if right.downcast_ref::<ExchangeExec>().is_none()

Review Comment:
   Done for small scans, a probe that `is_inlinable_scan_pipeline` accepts is 
now read directly in the join task. Null-aware joins share this code so they 
get it too. I skipped the one-partition check because some scans report one 
partition at planning time and split later, which could put a large scan in the 
single join task.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to