ewoodbury opened a new pull request, #2898:
URL: https://github.com/apache/datafusion-ballista/pull/2898

   ## Which issue does this PR close?
   
   Closes #2849.
   
   ## Rationale for this change
   
   A `NOT IN` subquery that stays a null-aware anti join has to read the whole 
subquery result in one task to be correct. The static planner gets that by 
putting a coalesce in front of the join, but when there's an aggregate above 
the join, the planner later removes that coalesce. The join then runs as 
several tasks, and each one returns its own copy of the rows it didn't match.
   
   Example: with the static planner and the `NOT IN` rewrite turned off, this 
query over a 4-file `t2` returns 26/26/23 instead of 5/5/5:
   
   ```sql
   select a % 3, count(*) from t1 where a not in (select b from t2) group by a 
% 3
   ```
   
   It only happens with the static planner (or pipelined mode, which always 
uses it). Default settings don't hit it.
   
   ## What changes are included in this PR?
   
   The static planner now ends the stage right after a null-aware join, so 
nothing planned above it can remove the coalesce. The subquery side is still 
read in parallel. The cost is one extra stage, carrying the join's output, 
which is never larger than the table the join already holds in memory.
   
   A one-partition coalesced shuffle reader would avoid the extra stage, but 
the static planner doesn't support those yet, so that's left as a follow-up.
   
   Tests: the `null_aware` client test gets a static-planner case and a `GROUP 
BY` query (that combination fails on main), and a planner test checks the new 
stage.
   
   ## Are there any user-facing changes?
   
   Correct results. `EXPLAIN` shows one more stage for null-aware anti joins 
under the static planner.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to