SubhamSinghal opened a new pull request, #27578:
URL: https://github.com/apache/datafusion/pull/27578

    ## Which issue does this PR close?
   
     - Part of #18221
   
   ## Rationale for this change
   
   `PiecewiseMergeJoinExec` has no embedded projection. It emits every column 
of both sides, join keys included, and a `ProjectionExec` above it drops the 
extra columns. On #18221's shape (`SELECT p, q ... ON l.a < r.b`, 100K × 100K) 
that doubles the bytes the join writes (`output_bytes` 74.5 → 37.3 GB with this 
PR).
   
   ## What changes are included in this PR?
   
   - `PiecewiseMergeJoinExec` embeds a parent projection 
(`try_swapping_with_projection`, `EmbeddedProjection`) for 
Inner/Left/Right/Full joins. Semi/Anti/Mark joins emit through their own 
streams and keep the `ProjectionExec`.
   - Each side is narrowed right after its join key is evaluated: the buffered 
batch in `build_buffered_data`, each streamed batch in `fetch_stream_batch`. 
Emission is unchanged. When the select list crosses sides (`SELECT r.q, l.p`), 
`OutputBuffer::push_columns` reorders the columns zero-copy.
   - A side can be narrowed to zero columns: a one-side-only select list works, 
and `count(*)` embeds `projection=[]`.
   - `with_new_children` (`Recompute`) re-applies the projection, which 
`try_new` alone would drop.
   - Proto: `PiecewiseMergeJoinExecNode.projection = 8`, encoded as on 
`NestedLoopJoinExecNode` (empty = none, `[u32::MAX]` = zero columns).
   
   ## Are these changes tested?
   
   - `piecewise_merge_join_matrix.slt`: each of the 4 join types × {partial and 
reordered, buffered side only, streamed side only, `count(*)`}, plus `>=` and 
identity cases. Each runs with PWMJ on and off across a batch-size sweep, with 
NLJ as the oracle.
   - `pwmj.slt`: plan shape. Unit tests: projections compose and survive 
`Recompute`, reordered output, existence joins rejected, equivalence properties 
projected. Proto round trip plus wire layout for all three states.
   - Mutants caught: no reorder, wrong `OutputBuffer` schema, unprojected 
never-matched streamed rows, unprojected buffered batch, wrong null-padding 
schema.
   
   ## Are there any user-facing changes?
   
   `EXPLAIN` shows `projection=[...]` on `PiecewiseMergeJoin`, and the 
`ProjectionExec` above it is gone. New public 
`PiecewiseMergeJoinExec::with_projection` / `projection` / 
`contains_projection`, plus a new optional proto field. No breaking change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to