[
https://issues.apache.org/jira/browse/BEAM-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933052#comment-16933052
]
Rui Wang commented on BEAM-7049:
--------------------------------
Recently there was a effort to separate multiple join implementation from one
single RelNode to mutiple RelNode: https://github.com/apache/beam/pull/9395
I am thinking UNION ALL and UNION can apply the idea: we separate UNION ALL and
UNION to different RelNode and to see if we can separate optimizations: one
shuffle implementation only is applied for UNION ALL RelNode and UNION will
still use existing implementation.
What do you think about this idea? It will start to touch SQL optimizer.
> Merge multiple input to one BeamUnionRel
> ----------------------------------------
>
> Key: BEAM-7049
> URL: https://issues.apache.org/jira/browse/BEAM-7049
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
> Reporter: Rui Wang
> Assignee: sridhar Reddy
> Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> BeamUnionRel assumes inputs are two and rejects more. So `a UNION b UNION c`
> will have to be created as UNION(a, UNION(b, c)) and have two shuffles. If
> BeamUnionRel can handle multiple shuffles, we will have only one shuffle
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
