[
https://issues.apache.org/jira/browse/SPARK-50559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk reassigned SPARK-50559:
--------------------------------
Assignee: Vladimir Golubev
> Store Union, Except and Intersect operator's output as lazy val
> ---------------------------------------------------------------
>
> Key: SPARK-50559
> URL: https://issues.apache.org/jira/browse/SPARK-50559
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Vladimir Golubev
> Assignee: Vladimir Golubev
> Priority: Major
> Labels: pull-request-available
>
> Currently `Union`'s (same is for `Except` and `Intersect`) `output` is a
> `def`. This creates performance issues for queries with large number of
> stacked `UNION`s because of rules like `WidenSetOperationTypes` that traverse
> the logical plan and call `output` on each `Union` node. This has quadratic
> complexity: O(number_of_unions * (1 + 2 + 3 + ... + number_of_unions)).
> The proposed solution is to use a `lazy val`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
