[
https://issues.apache.org/jira/browse/SPARK-46062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-46062.
---------------------------------
> CTE reference node does not inherit the flag `isStreaming` from CTE
> definition node
> -----------------------------------------------------------------------------------
>
> Key: SPARK-46062
> URL: https://issues.apache.org/jira/browse/SPARK-46062
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.2, 3.4.1, 3.5.0, 4.0.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.2, 3.5.1, 4.0.0
>
>
> Looks like this is a long standing bug.
> We figured out that CTE reference node would never set the isStreaming flag
> to true, regardless of the value for flag in CTE definition. The node cannot
> determine the right value of isStreaming flag by itself (likewise it cannot
> determine about resolution by itself) but it has no parameter in constructor,
> hence always takes the default (no children, so batch one).
> This may impact some rules which behaves differently depending on isStreaming
> flag. It would no longer be a problem once CTE reference is replaced with CTE
> definition at some point in "optimization phase", but all rules in analyzer
> and optimizer being triggered before the rule takes effect may be impacted.
> We probably couldn't sync the flag in real time, but we should sync the flag
> when we mark CTE reference to be "resolved". The rule `ResolveWithCTE` will
> be a good place to do that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
