[
https://issues.apache.org/jira/browse/SPARK-47305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun closed SPARK-47305.
---------------------------------
> PruneFilters incorrectly tags isStreaming flag when replacing child of Filter
> with LocalRelation
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-47305
> URL: https://issues.apache.org/jira/browse/SPARK-47305
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 3.5.1, 4.0.0
> Reporter: Jungtaek Lim
> Assignee: Jungtaek Lim
> Priority: Critical
> Labels: pull-request-available
> Fix For: 3.5.2, 3.4.3, 4.0.0
>
>
> This seems to be a very old bug in optimizer. Related ticket:
> https://issues.apache.org/jira/browse/SPARK-21765
> When filter is evaluated to be always false, PruneFilters replaces the filter
> with empty LocalRelation, which effectively prunes filter. The logic cares
> about migration of the isStreaming flag, but incorrectly migrated in some
> case, via picking up the value of isStreaming flag from root node rather than
> filter (or child).
> isStreaming flag is true if the value of isStreaming flag from any of
> children is true. Flipping the coin, some children might have isStreaming
> flag as "false". If the filter being pruned is a descendant to such children
> (in other word, ancestor of streaming node), LocalRelation is incorrectly
> tagged as streaming where it should be batch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
