Jungtaek Lim created SPARK-47305: ------------------------------------ Summary: PruneFilters incorrectly tags isStreaming flag when replacing child of Filter with LocalRelation Key: SPARK-47305 URL: https://issues.apache.org/jira/browse/SPARK-47305 Project: Spark Issue Type: Bug Components: Structured Streaming Affects Versions: 3.5.1, 3.5.0, 3.4.1, 3.4.0, 3.4.2, 4.0.0 Reporter: Jungtaek Lim
This seems to be a very old bug in optimizer. Related ticket: https://issues.apache.org/jira/browse/SPARK-21765 When filter is evaluated to be always false, PruneFilters replaces the filter with empty LocalRelation, which effectively prunes filter. The logic cares about migration of the isStreaming flag, but incorrectly migrated in some case, via picking up the value of isStreaming flag from root node rather than filter (or child). isStreaming flag is true if the value of isStreaming flag from any of children is true. Flipping the coin, some children might have isStreaming flag as "false". If the filter being pruned is a descendant to such children (in other word, ancestor of streaming node), LocalRelation is incorrectly tagged as streaming where it should be batch. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org