[ https://issues.apache.org/jira/browse/SPARK-22018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tathagata Das resolved SPARK-22018. ----------------------------------- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 19240 [https://github.com/apache/spark/pull/19240] > Catalyst Optimizer does not preserve top-level metadata while collapsing > projects > --------------------------------------------------------------------------------- > > Key: SPARK-22018 > URL: https://issues.apache.org/jira/browse/SPARK-22018 > Project: Spark > Issue Type: Bug > Components: Optimizer, Structured Streaming > Affects Versions: 2.1.1, 2.2.0 > Reporter: Tathagata Das > Assignee: Tathagata Das > Fix For: 3.0.0 > > > If there are two projects like as follows. > {code} > Project [a_with_metadata#27 AS b#26] > +- Project [a#0 AS a_with_metadata#27] > +- LocalRelation <empty>, [a#0, b#1] > {code} > Child Project has an output column with a metadata in it, and the parent > Project has an alias that implicitly forwards the metadata. So this metadata > is visible for higher operators. Upon applying CollapseProject optimizer > rule, the metadata is not preserved. > {code} > Project [a#0 AS b#26] > +- LocalRelation <empty>, [a#0, b#1] > {code} > This is incorrect, as downstream operators that expect certain metadata (e.g. > watermark in structured streaming) to identify certain fields will fail to do > so. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org