Tathagata Das created SPARK-22018: ------------------------------------- Summary: Catalyst Optimizer does not preserve top-level metadata while collapsing projects Key: SPARK-22018 URL: https://issues.apache.org/jira/browse/SPARK-22018 Project: Spark Issue Type: Bug Components: Optimizer, Structured Streaming Affects Versions: 2.2.0, 2.1.1 Reporter: Tathagata Das Assignee: Tathagata Das
If there are two projects like as follows. {code} Project [a_with_metadata#27 AS b#26] +- Project [a#0 AS a_with_metadata#27] +- LocalRelation <empty>, [a#0, b#1] {code} Child Project has an output column with a metadata in it, and the parent Project has an alias that implicitly forwards the metadata. So this metadata is visible for higher operators. Upon applying CollapseProject optimizer rule, the metadata is not preserved. {code} Project [a#0 AS b#26] +- LocalRelation <empty>, [a#0, b#1] {code} This is incorrect, as downstream operators that expect certain metadata (e.g. watermark in structured streaming) to identify certain fields will fail to do so. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org