Livia Zhu created SPARK-53625:
---------------------------------
Summary: Propagate metadata columns through projections to address
ApplyCharTypePadding incompatibility in streaming
Key: SPARK-53625
URL: https://issues.apache.org/jira/browse/SPARK-53625
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 4.1.0
Reporter: Livia Zhu
Currently, streams where the output DF access the `_metadata` column with a
char type in the schema and ApplyCharTypePadding rule applied will encounter an
AssertionError in streaming MicrobatchExecution. This is because the
ApplyCharTypePadding rule hides the metadata column with a Projection and this
code
([https://github.com/apache/spark/blob/87a71fabb097e1543a935fae8167bc47a29a127e/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala#L803)]
does not propagate the metadata column through the projection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
