[
https://issues.apache.org/jira/browse/SPARK-53625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim reassigned SPARK-53625:
------------------------------------
Assignee: Livia Zhu
> Propagate metadata columns through projections to address
> ApplyCharTypePadding incompatibility in streaming
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-53625
> URL: https://issues.apache.org/jira/browse/SPARK-53625
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.1.0
> Reporter: Livia Zhu
> Assignee: Livia Zhu
> Priority: Major
> Labels: pull-request-available
>
> Currently, streams where the output DF access the `_metadata` column with a
> char type in the schema and ApplyCharTypePadding rule applied will encounter
> an AssertionError in streaming MicrobatchExecution. This is because the
> ApplyCharTypePadding rule hides the metadata column with a Projection and
> this code
> ([https://github.com/apache/spark/blob/87a71fabb097e1543a935fae8167bc47a29a127e/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala#L803)]
> does not propagate the metadata column through the projection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
