[
https://issues.apache.org/jira/browse/BEAM-9451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17548347#comment-17548347
]
Danny McCormick commented on BEAM-9451:
---------------------------------------
This issue has been migrated to https://github.com/apache/beam/issues/19989
> Optimize translation when Schema information is available in Spark Structured
> Streaming runner
> ----------------------------------------------------------------------------------------------
>
> Key: BEAM-9451
> URL: https://issues.apache.org/jira/browse/BEAM-9451
> Project: Beam
> Issue Type: Improvement
> Components: runner-spark
> Reporter: Ismaël Mejía
> Priority: P3
> Labels: structured-streaming
>
> Spark Structured Streaming runner supports Datasets that already have Schema
> information. This is used by Spark to optimize jobs (via Catalyst). This
> issue is to implement optimized translations of the transforms for the runner
> so we can benefit of the performance improvements internally done by Spark.
> Notice that we also may need to map Beam's core internal representations like
> WindowedValue so we can have intermediary optimizations.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
