[
https://issues.apache.org/jira/browse/SPARK-39564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17557770#comment-17557770
]
Jungtaek Lim commented on SPARK-39564:
--------------------------------------
Will submit a PR soon.
> Expose the information of catalog table to the logical plan in streaming query
> ------------------------------------------------------------------------------
>
> Key: SPARK-39564
> URL: https://issues.apache.org/jira/browse/SPARK-39564
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> In Spark 3.1.0, we added ability to read and write from/to catalog table for
> streaming query. (Read:
> [SPARK-32885|https://issues.apache.org/jira/browse/SPARK-32885] / Write:
> [SPARK-32896|https://issues.apache.org/jira/browse/SPARK-32896])
> When the streaming query is being executed with such API, the table
> identifier for source and sink is "conditionally" exposed to the logical
> plan, depending on the DSv1 vs DSv2 and source vs sink.
> Since this is the only way to get the information of table, we would like to
> make some changes on relevant logical nodes to carry out and expose the
> information of table, specifically, table identifier.
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
