[
https://issues.apache.org/jira/browse/SPARK-46873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-46873:
------------------------------------
Assignee: Wei Liu
> PySpark spark.streams should not recreate new StreamingQueryManager
> -------------------------------------------------------------------
>
> Key: SPARK-46873
> URL: https://issues.apache.org/jira/browse/SPARK-46873
> Project: Spark
> Issue Type: Task
> Components: Connect, PySpark, SS
> Affects Versions: 4.0.0
> Reporter: Wei Liu
> Assignee: Wei Liu
> Priority: Major
> Labels: pull-request-available
>
> In Scala, there is only one streaming query manager for one spark session:
> ```
> scala> spark.streams
> val *res0*: *org.apache.spark.sql.streaming.StreamingQueryManager* =
> org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba
>
> scala> spark.streams
> val *res1*: *org.apache.spark.sql.streaming.StreamingQueryManager* =
> org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba
>
> scala> spark.streams
> val *res2*: *org.apache.spark.sql.streaming.StreamingQueryManager* =
> org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba
>
> scala> spark.streams
> val *res3*: *org.apache.spark.sql.streaming.StreamingQueryManager* =
> org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba
> ```
>
> In Python, this is currently false:
> ```
> >>> spark.streams
> <pyspark.sql.connect.streaming.query.StreamingQueryManager object at
> 0x1011f7c10>
> >>> spark.streams
> <pyspark.sql.connect.streaming.query.StreamingQueryManager object at
> 0x1011f71f0>
> >>> spark.streams
> <pyspark.sql.connect.streaming.query.StreamingQueryManager object at
> 0x1011f7be0>
> >>> spark.streams
> <pyspark.sql.connect.streaming.query.StreamingQueryManager object at
> 0x1011f7c40>
> ```
>
> Python should align scala behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
