Wei Liu created SPARK-46873:
-------------------------------
Summary: PySpark spark.streams should not recreate new
StreamingQueryManager
Key: SPARK-46873
URL: https://issues.apache.org/jira/browse/SPARK-46873
Project: Spark
Issue Type: Task
Components: Connect, PySpark, SS
Affects Versions: 4.0.0
Reporter: Wei Liu
In Scala, there is only one streaming query manager for one spark session:
```
scala> spark.streams
val *res0*: *org.apache.spark.sql.streaming.StreamingQueryManager* =
org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba
scala> spark.streams
val *res1*: *org.apache.spark.sql.streaming.StreamingQueryManager* =
org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba
scala> spark.streams
val *res2*: *org.apache.spark.sql.streaming.StreamingQueryManager* =
org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba
scala> spark.streams
val *res3*: *org.apache.spark.sql.streaming.StreamingQueryManager* =
org.apache.spark.sql.streaming.StreamingQueryManager@46bb8cba
```
In Python, this is currently false:
```
>>> spark.streams
<pyspark.sql.connect.streaming.query.StreamingQueryManager object at
0x1011f7c10>
>>> spark.streams
<pyspark.sql.connect.streaming.query.StreamingQueryManager object at
0x1011f71f0>
>>> spark.streams
<pyspark.sql.connect.streaming.query.StreamingQueryManager object at
0x1011f7be0>
>>> spark.streams
<pyspark.sql.connect.streaming.query.StreamingQueryManager object at
0x1011f7c40>
```
Python should align scala behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
