Chaoqin Li created SPARK-40492:
----------------------------------
Summary: Perform maintenance of StateStore instances when they
become inactive
Key: SPARK-40492
URL: https://issues.apache.org/jira/browse/SPARK-40492
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.3.0
Reporter: Chaoqin Li
Current the maintenance of StateStore is performed by a periodic task in the
management thread. If a streaming query become inactive before the next
maintenance task fire, its StateStore will be unloaded before cleanup.
There are 2 cases when a StateStore is unloaded.
# StateStoreProvider is not longer active in the system, for example, when a
query ends or the spark context terminates.
# There is other active StateStoreProvider in the system, for example, when a
partition is reassigned.
In case 1, we should do one last maintenance before unloading the instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
