Eric Marnadi created SPARK-44480:
------------------------------------
Summary: Add option for thread pool to perform maintenance for
RocksDB/HDFS State Store Providers
Key: SPARK-44480
URL: https://issues.apache.org/jira/browse/SPARK-44480
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.5.0
Reporter: Eric Marnadi
Maintenance tasks on StateStore was being done by a single background thread,
which is prone to straggling. In this change, the single background thread
would instead schedule maintenance tasks to a thread pool.
Introduce
{{spark.sql.streaming.stateStore.enableStateStoreMaintenanceThreadPool}} config
so that the user can enable a thread pool for maintenance manually.
Introduce {{spark.sql.streaming.stateStore.numStateStoreMaintenanceThreads}}
config so the thread pool size is configurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
