Pablo Langa Blanco created SPARK-56523:
------------------------------------------
Summary: RocksDB memory leak
Key: SPARK-56523
URL: https://issues.apache.org/jira/browse/SPARK-56523
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.2.0, 4.1.2, 3.5.9, 4.0.3
Reporter: Pablo Langa Blanco
Whilst running Spark tests in our environments, we detected unexpected memory
usage; upon analysing the issue in depth, we found that the problem lies in the
fact that the statistics object created here is not being released from memory
([https://github.com/apache/spark/blob/5d491f62748b4b9c34bc3b5bd7390f7b5ca75053/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala#L151]).
Only the options are closed at the close
([https://github.com/apache/spark/blob/5d491f62748b4b9c34bc3b5bd7390f7b5ca75053/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala#L2046]),
which does not release the statistics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
