[
https://issues.apache.org/jira/browse/SPARK-56523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18074225#comment-18074225
]
Pablo Langa Blanco commented on SPARK-56523:
--------------------------------------------
I will create a PR on this
> RocksDB memory leak
> -------------------
>
> Key: SPARK-56523
> URL: https://issues.apache.org/jira/browse/SPARK-56523
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.2.0, 4.1.2, 3.5.9, 4.0.3
> Reporter: Pablo Langa Blanco
> Priority: Major
>
> Whilst running Spark tests in our environments, we detected unexpected memory
> usage; upon analysing the issue in depth, we found that the problem lies in
> the fact that the statistics object created here is not being released from
> memory
> ([https://github.com/apache/spark/blob/5d491f62748b4b9c34bc3b5bd7390f7b5ca75053/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala#L151]).
> Only the options are closed at the close
> ([https://github.com/apache/spark/blob/5d491f62748b4b9c34bc3b5bd7390f7b5ca75053/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala#L2046]),
> which does not release the statistics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
