[
https://issues.apache.org/jira/browse/SPARK-41339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-41339:
------------------------------------
Assignee: Apache Spark
> RocksDB state store WriteBatch doesn't clean up native memory
> -------------------------------------------------------------
>
> Key: SPARK-41339
> URL: https://issues.apache.org/jira/browse/SPARK-41339
> Project: Spark
> Issue Type: Bug
> Components: SQL, Structured Streaming
> Affects Versions: 3.3.1
> Reporter: Adam Binford
> Assignee: Apache Spark
> Priority: Major
>
> The RocksDB state store uses a WriteBatch to hold updates that get written in
> a single transaction to commit. Somewhat indirectly abort is called after a
> successful task which calls writeBatch.clear(), but the data for a writeBatch
> is stored in a std::string in the native code. Not sure why it's stored as a
> string, but it is. [rocksdb/write_batch.h at main · facebook/rocksdb ·
> GitHub|https://github.com/facebook/rocksdb/blob/main/include/rocksdb/write_batch.h#L491]
> writeBatch.clear simply calls rep_.clear() and rep._resize()
> ([rocksdb/write_batch.cc at main · facebook/rocksdb ·
> GitHub|https://github.com/facebook/rocksdb/blob/main/db/write_batch.cc#L246-L247]),
> neither of which actually releases the memory built up by a std::string
> instance. The only way to actually release this memory is to delete the
> WriteBatch object itself.
> Currently, all memory taken by all write batches will remain until the
> RocksDB state store instance is closed, which never happens during the normal
> course of operation as all partitions remain loaded on an executor after a
> task completes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
