[ https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anish Shrigondekar updated SPARK-43311: --------------------------------------- Description: Today when RocksDB is used as a State Store provider, memory usage when writing using writeBatch is not capped. Also, a related issue is that the state store coordinator can create multiple RocksDB instances on a single node without enforcing a global limit on native memory usage. Due to these issues we could run into OOM issues and task failures. We are looking to improve this behavior by doing a series of improvements such as: * remove writeBatch and use native RocksDB operations * use writeBufferManager to manage global limit for all instances on a single node and accounting memtable + filter/index blocks usage as part of block cacheWith these changes we will be avoiding OOM issues around RocksDB native memory usage. was:RocksDB state store provider memory management enhancements > RocksDB state store provider memory management enhancements > ----------------------------------------------------------- > > Key: SPARK-43311 > URL: https://issues.apache.org/jira/browse/SPARK-43311 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 3.4.0 > Reporter: Anish Shrigondekar > Priority: Major > > Today when RocksDB is used as a State Store provider, memory usage when > writing using writeBatch is not capped. Also, a related issue is that the > state store coordinator can create multiple RocksDB instances on a single > node without enforcing a global limit on native memory usage. Due to these > issues we could run into OOM issues and task failures. > > We are looking to improve this behavior by doing a series of improvements > such as: > * remove writeBatch and use native RocksDB operations > * use writeBufferManager to manage global limit for all instances on a > single node and accounting memtable + filter/index blocks usage as part of > block cacheWith these changes we will be avoiding OOM issues around RocksDB > native memory usage. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org