[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing

Kimahriman commented on PR #38853: URL: https://github.com/apache/spark/pull/38853#issuecomment-1333651886 > Just to give you more context for your previous comment ([#38853 (comment)](https://github.com/apache/spark/pull/38853#issuecomment-1333073885))... > > We have

[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing

Kimahriman commented on PR #38853: URL: https://github.com/apache/spark/pull/38853#issuecomment-1333646333 > (@Kimahriman Looks like your git is not correctly set in your dev. I manually changed the author of commit based on your mail address in dev@ mailing list.) Thanks, I did this

[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing

Kimahriman commented on PR #38853: URL: https://github.com/apache/spark/pull/38853#issuecomment-1333073885 One question I did have, as I started digging into this I thought the problem was simply related to https://issues.apache.org/jira/browse/SPARK-38277, but as I dug into it I realized

[GitHub] [spark] Kimahriman commented on pull request #38853: [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing

Kimahriman commented on PR #38853: URL: https://github.com/apache/spark/pull/38853#issuecomment-1333068984 > 1. Have you run the fix with your production workload for a while and see there is no longer the same memory issue? Yes I confirmed that today. I have executors with a 25 GiB heap