Kent Yao created SPARK-55993:
--------------------------------
Summary: Flaky RocksDBStateStoreIntegrationSuite: bounded memory
usage calculation
Key: SPARK-55993
URL: https://issues.apache.org/jira/browse/SPARK-55993
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Kent Yao
The test 'bounded memory usage calculation' in
RocksDBStateStoreIntegrationSuite is flaky due to a shared global state race
condition in RocksDBMemoryManager (singleton object).
Line 423 asserts getNumRocksDBInstances(false) == 0 (no unbounded instances),
but the streaming query may briefly register unbounded instances during
initialization before the bounded-memory config takes effect. Since
RocksDBMemoryManager.instanceMemoryMap is a global ConcurrentHashMap shared
across all tests in the JVM, other concurrent state store operations can also
register instances.
The fix should wrap the assertion in eventually to tolerate transient state, or
move it after processAllAvailable completes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
