dwsmith1983 opened a new issue, #7174: URL: https://github.com/apache/datafusion-comet/issues/7174
`CometUnifiedMemoryPool` takes no lock, and `SparkMemory::try_acquire` reads the outstanding overcommit with a plain load before asking Spark for the request plus that debt ([spark_memory.rs#L111-L112](https://github.com/apache/datafusion-comet/blob/6a6d71c52f3452ac92ebe1c30c92c8515508f59e/native/core/src/execution/memory_pools/spark_memory.rs#L111-L112)). Two `try_grow` calls of one task that overlap both read the same debt and both put it into their Spark requests. Once the first repays it, the second request is larger than it needs to be, so Spark can refuse it or park it below the task's minimum share. Repro on main, in the `unified_pool.rs` tests. 50 bytes are owed and Spark has 60 free, enough for the debt plus both requests: ```rust let fake = FakeSpark::with(100); let pool: Arc<dyn MemoryPool> = Arc::new(CometUnifiedMemoryPool::with_spark(fake.memory())); let first = MemoryConsumer::new("first").register(&pool); let second = MemoryConsumer::new("second").register(&pool); first.grow(150); fake.set_limit(160); let (tx, rx) = std::sync::mpsc::channel(); // Lands while the first try_grow waits on Spark, after it has read the debt. fake.during_next_acquire(move || { let _ = tx.send(second.try_grow(1).map_err(|e| e.to_string())); }); let first_result = first.try_grow(5); rx.recv().unwrap().unwrap(); first_result.unwrap(); ``` The last line fails with `Task 0 failed to acquire 5 bytes plus 50 bytes overcommitted, only got 10`: Spark was asked for 106 bytes when 56 would have covered both. Found in #5613 ([review](https://github.com/apache/datafusion-comet/pull/5613#pullrequestreview-5629960270)). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
