dwsmith1983 opened a new issue, #7175: URL: https://github.com/apache/datafusion-comet/issues/7175
When `try_acquire` gets a short grant it hands the bytes back with `self.manager.release(granted)?` ([spark_memory.rs#L117](https://github.com/apache/datafusion-comet/blob/6a6d71c52f3452ac92ebe1c30c92c8515508f59e/native/core/src/execution/memory_pools/spark_memory.rs#L117)). If that JNI call fails, the `?` returns the error from `try_grow` ([fair_pool.rs#L180](https://github.com/apache/datafusion-comet/blob/6a6d71c52f3452ac92ebe1c30c92c8515508f59e/native/core/src/execution/memory_pools/fair_pool.rs#L180), [unified_pool.rs#L120](https://github.com/apache/datafusion-comet/blob/6a6d71c52f3452ac92ebe1c30c92c8515508f59e/native/core/src/execution/memory_pools/unified_pool.rs#L120)). Spark, and `CometTaskMemoryManager.used`, still hold the granted bytes until the task ends, but the pool counts none of them, so its `reserved()` is lower than what the task holds and later grows are checked against too small a total. Keeping the bytes charged when the hand-back fails, with a warning, would keep both sides equal while still returning the short-grant error so the operator spills. A test needs a `SparkMemoryManager` fake whose `release` can fail. Found in #5613 ([review](https://github.com/apache/datafusion-comet/pull/5613#pullrequestreview-5629960270)). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
