andygrove opened a new issue, #6257: URL: https://github.com/apache/datafusion-comet/issues/6257
### What is the problem the feature request solves? Native operators ask for memory with `try_grow` until they are refused, and then they spill. A partial grant from Spark is the normal spill signal, not an error. But `CometTaskMemoryManager.acquireMemory` logs a warning and calls `TaskMemoryManager.showMemoryUsage()` every time Spark grants less than was asked ([CometTaskMemoryManager.java#L74-L85](https://github.com/apache/datafusion-comet/blob/bc4be39964cbe9cdb5f2a949740a8164e6b5755b/spark/src/main/java/org/apache/spark/CometTaskMemoryManager.java#L74-L85)), and `showMemoryUsage()` logs another line per consumer at INFO. In a four-task aggregate with 96 MB of off-heap memory, that was 257 warnings in about five seconds. A query that spills heavily keeps producing them for its whole run. That buries real warnings and makes a healthy spill look like a failure. ### Describe the potential solution Log the partial grant at DEBUG, or at most once per task at INFO. Nothing is lost when a reservation really does fail: the error the pool returns already says how much was granted and lists the top consumers. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
