dwsmith1983 opened a new issue, #7244: URL: https://github.com/apache/datafusion-comet/issues/7244
### Describe the bug Found in #6336 ([review comment](https://github.com/apache/datafusion-comet/pull/6336#discussion_r4368030205)). Two native memory paths treat an error from Spark during a memory request as if the pool had refused it, so the task carries on instead of failing. - The multi-partition shuffle repartitioner grows its batch reservation with `self.reservation.try_grow(mem_growth).is_err()` and spills on any error ([multi_partition.rs#L577](https://github.com/apache/datafusion-comet/blob/c4a0b1f2b/native/shuffle/src/partitioners/multi_partition.rs#L577)). When a Spark-backed pool fails because another consumer's `spill()` threw, the repartitioner spills and keeps going, where only a `ResourcesExhausted` refusal should lead to a spill. - An infallible `MemoryReservation::grow` or `resize` on the Spark-backed pools goes through `SparkMemory::acquire`, which logs a failed call to Spark as a warning and books the whole size as overcommit ([spark_memory.rs#L135-L151](https://github.com/apache/datafusion-comet/blob/c4a0b1f2b/native/core/src/execution/memory_pools/spark_memory.rs#L135-L151)). The next `try_grow` asks Spark for that debt too, so a repeated exception does surface, but a single one is only logged. ### Steps to reproduce Run a native shuffle with more than one output partition on a Spark-backed pool where another memory consumer's `spill()` throws while the repartitioner asks for memory. The shuffle spills and finishes rather than failing with that exception. ### Expected behavior An error other than `ResourcesExhausted` fails the task, the way #6336 handles the shuffle writer's own buffer requests. For the infallible `grow` path, the exception should at least reach the task instead of only the log. ### Additional context #6336 fixes the shuffle writer's buffer requests and leaves these two paths as they are on main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
