andygrove opened a new issue, #6185: URL: https://github.com/apache/datafusion-comet/issues/6185
### Describe the bug Spark reads `spark.memory.offHeap.size` as bytes unless a unit is given. `CometExecIterator.getMemoryConfig` reads it with `SparkConf.getSizeAsMb`, which treats a bare number as MiB ([CometExecIterator.scala#L617](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/spark/src/main/scala/org/apache/comet/CometExecIterator.scala#L617)). The native memory usage log in the same file reads it correctly with `getSizeAsBytes` ([CometExecIterator.scala#L498](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/spark/src/main/scala/org/apache/comet/CometExecIterator.scala#L498)). With `spark.memory.offHeap.size=4294967296`, Spark's off-heap pool is 4 GiB, but Comet computes `memory_limit` as 4294967296 MiB, about 4 PiB. The `fair_unified` pool caps each task at `memory_limit / num_consumers`, so the cap never binds: `fair_unified` behaves like `greedy_unified`, and `spark.comet.exec.memoryPool.fraction` has no effect. Spark still enforces its own pool size, so Comet cannot acquire more than 4 GiB. ### Steps to reproduce Call `CometExecIterator.getMemoryConfig` with `spark.memory.offHeap.enabled=true` and `spark.memory.offHeap.size=4294967296`. `memoryLimit` comes back as 4294967296 x 2^20 bytes instead of 4294967296. ### Expected behavior `memoryLimit` is 4 GiB, matching Spark's pool. ### Additional context Fix: read the value with `getSizeAsBytes`, and add a test that uses a bare byte count. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
