andygrove commented on issue #6187: URL: https://github.com/apache/datafusion-comet/issues/6187#issuecomment-5850482265
Status after #6205. Item 1 is mostly resolved. `fair_unified` now really limits each consumer to its share, and the task's total to `pool_size`, so the per-consumer wording in the config doc and in both guides is accurate. What's left: - The "tighter whenever more than one task is running" sentences ([tuning/memory.md#L63-L65](https://github.com/apache/datafusion-comet/blob/bc4be39964cbe9cdb5f2a949740a8164e6b5755b/docs/source/user-guide/latest/tuning/memory.md?plain=1#L63-L65), [memory_management.md#L218-L221](https://github.com/apache/datafusion-comet/blob/bc4be39964cbe9cdb5f2a949740a8164e6b5755b/docs/source/contributor-guide/memory_management.md?plain=1#L218-L221)) are only true when N × fraction > 1. The contributor guide's version also compares Spark's per-task limit with the per-consumer share instead of the task's total. - [tuning/memory.md#L82-L84](https://github.com/apache/datafusion-comet/blob/bc4be39964cbe9cdb5f2a949740a8164e6b5755b/docs/source/user-guide/latest/tuning/memory.md?plain=1#L82-L84) and the config doc describe only the per-consumer share. They never say that `pool_size` is the fraction times `spark.memory.offHeap.size`, or that it also caps the task. Items 2 and 3 haven't changed. The deprecation warning still reads only the SparkConf, so it misses values set on the session and the extension-only path. `CometTaskMetricsSuite` still sets the fraction to 0.002 in three places. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
