sunchao commented on code in PR #6051: URL: https://github.com/apache/datafusion-comet/pull/6051#discussion_r4054250701
########## docs/source/user-guide/latest/tuning.md: ########## @@ -132,6 +138,31 @@ need to spill or have a single spillable operator. [shuffle]: #shuffle [Advanced Memory Tuning]: #advanced-memory-tuning +### Configuring Executor Memory Overhead + +Enabling off-heap memory is not sufficient on its own. It is recommended to also set +`spark.executor.memoryOverhead` when running Comet: + +``` +spark.executor.memoryOverhead=2g Review Comment: ### Correctness [P2] Could the `2g` recommendation preserve any larger calculated or configured overhead? In Spark 3.5 and 4.0, `ResourceProfile.calculateOverHeadMemory` uses the explicit value directly, so this replaces the default allowance rather than adding headroom. For example, a 32 GiB JVM executor has 3276 MiB overhead at the 0.1 factor. An 8 GiB Kubernetes cluster-mode Python executor also has 3276 MiB at its 0.4 default, which is an exception to the formula below. Setting `2g` reduces either allowance to 2048 MiB and reduces the calculated container allowance by 1228 MiB, while keeping heap and off-heap budgets unchanged. Comet's off-heap plugin path does not increase it afterward. A reader following the OOM guidance can therefore get less container headroom. Please scope `2g` to a small-executor example and tell readers to retain larger existing/default overhead and account for the applicable factor before adding their Comet margin. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
