andygrove commented on PR #6053: URL: https://github.com/apache/datafusion-comet/pull/6053#issuecomment-5744807609
Closing in favor of #6054. Extending the adjustment to off-heap mode is the wrong move once you look at whether the adjustment lands anywhere. `ResourceProfileManager` materializes the default `ResourceProfile` in its constructor, snapshotting `spark.executor.memoryOverhead` into it, and the cluster managers size executors from that profile rather than re-reading the conf. On Spark 3.4, 3.5 and 4.0 that happens before `SparkContext` creates the plugin container, so the driver plugin's mutation has never reached the executor container on those versions — it only works on 4.1 and 4.2. So rather than adding a second mode to a mechanism that silently works on two versions and silently does nothing on three, #6054 removes it and warns at startup instead, which is the only thing that works everywhere. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
