[ https://issues.apache.org/jira/browse/SPARK-43496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17750791#comment-17750791 ]
Laurenceau Julien commented on SPARK-43496: ------------------------------------------- Hi, I would like to suggest to go beyond that : As is spark on kubernetes always sets the following attributes to executors and driver pods: {code:java} requests: cpu: "1" memory: "1234M" limits: memory: "1234M" {code} with the values coming from the _[driver|executor]-memory_ and _[driver|executor]-cores_ options. This is good, except for resource utilization ! On most dev environment it's would be ok no to set any resource requests and limits. One could just rely on usual JVM options. Besides it would be nice if these settings could be user defined at different values than JVM values so that we would be able to manage overcommitment through it. Example: * options : _[driver|executor]-memory_ and _[driver|executor]-cores_ are used to set JVM options. If no custom definition it also sets pods requests & limits to stay consistent * new options: _spark.kubernetes.driver.requests.cpu_ _spark.kubernetes.driver.requests.memory_ _spark.kubernetes.driver.limits.cpu_ _spark.kubernetes.driver.limits.memory_ _spark.kubernetes.executor.requests.cpu_ _spark.kubernetes.executor.requests.memory_ _spark.kubernetes.executor.limits.cpu_ _spark.kubernetes.executor.limits.memory_ if unset then stay consistent with current behavior if set to 0 then disable this definition This would also solve the issue that driver/executor core is defined as an Integer and cannot be 0.5 for a driver. > Have a separate config for Memory limits for kubernetes pods > ------------------------------------------------------------ > > Key: SPARK-43496 > URL: https://issues.apache.org/jira/browse/SPARK-43496 > Project: Spark > Issue Type: Improvement > Components: Kubernetes > Affects Versions: 3.4.0 > Reporter: Alexander Yerenkow > Priority: Major > > Whole allocated memory to JVM is set into pod resources as both request and > limits. > This means there's not a way to use more memory for burst-like jobs in a > shared environment. > For example, if spark job uses external process (outside of JVM) to access > data, a bit of extra memory required for that, and having configured higher > limits for mem could be of use. > Another thought here - have a way to configure different JVM/ pod memory > request also could be a valid use case. > > Github PR: [https://github.com/apache/spark/pull/41067] > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org