[
https://issues.apache.org/jira/browse/SPARK-35723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17970192#comment-17970192
]
Laurenceau Julien commented on SPARK-35723:
-------------------------------------------
It seems to me that all Kubernetes cluster used as backend to run spark
workload have awful resources utilization.
Even with more ~ 150% requests (including pending pods), it is common to
observe real usage as : 25% cpu and 50% memory.
This is even more stringent since Kubernetes does not allow overcommit at node
level like suggested in this issue : [Overriding Total Cpu capacity. I.e.,
support cpu overcommit · Issue #44328 ·
kubernetes/kubernetes|https://github.com/kubernetes/kubernetes/issues/44328]
It would be really helpful if users could set on their own kubernetes request
and limits for spark jobs as floating value in order to be able to have request
matching real cpu usage and not the number of requested cores.
> [K8s] Spark executors memory request should be allowed to deviate from limit
> ----------------------------------------------------------------------------
>
> Key: SPARK-35723
> URL: https://issues.apache.org/jira/browse/SPARK-35723
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.1.2
> Reporter: Christian Thiel
> Priority: Major
>
> Currently the driver and executor memor requests always equals the limit.
> As stated in [SPARK-23825|https://issues.apache.org/jira/browse/SPARK-23825],
> this is a reasonable default and is especially important for the driver.
> For executors however, it might be usefull for users to deviate from this
> default for executors.
> In typical development environments on K8s, the namespace quotas are an upper
> bound to the memory request that is possible.
> The limits however can be much higher. For development, spark is often run in
> client mode. While the driver should request the memory it needs, we want to
> leverage all the resources of the cluster whith executors if they are free -
> and can live with an executor maybe beeing killed eventually.
> Thus I propose the introduction of {{spark.{driver,executor}.limit.memory}}
> similar to the {{spark.{driver,executor}.limit.cpu}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
