Re: spark executor pod has same memory value for request and limit

Talebzadeh Sent: Friday, March 10, 2023 20:25 To: Ismail Yenigul Cc: dev@spark.apache.org Subject: Re: spark executor pod has same memory value for request and limit EXTERNAL SENDER. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide

Re: spark executor pod has same memory value for request and limit

agreed. need to be enhanced! HTH view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other

Re: spark executor pod has same memory value for request and limit

Hi Mich, The issue is here there is no parameter to set executor pod request memory value. Currently we have only one parameter which is spark.executor.memory and it set pod resources limit and requests. Mich Talebzadeh , 10 Mar 2023 Cum, 22:04 tarihinde şunu yazdı: > Yes, both EKS and GKE

Re: spark executor pod has same memory value for request and limit

It means, my requested feature is not available in the latest branch too ;) Bjørn Jørgensen , 10 Mar 2023 Cum, 21:05 tarihinde şunu yazdı: > Strange to see that you are using spark 3.1.2 witch is EOL and you are > reading source files from 3.4.0-SNAPSHOT > > fre. 10. mar. 2023 kl. 19:01 skrev

Re: spark executor pod has same memory value for request and limit

I forgot top ask which k8s cluster are you using, assuming some clod vendor view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss,

Re: spark executor pod has same memory value for request and limit

Strange to see that you are using spark 3.1.2 witch is EOL and you are reading source files from 3.4.0-SNAPSHOT fre. 10. mar. 2023 kl. 19:01 skrev Ismail Yenigul : > and If you look at the code > > >

Re: spark executor pod has same memory value for request and limit

and If you look at the code https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194 .editOrNewResources() .addToRequests("memory",

Re: spark executor pod has same memory value for request and limit

Hi, using spark version v.3.1.2 spark.executor.memory is set. But the problem is not setting spark.executor.memory, the problem is that whatever value I set spark.executor.memory, spark executor pod has the same value for resources.limit.memory and resources.request.memory. I want to be able to

Re: spark executor pod has same memory value for request and limit

What are those currently set in spark-submit and which spark version on k8s --conf spark.driver.memory=2000m \ --conf spark.executor.memory=2000m \ HTH view my Linkedin profile

spark executor pod has same memory value for request and limit

Hi, There is a cpu parameter to set spark executor on k8s spark.kubernetes.executor.limit.cores and spark.kubernetes.executor.request.cores but there is no parameter to set memory request different then limits memory (such as spark.kubernetes.executor.request.memory) For that reason,