Re: Re: [VOTE] Release Apache Spark 3.4.0 (RC4)

Thank you @beliefer. On Sat, Mar 11, 2023 at 9:54 AM beliefer wrote: > There is a bug fix. > > https://issues.apache.org/jira/browse/SPARK-42740 > > > > 在 2023-03-10 20:48:30，"Xinrong Meng" 写道： > > https://issues.apache.org/jira/browse/SPARK-42745 can be a new release > blocker, thanks @Peter

Re:Re: [VOTE] Release Apache Spark 3.4.0 (RC4)

There is a bug fix. https://issues.apache.org/jira/browse/SPARK-42740 在 2023-03-10 20:48:30，"Xinrong Meng" 写道： https://issues.apache.org/jira/browse/SPARK-42745 can be a new release blocker, thanks @Peter Toth for reporting that. On Fri, Mar 10, 2023 at 8:21 PM Xinrong Meng wrote:

Re: spark executor pod has same memory value for request and limit

agreed. need to be enhanced! HTH view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other

Re: spark executor pod has same memory value for request and limit

Hi Mich, The issue is here there is no parameter to set executor pod request memory value. Currently we have only one parameter which is spark.executor.memory and it set pod resources limit and requests. Mich Talebzadeh , 10 Mar 2023 Cum, 22:04 tarihinde şunu yazdı: > Yes, both EKS and GKE

Re: spark executor pod has same memory value for request and limit

It means, my requested feature is not available in the latest branch too ;) Bjørn Jørgensen , 10 Mar 2023 Cum, 21:05 tarihinde şunu yazdı: > Strange to see that you are using spark 3.1.2 witch is EOL and you are > reading source files from 3.4.0-SNAPSHOT > > fre. 10. mar. 2023 kl. 19:01 skrev

Re: spark executor pod has same memory value for request and limit

I forgot top ask which k8s cluster are you using, assuming some clod vendor view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss,

Re: spark executor pod has same memory value for request and limit

Strange to see that you are using spark 3.1.2 witch is EOL and you are reading source files from 3.4.0-SNAPSHOT fre. 10. mar. 2023 kl. 19:01 skrev Ismail Yenigul : > and If you look at the code > > >

Re: spark executor pod has same memory value for request and limit

and If you look at the code https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194 .editOrNewResources() .addToRequests("memory",

Re: spark executor pod has same memory value for request and limit

Hi, using spark version v.3.1.2 spark.executor.memory is set. But the problem is not setting spark.executor.memory, the problem is that whatever value I set spark.executor.memory, spark executor pod has the same value for resources.limit.memory and resources.request.memory. I want to be able to

Re: spark executor pod has same memory value for request and limit

What are those currently set in spark-submit and which spark version on k8s --conf spark.driver.memory=2000m \ --conf spark.executor.memory=2000m \ HTH view my Linkedin profile

spark executor pod has same memory value for request and limit

Hi, There is a cpu parameter to set spark executor on k8s spark.kubernetes.executor.limit.cores and spark.kubernetes.executor.request.cores but there is no parameter to set memory request different then limits memory (such as spark.kubernetes.executor.request.memory) For that reason,

Unsubscribe

unsubscribe -- Gary Liu

Re: [VOTE] Release Apache Spark 3.4.0 (RC4)

https://issues.apache.org/jira/browse/SPARK-42745 can be a new release blocker, thanks @Peter Toth for reporting that. On Fri, Mar 10, 2023 at 8:21 PM Xinrong Meng wrote: > Please vote on releasing the following candidate(RC4) as Apache Spark > version 3.4.0. > > The vote is open until 11:59pm

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

Hi Peter, Thank you for raising that! Unfortunately, v3.4.0-rc4 has been cut already. On Fri, Mar 10, 2023 at 8:15 PM Peter Toth wrote: > Hi Xinrong, > > I've opened a PR to fix a regression from 3.3 to 3.4: > https://github.com/apache/spark/pull/40364 > Please wait with the RC4 cut if

[VOTE] Release Apache Spark 3.4.0 (RC4)

Please vote on releasing the following candidate(RC4) as Apache Spark version 3.4.0. The vote is open until 11:59pm Pacific time *March 15th* and passes if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. [ ] +1 Release this package as Apache Spark 3.4.0 [ ] -1 Do not release this

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

Hi Xinrong, I've opened a PR to fix a regression from 3.3 to 3.4: https://github.com/apache/spark/pull/40364 Please wait with the RC4 cut if possible. Thanks, Peter Xinrong Meng ezt írta (időpont: 2023. márc. 10., P, 0:07): > Thank you Hyukjin! :) > > I would prefer to cut v3.4.0-rc4 now if

Re: [VOTE] Release Apache Spark 3.4.0 (RC3)

Other than the tag issue, the sigs/artifacts/build/etc worked for me. So the next RC candidate looks promising ! Regards, Mridul On Thu, Mar 9, 2023 at 5:07 PM Xinrong Meng wrote: > Thank you Hyukjin! :) > > I would prefer to cut v3.4.0-rc4 now if there are no objections. > > On Fri, Mar 10,