It means, my requested feature is not available in the latest branch too ;)
Bjørn Jørgensen <bjornjorgen...@gmail.com>, 10 Mar 2023 Cum, 21:05 tarihinde şunu yazdı: > Strange to see that you are using spark 3.1.2 witch is EOL and you are > reading source files from 3.4.0-SNAPSHOT > > fre. 10. mar. 2023 kl. 19:01 skrev Ismail Yenigul <ismailyeni...@gmail.com > >: > >> and If you look at the code >> >> >> https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194 >> >> .editOrNewResources() >> .addToRequests("memory", executorMemoryQuantity) >> .addToLimits("memory", executorMemoryQuantity) >> .addToRequests("cpu", executorCpuQuantity) >> .addToLimits(executorResourceQuantities.asJava) >> .endResources() >> >> addToRequests and addToLimits for memory have the same value. >> maybe it is by design. but can I set custom values for them if I use >> podtemplate? >> >> >> >> Ismail Yenigul <ismailyeni...@gmail.com>, 10 Mar 2023 Cum, 20:52 >> tarihinde şunu yazdı: >> >>> Hi, >>> using spark version v.3.1.2 >>> >>> spark.executor.memory is set. >>> But the problem is not setting spark.executor.memory, the problem is >>> that whatever value I set spark.executor.memory, >>> spark executor pod has the same value for resources.limit.memory and >>> resources.request.memory. >>> I want to be able to set different values for them. >>> >>> >>> >>> >>> Mich Talebzadeh <mich.talebza...@gmail.com>, 10 Mar 2023 Cum, 20:44 >>> tarihinde şunu yazdı: >>> >>>> What are those currently set in spark-submit and which spark version on >>>> k8s >>>> >>>> --conf spark.driver.memory=2000m \ >>>> --conf spark.executor.memory=2000m \ >>>> >>>> HTH >>>> >>>> >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> >>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>> any loss, damage or destruction of data or any other property which may >>>> arise from relying on this email's technical content is explicitly >>>> disclaimed. The author will in no case be liable for any monetary damages >>>> arising from such loss, damage or destruction. >>>> >>>> >>>> >>>> >>>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <ismailyeni...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> There is a cpu parameter to set spark executor on k8s >>>>> spark.kubernetes.executor.limit.cores and >>>>> spark.kubernetes.executor.request.cores >>>>> but there is no parameter to set memory request different then limits >>>>> memory (such as spark.kubernetes.executor.request.memory) >>>>> For that reason, >>>>> spark.executor.memory is assigned to requests.memory and >>>>> limits.memory like the following >>>>> >>>>> Limits: >>>>> memory: 5734MiRequests: >>>>> cpu: 4 >>>>> memory: 5734Mi >>>>> >>>>> >>>>> Is there any special reason to not have >>>>> spark.kubernetes.executor.request.memory parameter? >>>>> and can I use spark.kubernetes.executor.podTemplateFile parameter to >>>>> set smaller memory request than the memory limit in pod template file? >>>>> >>>>> >>>>> Limits: >>>>> memory: 5734MiRequests: >>>>> cpu: 4 >>>>> memory: 1024Mi >>>>> >>>>> >>>>> Thanks >>>>> >>>>> > > -- > Bjørn Jørgensen > Vestre Aspehaug 4, 6010 Ålesund > Norge > > +47 480 94 297 >
