and If you look at the code
https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194
.editOrNewResources()
.addToRequests("memory", executorMemoryQuantity)
.addToLimits("memory", executorMemoryQuantity)
.addToRequests("cpu", executorCpuQuantity)
.addToLimits(executorResourceQuantities.asJava)
.endResources()
addToRequests and addToLimits for memory have the same value.
maybe it is by design. but can I set custom values for them if I use
podtemplate?
Ismail Yenigul <[email protected]>, 10 Mar 2023 Cum, 20:52 tarihinde
şunu yazdı:
> Hi,
> using spark version v.3.1.2
>
> spark.executor.memory is set.
> But the problem is not setting spark.executor.memory, the problem is that
> whatever value I set spark.executor.memory,
> spark executor pod has the same value for resources.limit.memory and
> resources.request.memory.
> I want to be able to set different values for them.
>
>
>
>
> Mich Talebzadeh <[email protected]>, 10 Mar 2023 Cum, 20:44
> tarihinde şunu yazdı:
>
>> What are those currently set in spark-submit and which spark version on
>> k8s
>>
>> --conf spark.driver.memory=2000m \
>> --conf spark.executor.memory=2000m \
>>
>> HTH
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <[email protected]>
>> wrote:
>>
>>> Hi,
>>>
>>> There is a cpu parameter to set spark executor on k8s
>>> spark.kubernetes.executor.limit.cores and
>>> spark.kubernetes.executor.request.cores
>>> but there is no parameter to set memory request different then limits
>>> memory (such as spark.kubernetes.executor.request.memory)
>>> For that reason,
>>> spark.executor.memory is assigned to requests.memory and limits.memory
>>> like the following
>>>
>>> Limits:
>>> memory: 5734MiRequests:
>>> cpu: 4
>>> memory: 5734Mi
>>>
>>>
>>> Is there any special reason to not have
>>> spark.kubernetes.executor.request.memory parameter?
>>> and can I use spark.kubernetes.executor.podTemplateFile parameter to set
>>> smaller memory request than the memory limit in pod template file?
>>>
>>>
>>> Limits:
>>> memory: 5734MiRequests:
>>> cpu: 4
>>> memory: 1024Mi
>>>
>>>
>>> Thanks
>>>
>>>
