hi Battini,
The limit is a k8s construct that tells k8s how much cpu/cores your driver
*can* consume.
when you have the same value for 'spark.driver.cores' and '
spark.kubernetes.driver.limit.cores' your driver then runs at the
'Guranteed' k8s quality of service class, which can make your driver less
chance gets evicted by the scheduler.
The same goes with the executor settings.
https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/
the QoS Guarantee is important when you are a mutitenant k8s cluster in
production.
Cheers,
Li
On Thu, Mar 7, 2019, 1:53 PM Battini Lakshman
wrote:
> Hello,
>
> I understand we need to specify the 'spark.kubernetes.driver.limit.cores'
> and 'spark.kubernetes.executor.limit.cores' config parameters while
> submitting spark on k8s namespace with resource quota applied.
>
> There are also other config parameters 'spark.driver.cores' and
> 'spark.executor.cores' mentioned in documentation. What is the difference
> between '' and 'spark.kubernetes.driver.limit.cores' please.
>
> Thanks!
>
> Best Regards,
> Lakshman B.
>
