Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

onfiguration property >> defaults to spark.executor.cores, which defaults to 1. >> >> On Fri, Mar 30, 2018, 3:03 PM Kimoon Kim <kim...@pepperdata.com> wrote: >> >>> David, glad it helped! And thanks for your clear example. >>> >>> > The o

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

r > *spark.kubernetes.executor.cores > *would be. Seeing that I wanted more than 1 and Yinan wants less, leaving > it at 1 night be best. > > > > Thanks, > > David > > > > *From: *Kimoon Kim <kim...@pepperdata.com> > *Date: *Friday, March 3

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

o be equal to spark.task.cpus. > > A relevant question is should Spark on Kubernetes really be opinionated on > how to set the cpu request and limit and even try to determine this > automatically? > > On Fri, Mar 30, 2018 at 11:40 AM, Kimoon Kim <kim...@pepperdata.com> &g

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

> Instead of requesting `[driver,executor].memory`, we should just request `[driver,executor].memory + [driver,executor].memoryOverhead `. I think this case is a bit clearer than the CPU case, so I went ahead and filed an issue with more details

Re: Toward an "API" for spark images used by the Kubernetes back-end

Thanks for starting this discussion. When I was troubleshooting Spark on K8s, I often faced a need to turn on debug messages on the driver and executor pods of my jobs, which would be possible if I somehow put the right log4j.properties file inside the pods. I know I can build custom Docker

Re: SPIP: Spark on Kubernetes

+1 (non-binding) Thanks, Kimoon On Tue, Aug 15, 2017 at 9:19 AM, Sean Suchter wrote: > +1 (non-binding) > > > > -- > View this message in context: http://apache-spark- > developers-list.1001551.n3.nabble.com/SPIP-Spark-on- > Kubernetes-tp22147p22150.html > Sent