Re: Spark on kubernetes : missing spark.kubernetes.driver.request.cores parameter ?
I am actually answering myself as I have check on master 3.x branch, and there is this feature ! https://issues.apache.org/jira/browse/SPARK-27754 So my understanding was correct. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Spark on kubernetes : missing spark.kubernetes.driver.request.cores parameter ?
Hello all, I am surprised that it is not possible to define "spark.kubernetes.driver.request.cores" when submitting a spark job on kubernetes. My understanding is that it would limit the CPU requests for the driver on the k8s cluster and we could still define how many cores (threads) we use in the driver with spark.driver.cores. In the 2.4 branch, the request is then defined by "spark.driver.cores", so we cannot throttle the driver. Is my understanding wrong or I missed something ? Thanks Christophe -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Spark Profiler
Hello Jack, You can also have a look at “Babar”, there is a nice “flame graph” feature too. I haven’t had the time to test it out. https://github.com/criteo/babar JC -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
