Hi Jacek, I am using this property spark.kubernetes.executor.deleteOnTermination=true only to troubleshoot else I am freeing up resources after executors complete their job. Now I want to use some Shared storage which can be shared by all executors to write the part files. Which Kubernetes Storage I should go for?
Regards Ranju From: Jacek Laskowski <ja...@japila.pl> Sent: Monday, March 8, 2021 4:14 PM To: Ranju Jain <ranju.j...@ericsson.com.invalid> Cc: Attila Zsolt Piros <piros.attila.zs...@gmail.com>; user@spark.apache.org Subject: Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage Hi, > as Executors terminates after their work completes. --conf spark.kubernetes.executor.deleteOnTermination=false ? Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books<https://protect2.fireeye.com/v1/url?k=901b36bb-cf800fbe-901b7620-86d2114eab2f-836d4bb779fe8f92&q=1&e=d3b471ef-b3ce-4cfd-9bcd-f42590d0f10b&u=https%3A%2F%2Fbooks.japila.pl%2F> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Sun, Mar 7, 2021 at 5:23 PM Ranju Jain <ranju.j...@ericsson.com.invalid<mailto:ranju.j...@ericsson.com.invalid>> wrote: Hi, I need to save the Executors processed data in the form of part files , but I think persistent Volume is not an option for this as Executors terminates after their work completes. So I am thinking to use shared volume across executor pods. Should I go with NFS or is there any other Volume option as well to explore? Regards Ranju
