Hi, On GCP I'd go for buckets in Google Storage. Not sure how reliable it is in production deployments though. Only demo experience here.
Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski "The Internals Of" Online Books <https://books.japila.pl/> Follow me on https://twitter.com/jaceklaskowski <https://twitter.com/jaceklaskowski> On Mon, Mar 8, 2021 at 12:33 PM Ranju Jain <ranju.j...@ericsson.com> wrote: > Hi Jacek, > > > > I am using this property > spark.kubernetes.executor.deleteOnTermination=true only to troubleshoot > else I am freeing up resources after executors complete their job. > > Now I want to use some Shared storage which can be shared by all executors > to write the part files. > > Which Kubernetes Storage I should go for? > > > > Regards > > Ranju > > *From:* Jacek Laskowski <ja...@japila.pl> > *Sent:* Monday, March 8, 2021 4:14 PM > *To:* Ranju Jain <ranju.j...@ericsson.com.invalid> > *Cc:* Attila Zsolt Piros <piros.attila.zs...@gmail.com>; > user@spark.apache.org > *Subject:* Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor > Part Files Storage > > > > Hi, > > > > > as Executors terminates after their work completes. > > > > --conf spark.kubernetes.executor.deleteOnTermination=false ? > > > Pozdrawiam, > > Jacek Laskowski > > ---- > > https://about.me/JacekLaskowski > > "The Internals Of" Online Books > <https://protect2.fireeye.com/v1/url?k=901b36bb-cf800fbe-901b7620-86d2114eab2f-836d4bb779fe8f92&q=1&e=d3b471ef-b3ce-4cfd-9bcd-f42590d0f10b&u=https%3A%2F%2Fbooks.japila.pl%2F> > > Follow me on https://twitter.com/jaceklaskowski > > > <https://twitter.com/jaceklaskowski> > > > > > > On Sun, Mar 7, 2021 at 5:23 PM Ranju Jain <ranju.j...@ericsson.com.invalid> > wrote: > > Hi, > > > > I need to save the Executors processed data in the form of part files , > but I think persistent Volume is not an option for this as Executors > terminates after their work completes. > > So I am thinking to use shared volume across executor pods. > > > > Should I go with NFS or is there any other Volume option as well to > explore? > > > > Regards > > Ranju > >
