RE: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage

; Attila Zsolt Piros ; user@spark.apache.org Subject: Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage Hi Ranju, In your statement: "What is the best shared storage can be used to collate all executors part files at one place." Are you looking for p

Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage

hat is the best shared storage can be used to collate all executors part > files at one place. > > > > Regards > > Ranju > > > > *From:* Mich Talebzadeh > *Sent:* Monday, March 8, 2021 8:06 PM > *To:* Ranju Jain > *Cc:* Attila Zsolt Piros ; > user@

RE: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage

Subject: Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage If the purpose is to use for temporary work and write put it in temporary sub-directory under a give bucket spark.conf.set("temporaryGcsBucket", config['GCPVariables']['tmp_bucket']) That dict

Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage

If the purpose is to use for temporary work and write put it in temporary sub-directory under a give bucket spark.conf.set("temporaryGcsBucket", config['GCPVariables']['tmp_bucket']) That dict reference is to this yml file entry CPVariables: tmp_bucket: "tmp_storage_bucket/tmp" just create

Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage

Monday, March 8, 2021 4:14 PM > *To:* Ranju Jain > *Cc:* Attila Zsolt Piros ; > user@spark.apache.org > *Subject:* Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor > Part Files Storage > > > > Hi, > > > > > as Executors terminates after t

RE: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage

Storage I should go for? Regards Ranju From: Jacek Laskowski Sent: Monday, March 8, 2021 4:14 PM To: Ranju Jain Cc: Attila Zsolt Piros ; user@spark.apache.org Subject: Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage Hi, > as Executors terminates after their w

Re: Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage

Hi, > as Executors terminates after their work completes. --conf spark.kubernetes.executor.deleteOnTermination=false ? Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski "The Internals Of" Online Books Follow me on https://twitter.com/jaceklaskowski

Spark 3.0.1 | Volume to use For Spark Kubernetes Executor Part Files Storage

Hi, I need to save the Executors processed data in the form of part files , but I think persistent Volume is not an option for this as Executors terminates after their work completes. So I am thinking to use shared volume across executor pods. Should I go with NFS or is there any other Volume