Re: How more than one spark job can write to same partition in the parquet file

Hey Chetan, I have not got your question. Are you trying to write to a partition from two actions ?? or you are looking for writing from two jobs. Except for maintaining the state for the dataset completeness in that case, I dont see any issues. We are writing data to a Partition using two

Re: How more than one spark job can write to same partition in the parquet file

We partitioned data logically for 2 different jobs...in our use case based on geography... On Thu, 12 Dec 2019 at 3:39 pm, Chetan Khatri wrote: > Thanks, If you can share alternative change in design. I would love to > hear from you. > > On Wed, Dec 11, 2019 at 9:34 PM ayan guha wrote: > >> No

Re: How more than one spark job can write to same partition in the parquet file

Thanks, If you can share alternative change in design. I would love to hear from you. On Wed, Dec 11, 2019 at 9:34 PM ayan guha wrote: > No we faced problem with that setup. > > On Thu, 12 Dec 2019 at 11:14 am, Chetan Khatri < > chetan.opensou...@gmail.com> wrote: > >> Hi Spark Users, >> would

Re: How more than one spark job can write to same partition in the parquet file

No we faced problem with that setup. On Thu, 12 Dec 2019 at 11:14 am, Chetan Khatri wrote: > Hi Spark Users, > would that be possible to write to same partition to the parquet file > through concurrent two spark jobs with different spark session. > > thanks > -- Best Regards, Ayan Guha

How more than one spark job can write to same partition in the parquet file

Hi Spark Users, would that be possible to write to same partition to the parquet file through concurrent two spark jobs with different spark session. thanks