Never mind, I noticed that mode=ignore doesn't write anything if target
path exists. Even if files previously exist only in different partitions
than the one's being written to.
So, ignore mode can't be used to mitigate the FileAlreadyExistsException
problem of append mode..
On Thu, May 9, 2019
Does spark handle 'ignore' mode on file level or partition level?
My code is like this:
df.write \
.option('mapreduce.fileoutputcommitter.algorithm.version', '2') \
.mode('ignore') \
.partitionBy('p') \
.orc(target_path)
When I used mode('append') my job
