Hello,
I have a DataFrame, with a date column which I want to use as a partition.
Each day I want to write the data for the same date in Parquet, and then
read a dataframe for a date range.
I'm using:
myDataframe.write().partitionBy("date").mode(SaveMode.Overwrite).parquet(parquetDir);
If I use SaveMode.Append, then writing data for the same partition adds the
same data there again.
If I use SaveMode.Overwrite, then writing data for a single partition
removes all the data for all partitions.
How can I overwrite only a given partition or manually remove a partition
before writing?
Many thanks!
Romi K.
