Hello Team, I am trying to write a DataSet as parquet file in Append mode partitioned by few columns. However since the job is time consuming, I would like to enable DirectFileOutputCommitter (i.e by-passing the writes to temporary folder).
Version of the spark i am using is 2.3.1. Can someone please help in enabling the configuration which allows direct write to S3 both in case of appending, writing new files and overwriting the files. Thanks, Padma CH
