[ https://issues.apache.org/jira/browse/SPARK-38115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17488445#comment-17488445 ]
kk commented on SPARK-38115: ---------------------------- Thanks [~hyukjin.kwon] for responding. Basically I am trying to write data to s3 from spark dataframe. And this will use FileOutputCommitter by spark. [https://stackoverflow.com/questions/46665299/spark-avoid-creating-temporary-directory-in-s3] Now my requirement is to either change the '{*}_temporary{*}' path to write to different s3 bucket and copy to original s3 by setting any spark conf or parameter part of write step. or stop creating *_temporary* when writing to s3. As we have version enabled bucket the _temporary is being stored in the version even though it is not physically present. Below is the write step: df.coalesce(1).write.format('parquet').mode('overwrite').save('{*}s3a{*}://outpath') > No spark conf to control the path of _temporary when writing to target > filesystem > --------------------------------------------------------------------------------- > > Key: SPARK-38115 > URL: https://issues.apache.org/jira/browse/SPARK-38115 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.4.8, 3.2.1 > Reporter: kk > Priority: Minor > Labels: spark, spark-conf, spark-sql, spark-submit > > No default spark conf or param to control the '_temporary' path when writing > to filesystem. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org