[ https://issues.apache.org/jira/browse/SPARK-38115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17496022#comment-17496022 ]
Steve Loughran commented on SPARK-38115: ---------------------------------------- bq. Is there any config as such to stop using FileOutputCommiter, because we didn't set any conf explicitly to use the committers. https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/committers.html bq. Just I am looking if I can use conf/options to manage temporary location as staging and have target path as primary no, because the commit-by-rename mechanism is broken on s3; tuning temp dir location isn't going to fix that > No spark conf to control the path of _temporary when writing to target > filesystem > --------------------------------------------------------------------------------- > > Key: SPARK-38115 > URL: https://issues.apache.org/jira/browse/SPARK-38115 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.4.8, 3.2.1 > Reporter: kk > Priority: Minor > Labels: spark, spark-conf, spark-sql, spark-submit > > No default spark conf or param to control the '_temporary' path when writing > to filesystem. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org