[ https://issues.apache.org/jira/browse/SPARK-11328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431275#comment-15431275 ]
Yin Huai commented on SPARK-11328: ---------------------------------- Actually, the JIRA is meant to provide more information in the error message (see https://github.com/apache/spark/pull/10080/files#diff-244a70a91841bbddf6fae17c14c18ce4R137). This jira does not aim to get rid of the file already exists error. > Provide more informative error message when direct parquet output committer > is used and there is a file already exists error. > ----------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-11328 > URL: https://issues.apache.org/jira/browse/SPARK-11328 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Yin Huai > Assignee: Nong Li > Priority: Critical > Fix For: 1.5.3, 1.6.0 > > > When saving data to S3 (e.g. saving to parquet), if there is an error during > the query execution, the partial file generated by the failed task will be > uploaded to S3 and the retries of this task will throw file already exist > error. It is very confusing to users because they may think that file already > exist error is the error causing the job failure. They can only find the real > error in the spark ui (in the stage page). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org