[ https://issues.apache.org/jira/browse/SPARK-11328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yin Huai updated SPARK-11328: ----------------------------- Summary: Provide more informative error message when direct parquet output committer is used and there is a file already exists error. (was: Correctly propagate error message in the case of failures when writing parquet) > Provide more informative error message when direct parquet output committer > is used and there is a file already exists error. > ----------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-11328 > URL: https://issues.apache.org/jira/browse/SPARK-11328 > Project: Spark > Issue Type: Improvement > Components: SQL > Reporter: Yin Huai > Assignee: Nong Li > Priority: Critical > Fix For: 1.5.3, 1.6.0 > > > When saving data to S3 (e.g. saving to parquet), if there is an error during > the query execution, the partial file generated by the failed task will be > uploaded to S3 and the retries of this task will throw file already exist > error. It is very confusing to users because they may think that file already > exist error is the error causing the job failure. They can only find the real > error in the spark ui (in the stage page). -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org