Chen He created SPARK-13628: ------------------------------- Summary: Temporary intermediate output file should be renamed before copying to destination filesystem Key: SPARK-13628 URL: https://issues.apache.org/jira/browse/SPARK-13628 Project: Spark Issue Type: Improvement Components: Input/Output Affects Versions: 1.6.0 Reporter: Chen He
Spark Executor will dump temporary file into local temp dir, copy it to destination filesystem, and then, rename it. It could be costly for Blobstore (such as openstack swift) which do the actual copy when file is renamed. If it does not affect other components, we may switch the sequence of copy and rename so that Spark can use Blobstore as final output destination. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org