[ https://issues.apache.org/jira/browse/SPARK-13628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15187419#comment-15187419 ]
Chen He commented on SPARK-13628: --------------------------------- Really appreciate your reply, Mr. Sean Owen. I am new to Spark and using it to talk to blobstore. We met a performance bottleneck that just as described in this issue. Would you mind provide more detail about the process or where can I find those details such as which class is doing the exactly rename and copy process. Thanks a lot! > Temporary intermediate output file should be renamed before copying to > destination filesystem > --------------------------------------------------------------------------------------------- > > Key: SPARK-13628 > URL: https://issues.apache.org/jira/browse/SPARK-13628 > Project: Spark > Issue Type: Improvement > Components: Input/Output > Affects Versions: 1.6.0 > Reporter: Chen He > > Spark Executor will dump temporary file into local temp dir, copy it to > destination filesystem, and then, rename it. It could be costly for Blobstore > (such as openstack swift) which do the actual copy when file is renamed. If > it does not affect other components, we may switch the sequence of copy and > rename so that Spark can use Blobstore as final output destination. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org