Whereas with s3://, the write takes 32 seconds and the rename takes 33 seconds:
14/05/06 20:23:08 INFO DAGScheduler: Stage 0 (saveAsTextFile at FileCopy.scala:17) finished in 32.208 s 14/05/06 20:23:08 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 14/05/06 20:23:08 INFO SparkContext: Job finished: saveAsTextFile at FileCopy.scala:17, took 32.430051749 s 14/05/06 20:23:41 INFO SparkDeploySchedulerBackend: Shutting down all executors -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-a-multipart-s3-file-tp5463p5474.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
