For example, this app just reads a 4GB file and writes a copy of it. It takes 41 seconds to write the file, then 3 more minutes to move all the temporary files.
I guess this is an issue with the hadoop / jets3t code layer, not Spark. 14/05/06 20:11:41 INFO TaskSetManager: Finished TID 63 in 8688 ms on ip-10-143-138-33.ec2.internal (progress: 63/63) 14/05/06 20:11:41 INFO DAGScheduler: Stage 0 (saveAsTextFile at FileCopy.scala:17) finished in 41.326 s 14/05/06 20:11:41 INFO SparkContext: Job finished: saveAsTextFile at FileCopy.scala:17, took 41.605480454 s 14/05/06 20:14:48 INFO NativeS3FileSystem: OutputStream for key 'dad-20140101-9M.copy/_SUCCESS' writing to tempfile '/tmp/hadoop-root/s3/output-1223846975509014265.tmp' 14/05/06 20:14:48 INFO NativeS3FileSystem: OutputStream for key 'dad-20140101-9M.copy/_SUCCESS' closed. Now beginning upload 14/05/06 20:14:48 INFO NativeS3FileSystem: OutputStream for key 'dad-20140101-9M.copy/_SUCCESS' upload complete 14/05/06 20:14:48 INFO SparkDeploySchedulerBackend: Shutting down all executors -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-a-multipart-s3-file-tp5463p5473.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
