Multipart uploads to Amazon S3 from Apache Spark
Cross posting an interesting question on Stack Overflow http://stackoverflow.com/questions/26321947/multipart-uploads-to-amazon-s3-from-apache-spark . Nick -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Multipart-uploads-to-Amazon-S3-from-Apache-Spark-tp16315.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Multipart uploads to Amazon S3 from Apache Spark
Not directly related, but FWIW, EMR seems to back away from s3n usage: Previously, Amazon EMR used the S3 Native FileSystem with the URI scheme, s3n. While this still works, we recommend that you use the s3 URI scheme for the best performance, security, and reliability. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-file-systems.html On Mon, Oct 13, 2014 at 1:42 PM, Nick Chammas nicholas.cham...@gmail.com wrote: Cross posting an interesting question on Stack Overflow http://stackoverflow.com/questions/26321947/multipart-uploads-to-amazon-s3-from-apache-spark . Nick -- View this message in context: Multipart uploads to Amazon S3 from Apache Spark http://apache-spark-user-list.1001560.n3.nabble.com/Multipart-uploads-to-Amazon-S3-from-Apache-Spark-tp16315.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
Re: Multipart uploads to Amazon S3 from Apache Spark
Oh, that's a straight reversal from their position up until earlier this year http://apache-spark-user-list.1001560.n3.nabble.com/How-to-read-a-multipart-s3-file-tp5463p5485.html . Was there an announcement explaining the change in recommendation? Nick On Mon, Oct 13, 2014 at 4:54 PM, Daniil Osipov daniil.osi...@shazam.com wrote: Not directly related, but FWIW, EMR seems to back away from s3n usage: Previously, Amazon EMR used the S3 Native FileSystem with the URI scheme, s3n. While this still works, we recommend that you use the s3 URI scheme for the best performance, security, and reliability. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-file-systems.html On Mon, Oct 13, 2014 at 1:42 PM, Nick Chammas nicholas.cham...@gmail.com wrote: Cross posting an interesting question on Stack Overflow http://stackoverflow.com/questions/26321947/multipart-uploads-to-amazon-s3-from-apache-spark . Nick -- View this message in context: Multipart uploads to Amazon S3 from Apache Spark http://apache-spark-user-list.1001560.n3.nabble.com/Multipart-uploads-to-Amazon-S3-from-Apache-Spark-tp16315.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
