I had similar problems with multi part uploads. In my case the real error was something else which was being masked by this issue https://issues.apache.org/jira/browse/SPARK-6560. In the end this bad digest exception was a side effect and not the original issue. For me it was some library version conflict on EMR.
Depending on the size of the output files, you might try to just disable the multipart upload using fs.s3n.multipart.uploads.enabled Cheers, Eugen 2016-02-07 15:05 GMT-08:00 Steve Loughran <ste...@hortonworks.com>: > > > On 7 Feb 2016, at 07:57, Dhimant <dhimant84.jays...@gmail.com> wrote: > > > > at > > > com.amazon.ws.emr.hadoop.fs.s3n.MultipartUploadOutputStream.uploadSinglePart(MultipartUploadOutputStream.java:245) > > ... 15 more > > Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The > > Content-MD5 you specified did not match what we received. (Service: > Amazon > > S3; Status Code: 400; Error Code: BadDigest; Request ID: > 5918216A5901FCC8), > > S3 Extended Request ID: > > > QSxtYln/yXqHYpdr4BWosin/TAFsGlK1FlKfE5PcuJkNrgoblGzTNt74kEhuNcrJCRZ3mXq0oUo= > > at > > > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) > > at > > > com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) > > at > > > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) > > at > > com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) > > at > > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3796) > > at > > > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1482) > > at > > > com.amazon.ws.emr.hadoop.fs.s3n.Jets3tNativeFileSystemStore.storeFile(Jets3tNativeFileSystemStore.java:140) > > ... 22 more > > > > This is amazon's own s3 client. nothing in the apache hadoop source tree. > Normally I'd say "use s3a to make s3n problems go away", but I don't know > what that does on Amazon's own EMR libraries > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
