[ https://issues.apache.org/jira/browse/SPARK-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved SPARK-11353. ------------------------------------ Resolution: Duplicate This is a duplicate of SPARK-13044; that's transitive a WONTFIX due to HADOOP-13325. The Hadoop project isn't going to upgrade jets3t as it will inevitably introduce a regression somewhere. Note also that moving to 0.9.4 lib needs changes in the org.apache.hadoop libraries, see HADOOP-11086. You cannot bump up the jets3t version and expect for all codepaths to work. If you want to auth with V4 APIs, use s3a:// URLs. > Writing to S3 buckets, which only support AWS4-HMAC-SHA256 fails with s3n URLs > ------------------------------------------------------------------------------ > > Key: SPARK-11353 > URL: https://issues.apache.org/jira/browse/SPARK-11353 > Project: Spark > Issue Type: Bug > Components: Input/Output > Affects Versions: 1.3.1, 1.5.1 > Reporter: Ćukasz Piepiora > > For certain regions like for example Frankfurt (eu-central-1) AWS supports > only [AWS Signature Version > 4|http://docs.aws.amazon.com/general/latest/gr/rande.html#d0e3788]. > Currently Spark is using jets3t library in version 0.9.3, which throws an > exception when code tries to save files in S3 in eu-central-1. > {code} > Caused by: java.lang.RuntimeException: Failed to automatically set required > header "x-amz-content-sha256" for request with entity > org.jets3t.service.impl.rest.httpclient.RepeatableRequestEntity@1e4bc601 > at > org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:238) > at > org.jets3t.service.impl.rest.httpclient.RestStorageService.authorizeHttpRequest(RestStorageService.java:762) > at > org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:324) > at > org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:277) > at > org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestPut(RestStorageService.java:1143) > at > org.jets3t.service.impl.rest.httpclient.RestStorageService.createObjectImpl(RestStorageService.java:1954) > at > org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectWithRequestEntityImpl(RestStorageService.java:1875) > at > org.jets3t.service.impl.rest.httpclient.RestStorageService.putObjectImpl(RestStorageService.java:1867) > at org.jets3t.service.StorageService.putObject(StorageService.java:840) > at org.jets3t.service.S3Service.putObject(S3Service.java:2212) > at org.jets3t.service.S3Service.putObject(S3Service.java:2356) > ... 23 more > Caused by: java.io.IOException: Stream closed > at > java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) > at java.io.BufferedInputStream.reset(BufferedInputStream.java:446) > at > org.jets3t.service.utils.SignatureUtils.awsV4GetOrCalculatePayloadHash(SignatureUtils.java:236) > ... 33 more > {code} > There is a newer version of jets3t 0.9.4, which seems to fix this issue > (http://www.jets3t.org/RELEASE_NOTES.html). > Therefore I suggest to upgrade jets3t dependency from 0.9.3 to 0.9.4 for > Hadoop profiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org