[
https://issues.apache.org/jira/browse/HADOOP-19938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095744#comment-18095744
]
Steve Loughran commented on HADOOP-19938:
-----------------------------------------
hadoop 3.4.x is on aws v2 sdk, that's a complete sdk rewrite, that'll be what
you need.
hadoop 3.4.0 doesn't ship a 1.x sdk at all, though if you add one it will try
to bridge across some parts of it.
for you, why not try swapping in the last v1 sdk to see if that helps
closing as cannot reproduce, sorry.
> S3A multipart upload fails at CompleteMultipartUpload when SSE-C encryption
> is enabled
> --------------------------------------------------------------------------------------
>
> Key: HADOOP-19938
> URL: https://issues.apache.org/jira/browse/HADOOP-19938
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.9.0, 3.0.0, 3.1.0
> Environment: Hadoop 3.1 Snapshot
> Reporter: Michael Johnson
> Assignee: Anis Elleuch
> Priority: Critical
>
> When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size
> to 5 Mb, storing data in AWS doesn't work anymore. For example, running the
> following code:
> {code}
> >>> df1 = spark.read.json('/home/user/people.json')
> >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json")
> {code}
> shows the following exception:
> {code:java}
> com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload
> initiate requested encryption. Subsequent part requests must include the
> appropriate encryption parameters.
> {code}
> After some investigation, I discovered that hadoop-aws doesn't send SSE-C
> headers in Put Object Part as stated in AWS specification:
> [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html]
> {code:java}
> If you requested server-side encryption using a customer-provided encryption
> key in your initiate multipart upload request, you must provide identical
> encryption information in each part upload using the following headers.
> {code}
>
> You can find a patch attached to this issue for a better clarification of the
> problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
