Casey,
I am not sure if this is related, but I cannot seem to retrieve files that
are 524288001 bytes (5MB + 1 byte) or 629145601 bytes (600MB + 1 byte) when
using server side encryption. Without encryption, these files store and
retrieve without issue. I'm sure there are other various permutations of
lengths that would result in this issue as well. I still find it puzzling
that after ingesting 167 million files in this manner, I am discovering
this issue.
-Dan
On Mon, Mar 18, 2019 at 12:38 PM Dan Smith <dan.smith.11221...@gmail.com>
wrote:
> Hi Casey,
>
> Thanks for the quick response. I have just confirmed that if I set the
> PartSize to 500MB the file uploads correctly. I am hesitant to do this in
> production but I think we are on the right track. Interestingly enough,
> when I set the PartSize to 5242880 the file did not store correctly (it
> retrieved with a different hash, and not even the same different hash I saw
> before).
>
> Here are my details of my client. Please forgive me for not seeing how to
> attach this to the tracker item you mentioned in your email.
>
> I am using the AWSSDK.S3 nuget package version 3.3.31.24, with ceph
> version "ceph version 12.2.10-551-gbb089269ea
> (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)"
>
> My relevant c# code is as follows:
>
> using (var client = new AmazonS3Client(AccessKey, SecretKey,
> GetS3Config()))
> using (var transferUtility = new
> Amazon.S3.Transfer.TransferUtility(client))
> {
> var transferRequest = new
> Amazon.S3.Transfer.TransferUtilityUploadRequest
> {
> BucketName = bucketName,
> Key = key,
> InputStream = stream,
> ServerSideEncryptionCustomerProvidedKey =
> EncryptionKey,
> ServerSideEncryptionCustomerMethod =
> string.IsNullOrWhiteSpace(EncryptionKey) ?
> ServerSideEncryptionCustomerMethod.None :
> ServerSideEncryptionCustomerMethod.AES256
> };
> transferUtility.Upload(transferRequest);
> }
>
> I am having hard time locating the default PartSize this library uses, but
> I think it is 5MB as the content-length header is a bit more than that
> size. Perhaps this sanitized header is useful:
>
> PUT
> [redacted]/delete-me?partNumber=18&uploadId=2~2dt4pYGY3vfKxBb9FcbVlAnbz_z3HTV
> HTTP/1.1
> Expect: 100-continue
> x-amz-server-side-encryption-customer-algorithm: AES256
> x-amz-server-side-encryption-customer-key: [redacted]
> x-amz-server-side-encryption-customer-key-MD5: [redacted]
> User-Agent: aws-sdk-dotnet-coreclr/3.3.31.24 aws-sdk-dotnet-core/3.3.32.2
> .NET_Core/4.6.27317.07 OS/Microsoft_Windows_10.0.17763 ClientAsync
> TransferManager/MultipartUploadCommand
> Host: [redacted]
> X-Amz-Date: [redacted]
> X-Amz-Decoded-Content-Length: 5242880
> X-Amz-Content-SHA256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
> Authorization: [redacted]
> Content-Length: 5248726
> Content-Type: text/plain
>
> 14000;chunk-signature=[redacted]
> [payload here]
>
> Thank you again for help on this matter. I'll point our vendor towards
> this bug as well.
>
> Cheers,
> Dan
>
> On Mon, Mar 18, 2019 at 12:05 PM Casey Bodley <cbod...@redhat.com> wrote:
>
>> Hi Dan,
>>
>> We just got a similar report about SSE-C in
>> http://tracker.ceph.com/issues/38700 that seems to be related to
>> multipart uploads. Could you please add some details there about your s3
>> client, its multipart chunk size, and your ceph version?
>>
>> On 3/18/19 2:38 PM, Dan Smith wrote:
>> > Hello,
>> >
>> > I have stored more than 167 million files in ceph using the S3 api.
>> > Out of those 167 million+ files, one file is not storing correctly.
>> >
>> > The file is 92MB in size. I have stored files much larger and much
>> > smaller. If I store the file WITHOUT using the Customer Provided
>> > 256-bit AES key using Server Side encryption, the file stores and
>> > retrieves just fine (SHA256 hashes match).
>> >
>> > If I store the file USING the 256-bit AES key using Server Side
>> > encryption, the file stores without error, however, when I retrieve
>> > the file and compare the hash of the file I retrieve from ceph against
>> > the hash of the original file, the hashes differ.
>> >
>> > If I store the file using Amazon S3, using the same AES key and their
>> > server side encryption the file stores are retrieves using out issue
>> > (hashes match).
>> >
>> > I can reproduce this issue in two different ceph environments.
>> > Thankfully, the file I am storing is not confidential, so I can share
>> > it out to anyone interested in this
>> > issue.(
>> https://s3.amazonaws.com/aws-website-afewgoodmenrankedfantasyfootballcom-j5gvt/delete-me
>> )
>> >
>> > I have opened a ticket with our vendor for support, but I am hoping
>> > someone might be able to give me some ideas on what might be going on
>> > as well.
>> >
>> > Cheers,
>> > Dan
>> >
>> > _______________________________________________
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
