shuan1026 opened a new pull request, #11979:
URL: https://github.com/apache/ozone/pull/11979

   ## What changes were proposed in this pull request?
   
   A multipart upload can be completed with non-contiguous part numbers, for 
example 1 and 3. AWS S3 then addresses the parts of the completed object as 1 
to N. Ozone keeps the part numbers used during upload: 
`S3MultipartUploadCompleteRequest` stamps them on the blocks of the completed 
key, and part-aware reads match on that stored number.
   
   For an object completed with parts 1 and 3:
   
   | `GetObject` with | Before | After |
   |---|---|---|
   | `partNumber=2` | `400 InvalidPart` | data uploaded as part 3, as on AWS S3 
|
   | `partNumber=3` | data uploaded as part 3 | `400 InvalidPart`, the object 
has 2 parts |
   
   `x-amz-mp-parts-count` is 2 in both cases. The AWS SDK for Java v2 reads 
`partNumber` 1 to N, so a `multipartEnabled` `S3AsyncClient` or 
`S3TransferManager` cannot download such objects.
   
   This patch stamps each block with the 1-based position of its part in the 
completion list instead. The list is already validated to be strictly 
ascending, so the order of the data does not change. `GetObject` and 
`HeadObject` with `partNumber`, and `ObjectParts` of `GetObjectAttributes`, 
read the stored number and need no change.
   
   Keys completed before this change keep their original part numbers. There is 
no proto or DB schema change. The `206` / `Content-Range` response that the SDK 
also expects is covered by HDDS-17662.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-17663
   
   ## How was this patch tested?
   
   - New 
`TestS3MultipartUploadCompleteRequest#testPartsRenumberedSequentiallyOnComplete`,
 also run for FSO by `TestS3MultipartUploadCompleteRequestWithFSO`. It 
completes an upload with parts 1, 3 and 5 and checks that the blocks carry part 
numbers 1, 2 and 3 in the original data order. It fails with `expected: <[1, 2, 
3]> but was: <[1, 3, 5]>` before the production change and passes after it.
   - 
`TestOzoneClientMultipartUploadWithFSO#testGetPartNumberWithNonContiguousParts` 
asserted the old behavior and is updated. It fails with `Cannot read part 2 of 
key ... because it does not exist` before the production change and passes 
after it.
   - `s3/MultipartUpload.robot` read part number 3 of an object completed with 
parts 1 and 3. It now reads part number 2. Not run locally.
   - ci: https://github.com/shuan1026/ozone/actions/runs/42367854407
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to