Jan Vermeulen created JCLOUDS-1608:
--------------------------------------
Summary: Slicing of large files can lead to exceed the 32 parts
limit of GCS
Key: JCLOUDS-1608
URL: https://issues.apache.org/jira/browse/JCLOUDS-1608
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 2.5.0, 2.2.1
Reporter: Jan Vermeulen
Fix For: 2.6.0, 2.2.1
MultipartUploadSlicingAlgorithm calculates slices for a large file by first
using the defaultPartSize. If the results of that slicing gives parts that
don't exceed the min/maxPartSizes (5MB and 5GB for GoogleCloudStorageBlobStore)
but that do exceed the maxNumberOfParts (32 for GoogleCloudStorageBlobStore),
the algoritm sets the number of parts to 32 and recalculates the size of the
parts. If there is any remainder after that, JClouds ends up uploading 33 parts
in total to GCS, causing the process to fail in completeMultipartUpload() when
recomposing the original content from the parts.
The following simple unitTest proves the case:
{{public class AlgoritmTest extends TestCase {}}
{{ public void testSlicing() {}}
{{ MultipartUploadSlicingAlgorithm algorithm = new
MultipartUploadSlicingAlgorithm(1024*1024*5,1024*1024*1024*5,32);}}
{{ algorithm.calculateChunkSize(1024*1024*1200+33);}}
{{ assertTrue(algorithm.getParts()+((algorithm.getRemaining() >
0)?(1):(0)) <= 32);}}
{{ }}}
{{}}}
It simulates the slicing of a file of 1.2GB+33 bytes (to make sure there is a
remainder).
The following patch fixes the issue:
{{ }}
{{ ...}}
{{ long remainder = length % unitPartSize; }}
{{ // SHB patch}}
{{ // remainder should be distributed over parts if we are at the
maximumNumberOfParts}}
{{ // (if not, an additional part is uploaded to GCS thus exceeding the
maximum allowed parts)}}
{{ // if (remainder == 0 && parts > 0) {}}
{{ // parts -= 1;}}
{{ // }}}
{{ if (remainder > 0 && parts == maximumNumberOfParts) {}}
{{ parts -= 1;}}
{{ partSize = length/parts;}}
{{ }}}
{{ // end of SHB patch
...}}
I also commented the code that reduces the number of parts when there is no
remainder, since that ends up creating a remaining part that is the same size
as the others.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
