An update on my previous request: I've tracked down in the code the
lines that
may cause the unexpected behaviour, precisely in the
AWSS3BlobStoreContextModule class:
protected void configure() {
super.configure();
bind(S3AsyncBlobStore.class).to(AWSS3AsyncBlobStore.class).in(Scopes.SINGLETON);
bind(S3BlobStore.class).to(AWSS3BlobStore.class).in(Scopes.SINGLETON);
bind(MultipartUploadStrategy.class).to(SequentialMultipartUploadStrategy.class);
bind(AsyncMultipartUploadStrategy.class).to(ParallelMultipartUploadStrategy.class);
}
So the MultipartUploadStrategy is associated with the
SequentialMultipartUploadStrategy,
instead of ParallelMultipartUploadStrategy: why..?
Also: can someone confirm that at the moment only Swift and S3 support
this kind of upload strategy?
Thank you in advance.
Paolo
On 02/08/2013 13:07, Paolo Viotti wrote:
Hello,
These days I am trying to test cloud storage performance using
different libraries and services.
One of the tests I am performing is about uploading large files.
So I've read about the nice multipart feature of jClouds [1], and I've
also seen examples ([2] and [3])
which tweak some properties (namely "jclouds.mpu.parallel.degree" and
"jclouds.mpu.parts.size")
to make the provider spawn several threads and do the multipart
uploads in parallel.
Since I was recording unexpected high latencies I wrote a quick test
[4] in which I enable the
HTTP header log and I perform a simple 50MB PUT to Amazon S3, setting
6000000 byte
parts and 5 threads.
The problem is: while the part size property is taken into account,
the thread property seems
to have no effect at all, so in the log [5] I see the same thread
("main") doing the upload serially, one by one.
Is there something wrong in my settings / what am I missing..?
Thanks in advance.
Paolo
PS: I am using jClouds version 1.6.1-incubating
[1]:
http://jclouds.incubator.apache.org/documentation/userguide/blobstore-guide/
"Multipart uploads paragraph"
[2]:
https://github.com/jclouds/jclouds-examples/blob/master/blobstore-largeblob/src/main/java/org/jclouds/examples/blobstore/largeblob/MainApp.java
[3]:
https://github.com/jclouds/jclouds-examples/blob/master/rackspace/src/main/java/org/jclouds/examples/rackspace/cloudfiles/UploadLargeObject.java
[4]: https://gist.github.com/pviotti/a431891d3176362719d6
[5]: https://gist.github.com/pviotti/f1345b6f2d4be25414ad
