[ https://issues.apache.org/jira/browse/JCLOUDS-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Gaul updated JCLOUDS-847: -------------------------------- Labels: performance s3 (was: performance) > S3 poor upload performance > -------------------------- > > Key: JCLOUDS-847 > URL: https://issues.apache.org/jira/browse/JCLOUDS-847 > Project: jclouds > Issue Type: Improvement > Components: jclouds-blobstore > Affects Versions: 1.8.1 > Environment: JDK 1.7.0_55, 64bit, Windows 7 > EU bucket, https > Reporter: Veit Guna > Assignee: Andrew Gaul > Priority: Major > Labels: performance, s3 > Fix For: 2.2.0 > > Attachments: s3-upload-test.zip > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Hi. > I'm using jclouds 1.8.1 together with the Apache HttpClient module to upload > files to S3. During tests, I encountered that upload performance is quite > poor in comparison to jets3t or windows tools like Cloudberry S3 Explorer. > Sending a 10MB binary file on a cable connection (100mbit down/5mbit up), to > an EU bucket (https, default endpoints), from a Windows 7 machine (JDK > 1.7.0_55, 64bit) gives the following results: > jclouds: ~55 secs > Amazon Java SDK: ~55 secs. > jets3t: ~18 secs > S3 Explorer: ~18 secs > Using a faster connection upload time increased up to 200 seconds with > jclouds/Amazon SDK. The rest kept the same around 18 secs. > So I wondered, where this difference comes from. I started digging into the > source code of jclouds, jets3t, httpclient and took a look at the network > packages which are send. > Long story short: too small buffer sizes! > Jclouds uses for the payload the "default" HttpEntities which HttpClient > provides. Such as FileEntity and InputStreamEntity. These are using an output > buffer size of hardcoded 4096 bytes. > This seems no problem, when the available upload bandwidth is quite small, > but slows down the connection on bigger bandwidth - as it seems. > For testing I simply created my own HttpClient module, based on the shipped > ones and made a simple change that adds a 128k buffer to the to-be-send > entity. The result is, that upload performance is now up to the other guys > like jets3t and S3 Explorer. > Please find attached a small maven project that can be used demonstrate the > difference. > To be honest, I'm not too deep into the jclouds code to provide a proper > patch, but my suggestion would be to provide an own (jclouds) implementation > of File- and InputStreamEntity that provide proper output buffer sizes. Maybe > with an option to overwrite them by configuration. > I also tried the HttpClient "http.socket.buffer-size", but that hadn't any > effect. Also the 2.0.0-SNAPSHOT version shows no difference. > What do you guys think? Is this a known problem? Or are there other settings > to increase the upload performance? > BTW: The same problem exists with the default > JavaUrlHttpCommandExecutorServiceModule which also > uses a 4k buffer. Also tried the OkHttp driver with the same results (1.8.1, > 2.0.0-SNAPHOT failed with an exception). > Thanks! > Veit -- This message was sent by Atlassian Jira (v8.3.4#803005)