[jira] [Commented] (JCLOUDS-849) Add the live test that extends the BaseComputeServiceLiveTest
[ https://issues.apache.org/jira/browse/JCLOUDS-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377586#comment-14377586 ] Ignasi Barrera commented on JCLOUDS-849: The DeploymentApi calls are direct calls to the provider so unless the response already contains the CloudService information you won't be able to do that. As you say, I see two options: * Perform the call to retrieve the CloudService information in the transformation function, as you already said, or * Change the ComputeService to return a DeploymentWithCloudService instead of a Deployment, and adapt the listNodes and getNode method to return such an object. You can make the calls you need in the adapter to populate that transport object with the fields you need. Add the live test that extends the BaseComputeServiceLiveTest - Key: JCLOUDS-849 URL: https://issues.apache.org/jira/browse/JCLOUDS-849 Project: jclouds Issue Type: Sub-task Components: jclouds-compute, jclouds-labs Reporter: Ignasi Barrera Assignee: fabio martelli Labels: azure Fix For: 2.0.0 The BaseComputeServiceLiveTest provides the contract of the ComputeService abstraction. All compute providers must have a test that extends the base one with all live tests passing. An example implemention can be the DigitalOcean one: https://github.com/jclouds/jclouds-labs/blob/master/digitalocean/src/main/java/org/jclouds/digitalocean/compute/strategy/DigitalOceanComputeServiceAdapter.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-769) Upload blob from stream
[ https://issues.apache.org/jira/browse/JCLOUDS-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1435#comment-1435 ] Akos Hajnal commented on JCLOUDS-769: - As far as I know, in S3, you can use part sizes less than 5MB as well (e.g., the last, fraction part uploaded can easily be even 1 byte-long), maybe 5MB is a good tradeoff. The content-length must be specified for the part only (its size) at uploading (because of PUT), and even uploaded parts may be of different lengths (I didn't try though). Moreover, in the case of output stream emulation, you don't have to do simultaneous uploads, you have the option to block output stream writes until the upload of the most recent part (buffer full) is done (if you want to save memory). Upload blob from stream --- Key: JCLOUDS-769 URL: https://issues.apache.org/jira/browse/JCLOUDS-769 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.1 Reporter: Akos Hajnal Labels: multipart, s3 Dear Developers, It was not easy, but using S3 API, it was possible to upload a large blob from stream - without knowing its size in advance (and storing all the data locally). I found solutions using jclouds' aws-s3 specific API (some async interface), but I really miss this feature from jclouds' general API. My dream is to have a method like: blob.getOutputStream() into which I can write as many data as I want, which pushes data to the storage simultaneously until I close the stream. (When I used S3, I created a wrapper class extending OutputStream, which initiates multipart upload, buffers data written to the output stream, writes a part when the buffer is full, and finalizes multipart upload on stream close.) I don't know it is possible for all providers, but I really miss it... Thank you, Akos Hajnal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (JCLOUDS-769) Upload blob from stream
[ https://issues.apache.org/jira/browse/JCLOUDS-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul updated JCLOUDS-769: Labels: multipart s3 (was: ) Upload blob from stream --- Key: JCLOUDS-769 URL: https://issues.apache.org/jira/browse/JCLOUDS-769 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Reporter: Akos Hajnal Labels: multipart, s3 Dear Developers, It was not easy, but using S3 API, it was possible to upload a large blob from stream - without knowing its size in advance (and storing all the data locally). I found solutions using jclouds' aws-s3 specific API (some async interface), but I really miss this feature from jclouds' general API. My dream is to have a method like: blob.getOutputStream() into which I can write as many data as I want, which pushes data to the storage simultaneously until I close the stream. (When I used S3, I created a wrapper class extending OutputStream, which initiates multipart upload, buffers data written to the output stream, writes a part when the buffer is full, and finalizes multipart upload on stream close.) I don't know it is possible for all providers, but I really miss it... Thank you, Akos Hajnal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-849) Add the live test that extends the BaseComputeServiceLiveTest
[ https://issues.apache.org/jira/browse/JCLOUDS-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377702#comment-14377702 ] Bhathiya Supun commented on JCLOUDS-849: Hi fabio, I think there won't be any conflict here as https://issues.apache.org/jira/browse/JCLOUDS-853 is on Create Operation and which involve DeploymentParams class than Deployment calss. Deployment class totally based on get deployment [https://msdn.microsoft.com/en-us/library/azure/ee460804.aspx] operation. Anyway I also think getting CloudServiceProperties information in the transformation function would be the best possible option. BTW I have added cloud service properties support here https://github.com/jclouds/jclouds-labs/pull/155 if you need. Add the live test that extends the BaseComputeServiceLiveTest - Key: JCLOUDS-849 URL: https://issues.apache.org/jira/browse/JCLOUDS-849 Project: jclouds Issue Type: Sub-task Components: jclouds-compute, jclouds-labs Reporter: Ignasi Barrera Assignee: fabio martelli Labels: azure Fix For: 2.0.0 The BaseComputeServiceLiveTest provides the contract of the ComputeService abstraction. All compute providers must have a test that extends the base one with all live tests passing. An example implemention can be the DigitalOcean one: https://github.com/jclouds/jclouds-labs/blob/master/digitalocean/src/main/java/org/jclouds/digitalocean/compute/strategy/DigitalOceanComputeServiceAdapter.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (JCLOUDS-769) Upload blob from stream
[ https://issues.apache.org/jira/browse/JCLOUDS-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul updated JCLOUDS-769: Affects Version/s: 1.8.1 Upload blob from stream --- Key: JCLOUDS-769 URL: https://issues.apache.org/jira/browse/JCLOUDS-769 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.1 Reporter: Akos Hajnal Labels: multipart, s3 Dear Developers, It was not easy, but using S3 API, it was possible to upload a large blob from stream - without knowing its size in advance (and storing all the data locally). I found solutions using jclouds' aws-s3 specific API (some async interface), but I really miss this feature from jclouds' general API. My dream is to have a method like: blob.getOutputStream() into which I can write as many data as I want, which pushes data to the storage simultaneously until I close the stream. (When I used S3, I created a wrapper class extending OutputStream, which initiates multipart upload, buffers data written to the output stream, writes a part when the buffer is full, and finalizes multipart upload on stream close.) I don't know it is possible for all providers, but I really miss it... Thank you, Akos Hajnal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-849) Add the live test that extends the BaseComputeServiceLiveTest
[ https://issues.apache.org/jira/browse/JCLOUDS-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377619#comment-14377619 ] fabio martelli commented on JCLOUDS-849: Hi Ignasi, thank you for your prompt reply. Yes, you are right, I won't be able to change into DeploymentAPI. Sorry for the mistake. I have taken a look into AzureComputeServiceAdapter about listNodes() and getNodes(). I could perform my changes into this class but I think that this won't be enogh since missing location comes from a deployment creation. Furthermore, I do think that this change will be in conflict with [~Bhash90] proposal (see JCLOUDS-853) so it would result in a waste of time. Since you seems agree with the option 1, I will retrieve the CloudService information in the transformation function. Thank you again for your very good support. Add the live test that extends the BaseComputeServiceLiveTest - Key: JCLOUDS-849 URL: https://issues.apache.org/jira/browse/JCLOUDS-849 Project: jclouds Issue Type: Sub-task Components: jclouds-compute, jclouds-labs Reporter: Ignasi Barrera Assignee: fabio martelli Labels: azure Fix For: 2.0.0 The BaseComputeServiceLiveTest provides the contract of the ComputeService abstraction. All compute providers must have a test that extends the base one with all live tests passing. An example implemention can be the DigitalOcean one: https://github.com/jclouds/jclouds-labs/blob/master/digitalocean/src/main/java/org/jclouds/digitalocean/compute/strategy/DigitalOceanComputeServiceAdapter.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-849) Add the live test that extends the BaseComputeServiceLiveTest
[ https://issues.apache.org/jira/browse/JCLOUDS-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377565#comment-14377565 ] fabio martelli commented on JCLOUDS-849: Hi, working on this issue I found a problem about deployment location retrieving: I need to populate NodeMetadata from a Deployment (see DeploymentToNodeMetadata class) but I cannot set location for the NodeMetadata because this information is at CloudService level but at Deployment's. I can see two alternatives: 1. perform a second request (when needed) to retrieve CloudService properties (including location) 2. change DeploymentApi.get() method in order to 2.a retrieve cloudservice information with deployments embedded into 2.b filter deployments set by name in order to retrieve the deployment with the given name 2.c include location among Deployment properties [~nacx], [~andreaturli], [~Bhash90], [~ilgrosso] what do you think? Please, let me know ASAP. Best Regards. Add the live test that extends the BaseComputeServiceLiveTest - Key: JCLOUDS-849 URL: https://issues.apache.org/jira/browse/JCLOUDS-849 Project: jclouds Issue Type: Sub-task Components: jclouds-compute, jclouds-labs Reporter: Ignasi Barrera Assignee: fabio martelli Labels: azure Fix For: 2.0.0 The BaseComputeServiceLiveTest provides the contract of the ComputeService abstraction. All compute providers must have a test that extends the base one with all live tests passing. An example implemention can be the DigitalOcean one: https://github.com/jclouds/jclouds-labs/blob/master/digitalocean/src/main/java/org/jclouds/digitalocean/compute/strategy/DigitalOceanComputeServiceAdapter.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-849) Add the live test that extends the BaseComputeServiceLiveTest
[ https://issues.apache.org/jira/browse/JCLOUDS-849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377735#comment-14377735 ] fabio martelli commented on JCLOUDS-849: Hi Bhathiya, the conflict would be on AzureComputeServiceAdapter. I was referring to the change suggested by you: Change AzureComputeServiceAdapterVirtualMachine, RoleList,OSImage,Location. Yes, the best option seems to be to get CloudService properties information in the transformation function. I need CloudService name instead. Thank you for your support. Add the live test that extends the BaseComputeServiceLiveTest - Key: JCLOUDS-849 URL: https://issues.apache.org/jira/browse/JCLOUDS-849 Project: jclouds Issue Type: Sub-task Components: jclouds-compute, jclouds-labs Reporter: Ignasi Barrera Assignee: fabio martelli Labels: azure Fix For: 2.0.0 The BaseComputeServiceLiveTest provides the contract of the ComputeService abstraction. All compute providers must have a test that extends the base one with all live tests passing. An example implemention can be the DigitalOcean one: https://github.com/jclouds/jclouds-labs/blob/master/digitalocean/src/main/java/org/jclouds/digitalocean/compute/strategy/DigitalOceanComputeServiceAdapter.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-769) Upload blob from stream
[ https://issues.apache.org/jira/browse/JCLOUDS-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377758#comment-14377758 ] Andrew Gaul commented on JCLOUDS-769: - This is tricky to implement since jclouds needs to satisfy Amazon's requirement for a Content-length per part. Depending on the chosen part size, this can require 5 MB to 5 GB of memory per part and per simultaneous upload or spooling the stream to a temporary file on-disk, as s3distcp does. Alternatively, we could expose the uploadPart method to the users who can do their own buffering and size computation. Upload blob from stream --- Key: JCLOUDS-769 URL: https://issues.apache.org/jira/browse/JCLOUDS-769 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.1 Reporter: Akos Hajnal Labels: multipart, s3 Dear Developers, It was not easy, but using S3 API, it was possible to upload a large blob from stream - without knowing its size in advance (and storing all the data locally). I found solutions using jclouds' aws-s3 specific API (some async interface), but I really miss this feature from jclouds' general API. My dream is to have a method like: blob.getOutputStream() into which I can write as many data as I want, which pushes data to the storage simultaneously until I close the stream. (When I used S3, I created a wrapper class extending OutputStream, which initiates multipart upload, buffers data written to the output stream, writes a part when the buffer is full, and finalizes multipart upload on stream close.) I don't know it is possible for all providers, but I really miss it... Thank you, Akos Hajnal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [jclouds-labs] BlobStore API for Orion based back-ends (#45)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/45#issuecomment-85718559
Re: [jclouds-site] Update compute.md (#130)
Go to http://8f9f01aa921ee2fd31c9-54b9bade39cf00b7396782e42750d03f.r67.cf5.rackcdn.com/ to review your changes. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-site/pull/130#issuecomment-85720357
Re: [jclouds-site] Update aws.md (#129)
Go to http://2824a28fd3dd3fddcb8d-dbd2c979c28bf73721663176c4e36725.r61.cf5.rackcdn.com/ to review your changes. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-site/pull/129#issuecomment-85720194
Re: [jclouds] Support for OpenStack OS-Services API (#573)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/573#issuecomment-85716682
Re: [jclouds] JCLOUDS-670 Option force no password when launching images on providers (#486)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/486#issuecomment-85715089
Re: [jclouds] Add support for Azure Copy Blob (#514)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/514#issuecomment-85715395
Re: [jclouds] Introduce BlobstoreCli (#487)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/487#issuecomment-85715196
Re: [jclouds] JCLOUDS-642: InternalUrl fallback (#462)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/462#issuecomment-85714738
Re: [jclouds] Token based authentication in openstack-keystone (#433)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/433#issuecomment-85714569
Re: [jclouds-examples] Add an example to create/delete/list Google Cloud Storage buckets. (#64)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-examples/pull/64#issuecomment-85717886
Re: [jclouds-examples] GCS-Example initial commit (#66)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-examples/pull/66#issuecomment-85717959
Re: [jclouds-labs-openstack] OS Ceilometer API support (#166)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-openstack/pull/166#issuecomment-85718143
Re: [jclouds-examples] Simplify interface of Google examples and fix the name and comment of userName variable (#63)
@demobox Can this be closed? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-examples/pull/63#issuecomment-85717848
[jira] [Updated] (JCLOUDS-631) Slowdown of putBlob operation for the FileSystem provider
[
https://issues.apache.org/jira/browse/JCLOUDS-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-631:
Labels: filesystem (was: )
Slowdown of putBlob operation for the FileSystem provider
-
Key: JCLOUDS-631
URL: https://issues.apache.org/jira/browse/JCLOUDS-631
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 1.8.0
Reporter: Nikola Knezevic
Assignee: Andrew Gaul
Labels: filesystem
The following (incomplete) piece of code when run for the filesystem provider
and swift, for uploading a 1GB file shows 2-3x slowdown for the filesystem
provider.
Note that the code only _writes_ through jclouds, there is no reading
involved.
{code}
public void sendLargeFilesOnlyJclouds() throws Exception
{
LOG.info(Working with {}, p);
Environment env =
Environment.builder().setName(SomeName).addProperties(p).build();
String providerName = env.getConnectionModules().get(0).getName();
String provider = checkNotNull(env.getProperty(providerName +
.provider), new ICStoreException(No provider specified));
Iterable? extends Module modules =
ImmutableSet.of(
new SLF4JLoggingModule()
);
if(provider.equals(s3))
{
provider = aws-s3;
modules =
ImmutableSet.Modulebuilder()
.addAll(modules)
.add(new ICStoreMultipartUploadModule())
.build();
}
Properties properties = new Properties();
// store provider credentials in properties
properties.setProperty(provider + .identity, env.getProperty(
providerName + .identity, ));
properties.setProperty(provider + .credential,
env.getProperty(providerName + .credential, ));
properties.putAll(env.getPropertiesForModule(providerName));
final BlobStoreContext blobStoreContext =
ContextBuilder.newBuilder(provider)
.overrides(properties)
.modules(modules)
.buildView(BlobStoreContext.class);
final BlobStore bs = blobStoreContext.getBlobStore();
bs.createContainerInLocation(null, mycontainer);
for (int i = 0; i num_invocations; i++)
{
InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
Blob blob = bs.blobBuilder(myblob).payload(fis).build();
LOG.info(Invocation number {}, i);
long startTime = System.currentTimeMillis();
bs.putBlob(mycontainer, blob);
long stopTime = System.currentTimeMillis();
LOG.info(Running time for one blob is {}, stopTime - startTime);
}
}
{code}
Where {{FakeInputStream}} is an input stream that produces a deterministic
output (though method {{read()}}, of a given size, and {{Environment}} is
just a smarter container for {{Properties}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jclouds-site] Update compute.md (#131)
Merged. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-site/pull/131#issuecomment-85720599
[jira] [Updated] (JCLOUDS-847) S3 poor upload performance
[ https://issues.apache.org/jira/browse/JCLOUDS-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul updated JCLOUDS-847: Labels: performance (was: ) S3 poor upload performance -- Key: JCLOUDS-847 URL: https://issues.apache.org/jira/browse/JCLOUDS-847 Project: jclouds Issue Type: Improvement Components: jclouds-drivers Affects Versions: 1.8.1 Environment: JDK 1.7.0_55, 64bit, Windows 7 EU bucket, https Reporter: Veit Guna Priority: Minor Labels: performance Attachments: s3-upload-test.zip Hi. I'm using jclouds 1.8.1 together with the Apache HttpClient module to upload files to S3. During tests, I encountered that upload performance is quite poor in comparison to jets3t or windows tools like Cloudberry S3 Explorer. Sending a 10MB binary file on a cable connection (100mbit down/5mbit up), to an EU bucket (https, default endpoints), from a Windows 7 machine (JDK 1.7.0_55, 64bit) gives the following results: jclouds: ~55 secs Amazon Java SDK: ~55 secs. jets3t: ~18 secs S3 Explorer: ~18 secs Using a faster connection upload time increased up to 200 seconds with jclouds/Amazon SDK. The rest kept the same around 18 secs. So I wondered, where this difference comes from. I started digging into the source code of jclouds, jets3t, httpclient and took a look at the network packages which are send. Long story short: too small buffer sizes! Jclouds uses for the payload the default HttpEntities which HttpClient provides. Such as FileEntity and InputStreamEntity. These are using an output buffer size of hardcoded 4096 bytes. This seems no problem, when the available upload bandwidth is quite small, but slows down the connection on bigger bandwidth - as it seems. For testing I simply created my own HttpClient module, based on the shipped ones and made a simple change that adds a 128k buffer to the to-be-send entity. The result is, that upload performance is now up to the other guys like jets3t and S3 Explorer. Please find attached a small maven project that can be used demonstrate the difference. To be honest, I'm not too deep into the jclouds code to provide a proper patch, but my suggestion would be to provide an own (jclouds) implementation of File- and InputStreamEntity that provide proper output buffer sizes. Maybe with an option to overwrite them by configuration. I also tried the HttpClient http.socket.buffer-size, but that hadn't any effect. Also the 2.0.0-SNAPSHOT version shows no difference. What do you guys think? Is this a known problem? Or are there other settings to increase the upload performance? BTW: The same problem exists with the default JavaUrlHttpCommandExecutorServiceModule which also uses a 4k buffer. Also tried the OkHttp driver with the same results (1.8.1, 2.0.0-SNAPHOT failed with an exception). Thanks! Veit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (JCLOUDS-631) Slowdown of putBlob operation for the FileSystem provider
[
https://issues.apache.org/jira/browse/JCLOUDS-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-631:
Labels: filesystem performance (was: filesystem)
Slowdown of putBlob operation for the FileSystem provider
-
Key: JCLOUDS-631
URL: https://issues.apache.org/jira/browse/JCLOUDS-631
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 1.8.0
Reporter: Nikola Knezevic
Assignee: Andrew Gaul
Labels: filesystem, performance
The following (incomplete) piece of code when run for the filesystem provider
and swift, for uploading a 1GB file shows 2-3x slowdown for the filesystem
provider.
Note that the code only _writes_ through jclouds, there is no reading
involved.
{code}
public void sendLargeFilesOnlyJclouds() throws Exception
{
LOG.info(Working with {}, p);
Environment env =
Environment.builder().setName(SomeName).addProperties(p).build();
String providerName = env.getConnectionModules().get(0).getName();
String provider = checkNotNull(env.getProperty(providerName +
.provider), new ICStoreException(No provider specified));
Iterable? extends Module modules =
ImmutableSet.of(
new SLF4JLoggingModule()
);
if(provider.equals(s3))
{
provider = aws-s3;
modules =
ImmutableSet.Modulebuilder()
.addAll(modules)
.add(new ICStoreMultipartUploadModule())
.build();
}
Properties properties = new Properties();
// store provider credentials in properties
properties.setProperty(provider + .identity, env.getProperty(
providerName + .identity, ));
properties.setProperty(provider + .credential,
env.getProperty(providerName + .credential, ));
properties.putAll(env.getPropertiesForModule(providerName));
final BlobStoreContext blobStoreContext =
ContextBuilder.newBuilder(provider)
.overrides(properties)
.modules(modules)
.buildView(BlobStoreContext.class);
final BlobStore bs = blobStoreContext.getBlobStore();
bs.createContainerInLocation(null, mycontainer);
for (int i = 0; i num_invocations; i++)
{
InputStream fis = new FakeInputStream(FAKE_IS_SEED, blobSize);
Blob blob = bs.blobBuilder(myblob).payload(fis).build();
LOG.info(Invocation number {}, i);
long startTime = System.currentTimeMillis();
bs.putBlob(mycontainer, blob);
long stopTime = System.currentTimeMillis();
LOG.info(Running time for one blob is {}, stopTime - startTime);
}
}
{code}
Where {{FakeInputStream}} is an input stream that produces a deterministic
output (though method {{read()}}, of a given size, and {{Environment}} is
just a smarter container for {{Properties}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (JCLOUDS-847) S3 poor upload performance
[ https://issues.apache.org/jira/browse/JCLOUDS-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul updated JCLOUDS-847: Priority: Major (was: Minor) S3 poor upload performance -- Key: JCLOUDS-847 URL: https://issues.apache.org/jira/browse/JCLOUDS-847 Project: jclouds Issue Type: Improvement Components: jclouds-drivers Affects Versions: 1.8.1 Environment: JDK 1.7.0_55, 64bit, Windows 7 EU bucket, https Reporter: Veit Guna Labels: performance Attachments: s3-upload-test.zip Hi. I'm using jclouds 1.8.1 together with the Apache HttpClient module to upload files to S3. During tests, I encountered that upload performance is quite poor in comparison to jets3t or windows tools like Cloudberry S3 Explorer. Sending a 10MB binary file on a cable connection (100mbit down/5mbit up), to an EU bucket (https, default endpoints), from a Windows 7 machine (JDK 1.7.0_55, 64bit) gives the following results: jclouds: ~55 secs Amazon Java SDK: ~55 secs. jets3t: ~18 secs S3 Explorer: ~18 secs Using a faster connection upload time increased up to 200 seconds with jclouds/Amazon SDK. The rest kept the same around 18 secs. So I wondered, where this difference comes from. I started digging into the source code of jclouds, jets3t, httpclient and took a look at the network packages which are send. Long story short: too small buffer sizes! Jclouds uses for the payload the default HttpEntities which HttpClient provides. Such as FileEntity and InputStreamEntity. These are using an output buffer size of hardcoded 4096 bytes. This seems no problem, when the available upload bandwidth is quite small, but slows down the connection on bigger bandwidth - as it seems. For testing I simply created my own HttpClient module, based on the shipped ones and made a simple change that adds a 128k buffer to the to-be-send entity. The result is, that upload performance is now up to the other guys like jets3t and S3 Explorer. Please find attached a small maven project that can be used demonstrate the difference. To be honest, I'm not too deep into the jclouds code to provide a proper patch, but my suggestion would be to provide an own (jclouds) implementation of File- and InputStreamEntity that provide proper output buffer sizes. Maybe with an option to overwrite them by configuration. I also tried the HttpClient http.socket.buffer-size, but that hadn't any effect. Also the 2.0.0-SNAPSHOT version shows no difference. What do you guys think? Is this a known problem? Or are there other settings to increase the upload performance? BTW: The same problem exists with the default JavaUrlHttpCommandExecutorServiceModule which also uses a 4k buffer. Also tried the OkHttp driver with the same results (1.8.1, 2.0.0-SNAPHOT failed with an exception). Thanks! Veit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [jclouds] Add copyBlob to portable abstraction and add S3-optimized variant (#511)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/511#issuecomment-85715297
Re: [jclouds] Updated the requests in unit test case to include globalIdentifier (#531)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/531#issuecomment-85715651
Re: [jclouds] Throw sane error if one of security groups is null (#515)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/515#issuecomment-85715464
Re: [jclouds-site] Update compute.md (#130)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-site/pull/130#issuecomment-85719600
Re: [jclouds-karaf] JCLOUDS-635 Always check for container existence before uploading blobs. (#50)
We should not address this with hacks in karaf. We have a bunch of JIRA issues tracking these and should address it in core: https://issues.apache.org/jira/browse/JCLOUDS-193 https://issues.apache.org/jira/browse/JCLOUDS-635 https://issues.apache.org/jira/browse/JCLOUDS-648 https://issues.apache.org/jira/browse/JCLOUDS-650 --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-karaf/pull/50#issuecomment-85719716
Re: [jclouds-site] Update aws.md (#129)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-site/pull/129#issuecomment-85719557
Re: [jclouds-site] add softlayer getting started (#134)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-site/pull/134#issuecomment-85720981
[jira] [Commented] (JCLOUDS-769) Upload blob from stream
[ https://issues.apache.org/jira/browse/JCLOUDS-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378753#comment-14378753 ] Andrew Gaul commented on JCLOUDS-769: - [~knl] You might want to push this for review as a strawman -- it might motivate progress on this important feature or at least give others a local workaround. If we expose the part size and document the per-part buffering does this help users? Personally I am inclined to expose portable initiateMPU/uploadPart/completeMPU/abortMPU operations and let people compose on top of those, although this suffers from explicit complexity. Upload blob from stream --- Key: JCLOUDS-769 URL: https://issues.apache.org/jira/browse/JCLOUDS-769 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.1 Reporter: Akos Hajnal Labels: multipart, s3 Dear Developers, It was not easy, but using S3 API, it was possible to upload a large blob from stream - without knowing its size in advance (and storing all the data locally). I found solutions using jclouds' aws-s3 specific API (some async interface), but I really miss this feature from jclouds' general API. My dream is to have a method like: blob.getOutputStream() into which I can write as many data as I want, which pushes data to the storage simultaneously until I close the stream. (When I used S3, I created a wrapper class extending OutputStream, which initiates multipart upload, buffers data written to the output stream, writes a part when the buffer is full, and finalizes multipart upload on stream close.) I don't know it is possible for all providers, but I really miss it... Thank you, Akos Hajnal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [jclouds] Expecting 2 of 3 or the first flaky test to succeed (#577)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/577#issuecomment-85717259
Re: [jclouds] Import rackspace-cloudfiles from labs (#565)
Superseded by #705 --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/565#issuecomment-85716224
Re: [jclouds-examples] Add layer 7 load balancer examples. (#61)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-examples/pull/61#issuecomment-85717726
Re: [jclouds-karaf] JCLOUDS-85: Improving the creation of instance of PropertyShellTableFact... (#18)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-karaf/pull/18#issuecomment-85718822
Re: [jclouds-labs] BlobStore API for Orion based back-ends (#45)
What do you mean by continue working on it? I assume you are not planning to merge it when we consider that the request is open since Nov 30, 2013. So does it make sense to continue working on it? --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/45#issuecomment-85735209
Re: [jclouds] JCLOUDS-599 ability to execute function locally before initScript runs. (#406)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/406#issuecomment-85714023
Re: [jclouds-labs] JCLOUDS-664: Implemented Role represenstaion in API (#118)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/118#issuecomment-85718608
Re: [jclouds-labs-aws] JCLOUDS-750: Refactor glacier domain classes using @AutoValue (#64)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-aws/pull/64#issuecomment-85718421
Re: [jclouds-labs-aws] Prefer TestUtils.randomByteSource (#42)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs-aws/pull/42#issuecomment-85718345
Re: [jclouds-karaf] JCLOUDS-716: Remove Rackspace First-Gen Cloud Servers (#63)
If no one who knows Karaf cares to merge this, I'll just go ahead and close it on April 3. I'd rather close it unmerged than risk breaking Karaf in some weird way I'm not prepared to deal with. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-karaf/pull/63#issuecomment-85719435
Re: [jclouds-karaf] Separates parent and top level pom. (#22)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-karaf/pull/22#issuecomment-85718867
Re: [jclouds-karaf] JCLOUDS-635 Always check for container existence before uploading blobs. (#50)
Thanks for the pull request but it's release week in jclouds and that means it's time to clean up the PR queue. This PR will be over 6 months old as of April 1. If you intend to continue work on it, please make a comment by April 2. Otherwise it will be closed on April 3. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-karaf/pull/50#issuecomment-85718919
Re: [jclouds] adding attache detach interface to nova api (#697)
@limorbortman Apologies for the lack of feedback. We're about releasing 1.9.0 and the master branches are frozen. We expect to have it out by the end of the week, and we'll get back to the pending PRs. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/697#issuecomment-85413637
[jira] [Commented] (JCLOUDS-769) Upload blob from stream
[ https://issues.apache.org/jira/browse/JCLOUDS-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378097#comment-14378097 ] Nikola Knezevic commented on JCLOUDS-769: - So, I mad some rudimentary, local support for S3 streaming by implementing my own SequentialMPUStrategy, and injecting it. However, it is far from satisfactory and looks like a hack. This would be impossible to do w/o Sequential strategy. Now, I could push this code for a review, but I think in order to include it, we first need to change what jclouds expose to the user. Upload blob from stream --- Key: JCLOUDS-769 URL: https://issues.apache.org/jira/browse/JCLOUDS-769 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.1 Reporter: Akos Hajnal Labels: multipart, s3 Dear Developers, It was not easy, but using S3 API, it was possible to upload a large blob from stream - without knowing its size in advance (and storing all the data locally). I found solutions using jclouds' aws-s3 specific API (some async interface), but I really miss this feature from jclouds' general API. My dream is to have a method like: blob.getOutputStream() into which I can write as many data as I want, which pushes data to the storage simultaneously until I close the stream. (When I used S3, I created a wrapper class extending OutputStream, which initiates multipart upload, buffers data written to the output stream, writes a part when the buffer is full, and finalizes multipart upload on stream close.) I don't know it is possible for all providers, but I really miss it... Thank you, Akos Hajnal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (JCLOUDS-769) Upload blob from stream
[ https://issues.apache.org/jira/browse/JCLOUDS-769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14378097#comment-14378097 ] Nikola Knezevic edited comment on JCLOUDS-769 at 3/24/15 4:28 PM: -- So, I made some rudimentary, local support for S3 (and Swift) streaming by implementing my own SequentialMPUStrategy, and injecting it. However, it is far from satisfactory and looks like a hack. This would be impossible to do w/o Sequential strategy. Now, I could push this code for a review, but I think in order to include it, we first need to change what jclouds expose to the user. was (Author: knl): So, I mad some rudimentary, local support for S3 streaming by implementing my own SequentialMPUStrategy, and injecting it. However, it is far from satisfactory and looks like a hack. This would be impossible to do w/o Sequential strategy. Now, I could push this code for a review, but I think in order to include it, we first need to change what jclouds expose to the user. Upload blob from stream --- Key: JCLOUDS-769 URL: https://issues.apache.org/jira/browse/JCLOUDS-769 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 1.8.1 Reporter: Akos Hajnal Labels: multipart, s3 Dear Developers, It was not easy, but using S3 API, it was possible to upload a large blob from stream - without knowing its size in advance (and storing all the data locally). I found solutions using jclouds' aws-s3 specific API (some async interface), but I really miss this feature from jclouds' general API. My dream is to have a method like: blob.getOutputStream() into which I can write as many data as I want, which pushes data to the storage simultaneously until I close the stream. (When I used S3, I created a wrapper class extending OutputStream, which initiates multipart upload, buffers data written to the output stream, writes a part when the buffer is full, and finalizes multipart upload on stream close.) I don't know it is possible for all providers, but I really miss it... Thank you, Akos Hajnal -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [jclouds-labs] JClouds Profitbricks provider - LoadBalancer API (#143)
@nacx No worries. Thanks. Good luck with the 1.9.0 release! --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds-labs/pull/143#issuecomment-85589488
Re: [jclouds] Forwarding and read-only BlobStores (#713)
+
+public abstract class ForwardingBlobStore extends ForwardingObject
+ implements BlobStore {
+ private final BlobStore blobStore;
+
+ public ForwardingBlobStore(BlobStore blobStore) {
+ this.blobStore = checkNotNull(blobStore);
+ }
+
+ protected BlobStore delegate() {
+ return blobStore;
+ }
+
+ @Override
+ public BlobStoreContext getContext() {
+ return delegate().getContext();
Note that there is an unintentional escape hatch from the `ForwardingObject`:
```java
BlobStore rwBlobStore = ...;
BlobStore roBlobStore = ReadOnlyBlobStore.newReadOnlyBlobStore(rwBlobStore);
roBlobStore.getContext().getBlobStore().putBlob(...);
```
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/713/files#r27006641
Re: [jclouds] adding attache detach interface to nova api (#697)
Hi @nacx, @everett-toews and @zack-shoylev can you please take a look? Thanks --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/697#issuecomment-85372414
[jclouds] Forwarding and read-only BlobStores (#713)
You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/713 -- Commit Summary -- * Add ForwardingBlobStore * Add ReadOnlyBlobStore -- File Changes -- A blobstore/src/main/java/org/jclouds/blobstore/util/ForwardingBlobStore.java (210) A blobstore/src/main/java/org/jclouds/blobstore/util/ReadOnlyBlobStore.java (124) M blobstore/src/test/java/org/jclouds/blobstore/util/BlobStoreUtilsTest.java (28) -- Patch Links -- https://github.com/jclouds/jclouds/pull/713.patch https://github.com/jclouds/jclouds/pull/713.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/713
