Re: [jclouds/jclouds] [JCLOUDS-1428] Support for SAS token based Authentication for Azure Blob Storage (#1270)

2019-01-29 Thread Aliaksandra Kharushka 
@ahorusz pushed 1 commit.

fa5106a4f31a2e8f7cb2654ffc768ce1dd134867  [JCLOUDS-1428] - Support for SAS 
token based Authentication for Azure Blob Storage


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1270/files/4145183ca3451b8f411f2147f55f97a4e14229ba..fa5106a4f31a2e8f7cb2654ffc768ce1dd134867

Re: [jclouds/jclouds] [JCLOUDS-1428] Support for SAS token based Authentication for Azure Blob Storage (#1270)

2019-01-29 Thread Aliaksandra Kharushka 
@ahorusz pushed 1 commit.

4145183ca3451b8f411f2147f55f97a4e14229ba  [JCLOUDS-1428] - Support for SAS 
token based Authentication for Azure Blob Storage


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1270/files/e80c6adc20ee363575dcd60b4a5fdc62402c5c1d..4145183ca3451b8f411f2147f55f97a4e14229ba

[jira] [Commented] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage

2019-01-29 Thread Alexandra Horuszko (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755790#comment-16755790
 ] 

Alexandra Horuszko commented on JCLOUDS-1428:
-

Hello, 

I have just opened a pull request #1270! 
h1. [JCLOUDS-1428] Support for SAS token based Authentication for Azure Blob 
Storage #1270

https://github.com/jclouds/jclouds/pull/1270

Best regards,

Alexandra 

> Support for SAS token based Authentication for Azure Blob Storage
> -
>
> Key: JCLOUDS-1428
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1428
> Project: jclouds
>  Issue Type: Improvement
>  Components: jclouds-blobstore
>Reporter: Himanshu Jain
>Priority: Major
>  Labels: azureblob
>
> Hi,
> We have one use case where we want to provide limited access to objects in 
> our storage accounts. We figured that the best way to do  this is by using 
> SAS token based authentication mechanism to upload/download objects to Azure 
> Blob Storage - [SAS based 
> Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1]
> We found that JClouds client library provides support for Azure Blob Storage 
> using account keys which might not fit our use case because of security 
> reasons.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jclouds/jclouds] [JCLOUDS-1428] Support for SAS token based Authentication for Azure Blob Storage (#1270)

2019-01-29 Thread Aliaksandra Kharushka 
Hi, 
This a pull request which corresponds to this Jira Issue: 
https://issues.apache.org/jira/browse/JCLOUDS-1428 

You can view, comment on, or merge this pull request online at:

  https://github.com/jclouds/jclouds/pull/1270

-- Commit Summary --

  * [JCLOUDS-1428] Support for SAS token based Authentication for Azure Blob 
Storage

-- File Changes --

M 
providers/azureblob/src/main/java/org/jclouds/azure/storage/filters/SharedKeyLiteAuthentication.java
 (124)
M 
providers/azureblob/src/main/java/org/jclouds/azureblob/blobstore/AzureBlobRequestSigner.java
 (103)
M 
providers/azureblob/src/test/java/org/jclouds/azureblob/blobstore/AzureBlobRequestSignerTest.java
 (6)

-- Patch Links --

https://github.com/jclouds/jclouds/pull/1270.patch
https://github.com/jclouds/jclouds/pull/1270.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1270

Jenkins build is back to normal : jclouds-labs #27

2019-01-29 Thread Apache Jenkins Server 
See

Build failed in Jenkins: jclouds #37

2019-01-29 Thread Apache Jenkins Server 
See 

Changes:

[gaul] Lazily open InputStream during complete MPU

--
[...truncated 1.00 MB...]
[INFO] allloadbalancer  SKIPPED
[INFO] all  SKIPPED
[INFO] Apache jclouds 2.2.0-SNAPSHOT .. SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 11:13 min
[INFO] Finished at: 2019-01-30T06:15:33Z
[INFO] 
Waiting for Jenkins to finish collecting data
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-checkstyle-plugin:2.15:check (default) on 
project jclouds-blobstore: You have 2 Checkstyle violations. -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.maven.plugins:maven-checkstyle-plugin:2.15:check (default) on 
project jclouds-blobstore: You have 2 Checkstyle violations.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:213)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.jvnet.hudson.maven3.launcher.Maven35Launcher.main 
(Maven35Launcher.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
at jenkins.maven3.agent.Maven35Main.launch (Maven35Main.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at hudson.maven.Maven3Builder.call (Maven3Builder.java:139)
at hudson.maven.Maven3Builder.call (Maven3Builder.java:70)
at hudson.remoting.UserRequest.perform (UserRequest.java:212)
at hudson.remoting.UserRequest.perform (UserRequest.java:54)
at hudson.remoting.Request$2.run (Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call 
(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run (FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker 
(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run 
(ThreadPoolExecutor.java:624)
at java.lang.Thread.run (Thread.java:748)
Caused by: org.apache.maven.plugin.MojoFailureException: You have 2 Checkstyle 
violations.
at org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute 
(CheckstyleViolationCheckMojo.java:580)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:208)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:154)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:146)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute

[jira] [Commented] (JCLOUDS-1367) Blob.getPayload.openStream() is inconsistent across implementations

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755719#comment-16755719
 ] 

ASF subversion and git services commented on JCLOUDS-1367:
--

Commit 13f32b28c90f4dda7ea21c95380d8c0879ba91fe in jclouds's branch 
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=13f32b2 ]

Lazily open InputStream during complete MPU

Previously the filesystem provider could exhaust file descriptors by
eagerly opening up to 10,000 parts.  This partially undoes
JCLOUDS-1367.


> Blob.getPayload.openStream() is inconsistent across implementations
> ---
>
> Key: JCLOUDS-1367
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1367
> Project: jclouds
>  Issue Type: Bug
>  Components: jclouds-blobstore
>Affects Versions: 2.1.0
>Reporter: Rishab Manocha
>Assignee: Andrew Gaul
>Priority: Minor
>  Labels: filesystem, transient
> Fix For: 2.2.0
>
>
> Blob.getPayload.openStream() returns a new InputStream object if the payload 
> is a sub type of  InputStream but returns the same stream if the payload is 
> of type InputStream.
> It would be desirable to have it return a new Stream object which can be read 
> independently.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755714#comment-16755714
 ] 

ASF subversion and git services commented on JCLOUDS-1371:
--

Commit aad98e6f9660fd7a4dd30fa72a2c16a41e1d8584 in jclouds-labs's branch 
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds-labs.git;h=aad98e6 ]

JCLOUDS-1371: JCLOUDS-1488: list optimize prefix

Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.


> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
>  Issue Type: Improvement
>  Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Priority: Major
>  Labels: filesystem
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire 
> container even when prefix and delimiter set.  The File API does not provide 
> a way to list a subset of files except for those within a specific directory 
> and the underlying filesystem makes no guarantees about enumeration order.  
> We can still optimize the case where prefix is set and delimiter is /.  
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755716#comment-16755716
 ] 

ASF subversion and git services commented on JCLOUDS-1371:
--

Commit 30b2ee9016a9f296a7e7ff5e219972f32db385dd in jclouds-labs's branch 
refs/heads/2.1.x from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds-labs.git;h=30b2ee9 ]

JCLOUDS-1371: JCLOUDS-1488: list optimize prefix

Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.


> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
>  Issue Type: Improvement
>  Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Priority: Major
>  Labels: filesystem
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire 
> container even when prefix and delimiter set.  The File API does not provide 
> a way to list a subset of files except for those within a specific directory 
> and the underlying filesystem makes no guarantees about enumeration order.  
> We can still optimize the case where prefix is set and delimiter is /.  
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (JCLOUDS-1488) Filesystem list call with prefix is slow in large containers

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755717#comment-16755717
 ] 

ASF subversion and git services commented on JCLOUDS-1488:
--

Commit 30b2ee9016a9f296a7e7ff5e219972f32db385dd in jclouds-labs's branch 
refs/heads/2.1.x from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds-labs.git;h=30b2ee9 ]

JCLOUDS-1371: JCLOUDS-1488: list optimize prefix

Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.


> Filesystem list call with prefix is slow in large containers
> 
>
> Key: JCLOUDS-1488
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1488
> Project: jclouds
>  Issue Type: Bug
>  Components: jclouds-blobstore
>Affects Versions: 2.1.1
> Environment: Java version: java version "1.8.0_131"
> Operating system: Fedora 27 x86_64
>Reporter: Lari Sinisalo
>Assignee: Andrew Gaul
>Priority: Major
>  Labels: filesystem
> Fix For: 2.2.0, 2.1.2
>
> Attachments: JCLOUDS1488.java
>
>
> When the filesystem blobstore is used, running the following code takes very 
> long if there are a lot of files in the container:
> {code:java}
>     ListContainerOptions options = new ListContainerOptions();
>     options.prefix("test-container-subdirectory/");
>     Set results =
>   blobStore.list("test-container",options);
> {code}
> See the attached Java source file [^JCLOUDS1488.java] for the full code.
> On my system, running the attached Java code takes over 10 seconds to list a 
> single file if there are 500,000 files in the container outside that prefix.
> Output from the attached code:
> {code:java}
> Number of blobs listed: 1
> First listed blob: test-container-subdirectory/file-to-list
> Time it took to list the blobs: 13256 ms
> {code}
> A more general version of this problem was reported previously in 
> JCLOUDS-1371.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (JCLOUDS-1488) Filesystem list call with prefix is slow in large containers

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755715#comment-16755715
 ] 

ASF subversion and git services commented on JCLOUDS-1488:
--

Commit aad98e6f9660fd7a4dd30fa72a2c16a41e1d8584 in jclouds-labs's branch 
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds-labs.git;h=aad98e6 ]

JCLOUDS-1371: JCLOUDS-1488: list optimize prefix

Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.


> Filesystem list call with prefix is slow in large containers
> 
>
> Key: JCLOUDS-1488
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1488
> Project: jclouds
>  Issue Type: Bug
>  Components: jclouds-blobstore
>Affects Versions: 2.1.1
> Environment: Java version: java version "1.8.0_131"
> Operating system: Fedora 27 x86_64
>Reporter: Lari Sinisalo
>Assignee: Andrew Gaul
>Priority: Major
>  Labels: filesystem
> Fix For: 2.2.0, 2.1.2
>
> Attachments: JCLOUDS1488.java
>
>
> When the filesystem blobstore is used, running the following code takes very 
> long if there are a lot of files in the container:
> {code:java}
>     ListContainerOptions options = new ListContainerOptions();
>     options.prefix("test-container-subdirectory/");
>     Set results =
>   blobStore.list("test-container",options);
> {code}
> See the attached Java source file [^JCLOUDS1488.java] for the full code.
> On my system, running the attached Java code takes over 10 seconds to list a 
> single file if there are 500,000 files in the container outside that prefix.
> Output from the attached code:
> {code:java}
> Number of blobs listed: 1
> First listed blob: test-container-subdirectory/file-to-list
> Time it took to list the blobs: 13256 ms
> {code}
> A more general version of this problem was reported previously in 
> JCLOUDS-1371.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: jclouds-labs #26

2019-01-29 Thread Apache Jenkins Server 
See 

--
[...truncated 767.65 KB...]
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
 (1.7 kB at 1.1 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-javadoc.jar
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-javadoc.jar
 (1.1 MB at 702 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
 (1.7 kB at 1.5 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-sources.jar
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-sources.jar
 (120 kB at 92 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
 (1.7 kB at 1.5 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-test-sources.jar
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-test-sources.jar
 (120 kB at 93 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
 (1.7 kB at 1.5 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-tests.jar
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-tests.jar
 (175 kB at 131 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
 (1.7 kB at 1.5 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-javadoc.jar
[INFO] Uploaded to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/dimensiondata-cloudcontrol-2.2.0-20190130.031627-106-javadoc.jar
 (1.1 MB at 696 kB/s)
[INFO] Uploading to apache.snapshots.https: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.2.0-SNAPSHOT/maven-metadata.xml
[INFO] Uploaded to apache.snapshots.https:

Build failed in Jenkins: 2.1.x » jclouds-labs-2.1.x #28

2019-01-29 Thread Apache Jenkins Server 
See 


--
[...truncated 599.91 KB...]
[WARNING] 
:176:
 warning - Tag @see: reference not found: 
org.jclouds.dimensiondata.cloudcontrol.domain.State.FAILED_ADD
[INFO] Building jar: 

[WARNING] Artifact 
org.apache.jclouds.labs:dimensiondata-cloudcontrol:javadoc:javadoc:2.1.2-SNAPSHOT
 already attached to project, ignoring duplicate
[INFO] 
[INFO] --- maven-source-plugin:2.2:jar-no-fork (attach-sources) @ 
dimensiondata-cloudcontrol ---
[INFO] Building jar: 

[WARNING] Artifact 
org.apache.jclouds.labs:dimensiondata-cloudcontrol:java-source:sources:2.1.2-SNAPSHOT
 already attached to project, ignoring duplicate
[INFO] 
[INFO] --- maven-source-plugin:2.2:test-jar-no-fork (attach-sources) @ 
dimensiondata-cloudcontrol ---
[INFO] Building jar: 

[WARNING] Artifact 
org.apache.jclouds.labs:dimensiondata-cloudcontrol:java-source:test-sources:2.1.2-SNAPSHOT
 already attached to project, ignoring duplicate
[INFO] 
[INFO] --- maven-surefire-plugin:2.17:test (integration) @ 
dimensiondata-cloudcontrol ---
[INFO] Toolchain in surefire-plugin: JDK[/home/jenkins/tools/java/latest1.7/]
[INFO] Skipping execution of surefire because it has already been run for this 
configuration
[JENKINS] Recording test results
[INFO] 
[INFO] --- duplicate-finder-maven-plugin:1.1.2:check (default) @ 
dimensiondata-cloudcontrol ---
[INFO] Checking compile classpath
[INFO] Checking runtime classpath
[INFO] Checking test classpath
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.15:check (default) @ 
dimensiondata-cloudcontrol ---
[INFO] 
[INFO] --- modernizer-maven-plugin:1.6.0:modernizer (modernizer) @ 
dimensiondata-cloudcontrol ---
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ 
dimensiondata-cloudcontrol ---
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.1.2-SNAPSHOT/dimensiondata-cloudcontrol-2.1.2-SNAPSHOT.jar
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.1.2-SNAPSHOT/dimensiondata-cloudcontrol-2.1.2-SNAPSHOT.pom
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.1.2-SNAPSHOT/dimensiondata-cloudcontrol-2.1.2-SNAPSHOT-tests.jar
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.1.2-SNAPSHOT/dimensiondata-cloudcontrol-2.1.2-SNAPSHOT-javadoc.jar
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.1.2-SNAPSHOT/dimensiondata-cloudcontrol-2.1.2-SNAPSHOT-sources.jar
[INFO] Installing 

 to 
/home/jenkins/jenkins-slave/maven-repositories/0/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.1.2-SNAPSHOT/dimensiondata-cloudcontrol-2.1.2-SNAPSHOT-test-sources.jar
[INFO] 
[INFO] --- maven-bundle-plugin:2.3.7:install (default-install) @ 
dimensiondata-cloudcontrol ---
[INFO] Local OBR update disabled (enable with -DobrRepository)
[INFO] 
[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ 
dimensiondata-cloudcontrol ---
Downloading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/jclouds/labs/dimensiondata-cloudcontrol/2.1.2-SNAPSHOT/maven-metadata.xml
Downloaded:

Re: [jclouds/jclouds] Lazily open InputStream during complete MPU (#1269)

2019-01-29 Thread Andrew Gaul 
> Is there anything about this change that would merit a test? I'm assuming 
> "does not open tons of file descriptors" is not something that's 
> easy/feasible to test for..?

Indeed this commit was ineffective due to JCLOUDS-1367.  I added a test and 
fixed the other issue.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1269#issuecomment-458789099

Re: [jclouds/jclouds] JCLOUDS-1371: JCLOUDS-1488: list optimize prefix (#1268)

2019-01-29 Thread Andrew Phillips 
demobox commented on this pull request.



> @@ -148,8 +149,13 @@ public boolean blobExists(final String containerName, 
> final String blobName) {
}
 
@Override
-   public Iterable getBlobKeysInsideContainer(final String 
containerName) {
-  return containerToBlobs.get(containerName).keySet();
+   public Iterable getBlobKeysInsideContainer(final String 
containerName, String prefix) {
+  ConcurrentSkipListMap blobs = 
containerToBlobs.get(containerName);
+  if (prefix == null) {
+ return blobs.keySet();
+  }
+  String lastPrefix = prefix + (char) 65535;  // TODO: better sentinel?

[minor] Perhaps create a constant `SENTINEL` or so just to make it clear that 
that's what the last character is supposed to be?

That might be confusing, especially if the TODO is removed in future?

-- 
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1268#discussion_r252093248

[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755560#comment-16755560
 ] 

ASF subversion and git services commented on JCLOUDS-1371:
--

Commit 29eec441e902162394a5dbceaf98c14d3b2bc87a in jclouds's branch 
refs/heads/local-blobstore/list-prefix from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=29eec44 ]

JCLOUDS-1371: JCLOUDS-1488: list optimize prefix

Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.


> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
>  Issue Type: Improvement
>  Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Priority: Major
>  Labels: filesystem
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire 
> container even when prefix and delimiter set.  The File API does not provide 
> a way to list a subset of files except for those within a specific directory 
> and the underlying filesystem makes no guarantees about enumeration order.  
> We can still optimize the case where prefix is set and delimiter is /.  
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jclouds/jclouds] JCLOUDS-1371: JCLOUDS-1488: list optimize prefix (#1268)

2019-01-29 Thread Andrew Gaul 
gaul commented on this pull request.



>   } else if (child.isDirectory()) {
-blobNames.add(function.apply(child.getAbsolutePath()) + 
File.separator); // TODO: undo if failures
-populateBlobKeysInContainer(child, blobNames, function);
+// Consider a prefix /a/b/c but we have only descended to path /a.
+// We need to match the path against the prefix to continue

Addressed suggestion via a more narrow filter.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1268#discussion_r252090379

[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container

2019-01-29 Thread Andrew Gaul (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755567#comment-16755567
 ] 

Andrew Gaul commented on JCLOUDS-1371:
--

This is partially fixed via filtering the prefix.  Leaving open to address the 
delimiter.

> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
>  Issue Type: Improvement
>  Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Priority: Major
>  Labels: filesystem
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire 
> container even when prefix and delimiter set.  The File API does not provide 
> a way to list a subset of files except for those within a specific directory 
> and the underlying filesystem makes no guarantees about enumeration order.  
> We can still optimize the case where prefix is set and delimiter is /.  
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (JCLOUDS-1488) Filesystem list call with prefix is slow in large containers

2019-01-29 Thread Andrew Gaul (JIRA) 


 [ 
https://issues.apache.org/jira/browse/JCLOUDS-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Gaul resolved JCLOUDS-1488.
--
   Resolution: Fixed
 Assignee: Andrew Gaul
Fix Version/s: 2.1.2
   2.2.0

> Filesystem list call with prefix is slow in large containers
> 
>
> Key: JCLOUDS-1488
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1488
> Project: jclouds
>  Issue Type: Bug
>  Components: jclouds-blobstore
>Affects Versions: 2.1.1
> Environment: Java version: java version "1.8.0_131"
> Operating system: Fedora 27 x86_64
>Reporter: Lari Sinisalo
>Assignee: Andrew Gaul
>Priority: Major
>  Labels: filesystem
> Fix For: 2.2.0, 2.1.2
>
> Attachments: JCLOUDS1488.java
>
>
> When the filesystem blobstore is used, running the following code takes very 
> long if there are a lot of files in the container:
> {code:java}
>     ListContainerOptions options = new ListContainerOptions();
>     options.prefix("test-container-subdirectory/");
>     Set results =
>   blobStore.list("test-container",options);
> {code}
> See the attached Java source file [^JCLOUDS1488.java] for the full code.
> On my system, running the attached Java code takes over 10 seconds to list a 
> single file if there are 500,000 files in the container outside that prefix.
> Output from the attached code:
> {code:java}
> Number of blobs listed: 1
> First listed blob: test-container-subdirectory/file-to-list
> Time it took to list the blobs: 13256 ms
> {code}
> A more general version of this problem was reported previously in 
> JCLOUDS-1371.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755565#comment-16755565
 ] 

ASF subversion and git services commented on JCLOUDS-1371:
--

Commit 7bf9c474c656926203a9ac34a1ed27db35c8515d in jclouds's branch 
refs/heads/2.1.x from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=7bf9c47 ]

JCLOUDS-1371: JCLOUDS-1488: list optimize prefix

Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.


> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
>  Issue Type: Improvement
>  Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Priority: Major
>  Labels: filesystem
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire 
> container even when prefix and delimiter set.  The File API does not provide 
> a way to list a subset of files except for those within a specific directory 
> and the underlying filesystem makes no guarantees about enumeration order.  
> We can still optimize the case where prefix is set and delimiter is /.  
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (JCLOUDS-1488) Filesystem list call with prefix is slow in large containers

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755566#comment-16755566
 ] 

ASF subversion and git services commented on JCLOUDS-1488:
--

Commit 7bf9c474c656926203a9ac34a1ed27db35c8515d in jclouds's branch 
refs/heads/2.1.x from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=7bf9c47 ]

JCLOUDS-1371: JCLOUDS-1488: list optimize prefix

Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.


> Filesystem list call with prefix is slow in large containers
> 
>
> Key: JCLOUDS-1488
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1488
> Project: jclouds
>  Issue Type: Bug
>  Components: jclouds-blobstore
>Affects Versions: 2.1.1
> Environment: Java version: java version "1.8.0_131"
> Operating system: Fedora 27 x86_64
>Reporter: Lari Sinisalo
>Priority: Major
>  Labels: filesystem
> Attachments: JCLOUDS1488.java
>
>
> When the filesystem blobstore is used, running the following code takes very 
> long if there are a lot of files in the container:
> {code:java}
>     ListContainerOptions options = new ListContainerOptions();
>     options.prefix("test-container-subdirectory/");
>     Set results =
>   blobStore.list("test-container",options);
> {code}
> See the attached Java source file [^JCLOUDS1488.java] for the full code.
> On my system, running the attached Java code takes over 10 seconds to list a 
> single file if there are 500,000 files in the container outside that prefix.
> Output from the attached code:
> {code:java}
> Number of blobs listed: 1
> First listed blob: test-container-subdirectory/file-to-list
> Time it took to list the blobs: 13256 ms
> {code}
> A more general version of this problem was reported previously in 
> JCLOUDS-1371.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container

2019-01-29 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755563#comment-16755563
 ] 

ASF subversion and git services commented on JCLOUDS-1371:
--

Commit 29eec441e902162394a5dbceaf98c14d3b2bc87a in jclouds's branch 
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=29eec44 ]

JCLOUDS-1371: JCLOUDS-1488: list optimize prefix

Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore.  Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider.  Further optimizations are
possible for delimiter.


> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
>  Issue Type: Improvement
>  Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Priority: Major
>  Labels: filesystem
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire 
> container even when prefix and delimiter set.  The File API does not provide 
> a way to list a subset of files except for those within a specific directory 
> and the underlying filesystem makes no guarantees about enumeration order.  
> We can still optimize the case where prefix is set and delimiter is /.  
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jclouds/jclouds] JCLOUDS-1371: JCLOUDS-1488: list optimize prefix (#1268)

2019-01-29 Thread Timur Alperovich 
timuralp approved this pull request.

Looks good to me!

>   } else if (child.isDirectory()) {
-blobNames.add(function.apply(child.getAbsolutePath()) + 
File.separator); // TODO: undo if failures
-populateBlobKeysInContainer(child, blobNames, function);
+// Consider a prefix /a/b/c but we have only descended to path /a.
+// We need to match the path against the prefix to continue

I was confused by the additional check here, but after removing it and running 
the tests, it makes sense: the directories will not include the trailing "/", 
so if the prefix is "dir/", then it wouldn't match the directory "dir" 
(excluding it from the list and not listing its contents). I think this comment 
helps, but maybe it could more explicitly call out this specific issues. As far 
as I can tell, this only affects prefix listings that end with the file 
separator.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1268#pullrequestreview-197775752

Re: [jclouds/jclouds] Lazily open InputStream during complete MPU (#1269)

2019-01-29 Thread Timur Alperovich 
timuralp approved this pull request.

Looks good to me!



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1269#pullrequestreview-197776920

Re: [jclouds/jclouds-examples] Added Google Cloud Storage parameterization to blobstore_basics (#97)

2019-01-29 Thread Andrew Gaul 
Sorry we moved to the apache/jclouds-examples repository so that this merge 
actually did not take place.  I manually committed it to the correct place.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-examples/pull/97#issuecomment-458694063

Re: [jclouds/jclouds] Lazily open InputStream during complete MPU (#1269)

2019-01-29 Thread Andrew Phillips 
Is there anything about this change that would merit a test? I'm assuming "does 
not open tons of file descriptors" is not something that's easy/feasible to 
test for..?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1269#issuecomment-458521805

Re: [jclouds/jclouds] JCLOUDS-1371: JCLOUDS-1488: list optimize prefix (#1268)

2019-01-29 Thread Ignasi Barrera 
nacx approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1268#pullrequestreview-197508785