[
https://issues.apache.org/jira/browse/JCLOUDS-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755715#comment-16755715
]
ASF subversion and git services commented on JCLOUDS-1488:
----------------------------------------------------------
Commit aad98e6f9660fd7a4dd30fa72a2c16a41e1d8584 in jclouds-labs's branch
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds-labs.git;h=aad98e6 ]
JCLOUDS-1371: JCLOUDS-1488: list optimize prefix
Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore. Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider. Further optimizations are
possible for delimiter.
> Filesystem list call with prefix is slow in large containers
> ------------------------------------------------------------
>
> Key: JCLOUDS-1488
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1488
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
> Affects Versions: 2.1.1
> Environment: Java version: java version "1.8.0_131"
> Operating system: Fedora 27 x86_64
> Reporter: Lari Sinisalo
> Assignee: Andrew Gaul
> Priority: Major
> Labels: filesystem
> Fix For: 2.2.0, 2.1.2
>
> Attachments: JCLOUDS1488.java
>
>
> When the filesystem blobstore is used, running the following code takes very
> long if there are a lot of files in the container:
> {code:java}
> ListContainerOptions options = new ListContainerOptions();
> options.prefix("test-container-subdirectory/");
> Set<? extends StorageMetadata> results =
> blobStore.list("test-container",options);
> {code}
> See the attached Java source file [^JCLOUDS1488.java] for the full code.
> On my system, running the attached Java code takes over 10 seconds to list a
> single file if there are 500,000 files in the container outside that prefix.
> Output from the attached code:
> {code:java}
> Number of blobs listed: 1
> First listed blob: test-container-subdirectory/file-to-list
> Time it took to list the blobs: 13256 ms
> {code}
> A more general version of this problem was reported previously in
> JCLOUDS-1371.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
