Re: [jclouds/jclouds] Handle "file" and "file/" collision (#1168)
kahing approved this pull request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1168#pullrequestreview-87742643
Re: [jclouds/jclouds] Handle "file" and "file/" collision (#1168)
what happens when I put key/ and get(key)? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1168#issuecomment-356496930
Re: [jclouds/jclouds] Handle "file" and "file/" collision (#1168)
kahing approved this pull request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1168#pullrequestreview-87730136
Re: [jclouds] Set xattr before rename to make blob create atomic (#913)
> }
> }
> - setBlobAccess(containerName, blobKey, BlobAccess.PRIVATE);
> +
> + setBlobAccess(containerName, tmpBlobName, BlobAccess.PRIVATE);
> +
> + if (!tmpFile.renameTo(outputFile)) {
> +throw new RuntimeException("Could not rename file " + tmpFile +
> " to " + outputFile);
if rename fails, tmpFile should be cleaned up
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/913/files#r51977958
Re: [jclouds] Set xattr before rename to make blob create atomic (#913)
:+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/913#issuecomment-180200490
Re: [jclouds] Fix filesystem api in OSX (#884)
:+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/884#issuecomment-159359936
[jclouds] don't remove directory if it's a blob (#881)
the filesystem blob store was always removing a/ when a/b was removed, even if the client explicitly created a/. this patch fixes that You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/881 -- Commit Summary -- * don't remove directory if it's a blob -- File Changes -- M apis/filesystem/src/main/java/org/jclouds/filesystem/strategy/internal/FilesystemStorageStrategyImpl.java (10) -- Patch Links -- https://github.com/jclouds/jclouds/pull/881.patch https://github.com/jclouds/jclouds/pull/881.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/881
Re: [jclouds] Always return ETag in filesystem object store get response (#850)
The case I am concerned about is when jclouds is used to expose files not written through jclouds. In those cases getUserDefinedFileAttributeView() would not return null but the extended attribute would not have been written. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/850#issuecomment-133762592
Re: [jclouds] Always return ETag in filesystem object store get response (#850)
`LocalBlobStore` calls getBlob on the storage strategy during listContainer, this change will make it unbearably slow for the cases when the md5 needs to be generated at run time. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/850#issuecomment-133578347
Re: [jclouds] JCLOUDS-826: Remove filesystem (#846)
The filesystem blobstore has the advantage of working with existing files. There maybe some advantages of a jdbc only blobstore, but I think people will vastly prefer one that is closer to the existing filesystem blobstore. The optimization that @rcoedo suggested (storing metadata in a database) is a great idea for the filesystem blobstore. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/846#issuecomment-133113001
[jclouds] replace old style asserts with AssertJ's assertThat (#849)
This is a straightforward replace of assert x -> assertThat(x).isTrue() It doesn't specialize cases like: assert x == null -> assertThat(x).isNull() You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/849 -- Commit Summary -- * replace old style asserts with AssertJ's assertThat -- File Changes -- M apis/atmos/src/test/java/org/jclouds/atmos/AtmosClientLiveTest.java (35) M apis/atmos/src/test/java/org/jclouds/atmos/blobstore/integration/AtmosIntegrationLiveTest.java (13) M apis/atmos/src/test/java/org/jclouds/atmos/options/ListOptionsTest.java (7) M apis/byon/src/test/java/org/jclouds/byon/BYONComputeServiceLiveTest.java (5) M apis/chef/src/test/java/org/jclouds/chef/handlers/ChefApiErrorRetryHandlerTest.java (7) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/compute/CloudStackComputeServiceAdapterLiveTest.java (5) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/compute/CloudStackExperimentLiveTest.java (5) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/AccountApiLiveTest.java (73) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/AddressApiLiveTest.java (17) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/AsyncJobApiLiveTest.java (21) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/ConfigurationApiLiveTest.java (6) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/DomainUserApiLiveTest.java (11) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/EventApiLiveTest.java (25) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/FirewallApiLiveTest.java (41) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalAlertApiLiveTest.java (3) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalConfigurationApiLiveTest.java (13) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalHostApiLiveTest.java (17) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalPodApiLiveTest.java (3) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalUserApiLiveTest.java (3) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GlobalVlanApiLiveTest.java (3) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/GuestOSApiLiveTest.java (15) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/HypervisorApiLiveTest.java (5) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/LimitApiLiveTest.java (8) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/LoadBalancerApiLiveTest.java (25) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/NATApiLiveTest.java (21) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/NetworkApiLiveTest.java (55) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/OfferingApiLiveTest.java (47) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/SSHKeyPairApiLiveTest.java (3) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/SecurityGroupApiLiveTest.java (49) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/TemplateApiLiveTest.java (39) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/VirtualMachineApiLiveTest.java (83) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/features/ZoneApiLiveTest.java (21) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/predicates/NetworkPredicatesTest.java (29) M apis/cloudstack/src/test/java/org/jclouds/cloudstack/predicates/PublicIPAddressPredicatesTest.java (11) M apis/ec2/src/test/java/org/jclouds/ec2/CloudApplicationArchitecturesEC2ApiLiveTest.java (19) M apis/ec2/src/test/java/org/jclouds/ec2/EBSBootEC2ApiLiveTest.java (17) M apis/ec2/src/test/java/org/jclouds/ec2/compute/EC2ComputeServiceLiveTest.java (9) M apis/ec2/src/test/java/org/jclouds/ec2/compute/EC2TemplateBuilderLiveTest.java (5) M apis/ec2/src/test/java/org/jclouds/ec2/compute/EC2TemplateBuilderTest.java (21) M apis/ec2/src/test/java/org/jclouds/ec2/compute/functions/RunningInstanceToNodeMetadataTest.java (3) M apis/ec2/src/test/java/org/jclouds/ec2/compute/options/EC2TemplateOptionsTest.java (7) M apis/ec2/src/test/java/org/jclouds/ec2/features/AMIApiLiveTest.java (3) M apis/ec2/src/test/java/org/jclouds/ec2/features/AvailabilityZoneAndRegionApiLiveTest.java (5) M apis/ec2/src/test/java/org/jclouds/ec2/features/ElasticBlockStoreApiLiveTest.java (11) M apis/ec2/src/test/java/org/jclouds/ec2/features/SecurityGroupApiLiveTest.java (5) M apis/ec2/src/test/java/org/jclouds/ec2/options/BundleInstanceS3StorageOptionsTest.java (5) M apis/ec2/src/test/java/org/jclouds/ec2/options/CreateImageOptionsTest.java (5) M apis/ec2/src/test/j
Re: [jclouds] JCLOUDS-883: Replace metadata during Swift copy (#840)
:+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/840#issuecomment-128469280
Re: [jclouds] marker should be null when there are no more items (#822)
> @@ -187,6 +187,8 @@ public void testListContainerMarker() throws
> InterruptedException {
> container);
> assert container.getNextMarker() == null;
>
> + container = view.getBlobStore().list(containerName,
> afterMarker("z"));
> + assert container.getNextMarker() == null;
done
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/822/files#r35145972
[jclouds] marker should be null when there are no more items (#822)
You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/822 -- Commit Summary -- * marker should be null when there are no more items -- File Changes -- M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/RegionScopedSwiftBlobStore.java (4) M apis/openstack-swift/src/test/java/org/jclouds/openstack/swift/v1/blobstore/integration/SwiftContainerIntegrationLiveTest.java (7) M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseContainerIntegrationTest.java (2) -- Patch Links -- https://github.com/jclouds/jclouds/pull/822.patch https://github.com/jclouds/jclouds/pull/822.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/822
[jclouds] range get of manifest object has no etag (#814)
You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/814 -- Commit Summary -- * range get of manifest object has no etag -- File Changes -- M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/functions/ToBlobMetadata.java (5) -- Patch Links -- https://github.com/jclouds/jclouds/pull/814.patch https://github.com/jclouds/jclouds/pull/814.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/814
[jira] [Updated] (JCLOUDS-957) openstack-swift provider throws different exceptions for copyBlob
[
https://issues.apache.org/jira/browse/JCLOUDS-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ka-Hing Cheung updated JCLOUDS-957:
---
Description:
S3 and LocalBlobStore both throw KeyNotFoundException if copyBlob is called
with an non-existent source key, but openstack-swift throws different
exceptions depends on how copyBlob is used.
{code:java}
@Test(groups = { "integration", "live" })
public void testCopy404Blob() throws Exception {
BlobStore blobStore = view.getBlobStore();
String container = getContainerName();
try {
blobStore.copyBlob(container, "blob", container, "blob2",
CopyOptions.NONE);
} catch (KeyNotFoundException e) {
} finally {
returnContainer(container);
}
}
@Test(groups = { "integration", "live" })
public void testCopy404BlobMeta() throws Exception {
BlobStore blobStore = view.getBlobStore();
String container = getContainerName();
try {
blobStore.copyBlob(container, "blob", container, "blob2",
CopyOptions.builder().userMetadata(ImmutableMap.of("x",
"1")).build());
} catch (KeyNotFoundException e) {
} finally {
returnContainer(container);
}
}
{code}
The first test throws NullPointerException:
{code}
java.lang.NullPointerException: null
at
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.copyBlob(RegionScopedSwiftBlobStore.java:287)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest.testCopy404Blob(BaseBlobIntegrationTest.java:992)
{code}
whereas the latter throws a swift specific exception:
{code}
org.jclouds.openstack.swift.v1.CopyObjectException: Either the source path
'/khc-blobstore0/blob' or the destination path '/khc-blobstore0/blob2' was not
found. (message: Not FoundThe resource could not be
found.)
at
org.jclouds.openstack.swift.v1.handlers.SwiftErrorHandler.handleError(SwiftErrorHandler.java:64)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:136)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy73.copy(Unknown Source)
at
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.copyBlob(RegionScopedSwiftBlobStore.java:307)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest.testCopy404BlobMeta(BaseBlobIntegrationTest.java:1004)
{code}
was:
S3 and LocalBlobStore both throw KeyNotFoundException if copyBlob is called
with an non-existent source key, but openstack-swift throws different
exceptions depends on how copyBlob is used.
@Test(groups = { "integration", "live" })
public void testCopy404Blob() throws Exception {
BlobStore blobStore = view.getBlobStore();
String container = getContainerName();
try {
blobStore.copyBlob(container, "blob", container, "blob2",
CopyOptions.NONE);
} catch (KeyNotFoundException e) {
} finally {
returnContainer(container);
}
}
@Test(groups = { "integration", "live" })
public void testCopy404BlobMeta() throws Exception {
BlobStore blobStore = view.getBlobStore();
String container = getContainerName();
try {
blobStore.copyBlob(container, "blob", container, "blob2",
CopyOptions.builder().userMetadata(ImmutableMap.of("x",
"1")).build());
} catch (KeyNotFoundException e) {
} finally {
returnContainer(container);
}
}
The first test throws NullPointerException:
java.lang.NullPointerException: null
at
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.copyBlob(RegionScopedSwiftBlobStore.java:287)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest.testCopy404Blob(BaseBlobIntegrationTest.java:992)
whereas the latter throws a swift specific exception:
org.jclouds.openstack.swift.v1.CopyObjectException: Either the source path
&#
[jira] [Created] (JCLOUDS-957) openstack-swift provider throws different exceptions for copyBlob
Ka-Hing Cheung created JCLOUDS-957:
--
Summary: openstack-swift provider throws different exceptions for
copyBlob
Key: JCLOUDS-957
URL: https://issues.apache.org/jira/browse/JCLOUDS-957
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Reporter: Ka-Hing Cheung
S3 and LocalBlobStore both throw KeyNotFoundException if copyBlob is called
with an non-existent source key, but openstack-swift throws different
exceptions depends on how copyBlob is used.
@Test(groups = { "integration", "live" })
public void testCopy404Blob() throws Exception {
BlobStore blobStore = view.getBlobStore();
String container = getContainerName();
try {
blobStore.copyBlob(container, "blob", container, "blob2",
CopyOptions.NONE);
} catch (KeyNotFoundException e) {
} finally {
returnContainer(container);
}
}
@Test(groups = { "integration", "live" })
public void testCopy404BlobMeta() throws Exception {
BlobStore blobStore = view.getBlobStore();
String container = getContainerName();
try {
blobStore.copyBlob(container, "blob", container, "blob2",
CopyOptions.builder().userMetadata(ImmutableMap.of("x",
"1")).build());
} catch (KeyNotFoundException e) {
} finally {
returnContainer(container);
}
}
The first test throws NullPointerException:
java.lang.NullPointerException: null
at
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.copyBlob(RegionScopedSwiftBlobStore.java:287)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest.testCopy404Blob(BaseBlobIntegrationTest.java:992)
whereas the latter throws a swift specific exception:
org.jclouds.openstack.swift.v1.CopyObjectException: Either the source path
'/khc-blobstore0/blob' or the destination path '/khc-blobstore0/blob2' was not
found. (message: Not FoundThe resource could not be
found.)
at
org.jclouds.openstack.swift.v1.handlers.SwiftErrorHandler.handleError(SwiftErrorHandler.java:64)
at
org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:136)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:105)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy73.copy(Unknown Source)
at
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.copyBlob(RegionScopedSwiftBlobStore.java:307)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest.testCopy404BlobMeta(BaseBlobIntegrationTest.java:1004)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (JCLOUDS-905) Expect: 100-Continue + SSL = timeout
[ https://issues.apache.org/jira/browse/JCLOUDS-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601961#comment-14601961 ] Ka-Hing Cheung commented on JCLOUDS-905: I wouldn't say this is a duplicate, although the workaround added by https://issues.apache.org/jira/browse/JCLOUDS-181 also works here. > Expect: 100-Continue + SSL = timeout > > > Key: JCLOUDS-905 > URL: https://issues.apache.org/jira/browse/JCLOUDS-905 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore, jclouds-core >Affects Versions: 2.0.0, 1.9.0 >Reporter: Svetoslav Neykov > > A bug in the state logic of HttpsUrlConnection used by the default http > driver will cause it to block and wait for headers at the wrong time when > used with "Expect: 100-Continue" header. When triggered the soTimeout value > is not applied on the socket which leads to indefinite block on a read call, > only to be cancelled by a server timeout. > The bug is triggered when using > * Expect: 100-Continue > * on Java 7 or Java 8 > * on SSL connection > * connection reuse is enabled (by default) > * the stream returned by the call is not closed by the caller > * but instead is closed by the GC > Closing the socket by GC trips the state logic of the "Expect: 100-Continue" > feature, eventually leading to failure to parse the http stream coming from > the server. > The problem is not specific to objectstore or even jclouds, but is triggered > here because of the use of "Expect: 100-Continue". Seems like using it in > combination with SSL connections will inevitably lead to failed requests. > A jclouds test case which reproduces the behaviour can be found at > https://github.com/apache/incubator-brooklyn/blob/master/locations/jclouds/src/test/java/brooklyn/entity/rebind/persister/jclouds/JcloudsExpect100ContinueTest.java > Note that if wire logging is enabled the problem can't be reproduced (closing > streams explicitly?) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (JCLOUDS-905) Expect: 100-Continue + SSL = timeout
[ https://issues.apache.org/jira/browse/JCLOUDS-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601948#comment-14601948 ] Ka-Hing Cheung edited comment on JCLOUDS-905 at 6/25/15 9:12 PM: - issue that added jclouds.strip-expect-header: https://issues.apache.org/jira/browse/JCLOUDS-181 was (Author: khc): issue that added jclouds.strip-expect-header > Expect: 100-Continue + SSL = timeout > > > Key: JCLOUDS-905 > URL: https://issues.apache.org/jira/browse/JCLOUDS-905 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore, jclouds-core >Affects Versions: 2.0.0, 1.9.0 >Reporter: Svetoslav Neykov > > A bug in the state logic of HttpsUrlConnection used by the default http > driver will cause it to block and wait for headers at the wrong time when > used with "Expect: 100-Continue" header. When triggered the soTimeout value > is not applied on the socket which leads to indefinite block on a read call, > only to be cancelled by a server timeout. > The bug is triggered when using > * Expect: 100-Continue > * on Java 7 or Java 8 > * on SSL connection > * connection reuse is enabled (by default) > * the stream returned by the call is not closed by the caller > * but instead is closed by the GC > Closing the socket by GC trips the state logic of the "Expect: 100-Continue" > feature, eventually leading to failure to parse the http stream coming from > the server. > The problem is not specific to objectstore or even jclouds, but is triggered > here because of the use of "Expect: 100-Continue". Seems like using it in > combination with SSL connections will inevitably lead to failed requests. > A jclouds test case which reproduces the behaviour can be found at > https://github.com/apache/incubator-brooklyn/blob/master/locations/jclouds/src/test/java/brooklyn/entity/rebind/persister/jclouds/JcloudsExpect100ContinueTest.java > Note that if wire logging is enabled the problem can't be reproduced (closing > streams explicitly?) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (JCLOUDS-905) Expect: 100-Continue + SSL = timeout
[ https://issues.apache.org/jira/browse/JCLOUDS-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14601948#comment-14601948 ] Ka-Hing Cheung commented on JCLOUDS-905: issue that added jclouds.strip-expect-header > Expect: 100-Continue + SSL = timeout > > > Key: JCLOUDS-905 > URL: https://issues.apache.org/jira/browse/JCLOUDS-905 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore, jclouds-core >Affects Versions: 2.0.0, 1.9.0 >Reporter: Svetoslav Neykov > > A bug in the state logic of HttpsUrlConnection used by the default http > driver will cause it to block and wait for headers at the wrong time when > used with "Expect: 100-Continue" header. When triggered the soTimeout value > is not applied on the socket which leads to indefinite block on a read call, > only to be cancelled by a server timeout. > The bug is triggered when using > * Expect: 100-Continue > * on Java 7 or Java 8 > * on SSL connection > * connection reuse is enabled (by default) > * the stream returned by the call is not closed by the caller > * but instead is closed by the GC > Closing the socket by GC trips the state logic of the "Expect: 100-Continue" > feature, eventually leading to failure to parse the http stream coming from > the server. > The problem is not specific to objectstore or even jclouds, but is triggered > here because of the use of "Expect: 100-Continue". Seems like using it in > combination with SSL connections will inevitably lead to failed requests. > A jclouds test case which reproduces the behaviour can be found at > https://github.com/apache/incubator-brooklyn/blob/master/locations/jclouds/src/test/java/brooklyn/entity/rebind/persister/jclouds/JcloudsExpect100ContinueTest.java > Note that if wire logging is enabled the problem can't be reproduced (closing > streams explicitly?) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (JCLOUDS-945) ListContainerOption's marker support in LocalBlobStore buggy
Ka-Hing Cheung created JCLOUDS-945:
--
Summary: ListContainerOption's marker support in LocalBlobStore
buggy
Key: JCLOUDS-945
URL: https://issues.apache.org/jira/browse/JCLOUDS-945
Project: jclouds
Issue Type: Bug
Reporter: Ka-Hing Cheung
adding this test fails
{code}
---
a/blobstore/src/test/java/org/jclouds/blobstore/integration/TransientBlobIntegrationTest.java
+++
b/blobstore/src/test/java/org/jclouds/blobstore/integration/TransientBlobIntegrationTest.java
@@ -16,12 +16,34 @@
*/
package org.jclouds.blobstore.integration;
+import org.jclouds.blobstore.BlobStore;
+import org.jclouds.blobstore.domain.PageSet;
+import org.jclouds.blobstore.domain.StorageMetadata;
import org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest;
+import org.jclouds.blobstore.options.ListContainerOptions;
+import org.jclouds.utils.TestUtils;
import org.testng.annotations.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
@Test(groups = { "integration" })
public class TransientBlobIntegrationTest extends BaseBlobIntegrationTest {
public TransientBlobIntegrationTest() {
provider = "transient";
}
+
+ @Test
+ public void testListMarker() throws Exception {
+ BlobStore blobStore = view.getBlobStore();
+ final String container = getContainerName();
+ try {
+ blobStore.createContainerInLocation(null, container);
+ blobStore.putBlob(container,
blobStore.blobBuilder("a/b").payload("").build());
+ ListContainerOptions options = new
ListContainerOptions().afterMarker("a/");
+ PageSet res = blobStore.list(container,
options);
+ assertThat(res).hasSize(1);
+ } finally {
+ returnContainer(container);
+ }
+ }
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (JCLOUDS-905) Expect: 100-Continue + SSL = timeout
[
https://issues.apache.org/jira/browse/JCLOUDS-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600496#comment-14600496
]
Ka-Hing Cheung edited comment on JCLOUDS-905 at 6/25/15 1:19 AM:
-
I've hit a (maybe) similar problem today. I am trying to run
S3BlobIntegrationLiveTest against a local S3 server, if I run the tests
serially everything is fine, but if I run them in parallel, I hit an expect
timeout. The backtrace when it's stuck is:
{code}
"pool-5-thread-10" #49 prio=5 os_prio=0 tid=0x7fb9ccd99800 nid=0x176f
runnable [0x7fb94a55]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0xfb6b6e50> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:818)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at
sun.net.www.protocol.http.HttpURLConnection.expect100Continue(HttpURLConnection.java:1192)
at
sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1296)
- locked <0xfb6a8900> (a
sun.net.www.protocol.http.HttpURLConnection)
at
sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
- locked <0xfb6a8900> (a
sun.net.www.protocol.http.HttpURLConnection)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:294)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:95)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy51.putObject(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:283)
at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:257)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest$1.call(BaseBlobIntegrationTest.java:138)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest$1.call(BaseBlobIntegrationTest.java:132)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
This is with HTTP only (not HTTPS). Setting
Constants.PROPERTY_STRIP_EXPECT_HEADER fixes it. From glancing at the code, it
seems like JDK itself should be consuming the 100 response, so it's not
possible to handle it from jclouds?
was (Author: khc):
I've hit a (maybe) similar problem today. I am trying to run
S3BlobIntegrationLiveTest against a local S3 server, if I run the tests
serially everything is fine, but if I run them in parallel, I hit an expect
timeout. The backtrace when it's stuck is:
{code}
"pool-5-thread-10" #49 prio=5 os_prio=0 tid=0x7fb9ccd99800 nid=0x176f
runnable [0x7fb94a55]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- loc
[jira] [Commented] (JCLOUDS-905) Expect: 100-Continue + SSL = timeout
[
https://issues.apache.org/jira/browse/JCLOUDS-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600496#comment-14600496
]
Ka-Hing Cheung commented on JCLOUDS-905:
I've hit a (maybe) similar problem today. I am trying to run
S3BlobIntegrationLiveTest against a local S3 server, if I run the tests
serially everything is fine, but if I run them in parallel, I hit an expect
timeout. The backtrace when it's stuck is:
{code}
"pool-5-thread-10" #49 prio=5 os_prio=0 tid=0x7fb9ccd99800 nid=0x176f
runnable [0x7fb94a55]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0xfb6b6e50> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:818)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at
sun.net.www.protocol.http.HttpURLConnection.expect100Continue(HttpURLConnection.java:1192)
at
sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1296)
- locked <0xfb6a8900> (a
sun.net.www.protocol.http.HttpURLConnection)
at
sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1257)
- locked <0xfb6a8900> (a
sun.net.www.protocol.http.HttpURLConnection)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:294)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:170)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:64)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:95)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at com.sun.proxy.$Proxy51.putObject(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:283)
at org.jclouds.s3.blobstore.S3BlobStore.putBlob(S3BlobStore.java:257)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest$1.call(BaseBlobIntegrationTest.java:138)
at
org.jclouds.blobstore.integration.internal.BaseBlobIntegrationTest$1.call(BaseBlobIntegrationTest.java:132)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
This is with HTTP only (not HTTPS). From glancing at the code, it seems like
JDK itself should be consuming the 100 response, so it's not possible to handle
it from jclouds?
> Expect: 100-Continue + SSL = timeout
>
>
> Key: JCLOUDS-905
> URL: https://issues.apache.org/jira/browse/JCLOUDS-905
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-core
>Affects Versions: 2.0.0, 1.9.0
>Reporter: Svetoslav Neykov
>
> A bug in the state logic of HttpsUrlConnection used by the default http
> driver will cause it to block and wait for headers at the wrong time when
> used with "Expect: 100-Continue" header. When triggered the soTimeout value
> is not applied on the socket which leads to indefinite block on a read call,
> only to be cancelled by a server timeout.
> The bug is triggered when using
> * Expect: 100-Continue
> * on Java 7 or Java 8
> * on SSL connection
> * connection reuse is enabled (by default)
> * the stream returned by the call is not closed by the caller
> * but instead is closed by t
[jclouds] list MPU parts with prefix listing instead of marker (#782)
we now use segment conventions that allow us to do more efficient listing You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/782 -- Commit Summary -- * list MPU parts with prefix listing instead of marker -- File Changes -- M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/RegionScopedSwiftBlobStore.java (5) -- Patch Links -- https://github.com/jclouds/jclouds/pull/782.patch https://github.com/jclouds/jclouds/pull/782.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/782
Re: [jclouds] object stores typically sort the list of containers (#781)
> @@ -83,7 +85,9 @@ public boolean containerExists(final String containerName) {
>
> @Override
> public Iterable getAllContainerNames() {
> - return containerToBlobs.keySet();
> + ArrayList containers = new
> ArrayList(containerToBlobs.keySet());
> + Collections.sort(containers);
> + return containers;
going to update this to sort in LocalBlobStore
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/781/files#r32997451
[jclouds] object stores typically sort the list of containers (#781)
You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/781 -- Commit Summary -- * object stores typically sort the list of containers -- File Changes -- M blobstore/src/main/java/org/jclouds/blobstore/TransientStorageStrategy.java (6) -- Patch Links -- https://github.com/jclouds/jclouds/pull/781.patch https://github.com/jclouds/jclouds/pull/781.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/781
Re: [jclouds] Handle lowercase user metadata headers (#773)
:+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/773#issuecomment-111377089
[jclouds] send HEAD instead of GET for blobMetadata (#772)
You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/772 -- Commit Summary -- * send HEAD instead of GET for blobMetadata -- File Changes -- M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/RegionScopedSwiftBlobStore.java (2) -- Patch Links -- https://github.com/jclouds/jclouds/pull/772.patch https://github.com/jclouds/jclouds/pull/772.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/772
Re: [jclouds] use segment names similar to those created by python-swiftclient (#770)
> @@ -410,7 +410,16 @@ public long countBlobs(String containerName) {
>
> @Override
> public MultipartUpload initiateMultipartUpload(String container,
> BlobMetadata blobMetadata) {
> - String uploadId = UUID.randomUUID().toString();
done
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/770/files#r32187887
[jclouds] use segment names similar to those created by python-swiftclient (#770)
Currently swift creates blob-1, blob-2... blob-n for multipart upload segments, which are very common names that can easily collide with normal objects. This changes the naming convention to that of python-swiftclient (the `swift' command line). python-swiftclient also uses a different segment container by default, that's not implemented by this patch You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/770 -- Commit Summary -- * use segment names similar to those created by python-swiftclient -- File Changes -- M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/RegionScopedSwiftBlobStore.java (30) -- Patch Links -- https://github.com/jclouds/jclouds/pull/770.patch https://github.com/jclouds/jclouds/pull/770.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/770
Re: [jclouds] fix range get for manifest object (#765)
> + @Test(groups = { "integration", "live" })
> + public void testGetRangeMultipart() throws InterruptedException,
> IOException {
> + String container = getContainerName();
> + try {
> + String name = "apples";
> + long length = getMinimumMultipartBlobSize();
> + ByteSource byteSource = TestUtils.randomByteSource().slice(0,
> length);
> + Blob blob = view.getBlobStore().blobBuilder(name)
> + .payload(byteSource)
> + .contentLength(length)
> + .build();
> + view.getBlobStore().putBlob(container, blob, new
> PutOptions().multipart(true));
> + blob = view.getBlobStore().getBlob(container, name, range(0, 5));
> + validateMetadata(blob.getMetadata(), container, name);
> + assertThat(blob.getPayload().openStream())
> + .hasContentEqualTo(byteSource.slice(0, 6).openStream());
done
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/765/files#r32163677
[jclouds] use awaitConsistency in more places so provider can override it (#767)
You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/767 -- Commit Summary -- * use awaitConsistency in more places so provider can override it -- File Changes -- M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java (11) M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobStoreIntegrationTest.java (11) -- Patch Links -- https://github.com/jclouds/jclouds/pull/767.patch https://github.com/jclouds/jclouds/pull/767.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/767
Re: [jclouds] fix range get for manifest object (#765)
> @@ -427,6 +427,25 @@ public void testGetTwoRanges() throws
> InterruptedException, IOException {
>}
> }
>
> + @Test(groups = { "integration", "live" })
> + public void testGetRangeMultipart() throws InterruptedException,
> IOException {
> + String container = getContainerName();
> + try {
> + String name = "apples";
> + long length = getMinimumMultipartBlobSize();
> + ByteSource byteSource = TestUtils.randomByteSource().slice(0,
> length);
> + Blob blob = view.getBlobStore().blobBuilder(name)
> + .payload(byteSource)
> + .contentLength(length)
> + .build();
> + view.getBlobStore().putBlob(container, blob, new
> PutOptions().multipart(true));
> + blob = view.getBlobStore().getBlob(container, name, range(0, 5));
> + validateMetadata(blob.getMetadata(), container, name);
done
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/765/files#r32081542
Re: [jclouds] fix multipart put of a blob with content md5 (#766)
> @@ -601,9 +603,16 @@ public void testPutInputStream() throws Exception {
>
> @Test(groups = { "integration", "live" })
> public void testPutMultipartByteSource() throws Exception {
> - long length = getMinimumMultipartBlobSize();
> + long length = 33554432 + 1; //
> MultipartUploadSlicingAlgorithm.DEFAULT_PART_SIZE + 1
done
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/766/files#r32080595
Re: [jclouds] fix multipart put of a blob with content md5 (#766)
>ByteSource byteSource = TestUtils.randomByteSource().slice(0, length); >Payload payload = new ByteSourcePayload(byteSource); > + > payload.getContentMetadata().setContentMD5(Hashing.md5().hashBytes(byteSource.read())); done --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/766/files#r32080582
Re: [jclouds] fix multipart put of a blob with content md5 (#766)
> @@ -649,6 +658,9 @@ private void testPut(Payload payload, Payload
> expectedPayload, long length, PutO
> .payload(payload)
> .contentLength(length);
>addContentMetadata(blobBuilder);
> + if (payload.getContentMetadata().getContentMD5AsHashCode() != null) {
> +
> blobBuilder.contentMD5(payload.getContentMetadata().getContentMD5AsHashCode());
done
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/766/files#r32080587
Re: [jclouds] fix multipart put of a blob with content md5 (#766)
> @@ -601,9 +603,16 @@ public void testPutInputStream() throws Exception {
>
> @Test(groups = { "integration", "live" })
> public void testPutMultipartByteSource() throws Exception {
> - long length = getMinimumMultipartBlobSize();
> + long length = 33554432 + 1; //
> MultipartUploadSlicingAlgorithm.DEFAULT_PART_SIZE + 1
> + BlobStore blobStore = view.getBlobStore();
> + MultipartUploadSlicingAlgorithm algorithm = new
> MultipartUploadSlicingAlgorithm(
> + getMinimumMultipartBlobSize(),
> blobStore.getMaximumMultipartPartSize(),
done
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/766/files#r32080602
[jclouds] fix range get for manifest object (#765)
swift doesn't return the etag if the client is doing a range get of a manifest (SLO or DLO), this fixes SwiftObject so etag is optional You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/765 -- Commit Summary -- * fix range get for manifest object -- File Changes -- M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/domain/SwiftObject.java (2) M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java (19) -- Patch Links -- https://github.com/jclouds/jclouds/pull/765.patch https://github.com/jclouds/jclouds/pull/765.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/765
[jclouds] fix multipart put of a blob with content md5 (#766)
previously the parts will carry the content md5 of the entire object, because unsetting the md5 of a ContentMetadata actually didn't do anything You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/766 -- Commit Summary -- * fix multipart put of a blob with content md5 -- File Changes -- M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java (14) M core/src/main/java/org/jclouds/io/ContentMetadataBuilder.java (2) -- Patch Links -- https://github.com/jclouds/jclouds/pull/766.patch https://github.com/jclouds/jclouds/pull/766.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/766
[jclouds] remove blob uri from transient blobstore (#760)
container may not be valid hostnames and the current blob uri code can fail. there isn't really a way to use uri for transient blobstore anyway, and removing it is the easier option. for reference, filesystem blobstore also doesn't do this You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/760 -- Commit Summary -- * remove blob uri from transient blobstore -- File Changes -- M blobstore/src/main/java/org/jclouds/blobstore/TransientStorageStrategy.java (3) -- Patch Links -- https://github.com/jclouds/jclouds/pull/760.patch https://github.com/jclouds/jclouds/pull/760.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/760
Re: [jclouds] tighten the isUrlEncoded check (#758)
Ugh this is messier than I thought.
The original motivation was to make https://github.com/bouncestorage/swiftproxy
pass swift's functional tests, which tries to put a blob named "dealde%2Fl04
011e%204c8df/flash.png", which won't be escaped by jclouds and causes a number
of failures. I ended up escaping blobs before passing to jclouds, which causes
other problems.
In particular, when we call
putBlob("dealde%252Fl04%20011e%25204c8df/flash.png") jclouds would send the
path as is, which causes the server side to interpret it as "dealde%2Fl04
011e%204c8df/flash.png", and so the latter is what what would be returned from
list(). So now what we put() and what we get from list() are different.
Some o this is captured in https://issues.apache.org/jira/browse/JCLOUDS-918
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/758#issuecomment-107128542
Re: [jclouds] tighten the isUrlEncoded check (#758)
tried running the core tests and s3 integration test, everything passed for me --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/758#issuecomment-106964378
[jclouds] tighten the isUrlEncoded check (#758)
ideally we shouldn't need this function and instead never double
encode strings, but auditing for that is beyond what I have time
for. currently, putBlob(" ") and putBlob("%20") behave the same
way which is arguably incorrect
Old PR is https://github.com/jclouds/jclouds/pull/755 . This worksaround some
of the test failures
You can view, comment on, or merge this pull request online at:
https://github.com/jclouds/jclouds/pull/758
-- Commit Summary --
* tighten the isUrlEncoded check
-- File Changes --
M core/src/main/java/org/jclouds/util/Strings2.java (31)
M core/src/test/java/org/jclouds/util/Strings2Test.java (4)
-- Patch Links --
https://github.com/jclouds/jclouds/pull/758.patch
https://github.com/jclouds/jclouds/pull/758.diff
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/758
[jira] [Commented] (JCLOUDS-918) putting blobs with % encoded names gives unexpected results
[
https://issues.apache.org/jira/browse/JCLOUDS-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565414#comment-14565414
]
Ka-Hing Cheung commented on JCLOUDS-918:
I misspoke a bit, https://github.com/jclouds/jclouds/pull/755 addresses other
similar issues but does not address the specific examples above.
> putting blobs with % encoded names gives unexpected results
> ---
>
> Key: JCLOUDS-918
> URL: https://issues.apache.org/jira/browse/JCLOUDS-918
> Project: jclouds
> Issue Type: Bug
> Reporter: Ka-Hing Cheung
>
> adding this test case causes Swift (but probably all http based providers) to
> fail:
> {code:title=BaseBlobIntegrationTest.java|borderStyle=solid}
> diff --git
> a/blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java
>
> b/blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java
> index 730d935..c9275ad 100644
> ---
> a/blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java
> +++
> b/blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java
> @@ -529,7 +529,8 @@ public class BaseBlobIntegrationTest extends
> BaseBlobStoreIntegrationTest {
>
>return new Object[][] { { "file", "text/xml", file, realObject },
> { "string", "text/xml", realObject, realObject },
> - { "bytes", "application/octet-stream", realObject.getBytes(),
> realObject } };
> + { "bytes", "application/octet-stream", realObject.getBytes(),
> realObject },
> + { "%20", "text/xml", file, realObject } };
> }
>
> @Test(groups = { "integration", "live" }, dataProvider = "putTests")
> @@ -552,6 +553,7 @@ public class BaseBlobIntegrationTest extends
> BaseBlobStoreIntegrationTest {
> assertEquals(returnedString, realObject);
> PageSet set =
> view.getBlobStore().list(container);
> assert set.size() == 1 : set;
> + assertThat(set.iterator().next().getName()).isEqualTo(name);
>} finally {
> returnContainer(container);
>}
> {code}
> The issue is that because the blob name is encoded, Strings2.urlEncode won't
> encode the name again, so the server side thinks we are trying to put a blob
> named " ". So on list(), a blob named " " would be returned.
> Similarly, if putBlob("%2520") is called, jclouds won't escape the name, so
> the server sees it as "%20", and will return that as the blob name. However,
> any jclouds operations on blob "%20" will fail, because again jclouds won't
> escape "%20", and the server will think that the request is on blob " ",
> which doesn't exist.
> This is the same family of issues as
> https://issues.apache.org/jira/browse/JCLOUDS-217 . I attempted to address
> this with https://github.com/jclouds/jclouds/pull/755 but turns out other
> code has hidden dependencies on broken behaviors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (JCLOUDS-918) putting blobs with % encoded names gives unexpected results
Ka-Hing Cheung created JCLOUDS-918:
--
Summary: putting blobs with % encoded names gives unexpected
results
Key: JCLOUDS-918
URL: https://issues.apache.org/jira/browse/JCLOUDS-918
Project: jclouds
Issue Type: Bug
Reporter: Ka-Hing Cheung
adding this test case causes Swift (but probably all http based providers) to
fail:
{code:title=BaseBlobIntegrationTest.java|borderStyle=solid}
diff --git
a/blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java
b/blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java
index 730d935..c9275ad 100644
---
a/blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java
+++
b/blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java
@@ -529,7 +529,8 @@ public class BaseBlobIntegrationTest extends
BaseBlobStoreIntegrationTest {
return new Object[][] { { "file", "text/xml", file, realObject },
{ "string", "text/xml", realObject, realObject },
- { "bytes", "application/octet-stream", realObject.getBytes(),
realObject } };
+ { "bytes", "application/octet-stream", realObject.getBytes(),
realObject },
+ { "%20", "text/xml", file, realObject } };
}
@Test(groups = { "integration", "live" }, dataProvider = "putTests")
@@ -552,6 +553,7 @@ public class BaseBlobIntegrationTest extends
BaseBlobStoreIntegrationTest {
assertEquals(returnedString, realObject);
PageSet set =
view.getBlobStore().list(container);
assert set.size() == 1 : set;
+ assertThat(set.iterator().next().getName()).isEqualTo(name);
} finally {
returnContainer(container);
}
{code}
The issue is that because the blob name is encoded, Strings2.urlEncode won't
encode the name again, so the server side thinks we are trying to put a blob
named " ". So on list(), a blob named " " would be returned.
Similarly, if putBlob("%2520") is called, jclouds won't escape the name, so the
server sees it as "%20", and will return that as the blob name. However, any
jclouds operations on blob "%20" will fail, because again jclouds won't escape
"%20", and the server will think that the request is on blob " ", which doesn't
exist.
This is the same family of issues as
https://issues.apache.org/jira/browse/JCLOUDS-217 . I attempted to address this
with https://github.com/jclouds/jclouds/pull/755 but turns out other code has
hidden dependencies on broken behaviors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jclouds] tighten the isUrlEncoded check (#755)
giving up, the code is buggy but there are many dependencies on its buggy behaviors --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/755#issuecomment-106895525
Re: [jclouds] tighten the isUrlEncoded check (#755)
Closed #755. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/755#event-317495999
[jclouds] metadata and content metadata should have the same length (#757)
for range get in a LocalBlobStore, metadata.getSize() has the original blob size and contentMetadata.getContentLength() has the actual payload size. Other blobstores have the same size in both You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/757 -- Commit Summary -- * metadata and content metadata should have the same length -- File Changes -- M blobstore/src/main/java/org/jclouds/blobstore/TransientStorageStrategy.java (1) M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (4) M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobStoreIntegrationTest.java (1) -- Patch Links -- https://github.com/jclouds/jclouds/pull/757.patch https://github.com/jclouds/jclouds/pull/757.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/757
[jclouds] tighten the isUrlEncoded check (#755)
ideally we shouldn't need this function and instead never double
encode strings, but auditing for that is beyond what I have time
for. currently, putBlob(" ") and putBlob("%20") behave the same
way which is arguably incorrect
You can view, comment on, or merge this pull request online at:
https://github.com/jclouds/jclouds/pull/755
-- Commit Summary --
* tighten the isUrlEncoded check
-- File Changes --
M core/src/main/java/org/jclouds/util/Strings2.java (29)
M core/src/test/java/org/jclouds/util/Strings2Test.java (4)
-- Patch Links --
https://github.com/jclouds/jclouds/pull/755.patch
https://github.com/jclouds/jclouds/pull/755.diff
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/755
[jclouds] implemented some basic range validations in local blob store (#752)
also fixed a range get bug in local blob store You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/752 -- Commit Summary -- * implemented some basic range validations in local blob store -- File Changes -- M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (5) M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java (27) -- Patch Links -- https://github.com/jclouds/jclouds/pull/752.patch https://github.com/jclouds/jclouds/pull/752.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/752
[jclouds] metadata and content metadata should have the same length (#751)
for range get in a LocalBlobStore, metadata.getSize() has the original blob size and contentMetadata.getContentLength() has the actual payload size. Other blobstores have the same size in both You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/751 -- Commit Summary -- * metadata and content metadata should have the same length -- File Changes -- M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (4) -- Patch Links -- https://github.com/jclouds/jclouds/pull/751.patch https://github.com/jclouds/jclouds/pull/751.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/751
[jira] [Commented] (JCLOUDS-883) Swift portable copy blob cannot replace user metadata
[
https://issues.apache.org/jira/browse/JCLOUDS-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14490340#comment-14490340
]
Ka-Hing Cheung commented on JCLOUDS-883:
You can replace user metadata in Swift with POST:
http://developer.openstack.org/api-ref-objectstorage-v1.html#storage_object_services
> Swift portable copy blob cannot replace user metadata
> -
>
> Key: JCLOUDS-883
> URL: https://issues.apache.org/jira/browse/JCLOUDS-883
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.0
>Reporter: Andrew Gaul
>Assignee: Zack Shoylev
>Priority: Minor
> Labels: openstack-swift
>
> Swift only allows appending to user metadata, different from other object
> stores, such that the following integration test fails:
> {noformat}
> Failed tests:
>
> CloudFilesUSBlobIntegrationLiveTest>BaseBlobIntegrationTest.testCopyBlobReplaceMetadata:812->BaseBlobIntegrationTest.checkUserMetadata:606
>
> Expecting:
> <{"key2"="value2", "key3"="value3", "key1"="value1", "key4"="value4"}>
> to be equal to:
> <{"key3"="value3", "key4"="value4"}>
> but was not.
> {noformat}
> Unfortunately the provider API does not appear to provide a way to set the
> combination of content and user metadata to fix this up after the fact.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (JCLOUDS-856) openstack-swift provider mingles blob names that end with slash
[
https://issues.apache.org/jira/browse/JCLOUDS-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ka-Hing Cheung updated JCLOUDS-856:
---
Description:
Swift (both the old swift and the new openstack-swift providers) deletes
trailing slash on listContainer. deleteContainer() also fails if you have a
blob with trailing slash.
The mingling is done in MarkersIfDirectoryReturnNameStrategy. Working around
that in application is not ideal because metadata.getType() is RELATIVE_PATH
could mean a slash was deleted from the end, or the blob's content type is
application/directory. It's not clear how to figure out what the real name of
the returned objects without probing the store again.
{code:java}
@Test
public void testListObjectWithTrailingSlash() throws InterruptedException {
BlobStore blobStore = view.getBlobStore();
String containerName = getContainerName();
try {
String key = "a/";
String etag = blobStore.putBlob(containerName,
blobStore.blobBuilder(key).payload("content").build());
assertThat(etag).isNotNull();
PageSet res =
blobStore.list(containerName, new
ListContainerOptions().recursive());
assertThat(res).hasSize(1);
StorageMetadata meta = res.iterator().next();
assertThat(meta.getName()).isEqualTo(key);
} finally {
returnContainer(containerName);
}
}
{code}
was:
Swift (both the old swift and the new openstack-swift providers) deletes
trailing slash on listContainer. deleteContainer() also fails if you have a
blob with trailing slash.
The mingling is done in MarkersIfDirectoryReturnNameStrategy. Working around
that in application is not ideal because metadata.getType() is RELATIVE_PATH
could mean a slash was deleted from the end, or the blob's content type is
application/directory. It's not clear how to figure out what the real name of
the returned objects without probing the store again.
@Test
public void testListObjectWithTrailingSlash() throws InterruptedException {
BlobStore blobStore = view.getBlobStore();
String containerName = getContainerName();
try {
String key = "a/";
String etag = blobStore.putBlob(containerName,
blobStore.blobBuilder(key).payload("content").build());
assertThat(etag).isNotNull();
PageSet res =
blobStore.list(containerName, new
ListContainerOptions().recursive());
assertThat(res).hasSize(1);
StorageMetadata meta = res.iterator().next();
assertThat(meta.getName()).isEqualTo(key);
} finally {
returnContainer(containerName);
}
}
> openstack-swift provider mingles blob names that end with slash
> ---
>
> Key: JCLOUDS-856
> URL: https://issues.apache.org/jira/browse/JCLOUDS-856
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.0
>Reporter: Ka-Hing Cheung
> Labels: openstack-swift, swift
>
> Swift (both the old swift and the new openstack-swift providers) deletes
> trailing slash on listContainer. deleteContainer() also fails if you have a
> blob with trailing slash.
> The mingling is done in MarkersIfDirectoryReturnNameStrategy. Working around
> that in application is not ideal because metadata.getType() is RELATIVE_PATH
> could mean a slash was deleted from the end, or the blob's content type is
> application/directory. It's not clear how to figure out what the real name of
> the returned objects without probing the store again.
> {code:java}
> @Test
> public void testListObjectWithTrailingSlash() throws InterruptedException {
>BlobStore blobStore = view.getBlobStore();
>String containerName = getContainerName();
>try {
>String key = "a/";
>String etag = blobStore.putBlob(containerName,
>blobStore.blobBuilder(key).payload("content").build());
>assertThat(etag).isNotNull();
>PageSet res =
>blobStore.list(containerName, new
> ListContainerOptions().recursive());
>assertThat(res).hasSize(1);
>StorageMetadata meta = res.iterator().next();
>assertThat(meta.getName()).isEqualTo(key);
>} finally {
>returnContainer(containerName);
>}
>}
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (JCLOUDS-856) openstack-swift provider mingles blob names that end with slash
Ka-Hing Cheung created JCLOUDS-856:
--
Summary: openstack-swift provider mingles blob names that end with
slash
Key: JCLOUDS-856
URL: https://issues.apache.org/jira/browse/JCLOUDS-856
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 2.0.0
Reporter: Ka-Hing Cheung
Swift (both the old swift and the new openstack-swift providers) deletes
trailing slash on listContainer. deleteContainer() also fails if you have a
blob with trailing slash.
The mingling is done in MarkersIfDirectoryReturnNameStrategy. Working around
that in application is not ideal because metadata.getType() is RELATIVE_PATH
could mean a slash was deleted from the end, or the blob's content type is
application/directory. It's not clear how to figure out what the real name of
the returned objects without probing the store again.
@Test
public void testListObjectWithTrailingSlash() throws InterruptedException {
BlobStore blobStore = view.getBlobStore();
String containerName = getContainerName();
try {
String key = "a/";
String etag = blobStore.putBlob(containerName,
blobStore.blobBuilder(key).payload("content").build());
assertThat(etag).isNotNull();
PageSet res =
blobStore.list(containerName, new
ListContainerOptions().recursive());
assertThat(res).hasSize(1);
StorageMetadata meta = res.iterator().next();
assertThat(meta.getName()).isEqualTo(key);
} finally {
returnContainer(containerName);
}
}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [jclouds] Add utility to crawl a blobstore (#702)
:+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/702#issuecomment-78163635
[jclouds] fix non-recursive list of dir with empty prefix (#694)
You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/694 -- Commit Summary -- * fix non-recursive list of dir with empty prefix -- File Changes -- M apis/filesystem/src/test/java/org/jclouds/filesystem/FilesystemBlobStoreTest.java (18) M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (2) -- Patch Links -- https://github.com/jclouds/jclouds/pull/694.patch https://github.com/jclouds/jclouds/pull/694.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/694
[jclouds] fix non-recursive list with emtpy prefix (#693)
You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/693 -- Commit Summary -- * fix non-recursive list with emtpy prefix -- File Changes -- M apis/filesystem/src/test/java/org/jclouds/filesystem/FilesystemBlobStoreTest.java (15) M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (2) -- Patch Links -- https://github.com/jclouds/jclouds/pull/693.patch https://github.com/jclouds/jclouds/pull/693.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/693
Re: [jclouds] fix usermetadata check on OS X (#648)
@jdaggett I just did a `mvn clean install` with xattrs disabled and TestUtils.isMacOSX() return true, and everything passed. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/648#issuecomment-70746966
Re: [jclouds] fix usermetadata check on OS X (#648)
@jdaggett how did you run the test? I don't have OS X but locally I verified with disabling xattrs and faking TestUtils.isMacOSX() to always return true. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/648#issuecomment-70741593
[jclouds] fix usermetadata check on OS X (#648)
Java on OS X does not support extended attributes, which the filesystem blobstore uses to implement user metadata. This disable the relevant test on OS X You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/648 -- Commit Summary -- * fix usermetadata check on OS X -- File Changes -- M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java (12) -- Patch Links -- https://github.com/jclouds/jclouds/pull/648.patch https://github.com/jclouds/jclouds/pull/648.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/648
Re: [jclouds] Refactor blobstore filesystem tests for Mac OS X (#643)
sure thing I will look at this now --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/643#issuecomment-70354401
Re: [jclouds] Add @Nullable attribute to BlobStore methods (#645)
:+1: --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/645#issuecomment-69979089
[jclouds] make xattr work in docker volume (#642)
when you bind a host volume into docker, java does not correctly detect that and checks the xattr support on the root fs instead of the host fs. The root fs often does not support xattr, so the check would fail even if the target really does support xattr. the fix is just try setting the xattrs anyway, and let them fail if there really isn't xattrs support You can merge this Pull Request by running: git pull https://github.com/kahing/jclouds filesystem-in-docker-xattr Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/642 -- Commit Summary -- * make xattr work in docker volume -- File Changes -- M apis/filesystem/src/main/java/org/jclouds/filesystem/strategy/internal/FilesystemStorageStrategyImpl.java (30) -- Patch Links -- https://github.com/jclouds/jclouds/pull/642.patch https://github.com/jclouds/jclouds/pull/642.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/642
[jclouds] disable directory blob tests in OS X (#640)
On OS X, java doesn't support xattrs, which is required by directory blobs. Disable those tests on OS X should fix @jdaggett 's problem with #637 You can merge this Pull Request by running: git pull https://github.com/kahing/jclouds filesystem-directory-blobs Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/640 -- Commit Summary -- * support directory blobs * disable directory blob tests in OS X -- File Changes -- M apis/filesystem/src/main/java/org/jclouds/filesystem/predicates/validators/internal/FilesystemBlobKeyValidatorImpl.java (5) M apis/filesystem/src/main/java/org/jclouds/filesystem/strategy/internal/FilesystemStorageStrategyImpl.java (142) M apis/filesystem/src/test/java/org/jclouds/filesystem/FilesystemBlobStoreTest.java (61) M apis/filesystem/src/test/java/org/jclouds/filesystem/predicates/validators/internal/FilesystemBlobKeyValidatorTest.java (5) M apis/filesystem/src/test/java/org/jclouds/filesystem/strategy/internal/FilesystemStorageStrategyImplTest.java (67) M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (21) M blobstore/src/main/java/org/jclouds/blobstore/reference/BlobStoreConstants.java (1) -- Patch Links -- https://github.com/jclouds/jclouds/pull/640.patch https://github.com/jclouds/jclouds/pull/640.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/640
Re: [jclouds] support directory blobs (#637)
I know what the problem is and working on a fix. --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/637#issuecomment-69625983
Re: [jclouds] support directory blobs (#637)
> @@ -328,7 +376,20 @@ public void removeBlob(final String container, final
> String blobKey) {
>logger.debug("Deleting blob %s", fileName);
>File fileToBeDeleted = new File(fileName);
>if (!fileToBeDeleted.delete()) {
> - logger.debug("Could not delete %s", fileToBeDeleted);
> + if (fileToBeDeleted.isDirectory()) {
the proposed change modifies the intended semantic
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/637/files#r22750668
Re: [jclouds] support directory blobs (#637)
> @@ -328,7 +376,20 @@ public void removeBlob(final String container, final
> String blobKey) {
>logger.debug("Deleting blob %s", fileName);
>File fileToBeDeleted = new File(fileName);
>if (!fileToBeDeleted.delete()) {
> - logger.debug("Could not delete %s", fileToBeDeleted);
> + if (fileToBeDeleted.isDirectory()) {
> +try {
> + UserDefinedFileAttributeView view =
> getUserDefinedFileAttributeView(fileToBeDeleted.toPath());
> + if (view != null) {
> + for (String s : view.list()) {
> + view.delete(s);
No, delete() would fail if it's not empty, in that case we need to delete all
the attributes, which will cause us to no longer treat it as a blob
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/637/files#r22750631
[jclouds] support directory blobs (#637)
make the filesystem blob store distinguish between a/ and a. a/ must be a directory blob with no content and only metadata on listing, only directories with md5 attribute is considered a blob and returned You can merge this Pull Request by running: git pull https://github.com/kahing/jclouds filesystem-directory-blobs Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/637 -- Commit Summary -- * support directory blobs -- File Changes -- M apis/filesystem/src/main/java/org/jclouds/filesystem/predicates/validators/internal/FilesystemBlobKeyValidatorImpl.java (5) M apis/filesystem/src/main/java/org/jclouds/filesystem/strategy/internal/FilesystemStorageStrategyImpl.java (135) M apis/filesystem/src/test/java/org/jclouds/filesystem/FilesystemBlobStoreTest.java (54) M apis/filesystem/src/test/java/org/jclouds/filesystem/predicates/validators/internal/FilesystemBlobKeyValidatorTest.java (5) M apis/filesystem/src/test/java/org/jclouds/filesystem/strategy/internal/FilesystemStorageStrategyImplTest.java (58) M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (19) M blobstore/src/main/java/org/jclouds/blobstore/reference/BlobStoreConstants.java (1) -- Patch Links -- https://github.com/jclouds/jclouds/pull/637.patch https://github.com/jclouds/jclouds/pull/637.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/637
Re: [jclouds] Handle listing with marker after the last key (#636)
> @@ -262,7 +269,7 @@ public boolean apply(StorageMetadata o) {
> if (!contents.isEmpty()) {
> StorageMetadata lastElement = contents.last();
> contents = newTreeSet(Iterables.limit(contents, maxResults));
> -if (!contents.contains(lastElement)) {
> +if (maxResults != 0 && !contents.contains(lastElement)) {
can `maxResults` be negative?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/636/files#r22695918
[jclouds] fix fs container validator test (#634)
previoisly it was testing blob key validator instead of container name validator You can merge this Pull Request by running: git pull https://github.com/kahing/jclouds fix-fs-container-validator-test Or you can view, comment on it, or merge it online at: https://github.com/jclouds/jclouds/pull/634 -- Commit Summary -- * fix fs container validator test -- File Changes -- M apis/filesystem/src/test/java/org/jclouds/filesystem/predicates/validators/internal/FilesystemContainerNameValidatorTest.java (6) -- Patch Links -- https://github.com/jclouds/jclouds/pull/634.patch https://github.com/jclouds/jclouds/pull/634.diff --- Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/634
