Re: [jclouds/jclouds] JCLOUDS-1263: URL encode object names from list object response before creating URIs with them (#1084)
@matmannion Can you look at some of the earlier URL encoding commits to see if there is a better solution? @timuralp fixed a lot of these issues in [JCLOUDS-217](https://issues.apache.org/jira/browse/JCLOUDS-217). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1084#issuecomment-291394897
Re: [jclouds/jclouds] JCLOUDS-1264: Use core BindToJsonPayload to send OpenStack Swift multipart manifests (#1083)
Ditto about integration tests, this time in `StaticLargeObjectApiLiveTest`. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1083#issuecomment-291394650
Re: [jclouds/jclouds] JCLOUDS-1263: URL encode object names from list object response before creating URIs with them (#1084)
@matmannion Can you add a test to `ContainerApiLiveTest`? If you haven't run
the integration tests before, you can do so via:
```
mvn integration-test -pl :openstack-swift -Plive
-Dtest.rackspace-cloudfiles-us.identity="${JCLOUDS_IDENTITY}"
-Dtest.rackspace-cloudfiles-us.credential="${JCLOUDS_CREDENTIAL}"
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1084#issuecomment-291394519
Re: [jclouds/jclouds] JCLOUDS-1263: URL encode object names from list object response before creating URIs with them (#1084)
Looks good -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1084#issuecomment-291242800
[jira] [Commented] (JCLOUDS-1263) Listing objects in Openstack Swift blob store containers fails if a key has an invalid URL encode sequence
[
https://issues.apache.org/jira/browse/JCLOUDS-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953515#comment-15953515
]
Mat Mannion commented on JCLOUDS-1263:
--
Potential fix for this in review: https://github.com/jclouds/jclouds/pull/1084
> Listing objects in Openstack Swift blob store containers fails if a key has
> an invalid URL encode sequence
> --
>
> Key: JCLOUDS-1263
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1263
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.1
> Environment: Devstack on Ubuntu 14.04
>Reporter: Mat Mannion
>
> If an object is PUT to a container that has an invalid URL encode sequence,
> subsequence calls to list objects in that container fail due to an attempt to
> double-URLDecode the object name.
> A small project that reproduces the issue against the latest Devstack Swift
> can be found here:
> https://github.com/UniversityofWarwick/jclouds-urlencode-bug
> In short:
> {noformat}
> >> PUT
> >> http://137.205.194.8:8080/v1/AUTH_d92af3a8d7f44a1091fe38f863ebb69e/encode-test/Files/OpenOffice.org%203.3%20%28en-GB%29%20Installation%20Files/openofficeorg1%25%24.cab
> >> HTTP/1.1
> << HTTP/1.1 201 Created
> >> GET
> >> http://137.205.194.8:8080/v1/AUTH_d92af3a8d7f44a1091fe38f863ebb69e/encode-test?format=json&prefix=Files/
> >> HTTP/1.1
> << HTTP/1.1 200 OK
> << "[{"hash": "0cbc6611f5540bd0809a388dc95a615b", "last_modified":
> "2017-03-31T09:34:41.522970", "bytes": 4, "name": "Files/OpenOffice.org 3.3
> (en-GB) Installation Files/openofficeorg1%$.cab", "content_type":
> "application/unknown"}]"
> {noformat}
> In the JSON that Swift returns the key isn't URL encoded
> {{Files/OpenOffice.org 3.3 (en-GB) Installation Files/openofficeorg1%$.cab}},
> but when JClouds parses the response in
> {{ParseObjectListFromResponse$ToSwiftObject}}, it builds a URI on the
> assumption that the path is URL-encoded and this throws an exception due to
> the invalid escape sequence:
> {noformat}
> java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in
> escape (%) pattern - For input string: "$."
> at java.net.URLDecoder.decode(URLDecoder.java:194)
> at org.jclouds.util.Strings2.urlDecode(Strings2.java:131)
> at org.jclouds.http.Uris$UriBuilder.path(Uris.java:143)
> at org.jclouds.http.Uris$UriBuilder.appendPath(Uris.java:151)
> at
> org.jclouds.openstack.swift.v1.functions.ParseObjectListFromResponse$ToSwiftObject.apply(ParseObjectListFromResponse.java:99)
> at
> org.jclouds.openstack.swift.v1.functions.ParseObjectListFromResponse$ToSwiftObject.apply(ParseObjectListFromResponse.java:78)
> at
> com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:582)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:343)
> at com.google.common.collect.Iterables.addAll(Iterables.java:348)
> at
> org.jclouds.blobstore.domain.internal.PageSetImpl.(PageSetImpl.java:31)
> at
> org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.list(RegionScopedSwiftBlobStore.java:254)
> at
> uk.ac.warwick.urlencoding.AbstractJCloudsURLEncodingTest.assertCanPutAndList(AbstractJCloudsURLEncodingTest.java:62)
> at
> uk.ac.warwick.urlencoding.AbstractJCloudsURLEncodingTest.notEncodedWithIllegalEncode(AbstractJCloudsURLEncodingTest.java:102)
> at
> uk.ac.warwick.urlencoding.SwiftJCloudsURLEncodingTest.notEncodedWithIllegalEncode(SwiftJCloudsURLEncodingTest.java:7)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jclouds/jclouds] JCLOUDS-1263: URL encode object names from list object response before creating URIs with them (#1084)
This fix calls urlEncode on the path before appending them to the existing path, as the core UriBuilder class expects them to be URL-encoded. I'm not overly enamoured with this fix, it seems wasteful to encode something for it to be immediately decoded, but equally I'm nervous about making changes to jclouds-core and this seemed to be the easiest drop-in fix for the underlying issue. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1084 -- Commit Summary -- * JCLOUDS-1263: URL encode object names from list object response before creating URIs with them -- File Changes -- M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/functions/ParseObjectListFromResponse.java (5) M apis/openstack-swift/src/test/java/org/jclouds/openstack/swift/v1/features/ObjectApiMockTest.java (4) M apis/openstack-swift/src/test/resources/object_list.json (2) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1084.patch https://github.com/jclouds/jclouds/pull/1084.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1084
[jclouds/jclouds] JCLOUDS-1264: Use core BindToJsonPayload to send OpenStack Swift multipart manifests (#1083)
This fixes a bug where previously BindManifestToJsonPayload used the character length as the ContentLength, instead of the byte length, which caused issues if the JSON contained multi-byte Unicode characters. You can view, comment on, or merge this pull request online at: https://github.com/jclouds/jclouds/pull/1083 -- Commit Summary -- * JCLOUDS-1264: Use core BindToJsonPayload to send OpenStack Swift multipart manifests -- File Changes -- D apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/binders/BindManifestToJsonPayload.java (65) M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/RegionScopedSwiftBlobStore.java (2) M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/features/StaticLargeObjectApi.java (3) M apis/openstack-swift/src/test/java/org/jclouds/openstack/swift/v1/features/StaticLargeObjectApiMockTest.java (48) -- Patch Links -- https://github.com/jclouds/jclouds/pull/1083.patch https://github.com/jclouds/jclouds/pull/1083.diff -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/jclouds/jclouds/pull/1083
[jira] [Commented] (JCLOUDS-1264) Multipart SLO uploads fail if key contains multibyte Unicode characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15953303#comment-15953303
]
Mat Mannion commented on JCLOUDS-1264:
--
A fix for this is in review here https://github.com/jclouds/jclouds/pull/1083
> Multipart SLO uploads fail if key contains multibyte Unicode characters
> ---
>
> Key: JCLOUDS-1264
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1264
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.0.1
> Environment: Devstack on Ubuntu 14.04
>Reporter: Mat Mannion
>
> If a large object is PUT to a key that contains multibyte Unicode characters,
> the manifest upload fails because the {{Content-Length}} header sent is
> incorrect.
> A small project that reproduces the issue against the latest Devstack Swift
> can be found here:
> https://github.com/UniversityofWarwick/jclouds-highbyte-slo-bug
> In short:
> {noformat}
> >> "[{"path":"slo-test/Ni[0xc5][0x9f]an.rar/slo/1491214092.299000/38547913/0/0001","etag":"58f06dd588d8ffb3beb46ada6309436b","size_bytes":33554432},{"path":"slo-test/Ni[0xc5][0x9f]an.rar/slo/1491214092.299000/38547913/0/0002","etag":"2b4b81733d0a2e4abe89516639627408","size_bytes":4993481}]"
> >> PUT
> >> http://137.205.194.8:8080/v1/AUTH_c7a3e66567ec442080a360d6d23f2dbe/slo-test/Ni%C5%9Fan.rar?multipart-manifest=put
> >> HTTP/1.1
> >> Content-Length: 272
> 14657 ERROR org.jclouds.http.internal.JavaUrlHttpCommandExecutorService error
> after writing 0/272 bytes to
> http://137.205.194.8:8080/v1/AUTH_c7a3e66567ec442080a360d6d23f2dbe/slo-test/Ni%C5%9Fan.rar?multipart-manifest=put
> java.io.IOException: too many bytes written
> at
> sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3505)
> at
> com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:53)
> at com.google.common.io.ByteStreams.copy(ByteStreams.java:179)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:298)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:171)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:65)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:99)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at com.sun.proxy.$Proxy77.replaceManifest(Unknown Source)
> at
> org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.completeMultipartUpload(RegionScopedSwiftBlobStore.java:522)
> at
> uk.ac.warwick.slo.AbstractJCloudsSLOTest.putSLO(AbstractJCloudsSLOTest.java:70)
> at
> uk.ac.warwick.slo.AbstractJCloudsSLOTest.assertCanPutAndGetSLO(AbstractJCloudsSLOTest.java:74)
> at
> uk.ac.warwick.slo.AbstractJCloudsSLOTest.sloWithHighByteChars(AbstractJCloudsSLOTest.java:91)
> at
> uk.ac.warwick.slo.SwiftJCloudsSLOTest.sloWithHighByteChars(SwiftJCloudsSLOTest.java:7)
> {noformat}
> The cause of this is
> {{org.jclouds.openstack.swift.v1.binders.BindManifestToJsonPayload:61}},
> where the ContentLength is set to the length of the JSON String in Java,
> which is not the same as the String byte length. This is actually
> unnecessary, as {{request.setPayload(String)}} creates a {{StringPayload}},
> and the constructor here sets the content length to the correct byte length -
> so I think this code is left-over from a previous version where this isn't
> the case.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (JCLOUDS-1264) Multipart SLO uploads fail if key contains multibyte Unicode characters
Mat Mannion created JCLOUDS-1264:
Summary: Multipart SLO uploads fail if key contains multibyte
Unicode characters
Key: JCLOUDS-1264
URL: https://issues.apache.org/jira/browse/JCLOUDS-1264
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 2.0.1
Environment: Devstack on Ubuntu 14.04
Reporter: Mat Mannion
If a large object is PUT to a key that contains multibyte Unicode characters,
the manifest upload fails because the {{Content-Length}} header sent is
incorrect.
A small project that reproduces the issue against the latest Devstack Swift can
be found here: https://github.com/UniversityofWarwick/jclouds-highbyte-slo-bug
In short:
{noformat}
>> "[{"path":"slo-test/Ni[0xc5][0x9f]an.rar/slo/1491214092.299000/38547913/0/0001","etag":"58f06dd588d8ffb3beb46ada6309436b","size_bytes":33554432},{"path":"slo-test/Ni[0xc5][0x9f]an.rar/slo/1491214092.299000/38547913/0/0002","etag":"2b4b81733d0a2e4abe89516639627408","size_bytes":4993481}]"
>> PUT
>> http://137.205.194.8:8080/v1/AUTH_c7a3e66567ec442080a360d6d23f2dbe/slo-test/Ni%C5%9Fan.rar?multipart-manifest=put
>> HTTP/1.1
>> Content-Length: 272
14657 ERROR org.jclouds.http.internal.JavaUrlHttpCommandExecutorService error
after writing 0/272 bytes to
http://137.205.194.8:8080/v1/AUTH_c7a3e66567ec442080a360d6d23f2dbe/slo-test/Ni%C5%9Fan.rar?multipart-manifest=put
java.io.IOException: too many bytes written
at
sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3505)
at
com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:53)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:179)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:298)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:171)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:65)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:99)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy77.replaceManifest(Unknown Source)
at
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.completeMultipartUpload(RegionScopedSwiftBlobStore.java:522)
at
uk.ac.warwick.slo.AbstractJCloudsSLOTest.putSLO(AbstractJCloudsSLOTest.java:70)
at
uk.ac.warwick.slo.AbstractJCloudsSLOTest.assertCanPutAndGetSLO(AbstractJCloudsSLOTest.java:74)
at
uk.ac.warwick.slo.AbstractJCloudsSLOTest.sloWithHighByteChars(AbstractJCloudsSLOTest.java:91)
at
uk.ac.warwick.slo.SwiftJCloudsSLOTest.sloWithHighByteChars(SwiftJCloudsSLOTest.java:7)
{noformat}
The cause of this is
{{org.jclouds.openstack.swift.v1.binders.BindManifestToJsonPayload:61}}, where
the ContentLength is set to the length of the JSON String in Java, which is not
the same as the String byte length. This is actually unnecessary, as
{{request.setPayload(String)}} creates a {{StringPayload}}, and the constructor
here sets the content length to the correct byte length - so I think this code
is left-over from a previous version where this isn't the case.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
