[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1638:
-
Labels: s3 (was: )
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
> Labels: s3
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; error at 586:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 586; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
> {noformat}
> When there's a control character in the folder path in S3, we can't parse it
> from the response because it throws SAXParseException.
> Can there be an option that at least lets us forward the encoding-type param?
> https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
> And url decode it for us so that listing can be possible? This bug currently
> doesn't allow us to list any children of a root folder if one of the children
> contains control characters.
> Here's an example XML response from S3 when listing objects from cURL:
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
> {noformat}
> Child folder of 'some' contains
> {noformat}
> some/test/
> {noformat}
> which can't be parsed.
> But with the urlParam &encoding-type=url :
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
> {noformat}
> {noformat}
> some/test%10/
> {noformat}
> Can probably be parsed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860905#comment-17860905
]
Andrew Gaul commented on JCLOUDS-1638:
--
This seems like a well-scoped feature to add. Could you submit a PR at
https://github.com/apache/jclouds/ ? It might be best to start with a unit
test which demonstrates the behavior that is broken. The encoding-type
parameter was introduced a long time ago so I think we can assume all
implementations should support this now so it should be the default behavior,
optionally with a config to opt into the older behavior.
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; error at 586:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 586; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
> {noformat}
> When there's a control character in the folder path in S3, we can't parse it
> from the response because it throws SAXParseException.
> Can there be an option that at least lets us forward the encoding-type param?
> https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
> And url decode it for us so that listing can be possible? This bug currently
> doesn't allow us to list any children of a root folder if one of the children
> contains control characters.
> Here's an example XML response from S3 when listing objects from cURL:
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
> {noformat}
> Child folder of 'some' contains
> {noformat}
> some/test/
> {noformat}
> which can't be parsed.
> But with the urlParam &encoding-type=url :
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
> {noformat}
> {noformat}
> some/test%10/
> {noformat}
> Can probably be parsed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1638:
-
Component/s: jclouds-blobstore
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; error at 586:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 586; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
> {noformat}
> When there's a control character in the folder path in S3, we can't parse it
> from the response because it throws SAXParseException.
> Can there be an option that at least lets us forward the encoding-type param?
> https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
> And url decode it for us so that listing can be possible? This bug currently
> doesn't allow us to list any children of a root folder if one of the children
> contains control characters.
> Here's an example XML response from S3 when listing objects from cURL:
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
> {noformat}
> Child folder of 'some' contains
> {noformat}
> some/test/
> {noformat}
> which can't be parsed.
> But with the urlParam &encoding-type=url :
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
> {noformat}
> {noformat}
> some/test%10/
> {noformat}
> Can probably be parsed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860666#comment-17860666
]
Jacob Nguyen edited comment on JCLOUDS-1638 at 6/28/24 12:07 AM:
-
Extra thing to note, S3 SDK now uses encoding-type = url by default in their
newer versions now.
Not sure whether JClouds should do the same thing.
https://github.com/aws/aws-sdk-java/issues/333#issuecomment-213096411
was (Author: JIRAUSER306001):
Extra thing to note, S3 SDK now uses encoding-type = url by default in their
newer versions now.
Not sure whether JClouds should do the same thing.
https://github.com/aws/aws-sdk-java/issues/460#issuecomment-240296956
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; error at 586:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 586; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
> {noformat}
> When there's a control character in the folder path in S3, we can't parse it
> from the response because it throws SAXParseException.
> Can there be an option that at least lets us forward the encoding-type param?
> https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
> And url decode it for us so that listing can be possible? This bug currently
> doesn't allow us to list any children of a root folder if one of the children
> contains control characters.
> Here's an example XML response from S3 when listing objects from cURL:
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
> {noformat}
> Child folder of 'some' contains
> {noformat}
> some/test/
> {noformat}
> which can't be parsed.
> But with the urlParam &encoding-type=url :
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
> {noformat}
> {noformat}
> some/test%10/
> {noformat}
> Can probably be parsed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17860666#comment-17860666
]
Jacob Nguyen commented on JCLOUDS-1638:
---
Extra thing to note, S3 SDK now uses encoding-type = url by default in their
newer versions now.
Not sure whether JClouds should do the same thing.
https://github.com/aws/aws-sdk-java/issues/460#issuecomment-240296956
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; error at 586:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 586; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
> {noformat}
> When there's a control character in the folder path in S3, we can't parse it
> from the response because it throws SAXParseException.
> Can there be an option that at least lets us forward the encoding-type param?
> https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
> And url decode it for us so that listing can be possible? This bug currently
> doesn't allow us to list any children of a root folder if one of the children
> contains control characters.
> Here's an example XML response from S3 when listing objects from cURL:
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
> {noformat}
> Child folder of 'some' contains
> {noformat}
> some/test/
> {noformat}
> which can't be parsed.
> But with the urlParam &encoding-type=url :
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
> {noformat}
> {noformat}
> some/test%10/
> {noformat}
> Can probably be parsed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Affects Version/s: 2.6.0
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
>Affects Versions: 2.5.0, 2.6.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
> HTTP/1.1; error at 586:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 586; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
> {noformat}
> When there's a control character in the folder path in S3, we can't parse it
> from the response because it throws SAXParseException.
> Can there be an option that at least lets us forward the encoding-type param?
> https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
> And url decode it for us so that listing can be possible? This bug currently
> doesn't allow us to list any children of a root folder if one of the children
> contains control characters.
> Here's an example XML response from S3 when listing objects from cURL:
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
> {noformat}
> Child folder of 'some' contains
> {noformat}
> some/test/
> {noformat}
> which can't be parsed.
> But with the urlParam &encoding-type=url :
> {noformat}
>
> xmlns="http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
> {noformat}
> {noformat}
> some/test%10/
> {noformat}
> Can probably be parsed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Description:
{noformat}
java.lang.RuntimeException: request: GET
https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
HTTP/1.1; error at 586:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 586; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
{noformat}
Child folder of 'some' contains
{noformat}
some/test/
{noformat}
which can't be parsed.
But with the urlParam &encoding-type=url :
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
{noformat}
{noformat}
some/test%10/
{noformat}
Can probably be parsed.
was:
{noformat}
java.lang.RuntimeException: request: GET
https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
{noformat}
Child f
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Description:
{noformat}
java.lang.RuntimeException: request: GET
https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://cloudsync-performance-tests.s3.amazonaws.com/?delimiter=/&prefix=some/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
{noformat}
Child folder of 'some' contains
{noformat}
some/test/
{noformat}
which can't be parsed.
But with the urlParam &encoding-type=url :
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
{noformat}
{noformat}
some/test%10/
{noformat}
Can probably be parsed.
was:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/fals
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Description:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
{noformat}
Child folder of 'some' contains
{noformat}
some/test/
{noformat}
which can't be parsed.
But with the urlParam &encoding-type=url :
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
{noformat}
{noformat}
some/test%10/
{noformat}
Can probably be parsed.
was:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Description:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/falsesome/test/
{noformat}
Child folder of 'some' contains
{noformat}
some/test/
{noformat}
which can't be parsed.
But with the urlParam:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
{noformat}
{noformat}
some/test%10/
{noformat}
Can probably be parsed.
was:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>different-bucket-namesome/1000
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Description:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type param?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
Here's an example XML response from S3 when listing objects from cURL:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>different-bucket-namesome/1000/falsesome/test/
{noformat}
Child folder of 'some' contains
{noformat}
some/test/
{noformat}
which can't be parsed.
But with the urlParam:
{noformat}
http://s3.amazonaws.com/doc/2006-03-01/";>cloudsync-performance-testssome/1000/urlfalsesome/test%10/
{noformat}
{noformat}
some/test%10/
{noformat}
Can probably be parsed.
was:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type header?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/br
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Description:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type header?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder if one of the children
contains control characters.
was:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type header?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder.
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://sclas-cloud-storage-master.s3.amazonaws.com/?delimit
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Description:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
When there's a control character in the folder path in S3, we can't parse it
from the response because it throws SAXParseException.
Can there be an option that at least lets us forward the encoding-type header?
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_RequestSyntax
And url decode it for us so that listing can be possible? This bug currently
doesn't allow us to list any children of a root folder.
was:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
> HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 323; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
[jira] [Created] (JCLOUDS-1638) SAXParseException on S3 Listing
Jacob Nguyen created JCLOUDS-1638:
-
Summary: SAXParseException on S3 Listing
Key: JCLOUDS-1638
URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
Project: jclouds
Issue Type: Bug
Reporter: Jacob Nguyen
Assignee: Andrew Gaul
{noformat}
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Description:
{noformat}
java.lang.RuntimeException: request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
request: GET
https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
lineNumber: 2; columnNumber: 323; Character reference "" is an invalid XML
character.
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
at
org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
at
org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
{noformat}
was:
{noformat}
{noformat}
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
> HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 323; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1638) SAXParseException on S3 Listing
[
https://issues.apache.org/jira/browse/JCLOUDS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jacob Nguyen updated JCLOUDS-1638:
--
Affects Version/s: 2.5.0
> SAXParseException on S3 Listing
> ---
>
> Key: JCLOUDS-1638
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1638
> Project: jclouds
> Issue Type: Bug
>Affects Versions: 2.5.0
>Reporter: Jacob Nguyen
>Assignee: Andrew Gaul
>Priority: Major
>
> {noformat}
> java.lang.RuntimeException: request: GET
> https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
> HTTP/1.1; response: HTTP/1.1 200 OK; cause: java.lang.RuntimeException:
> request: GET
> https://sclas-cloud-storage-master.s3.amazonaws.com/?delimiter=/&prefix=Data/57-2943/10-8-20/&max-keys=1000
> HTTP/1.1; error at 323:2 in document ; cause: org.xml.sax.SAXParseException;
> lineNumber: 2; columnNumber: 323; Character reference "" is an invalid
> XML character.
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:174)
> at
> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:146)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:86)
> at org.jclouds.http.functions.ParseSax.apply(ParseSax.java:52)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
> at
> org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
> at jdk.proxy2/jdk.proxy2.$Proxy235.listBucket(Unknown Source)
> at org.jclouds.s3.blobstore.S3BlobStore.list(S3BlobStore.java:177)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
[
https://issues.apache.org/jira/browse/JCLOUDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841080#comment-17841080
]
Philipp Nanz commented on JCLOUDS-1637:
---
I have tried it with 2.6.1-SNAPSHOT and it works now, thank you!
> JClouds does not work with Jakarta XML bind on classpath
>
>
> Key: JCLOUDS-1637
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.6.0
>Reporter: Philipp Nanz
>Assignee: Andrew Gaul
>Priority: Major
> Fix For: 2.6.1
>
>
> This is kind-of a follow up to JCLOUDS-1627:
> When you have Spring Boot 3.2 powered environment/classpath, JClouds will
> fail to start with {{{}java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException{}}}.
> The issue basically stems from
> https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
> which is still pointing to {{javax.xml.bind}} classes.
> The most simplistic solution probably would be to just replace the package
> names with {{{}jakarta.xml.bind{}}}.
> However, if you want to continue supporting {{{}javax.xml.bind{}}}, a
> possible solution would be to have two different XMLParser implementations
> and then load either of them, depending on which JAXB variant is available on
> the classpath.
> For reference, I have created a simple demo application that showcases the
> problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
> Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
[
https://issues.apache.org/jira/browse/JCLOUDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840826#comment-17840826
]
Andrew Gaul commented on JCLOUDS-1637:
--
It's possible the snapshot publication is broken. Can you try just {{mvn
install}} locally?
> JClouds does not work with Jakarta XML bind on classpath
>
>
> Key: JCLOUDS-1637
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.6.0
>Reporter: Philipp Nanz
>Assignee: Andrew Gaul
>Priority: Major
> Fix For: 2.6.1
>
>
> This is kind-of a follow up to JCLOUDS-1627:
> When you have Spring Boot 3.2 powered environment/classpath, JClouds will
> fail to start with {{{}java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException{}}}.
> The issue basically stems from
> https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
> which is still pointing to {{javax.xml.bind}} classes.
> The most simplistic solution probably would be to just replace the package
> names with {{{}jakarta.xml.bind{}}}.
> However, if you want to continue supporting {{{}javax.xml.bind{}}}, a
> possible solution would be to have two different XMLParser implementations
> and then load either of them, depending on which JAXB variant is available on
> the classpath.
> For reference, I have created a simple demo application that showcases the
> problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
> Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
[
https://issues.apache.org/jira/browse/JCLOUDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840666#comment-17840666
]
Philipp Nanz commented on JCLOUDS-1637:
---
Thanks for the quick fix! I will be happy to test it, I couldn't find it
though. Is it supposed to be published in
[https://repository.apache.org/content/repositories/snapshots?|https://repository.apache.org/content/repositories/snapshots]
> JClouds does not work with Jakarta XML bind on classpath
>
>
> Key: JCLOUDS-1637
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.6.0
>Reporter: Philipp Nanz
>Assignee: Andrew Gaul
>Priority: Major
> Fix For: 2.6.1
>
>
> This is kind-of a follow up to JCLOUDS-1627:
> When you have Spring Boot 3.2 powered environment/classpath, JClouds will
> fail to start with {{{}java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException{}}}.
> The issue basically stems from
> https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
> which is still pointing to {{javax.xml.bind}} classes.
> The most simplistic solution probably would be to just replace the package
> names with {{{}jakarta.xml.bind{}}}.
> However, if you want to continue supporting {{{}javax.xml.bind{}}}, a
> possible solution would be to have two different XMLParser implementations
> and then load either of them, depending on which JAXB variant is available on
> the classpath.
> For reference, I have created a simple demo application that showcases the
> problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
> Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
[
https://issues.apache.org/jira/browse/JCLOUDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840407#comment-17840407
]
ASF subversion and git services commented on JCLOUDS-1637:
--
Commit c73660dac8303266d875a4fabe63cc731ebdd437 in jclouds's branch
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=c73660dac8 ]
JCLOUDS-1637: Use glassfish jaxb implementation
Required by jakarta.xml.bind-api.
> JClouds does not work with Jakarta XML bind on classpath
>
>
> Key: JCLOUDS-1637
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.6.0
>Reporter: Philipp Nanz
>Assignee: Andrew Gaul
>Priority: Major
> Fix For: 2.6.1
>
>
> This is kind-of a follow up to JCLOUDS-1627:
> When you have Spring Boot 3.2 powered environment/classpath, JClouds will
> fail to start with {{{}java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException{}}}.
> The issue basically stems from
> https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
> which is still pointing to {{javax.xml.bind}} classes.
> The most simplistic solution probably would be to just replace the package
> names with {{{}jakarta.xml.bind{}}}.
> However, if you want to continue supporting {{{}javax.xml.bind{}}}, a
> possible solution would be to have two different XMLParser implementations
> and then load either of them, depending on which JAXB variant is available on
> the classpath.
> For reference, I have created a simple demo application that showcases the
> problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
> Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
[
https://issues.apache.org/jira/browse/JCLOUDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1637.
--
Fix Version/s: 2.6.1
Resolution: Fixed
Please wait a few hours for CI then test 2.6.1-SNAPSHOT. I wonder if there are
any other stale javax references? For example s3 has
javax.xml.parsers.DocumentBuilderFactory.
> JClouds does not work with Jakarta XML bind on classpath
>
>
> Key: JCLOUDS-1637
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.6.0
>Reporter: Philipp Nanz
>Assignee: Andrew Gaul
>Priority: Major
> Fix For: 2.6.1
>
>
> This is kind-of a follow up to JCLOUDS-1627:
> When you have Spring Boot 3.2 powered environment/classpath, JClouds will
> fail to start with {{{}java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException{}}}.
> The issue basically stems from
> https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
> which is still pointing to {{javax.xml.bind}} classes.
> The most simplistic solution probably would be to just replace the package
> names with {{{}jakarta.xml.bind{}}}.
> However, if you want to continue supporting {{{}javax.xml.bind{}}}, a
> possible solution would be to have two different XMLParser implementations
> and then load either of them, depending on which JAXB variant is available on
> the classpath.
> For reference, I have created a simple demo application that showcases the
> problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
> Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
[
https://issues.apache.org/jira/browse/JCLOUDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1637:
-
Issue Type: Improvement (was: Bug)
> JClouds does not work with Jakarta XML bind on classpath
>
>
> Key: JCLOUDS-1637
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.6.0
>Reporter: Philipp Nanz
>Assignee: Andrew Gaul
>Priority: Major
>
> This is kind-of a follow up to JCLOUDS-1627:
> When you have Spring Boot 3.2 powered environment/classpath, JClouds will
> fail to start with {{{}java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException{}}}.
> The issue basically stems from
> https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
> which is still pointing to {{javax.xml.bind}} classes.
> The most simplistic solution probably would be to just replace the package
> names with {{{}jakarta.xml.bind{}}}.
> However, if you want to continue supporting {{{}javax.xml.bind{}}}, a
> possible solution would be to have two different XMLParser implementations
> and then load either of them, depending on which JAXB variant is available on
> the classpath.
> For reference, I have created a simple demo application that showcases the
> problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
> Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
[
https://issues.apache.org/jira/browse/JCLOUDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840401#comment-17840401
]
ASF subversion and git services commented on JCLOUDS-1637:
--
Commit 7a438ceebdc298a707d4ce8e1ec22bea2834bf3d in jclouds's branch
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=7a438ceebd ]
JCLOUDS-1637: Replace java.xml.bind uses
> JClouds does not work with Jakarta XML bind on classpath
>
>
> Key: JCLOUDS-1637
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-core
>Affects Versions: 2.6.0
>Reporter: Philipp Nanz
>Assignee: Andrew Gaul
>Priority: Major
>
> This is kind-of a follow up to JCLOUDS-1627:
> When you have Spring Boot 3.2 powered environment/classpath, JClouds will
> fail to start with {{{}java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException{}}}.
> The issue basically stems from
> https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
> which is still pointing to {{javax.xml.bind}} classes.
> The most simplistic solution probably would be to just replace the package
> names with {{{}jakarta.xml.bind{}}}.
> However, if you want to continue supporting {{{}javax.xml.bind{}}}, a
> possible solution would be to have two different XMLParser implementations
> and then load either of them, depending on which JAXB variant is available on
> the classpath.
> For reference, I have created a simple demo application that showcases the
> problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
> Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
[
https://issues.apache.org/jira/browse/JCLOUDS-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Nanz updated JCLOUDS-1637:
--
Description:
This is kind-of a follow up to JCLOUDS-1627:
When you have Spring Boot 3.2 powered environment/classpath, JClouds will fail
to start with {{{}java.lang.NoClassDefFoundError:
javax/xml/bind/JAXBException{}}}.
The issue basically stems from
https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
which is still pointing to {{javax.xml.bind}} classes.
The most simplistic solution probably would be to just replace the package
names with {{{}jakarta.xml.bind{}}}.
However, if you want to continue supporting {{{}javax.xml.bind{}}}, a possible
solution would be to have two different XMLParser implementations and then load
either of them, depending on which JAXB variant is available on the classpath.
For reference, I have created a simple demo application that showcases the
problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
Thanks in advance for looking into it!
was:
This is kind-of a follow up to JCLOUDS-1627:
When you have Spring Boot 3.2 powered environment/classpath, JClouds will fail
to start with {{java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException}}.
The issue basically stems from
[https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java|JAXBParser.java],
which is still pointing to {{javax.xml.bind}} classes.
The most simplistic solution probably would be to just replace the package
names with {{jakarta.xml.bind}}.
However, if you want to continue supporting {{javax.xml.bind}}, a possible
solution would be to have two different XMLParser implementations and then load
either of them, depending on which JAXB variant is available on the classpath.
For reference, I have created a simple demo application that showcases the
problem: https://github.com/philippn/jclouds-vs-jakarta-xml-bind
Thanks in advance for looking into it!
> JClouds does not work with Jakarta XML bind on classpath
>
>
> Key: JCLOUDS-1637
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-core
>Affects Versions: 2.6.0
>Reporter: Philipp Nanz
>Assignee: Andrew Gaul
>Priority: Major
>
> This is kind-of a follow up to JCLOUDS-1627:
> When you have Spring Boot 3.2 powered environment/classpath, JClouds will
> fail to start with {{{}java.lang.NoClassDefFoundError:
> javax/xml/bind/JAXBException{}}}.
> The issue basically stems from
> https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java,
> which is still pointing to {{javax.xml.bind}} classes.
> The most simplistic solution probably would be to just replace the package
> names with {{{}jakarta.xml.bind{}}}.
> However, if you want to continue supporting {{{}javax.xml.bind{}}}, a
> possible solution would be to have two different XMLParser implementations
> and then load either of them, depending on which JAXB variant is available on
> the classpath.
> For reference, I have created a simple demo application that showcases the
> problem: [https://github.com/philippn/jclouds-vs-jakarta-xml-bind]
> Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (JCLOUDS-1637) JClouds does not work with Jakarta XML bind on classpath
Philipp Nanz created JCLOUDS-1637:
-
Summary: JClouds does not work with Jakarta XML bind on classpath
Key: JCLOUDS-1637
URL: https://issues.apache.org/jira/browse/JCLOUDS-1637
Project: jclouds
Issue Type: Bug
Components: jclouds-core
Affects Versions: 2.6.0
Reporter: Philipp Nanz
Assignee: Andrew Gaul
This is kind-of a follow up to JCLOUDS-1627:
When you have Spring Boot 3.2 powered environment/classpath, JClouds will fail
to start with {{java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException}}.
The issue basically stems from
[https://github.com/apache/jclouds/blob/master/core/src/main/java/org/jclouds/xml/internal/JAXBParser.java|JAXBParser.java],
which is still pointing to {{javax.xml.bind}} classes.
The most simplistic solution probably would be to just replace the package
names with {{jakarta.xml.bind}}.
However, if you want to continue supporting {{javax.xml.bind}}, a possible
solution would be to have two different XMLParser implementations and then load
either of them, depending on which JAXB variant is available on the classpath.
For reference, I have created a simple demo application that showcases the
problem: https://github.com/philippn/jclouds-vs-jakarta-xml-bind
Thanks in advance for looking into it!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (JCLOUDS-1636) Allow arbitrary strings for storage tier
Andrew Gaul created JCLOUDS-1636: Summary: Allow arbitrary strings for storage tier Key: JCLOUDS-1636 URL: https://issues.apache.org/jira/browse/JCLOUDS-1636 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 2.6.0 Reporter: Andrew Gaul Assignee: Andrew Gaul This will allow clients to express any storage tier instead of only the jclouds enum values. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1635) Add another blobstore Tier
[ https://issues.apache.org/jira/browse/JCLOUDS-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul resolved JCLOUDS-1635. -- Fix Version/s: 2.6.1 Assignee: Andrew Gaul Resolution: Fixed > Add another blobstore Tier > -- > > Key: JCLOUDS-1635 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1635 > Project: jclouds > Issue Type: New Feature > Components: jclouds-blobstore >Affects Versions: 2.6.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Major > Fix For: 2.6.1 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently jclouds supports STANDARD, INFREQUENT, and ARCHIVE. But AzureBlob, > GCS, and S3 all support at least 4 tiers. So let's add something cheaper > than INFREQUENT but faster than ARCHIVE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1635) Add another blobstore Tier
[ https://issues.apache.org/jira/browse/JCLOUDS-1635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839205#comment-17839205 ] ASF subversion and git services commented on JCLOUDS-1635: -- Commit da1bc06f9efb626c03eb3119e9c77adf5b12f179 in jclouds's branch refs/heads/master from Andrew Gaul [ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=da1bc06f9e ] JCLOUDS-1635: Add COOL and COLD to Tier The former replaces INFREQUENT. References gaul/s3proxy#625. > Add another blobstore Tier > -- > > Key: JCLOUDS-1635 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1635 > Project: jclouds > Issue Type: New Feature > Components: jclouds-blobstore >Affects Versions: 2.6.0 >Reporter: Andrew Gaul >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Currently jclouds supports STANDARD, INFREQUENT, and ARCHIVE. But AzureBlob, > GCS, and S3 all support at least 4 tiers. So let's add something cheaper > than INFREQUENT but faster than ARCHIVE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (JCLOUDS-1635) Add another blobstore Tier
Andrew Gaul created JCLOUDS-1635: Summary: Add another blobstore Tier Key: JCLOUDS-1635 URL: https://issues.apache.org/jira/browse/JCLOUDS-1635 Project: jclouds Issue Type: New Feature Components: jclouds-blobstore Affects Versions: 2.6.0 Reporter: Andrew Gaul Currently jclouds supports STANDARD, INFREQUENT, and ARCHIVE. But AzureBlob, GCS, and S3 all support at least 4 tiers. So let's add something cheaper than INFREQUENT but faster than ARCHIVE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1634) Add AzureBlob cold access tier
[ https://issues.apache.org/jira/browse/JCLOUDS-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul resolved JCLOUDS-1634. -- Fix Version/s: 2.6.1 Resolution: Fixed > Add AzureBlob cold access tier > -- > > Key: JCLOUDS-1634 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1634 > Project: jclouds > Issue Type: Improvement > Components: jclouds-blobstore >Affects Versions: 2.6.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Minor > Labels: azureblob > Fix For: 2.6.1 > > > Currently jclouds only supports hot, cool, and archive: > https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1634) Add AzureBlob cold access tier
[ https://issues.apache.org/jira/browse/JCLOUDS-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836911#comment-17836911 ] ASF subversion and git services commented on JCLOUDS-1634: -- Commit e1f34bbfa77619871fdf1af9b3090c8ff695869d in jclouds's branch refs/heads/master from Andrew Gaul [ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=e1f34bbfa7 ] JCLOUDS-1634: Add AzureBlob COLD access tier References gaul/s3proxy#625. > Add AzureBlob cold access tier > -- > > Key: JCLOUDS-1634 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1634 > Project: jclouds > Issue Type: Improvement > Components: jclouds-blobstore >Affects Versions: 2.6.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Minor > Labels: azureblob > > Currently jclouds only supports hot, cool, and archive: > https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1634) Add AzureBlob cold access tier
[ https://issues.apache.org/jira/browse/JCLOUDS-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul updated JCLOUDS-1634: - Labels: azureblob (was: ) > Add AzureBlob cold access tier > -- > > Key: JCLOUDS-1634 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1634 > Project: jclouds > Issue Type: Improvement > Components: jclouds-blobstore >Affects Versions: 2.6.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Minor > Labels: azureblob > > Currently jclouds only supports hot, cool, and archive: > https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (JCLOUDS-1634) Add AzureBlob cold access tier
Andrew Gaul created JCLOUDS-1634: Summary: Add AzureBlob cold access tier Key: JCLOUDS-1634 URL: https://issues.apache.org/jira/browse/JCLOUDS-1634 Project: jclouds Issue Type: Improvement Components: jclouds-blobstore Affects Versions: 2.6.0 Reporter: Andrew Gaul Assignee: Andrew Gaul Currently jclouds only supports hot, cool, and archive: https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1631) AWS S3, sign for authorization header failed if query part contains special chars
[ https://issues.apache.org/jira/browse/JCLOUDS-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul resolved JCLOUDS-1631. -- Fix Version/s: 2.6.1 Assignee: Andrew Gaul Resolution: Fixed > AWS S3, sign for authorization header failed if query part contains special > chars > - > > Key: JCLOUDS-1631 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1631 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.5.0 >Reporter: Maksim Hadalau >Assignee: Andrew Gaul >Priority: Major > Fix For: 2.6.1 > > Time Spent: 50m > Remaining Estimate: 0h > > Problem description: > Can not list blobs for aws-s3 provider if prefix contains special chars %/& > Steps to reproduce: > try to list blobs with following prefix: > "Folder (`~!@#$%^&*-_+[]'|<>.?) Name/" > Actual behavior: > Error: URLDecoder: Incomplete trailing escape (%) pattern > Expected behavior: > provided prefix must be listed > Problem location: > `AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader()` > > Multimap queryMap = > queryParser().apply(request.getEndpoint().getQuery()); > request.getEndpoint().getQuery() - returns a decoded query string > however queryParser() require encoded one > Fix: > Multimap queryMap = > queryParser().apply(request.getEndpoint().getRawQuery()); > When jclouds generates a request to the AWS it encodes prefix (encoding all > special chars in it, including % and &), however calling `getQuery()` returns > decoded version of query string which lead to unpredictable behavior. > P.S. required patch in 2.5.x if possible > PR https://github.com/apache/jclouds/pull/200 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1631) AWS S3, sign for authorization header failed if query part contains special chars
[ https://issues.apache.org/jira/browse/JCLOUDS-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836909#comment-17836909 ] ASF subversion and git services commented on JCLOUDS-1631: -- Commit 0688553087aabdaa7f503265bbc4a4cb60adef55 in jclouds's branch refs/heads/master from Maksim_Hadalau [ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=0688553087 ] JCLOUDS-1631: fix AWSRequestAuthorizeSignatureV4 when prefix contains special chars > AWS S3, sign for authorization header failed if query part contains special > chars > - > > Key: JCLOUDS-1631 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1631 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.5.0 >Reporter: Maksim Hadalau >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Problem description: > Can not list blobs for aws-s3 provider if prefix contains special chars %/& > Steps to reproduce: > try to list blobs with following prefix: > "Folder (`~!@#$%^&*-_+[]'|<>.?) Name/" > Actual behavior: > Error: URLDecoder: Incomplete trailing escape (%) pattern > Expected behavior: > provided prefix must be listed > Problem location: > `AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader()` > > Multimap queryMap = > queryParser().apply(request.getEndpoint().getQuery()); > request.getEndpoint().getQuery() - returns a decoded query string > however queryParser() require encoded one > Fix: > Multimap queryMap = > queryParser().apply(request.getEndpoint().getRawQuery()); > When jclouds generates a request to the AWS it encodes prefix (encoding all > special chars in it, including % and &), however calling `getQuery()` returns > decoded version of query string which lead to unpredictable behavior. > P.S. required patch in 2.5.x if possible > PR https://github.com/apache/jclouds/pull/200 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (JCLOUDS-1633) Google cloud storage handles content-encoding incorrectly
Maksim Hadalau created JCLOUDS-1633:
---
Summary: Google cloud storage handles content-encoding incorrectly
Key: JCLOUDS-1633
URL: https://issues.apache.org/jira/browse/JCLOUDS-1633
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore
Affects Versions: 2.5.0
Reporter: Maksim Hadalau
Steps to reproduce:
1. upload blob with Content-Encoding: gzip (content must be compressed on user
side)
2. verify that both getBlobMetadata and GCS UI shows correct object (stored
size == compressed size) and metadata returns Content-Encoding: gzip
3. Download blob using getBlob() and try to decompress payload
As result - decompression - failed because payload is not GZIP
Problem location
GoogleCloudStorageBlobStore.java
{code:java}
@Override
public Blob getBlob(String container, String name, GetOptions options) {
GoogleCloudStorageObject gcsObject = api.getObjectApi().getObject(container,
Strings2.urlEncode(name));
// here metadata contains a correct value for Content-Encoding: gzip
if (gcsObject == null) {
return null;
}
org.jclouds.http.options.GetOptions httpOptions =
blob2ObjectGetOptions.apply(options);
MutableBlobMetadata metadata = objectToBlobMetadata.apply(gcsObject);
Blob blob = new BlobImpl(metadata);
// TODO: Does getObject not get the payload?!
// Payload metadata returns Content-Encoding: null, due to google cloud
storage API, by default they do auto-decompression. If user want to download
compressed content - we should provide Accept-Encoding: gzip header. Which is
not possible right now (see GetOptions.class)
Payload payload = api.getObjectApi().download(container,
Strings2.urlEncode(name), httpOptions).getPayload();
// replacing payload metadata with object metadata is incorrect
// metadata contains Content-Encoding:gzip, but without providing
Accept-Encoding header payload metadata returned in decompressed form
(Content-Encoding: null)
payload.setContentMetadata(metadata.getContentMetadata()); // Doing this
first retains it on setPayload.
blob.setPayload(payload);
return blob;
} {code}
How to fix?
We should not replace payload metadata with object metadata(content encoding at
least) and extend GetOptions class with ability to provide
headers(Accept-Encoding in my case), or write a logic that will add
Accept-Encoding header if object metadata contains Content-Encoding header.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1623) Caused by: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
[
https://issues.apache.org/jira/browse/JCLOUDS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Dimitrov updated JCLOUDS-1623:
---
Description:
Hello,
We are observing an issue with JClouds while transferring a file with an input
stream only.
We are opening an input stream to a remote file and we are passing the stream
to the JClouds library, but after some time it fails with the following error:
{noformat}
Exception in thread "main" java.lang.RuntimeException: java.io.EOFException:
reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
at com.google.common.base.Throwables.propagate(Throwables.java:241)
at
org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
at
org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:385)
at
org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
at
org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
at org.example.Main.main(Main.java:50)
Caused by: java.io.EOFException: reached end of stream after skipping 14933328
bytes; 67108864 bytes expected
at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:807)
at
org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
... 5 more
{noformat}
Note: The issue occurs only for larger files (more than 32 mb).
Java version: sapmachine-jdk-11.0.15.0.1.jdk
IAAS: AWS-s3
If we pass a BufferedInputStream the code works, so I guess that there is some
issue with the processing of the stream. The stream returned by the Java Http
Client is HttpResponseInputStream.
Code:
{code:title=Main.java|borderStyle=solid}
package org.example;
import com.google.common.collect.ImmutableSet;
import com.google.inject.Module;
import org.jclouds.ContextBuilder;
import org.jclouds.blobstore.BlobStore;
import org.jclouds.blobstore.BlobStoreContext;
import org.jclouds.blobstore.domain.Blob;
import org.jclouds.blobstore.options.PutOptions;
import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;
import java.io.BufferedInputStream;
import java.io.InputStream;
import java.net.Authenticator;
import java.net.PasswordAuthentication;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.UUID;
public class Main {
public static void main(String[] args) throws Exception {
Iterable modules = ImmutableSet.of(
new SLF4JLoggingModule());
ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-s3")
.credentials("", "")
.modules(modules);
BlobStoreContext blobStoreContext =
contextBuilder.buildView(BlobStoreContext.class);
BlobStore blobStore = blobStoreContext.getBlobStore();
String decodedUrl = "";
HttpClient client = buildHttpClient(decodedUrl);
HttpResponse response =
callRemoteEndpointWithRetry(client, decodedUrl);
long fileSize = response.headers()
.firstValueAsLong("Content-Length")
.orElseThrow(() -> new IllegalArgumentException("No
Content-Length"));
System.out.println("Bytes: " + fileSize);
InputStream inputStream = response.body();
String container = "";
Blob blob = blobStore.blobBuilder("TEST.zip")
//.payload(new BufferedInputStream(inputStream, 8 * 1024))
.payload(inputStream)
.contentDisposition(UUID.randomUUID().toString())
.contentType("application/octet-stream")
.contentLength(fileSize)
.build();
blobStore.putBlob(container, blob, PutOptions.Builder.multipart());
}
private static HttpClient buildHttpClient(String decodedUrl) {
return HttpClient.newBuilder()
.version(HttpClient.Version.HTTP_1_1)
.connectTimeout(Duration.ofMinutes(60))
.followRedirects(HttpClient.Redirect.NORMAL)
.authenticator(buildPasswordAuthenticator(decodedUrl))
.build();
}
private static Authenticator buildPasswordAuthenticator(String decodedUrl) {
return new Authenticator() {
@Override
protected PasswordAuthentication getPasswordAuthentication() {
var uri = URI.create(decodedUrl);
var userInfo = uri.getUserInfo();
if (userInfo != null) {
var separatorIndex = userInfo.indexOf(':');
var
[jira] [Commented] (JCLOUDS-1630) UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve bracket) ocured
[
https://issues.apache.org/jira/browse/JCLOUDS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828964#comment-17828964
]
ASF subversion and git services commented on JCLOUDS-1630:
--
Commit bc43572d657956e013b094dfa08618c61a795e3d in jclouds's branch
refs/heads/master from Maxim
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=bc43572d65 ]
JCLOUDS-1630: Handle URI template properly with opened curve bracket (#199)
Co-authored-by: Maksim_Hadalau
> UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve
> bracket) ocured
> -
>
> Key: JCLOUDS-1630
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1630
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-core
>Affects Versions: 2.5.0
>Reporter: Maksim Hadalau
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Steps to reproduce:
> Try to create a blob with opened curve in a storage key
>
> {code:java}
> // below test fails because actual result is: "/repos/folder with "
> public void testIncorrectExpand() {
> assertEquals(expand("/repos/folder with { brackets in a key",
> ImmutableMap.of()),
> "/repos/folder with { brackets in a key"); }
>
> {code}
> Solution, if no variables provided - return template as is.
> patch to 2.5.x if possible
> PR https://github.com/apache/jclouds/pull/199
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aliaksandr updated JCLOUDS-1632:
Description:
*Steps to reproduce*:
# Configure blob store using `google-cloud-storage`
# Try to put a blob with name containing any Unicode character
# The error occurred something like below
{noformat}
SEVERE: error after writing 389/409 bytes to
https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
java.io.IOException: too many bytes written
at
java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
at
com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
at
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
at
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
{noformat}
*Root cause*
The constructor
org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
java.lang.Iterable) doesn't calculate
content's length properly.
Specifically the constructor counts the length of
org.jclouds.io.payloads.MultipartForm#createHeaders string instead of byte
array length of that string to UTF-8
AS a result a wrong content length is assigned to output stream causing the
error above.
*Solution*
Use the length of a byte array instead of string length obtained from the
method org.jclouds.io.payloads.MultipartForm#createHeaders
See the PR https://github.com/apache/jclouds/pull/201
was:
*Steps to reproduce*:
# Configure blob store using `google-cloud-storage`
# Try to put a blob with name containing any Unicode character
# The error occurred something like below
{noformat}
SEVERE: error after writing 389/409 bytes to
https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
java.io.IOException: too many bytes written
at
java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
at
com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHand
[jira] [Updated] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1632:
-
Flags: (was: Important)
> GCP BlobStore fails to put a blob if blob name contains non ASCII characters
>
>
> Key: JCLOUDS-1632
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1632
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Aliaksandr
>Priority: Major
> Labels: GCP
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> *Steps to reproduce*:
> # Configure blob store using `google-cloud-storage`
> # Try to put a blob with name containing any Unicode character
> # The error occurred something like below
> {noformat}
> SEVERE: error after writing 389/409 bytes to
> https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
> java.io.IOException: too many bytes written
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
> at
> com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
> at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
> at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
> {noformat}
> *Root cause*
> The contractor
> org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
> java.lang.Iterable) doesn't calculate
> content's length properly.
> Specifically the constructor counts the length of
> org.jclouds.io.payloads.MultipartForm#createHeaders string instead of byte
> array length of that string to UTF-8
> AS a result a wrong content length is assigned to output stream causing the
> error above.
> *Solution*
> Use the length of a byte array instead of string length obtained from the
> method org.jclouds.io.payloads.MultipartForm#createHeaders
> See the PR https://github.com/apache/jclouds/pull/201
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1632.
--
Fix Version/s: 2.6.1
Assignee: Andrew Gaul
Resolution: Fixed
> GCP BlobStore fails to put a blob if blob name contains non ASCII characters
>
>
> Key: JCLOUDS-1632
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1632
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Aliaksandr
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> *Steps to reproduce*:
> # Configure blob store using `google-cloud-storage`
> # Try to put a blob with name containing any Unicode character
> # The error occurred something like below
> {noformat}
> SEVERE: error after writing 389/409 bytes to
> https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
> java.io.IOException: too many bytes written
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
> at
> com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
> at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
> at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
> {noformat}
> *Root cause*
> The contractor
> org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
> java.lang.Iterable) doesn't calculate
> content's length properly.
> Specifically the constructor counts the length of
> org.jclouds.io.payloads.MultipartForm#createHeaders string instead of byte
> array length of that string to UTF-8
> AS a result a wrong content length is assigned to output stream causing the
> error above.
> *Solution*
> Use the length of a byte array instead of string length obtained from the
> method org.jclouds.io.payloads.MultipartForm#createHeaders
> See the PR https://github.com/apache/jclouds/pull/201
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1632:
-
Labels: google-cloud-storage (was: GCP)
> GCP BlobStore fails to put a blob if blob name contains non ASCII characters
>
>
> Key: JCLOUDS-1632
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1632
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Aliaksandr
>Priority: Major
> Labels: google-cloud-storage
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> *Steps to reproduce*:
> # Configure blob store using `google-cloud-storage`
> # Try to put a blob with name containing any Unicode character
> # The error occurred something like below
> {noformat}
> SEVERE: error after writing 389/409 bytes to
> https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
> java.io.IOException: too many bytes written
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
> at
> com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
> at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
> at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
> {noformat}
> *Root cause*
> The contractor
> org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
> java.lang.Iterable) doesn't calculate
> content's length properly.
> Specifically the constructor counts the length of
> org.jclouds.io.payloads.MultipartForm#createHeaders string instead of byte
> array length of that string to UTF-8
> AS a result a wrong content length is assigned to output stream causing the
> error above.
> *Solution*
> Use the length of a byte array instead of string length obtained from the
> method org.jclouds.io.payloads.MultipartForm#createHeaders
> See the PR https://github.com/apache/jclouds/pull/201
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1632:
-
Component/s: (was: jclouds-core)
> GCP BlobStore fails to put a blob if blob name contains non ASCII characters
>
>
> Key: JCLOUDS-1632
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1632
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Aliaksandr
>Priority: Major
> Labels: GCP
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> *Steps to reproduce*:
> # Configure blob store using `google-cloud-storage`
> # Try to put a blob with name containing any Unicode character
> # The error occurred something like below
> {noformat}
> SEVERE: error after writing 389/409 bytes to
> https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
> java.io.IOException: too many bytes written
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
> at
> com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
> at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
> at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
> {noformat}
> *Root cause*
> The contractor
> org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
> java.lang.Iterable) doesn't calculate
> content's length properly.
> Specifically the constructor counts the length of
> org.jclouds.io.payloads.MultipartForm#createHeaders string instead of byte
> array length of that string to UTF-8
> AS a result a wrong content length is assigned to output stream causing the
> error above.
> *Solution*
> Use the length of a byte array instead of string length obtained from the
> method org.jclouds.io.payloads.MultipartForm#createHeaders
> See the PR https://github.com/apache/jclouds/pull/201
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828950#comment-17828950
]
ASF subversion and git services commented on JCLOUDS-1632:
--
Commit b379c17156237de96cedbb19d82bc86168fe83af in jclouds's branch
refs/heads/master from Aliaksandr Stsiapanay
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=b379c17156 ]
JCLOUDS-1632: GCP BlobStore fails to put a blob if blob name contains non ASCII
characters
> GCP BlobStore fails to put a blob if blob name contains non ASCII characters
>
>
> Key: JCLOUDS-1632
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1632
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-core
>Affects Versions: 2.5.0
>Reporter: Aliaksandr
>Priority: Major
> Labels: GCP
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Steps to reproduce*:
> # Configure blob store using `google-cloud-storage`
> # Try to put a blob with name containing any Unicode character
> # The error occurred something like below
> {noformat}
> SEVERE: error after writing 389/409 bytes to
> https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
> java.io.IOException: too many bytes written
> at
> java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
> at
> com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
> at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
> at
> org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
> at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
> at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
> at
> org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> at
> com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
> at
> org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> at
> com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
> at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
> {noformat}
> *Root cause*
> The contractor
> org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
> java.lang.Iterable) doesn't calculate
> content's length properly.
> Specifically the constructor counts the length of
> org.jclouds.io.payloads.MultipartForm#createHeaders string instead of byte
> array length of that string to UTF-8
> AS a result a wrong content length is assigned to output stream causing the
> error above.
> *Solution*
> Use the length of a byte array instead of string length obtained from the
> method org.jclouds.io.payloads.MultipartForm#createHeaders
> See the PR https://github.com/apache/jclouds/pull/201
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aliaksandr updated JCLOUDS-1632:
Description:
*Steps to reproduce*:
# Configure blob store using `google-cloud-storage`
# Try to put a blob with name containing any Unicode character
# The error occurred something like below
{noformat}
SEVERE: error after writing 389/409 bytes to
https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
java.io.IOException: too many bytes written
at
java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
at
com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
at
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
at
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
{noformat}
*Root cause*
The contractor
org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
java.lang.Iterable) doesn't calculate
content's length properly.
Specifically the constructor counts the length of
org.jclouds.io.payloads.MultipartForm#createHeaders string instead of byte
array length of that string to UTF-8
AS a result a wrong content length is assigned to output stream causing the
error above.
*Solution*
Use the length of a byte array instead of string length obtained from the
method org.jclouds.io.payloads.MultipartForm#createHeaders
See the PR https://github.com/apache/jclouds/pull/201
was:
*Steps to reproduce*:
# Configure blob store using `google-cloud-storage`
# Try to put a blob with name containing any Unicode character
# The error occurred something like below
{noformat}
SEVERE: error after writing 389/409 bytes to
https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
java.io.IOException: too many bytes written
at
java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
at
com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHand
[jira] [Updated] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
[
https://issues.apache.org/jira/browse/JCLOUDS-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aliaksandr updated JCLOUDS-1632:
Description:
*Steps to reproduce*:
# Configure blob store using `google-cloud-storage`
# Try to put a blob with name containing any Unicode character
# The error occurred something like below
{noformat}
SEVERE: error after writing 389/409 bytes to
https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
java.io.IOException: too many bytes written
at
java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
at
com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
at
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
at
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
{noformat}
*Root cause*
The contractor
org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
java.lang.Iterable) doesn't calculate
content's length properly.
Specifically the contractor counts the length of
org.jclouds.io.payloads.MultipartForm#createHeaders string instead of encoded
byte array length to UTF-8.
AS a result a wrong content length is assigned to output stream causing the
error above.
*Solution*
Use the length of byte array encoded to UTF-8 instead of string length obtained
from the method org.jclouds.io.payloads.MultipartForm#createHeaders
See the PR https://github.com/apache/jclouds/pull/201
was:
*Steps to reproduce*:
# Configure blob store using `google-cloud-storage`
# Try to put a blob with name containing any Unicode character
# The error occurred something like below
{noformat}
SEVERE: error after writing 389/409 bytes to
https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
java.io.IOException: too many bytes written
at
java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
at
com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
[jira] [Created] (JCLOUDS-1632) GCP BlobStore fails to put a blob if blob name contains non ASCII characters
Aliaksandr created JCLOUDS-1632:
---
Summary: GCP BlobStore fails to put a blob if blob name contains
non ASCII characters
Key: JCLOUDS-1632
URL: https://issues.apache.org/jira/browse/JCLOUDS-1632
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore, jclouds-core
Affects Versions: 2.5.0
Reporter: Aliaksandr
*Steps to reproduce*:
# Configure blob store using `google-cloud-storage`
# Try to put a blob with name containing any Unicode character
# The error occurred something like below
{noformat}
SEVERE: error after writing 389/409 bytes to
https://www.googleapis.com/upload/storage/v1/b/gcp-dev/o?uploadType=multipart
java.io.IOException: too many bytes written
at
java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3812)
at
com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
at org.jclouds.io.ByteStreams2.copy(ByteStreams2.java:73)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:302)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:175)
at
org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:66)
at
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
at
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
at
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
at
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at jdk.proxy3/jdk.proxy3.$Proxy86.multipartUpload(Unknown Source)
at
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:235)
at
org.jclouds.googlecloudstorage.blobstore.GoogleCloudStorageBlobStore.putBlob(GoogleCloudStorageBlobStore.java:207)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at
com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50)
at jdk.proxy3/jdk.proxy3.$Proxy78.putBlob(Unknown Source)
{noformat}
*Root cause*
The contractor
org.jclouds.io.payloads.MultipartForm#MultipartForm(java.lang.String,
java.lang.Iterable) doesn't calculate
content's length properly.
Specifically the contractor counts the length of
org.jclouds.io.payloads.MultipartForm#createHeaders string instead of encoded
byte array length to UTF-8.
AS a result a wrong content length is assigned to output stream causing the
error above.
*Solution*
Use the length of byte array encoded to UTF-8 instead of string length obtained
from the method org.jclouds.io.payloads.MultipartForm#createHeaders
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1631) AWS S3, sign for authorization header failed if query part contains special chars
[ https://issues.apache.org/jira/browse/JCLOUDS-1631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maksim Hadalau updated JCLOUDS-1631: Description: Problem description: Can not list blobs for aws-s3 provider if prefix contains special chars %/& Steps to reproduce: try to list blobs with following prefix: "Folder (`~!@#$%^&*-_+[]'|<>.?) Name/" Actual behavior: Error: URLDecoder: Incomplete trailing escape (%) pattern Expected behavior: provided prefix must be listed Problem location: `AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader()` Multimap queryMap = queryParser().apply(request.getEndpoint().getQuery()); request.getEndpoint().getQuery() - returns a decoded query string however queryParser() require encoded one Fix: Multimap queryMap = queryParser().apply(request.getEndpoint().getRawQuery()); When jclouds generates a request to the AWS it encodes prefix (encoding all special chars in it, including % and &), however calling `getQuery()` returns decoded version of query string which lead to unpredictable behavior. P.S. required patch in 2.5.x if possible PR https://github.com/apache/jclouds/pull/200 was: Problem description: Can not list blobs for aws-s3 provider if prefix contains special chars %/& Steps to reproduce: try to list blobs with following prefix: "Folder (`~!@#$%^&*-_+[]'|<>.?) Name/" Actual behavior: Error: URLDecoder: Incomplete trailing escape (%) pattern Expected behavior: provided prefix must be listed Problem location: `AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader()` Multimap queryMap = queryParser().apply(request.getEndpoint().getQuery()); request.getEndpoint().getQuery() - returns a decoded query string however queryParser() require encoded one Fix: Multimap queryMap = queryParser().apply(request.getEndpoint().getRawQuery()); When jclouds generates a request to the AWS it encodes prefix (encoding all special chars in it, including % and &), however calling `getQuery()` returns decoded version of query string which lead to unpredictable behavior. P.S. required patch in 2.5.x if possible > AWS S3, sign for authorization header failed if query part contains special > chars > - > > Key: JCLOUDS-1631 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1631 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.5.0 >Reporter: Maksim Hadalau >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Problem description: > Can not list blobs for aws-s3 provider if prefix contains special chars %/& > Steps to reproduce: > try to list blobs with following prefix: > "Folder (`~!@#$%^&*-_+[]'|<>.?) Name/" > Actual behavior: > Error: URLDecoder: Incomplete trailing escape (%) pattern > Expected behavior: > provided prefix must be listed > Problem location: > `AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader()` > > Multimap queryMap = > queryParser().apply(request.getEndpoint().getQuery()); > request.getEndpoint().getQuery() - returns a decoded query string > however queryParser() require encoded one > Fix: > Multimap queryMap = > queryParser().apply(request.getEndpoint().getRawQuery()); > When jclouds generates a request to the AWS it encodes prefix (encoding all > special chars in it, including % and &), however calling `getQuery()` returns > decoded version of query string which lead to unpredictable behavior. > P.S. required patch in 2.5.x if possible > PR https://github.com/apache/jclouds/pull/200 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (JCLOUDS-1631) AWS S3, sign for authorization header failed if query part contains special chars
Maksim Hadalau created JCLOUDS-1631: --- Summary: AWS S3, sign for authorization header failed if query part contains special chars Key: JCLOUDS-1631 URL: https://issues.apache.org/jira/browse/JCLOUDS-1631 Project: jclouds Issue Type: Bug Components: jclouds-blobstore Affects Versions: 2.5.0 Reporter: Maksim Hadalau Problem description: Can not list blobs for aws-s3 provider if prefix contains special chars %/& Steps to reproduce: try to list blobs with following prefix: "Folder (`~!@#$%^&*-_+[]'|<>.?) Name/" Actual behavior: Error: URLDecoder: Incomplete trailing escape (%) pattern Expected behavior: provided prefix must be listed Problem location: `AWSRequestAuthorizeSignatureV4.signForAuthorizationHeader()` Multimap queryMap = queryParser().apply(request.getEndpoint().getQuery()); request.getEndpoint().getQuery() - returns a decoded query string however queryParser() require encoded one Fix: Multimap queryMap = queryParser().apply(request.getEndpoint().getRawQuery()); When jclouds generates a request to the AWS it encodes prefix (encoding all special chars in it, including % and &), however calling `getQuery()` returns decoded version of query string which lead to unpredictable behavior. P.S. required patch in 2.5.x if possible -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1630) UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve bracket) ocured
[
https://issues.apache.org/jira/browse/JCLOUDS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maksim Hadalau updated JCLOUDS-1630:
Description:
Steps to reproduce:
Try to create a blob with opened curve in a storage key
{code:java}
// below test fails because actual result is: "/repos/folder with "
public void testIncorrectExpand() {
assertEquals(expand("/repos/folder with { brackets in a key",
ImmutableMap.of()),
"/repos/folder with { brackets in a key"); }
{code}
Solution, if no variables provided - return template as is.
patch to 2.5.x if possible
PR https://github.com/apache/jclouds/pull/199
was:
Steps to reproduce:
Try to create a blob with opened curve in a storage key
{code:java}
// below test fails because actual result is: "/repos/folder with "
public void testIncorrectExpand() {
assertEquals(expand("/repos/folder with { brackets in a key",
ImmutableMap.of()),
"/repos/folder with { brackets in a key"); }
{code}
Solution, if no variables provided - return template as is.
patch to 2.5.x if possible
> UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve
> bracket) ocured
> -
>
> Key: JCLOUDS-1630
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1630
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-core
>Affects Versions: 2.5.0
>Reporter: Maksim Hadalau
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps to reproduce:
> Try to create a blob with opened curve in a storage key
>
> {code:java}
> // below test fails because actual result is: "/repos/folder with "
> public void testIncorrectExpand() {
> assertEquals(expand("/repos/folder with { brackets in a key",
> ImmutableMap.of()),
> "/repos/folder with { brackets in a key"); }
>
> {code}
> Solution, if no variables provided - return template as is.
> patch to 2.5.x if possible
> PR https://github.com/apache/jclouds/pull/199
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1630) UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve bracket) ocured
[
https://issues.apache.org/jira/browse/JCLOUDS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maksim Hadalau updated JCLOUDS-1630:
Description:
Steps to reproduce:
Try to create a blob with opened curve in a storage key
{code:java}
// blow test fails because actual result is: "/repos/folder with "
public void testIncorrectExpand() {
assertEquals(expand("/repos/folder with { brackets in a key",
ImmutableMap.of()),
"/repos/folder with { brackets in a key"); }
{code}
Solution, if no variables provided - return template as is.
patch to 2.5.x if possible
was:
Steps to reproduce:
Try to create a blob with opened curve in a storage key
// blow test fails because actual result is: "/repos/folder with "
public void testIncorrectExpand() {
assertEquals(expand("/repos/folder with { brackets in a key",
ImmutableMap.of()),
"/repos/folder with { brackets in a key");
}
Solution, if no variables provided - return template as is.
patch to 2.5.x if possible
> UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve
> bracket) ocured
> -
>
> Key: JCLOUDS-1630
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1630
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-core
>Affects Versions: 2.5.0
>Reporter: Maksim Hadalau
>Priority: Major
>
> Steps to reproduce:
> Try to create a blob with opened curve in a storage key
>
> {code:java}
> // blow test fails because actual result is: "/repos/folder with "
> public void testIncorrectExpand() {
> assertEquals(expand("/repos/folder with { brackets in a key",
> ImmutableMap.of()),
> "/repos/folder with { brackets in a key"); }
>
> {code}
> Solution, if no variables provided - return template as is.
> patch to 2.5.x if possible
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1630) UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve bracket) ocured
[
https://issues.apache.org/jira/browse/JCLOUDS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maksim Hadalau updated JCLOUDS-1630:
Description:
Steps to reproduce:
Try to create a blob with opened curve in a storage key
{code:java}
// below test fails because actual result is: "/repos/folder with "
public void testIncorrectExpand() {
assertEquals(expand("/repos/folder with { brackets in a key",
ImmutableMap.of()),
"/repos/folder with { brackets in a key"); }
{code}
Solution, if no variables provided - return template as is.
patch to 2.5.x if possible
was:
Steps to reproduce:
Try to create a blob with opened curve in a storage key
{code:java}
// blow test fails because actual result is: "/repos/folder with "
public void testIncorrectExpand() {
assertEquals(expand("/repos/folder with { brackets in a key",
ImmutableMap.of()),
"/repos/folder with { brackets in a key"); }
{code}
Solution, if no variables provided - return template as is.
patch to 2.5.x if possible
> UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve
> bracket) ocured
> -
>
> Key: JCLOUDS-1630
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1630
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore, jclouds-core
>Affects Versions: 2.5.0
>Reporter: Maksim Hadalau
>Priority: Major
>
> Steps to reproduce:
> Try to create a blob with opened curve in a storage key
>
> {code:java}
> // below test fails because actual result is: "/repos/folder with "
> public void testIncorrectExpand() {
> assertEquals(expand("/repos/folder with { brackets in a key",
> ImmutableMap.of()),
> "/repos/folder with { brackets in a key"); }
>
> {code}
> Solution, if no variables provided - return template as is.
> patch to 2.5.x if possible
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (JCLOUDS-1630) UriTemplates.expand() silently cut storage prefix (blob key) if { (open curve bracket) ocured
Maksim Hadalau created JCLOUDS-1630:
---
Summary: UriTemplates.expand() silently cut storage prefix (blob
key) if { (open curve bracket) ocured
Key: JCLOUDS-1630
URL: https://issues.apache.org/jira/browse/JCLOUDS-1630
Project: jclouds
Issue Type: Bug
Components: jclouds-blobstore, jclouds-core
Affects Versions: 2.5.0
Reporter: Maksim Hadalau
Steps to reproduce:
Try to create a blob with opened curve in a storage key
// blow test fails because actual result is: "/repos/folder with "
public void testIncorrectExpand() {
assertEquals(expand("/repos/folder with { brackets in a key",
ImmutableMap.of()),
"/repos/folder with { brackets in a key");
}
Solution, if no variables provided - return template as is.
patch to 2.5.x if possible
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1626) `FileInputStream` leak when using `filesystem` provider and performing a multipart upload
[
https://issues.apache.org/jira/browse/JCLOUDS-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1626.
--
Assignee: Andrew Gaul
Resolution: Fixed
This is a problem with S3Proxy, not jclouds and probably addressed by
https://github.com/gaul/s3proxy/commit/ec12ae0fe502aaae858ee60c256376c604b07093.
Please provide a proper test case in the future.
> `FileInputStream` leak when using `filesystem` provider and performing a
> multipart upload
> -
>
> Key: JCLOUDS-1626
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1626
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Reporter: Ji Xinchi
>Assignee: Andrew Gaul
>Priority: Critical
> Labels: filesystem
>
> I'm using gaul/s3proxy built on the latest jclouds. The provider
> configuration is "filesystem", and basedir configuration is a directory where
> a NFS is mounted.
>
> When I performed a multipart upload, I found that files such as
> ".nfs10bf0005" remained in the directory after uploading.
> Each file corresponded to one part and the files could not be deleted. The
> error "Device Busy" was reported. These files will disappear automatically
> after restarting s3proxy.
>
> I found this PR, and tried to restore following snippet and it worked. But
> this is just a temporary fix.
> {code:java}
> InputStream is = blob.getPayload().openStream();
> if (is instanceof FileInputStream) {
> // except for FileInputStream since large MPU can open too many
> fds
> is.close();
> payload = blob.getPayload();
> } else {
> blob.resetPayload(/*release=*/ false);
> payload = new InputStreamPayload(is);
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1626) `FileInputStream` leak when using `filesystem` provider and performing a multipart upload
[
https://issues.apache.org/jira/browse/JCLOUDS-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822884#comment-17822884
]
ASF subversion and git services commented on JCLOUDS-1626:
--
Commit 4c7fb2c8b9f36207cd0b04f6266521412e6678f3 in jclouds's branch
refs/heads/master from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=4c7fb2c8b9 ]
JCLOUDS-1626: Close stream in MultiBlobInputStream
Otherwise the inner FileInputStream will leak if the caller only reads
part of the stream before closing the outer MultiBlobInputStream.
> `FileInputStream` leak when using `filesystem` provider and performing a
> multipart upload
> -
>
> Key: JCLOUDS-1626
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1626
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Reporter: Ji Xinchi
>Priority: Critical
> Labels: filesystem
>
> I'm using gaul/s3proxy built on the latest jclouds. The provider
> configuration is "filesystem", and basedir configuration is a directory where
> a NFS is mounted.
>
> When I performed a multipart upload, I found that files such as
> ".nfs10bf0005" remained in the directory after uploading.
> Each file corresponded to one part and the files could not be deleted. The
> error "Device Busy" was reported. These files will disappear automatically
> after restarting s3proxy.
>
> I found this PR, and tried to restore following snippet and it worked. But
> this is just a temporary fix.
> {code:java}
> InputStream is = blob.getPayload().openStream();
> if (is instanceof FileInputStream) {
> // except for FileInputStream since large MPU can open too many
> fds
> is.close();
> payload = blob.getPayload();
> } else {
> blob.resetPayload(/*release=*/ false);
> payload = new InputStreamPayload(is);
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1626) `FileInputStream` leak when using `filesystem` provider and performing a multipart upload
[
https://issues.apache.org/jira/browse/JCLOUDS-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822883#comment-17822883
]
Andrew Gaul commented on JCLOUDS-1626:
--
Apparently this is referencing b7f28f1e6a2632fce579360c7900b6dc548ff25f.
> `FileInputStream` leak when using `filesystem` provider and performing a
> multipart upload
> -
>
> Key: JCLOUDS-1626
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1626
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Reporter: Ji Xinchi
>Priority: Critical
> Labels: filesystem
>
> I'm using gaul/s3proxy built on the latest jclouds. The provider
> configuration is "filesystem", and basedir configuration is a directory where
> a NFS is mounted.
>
> When I performed a multipart upload, I found that files such as
> ".nfs10bf0005" remained in the directory after uploading.
> Each file corresponded to one part and the files could not be deleted. The
> error "Device Busy" was reported. These files will disappear automatically
> after restarting s3proxy.
>
> I found this PR, and tried to restore following snippet and it worked. But
> this is just a temporary fix.
> {code:java}
> InputStream is = blob.getPayload().openStream();
> if (is instanceof FileInputStream) {
> // except for FileInputStream since large MPU can open too many
> fds
> is.close();
> payload = blob.getPayload();
> } else {
> blob.resetPayload(/*release=*/ false);
> payload = new InputStreamPayload(is);
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1585) Unable to use Google Cloud Storage provider inside a Google Cloud environment
[ https://issues.apache.org/jira/browse/JCLOUDS-1585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821250#comment-17821250 ] Aliaksandr commented on JCLOUDS-1585: - [~gaul] Are there any plans to fix the issue soon? > Unable to use Google Cloud Storage provider inside a Google Cloud environment > - > > Key: JCLOUDS-1585 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1585 > Project: jclouds > Issue Type: New Feature > Components: jclouds-blobstore >Affects Versions: 2.4.0 > Environment: Google dataproc cluster with 1.5-ubuntu18 image version >Reporter: Alexis BRENON >Priority: Major > Labels: google-cloud-storage, help-wanted > > To generate a Google Storage blobstore, user needs to pass credentials > composed of identity and private key. > However, when running the application inside a Google Cloud environment > (Compute Engine, Dataproc, etc.) these credentials are not available and > suitable creadentials can be fetched through [Application Default Credentials > library|https://cloud.google.com/docs/authentication/production#automatically]. > This bug prevents anyone to use the GCS blobstore inside GCP environment as > already reported on > [StackOverflow|https://stackoverflow.com/q/56279711/4373898]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1629) Upgrade to Guice 7
[ https://issues.apache.org/jira/browse/JCLOUDS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul resolved JCLOUDS-1629. -- Fix Version/s: 2.6.0 Resolution: Fixed > Upgrade to Guice 7 > -- > > Key: JCLOUDS-1629 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1629 > Project: jclouds > Issue Type: Improvement > Components: jclouds-core >Affects Versions: 2.5.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Major > Labels: guice > Fix For: 2.6.0 > > Time Spent: 50m > Remaining Estimate: 0h > > [~basil] JCLOUDS-1627 upgrading some of the packages to jakarta but not the > important annotations one for Guice. When changing these to Jakarta I could > get Guice 7 working but not 6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1628) Netty 3.10.* has multiple security vulnerabilities
[ https://issues.apache.org/jira/browse/JCLOUDS-1628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul resolved JCLOUDS-1628. -- Fix Version/s: 2.6.0 Assignee: Andrew Gaul Resolution: Fixed > Netty 3.10.* has multiple security vulnerabilities > -- > > Key: JCLOUDS-1628 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1628 > Project: jclouds > Issue Type: Improvement > Components: jclouds-drivers >Affects Versions: 2.5.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Major > Labels: netty > Fix For: 2.6.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Need to upgrade to 4.x to resolve this but this has multiple API changes. > Propose removing the driver entirely. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1628) Netty 3.10.* has multiple security vulnerabilities
[ https://issues.apache.org/jira/browse/JCLOUDS-1628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820654#comment-17820654 ] ASF subversion and git services commented on JCLOUDS-1628: -- Commit 4f3955799bb8e419556c9590dd60faed5d3122dd in jclouds's branch refs/heads/master from Andrew Gaul [ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=4f3955799b ] JCLOUDS-1628: Remove Netty driver 3.x has multiple security vulnerabilities but upgrading to 4.x is API incompatible. Remove due to lack of known users. > Netty 3.10.* has multiple security vulnerabilities > -- > > Key: JCLOUDS-1628 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1628 > Project: jclouds > Issue Type: Improvement > Components: jclouds-drivers >Affects Versions: 2.5.0 >Reporter: Andrew Gaul >Priority: Major > Labels: netty > Time Spent: 20m > Remaining Estimate: 0h > > Need to upgrade to 4.x to resolve this but this has multiple API changes. > Propose removing the driver entirely. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1629) Upgrade to Guice 7
[ https://issues.apache.org/jira/browse/JCLOUDS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820562#comment-17820562 ] ASF subversion and git services commented on JCLOUDS-1629: -- Commit 107741f786a6aabfd68617b4e2e4ff59efac675d in jclouds's branch refs/heads/master from Andrew Gaul [ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=107741f786 ] JCLOUDS-1629: Upgrade to Guice 7.0.0 This also changes from javax to jakarta annotations. > Upgrade to Guice 7 > -- > > Key: JCLOUDS-1629 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1629 > Project: jclouds > Issue Type: Improvement > Components: jclouds-core >Affects Versions: 2.5.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Major > Labels: guice > Time Spent: 50m > Remaining Estimate: 0h > > [~basil] JCLOUDS-1627 upgrading some of the packages to jakarta but not the > important annotations one for Guice. When changing these to Jakarta I could > get Guice 7 working but not 6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1606) Cannot upload more than 32 parts to GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820546#comment-17820546
]
Lari Hotari commented on JCLOUDS-1606:
--
Thanks for fixing this!
> Cannot upload more than 32 parts to GCS
> ---
>
> Key: JCLOUDS-1606
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1606
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Lari Hotari
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> There's currently a limitation in JClouds that it cannot upload more than 32
> parts to GCS.
> {code:java}
> org.jclouds.http.HttpResponseException: command: POST
> https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose
> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
> "error": {
> "code": 400,
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "errors": [
> {
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "domain": "global",
> "reason": "invalid"
> }
> ]
> }
> } {code}
> The limitation of 32 parts is per API call to the compose endpoint.
>
> When there are more than 32 parts, the endpoint should be called multiple
> times. The total limit is 1 parts in GCS.
> [https://cloud.google.com/storage/docs/composite-objects]
>
> {quote}When you perform a composition:
> * The source objects are unaffected.
> * You can use between {*}1 and 32 source objects{*}.
> * {*}Source objects can themselves be composite objects{*}.{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (JCLOUDS-1606) Cannot upload more than 32 parts to GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lari Hotari closed JCLOUDS-1606.
Resolution: Fixed
> Cannot upload more than 32 parts to GCS
> ---
>
> Key: JCLOUDS-1606
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1606
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Lari Hotari
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> There's currently a limitation in JClouds that it cannot upload more than 32
> parts to GCS.
> {code:java}
> org.jclouds.http.HttpResponseException: command: POST
> https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose
> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
> "error": {
> "code": 400,
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "errors": [
> {
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "domain": "global",
> "reason": "invalid"
> }
> ]
> }
> } {code}
> The limitation of 32 parts is per API call to the compose endpoint.
>
> When there are more than 32 parts, the endpoint should be called multiple
> times. The total limit is 1 parts in GCS.
> [https://cloud.google.com/storage/docs/composite-objects]
>
> {quote}When you perform a composition:
> * The source objects are unaffected.
> * You can use between {*}1 and 32 source objects{*}.
> * {*}Source objects can themselves be composite objects{*}.{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1606) Cannot upload more than 32 parts to GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820545#comment-17820545
]
Lari Hotari commented on JCLOUDS-1606:
--
[~gaul] Yes, that should be fine. I'll close this issue.
> Cannot upload more than 32 parts to GCS
> ---
>
> Key: JCLOUDS-1606
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1606
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Lari Hotari
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> There's currently a limitation in JClouds that it cannot upload more than 32
> parts to GCS.
> {code:java}
> org.jclouds.http.HttpResponseException: command: POST
> https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose
> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
> "error": {
> "code": 400,
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "errors": [
> {
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "domain": "global",
> "reason": "invalid"
> }
> ]
> }
> } {code}
> The limitation of 32 parts is per API call to the compose endpoint.
>
> When there are more than 32 parts, the endpoint should be called multiple
> times. The total limit is 1 parts in GCS.
> [https://cloud.google.com/storage/docs/composite-objects]
>
> {quote}When you perform a composition:
> * The source objects are unaffected.
> * You can use between {*}1 and 32 source objects{*}.
> * {*}Source objects can themselves be composite objects{*}.{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1606) Cannot upload more than 32 parts to GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820433#comment-17820433
]
Andrew Gaul commented on JCLOUDS-1606:
--
The 32-part limit exists but now jclouds just uses bigger parts. While a
smarter strategy could compose bigger parts from smaller parts, please close
this if your symptoms are resolved.
> Cannot upload more than 32 parts to GCS
> ---
>
> Key: JCLOUDS-1606
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1606
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Lari Hotari
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> There's currently a limitation in JClouds that it cannot upload more than 32
> parts to GCS.
> {code:java}
> org.jclouds.http.HttpResponseException: command: POST
> https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose
> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
> "error": {
> "code": 400,
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "errors": [
> {
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "domain": "global",
> "reason": "invalid"
> }
> ]
> }
> } {code}
> The limitation of 32 parts is per API call to the compose endpoint.
>
> When there are more than 32 parts, the endpoint should be called multiple
> times. The total limit is 1 parts in GCS.
> [https://cloud.google.com/storage/docs/composite-objects]
>
> {quote}When you perform a composition:
> * The source objects are unaffected.
> * You can use between {*}1 and 32 source objects{*}.
> * {*}Source objects can themselves be composite objects{*}.{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1629) Upgrade to Guice 7
[ https://issues.apache.org/jira/browse/JCLOUDS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820255#comment-17820255 ] Basil Crow commented on JCLOUDS-1629: - bq. When changing these to Jakarta I could get Guice 7 working but not 6. I don't have any ideas offhand. If Guice 7 is working but not Guice 6, then maybe it would make sense to file a Guice ticket about this. > Upgrade to Guice 7 > -- > > Key: JCLOUDS-1629 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1629 > Project: jclouds > Issue Type: Improvement > Components: jclouds-core >Affects Versions: 2.5.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Major > Labels: guice > Time Spent: 20m > Remaining Estimate: 0h > > [~basil] JCLOUDS-1627 upgrading some of the packages to jakarta but not the > important annotations one for Guice. When changing these to Jakarta I could > get Guice 7 working but not 6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container
[
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820254#comment-17820254
]
Basil Crow commented on JCLOUDS-1371:
-
Our usage is at
https://github.com/jenkinsci/artifact-manager-s3-plugin/blob/1239f6b0f5820075ebf8f37096f86e198f22adf6/src/test/java/io/jenkins/plugins/artifact_manager_jclouds/MockApiMetadata.java#L193-L202.
I am not the original author of the code, so I cannot comment on why it was
done this way.
> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Assignee: Andrew Gaul
>Priority: Major
> Labels: filesystem
> Fix For: 2.5.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire
> container even when prefix and delimiter set. The File API does not provide
> a way to list a subset of files except for those within a specific directory
> and the underlying filesystem makes no guarantees about enumeration order.
> We can still optimize the case where prefix is set and delimiter is /.
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1629) Upgrade to Guice 7
[ https://issues.apache.org/jira/browse/JCLOUDS-1629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820242#comment-17820242 ] Andrew Gaul commented on JCLOUDS-1629: -- Could use some help here: https://github.com/apache/jclouds/pull/197 > Upgrade to Guice 7 > -- > > Key: JCLOUDS-1629 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1629 > Project: jclouds > Issue Type: Improvement > Components: jclouds-core >Affects Versions: 2.5.0 >Reporter: Andrew Gaul >Assignee: Andrew Gaul >Priority: Major > Labels: guice > Time Spent: 10m > Remaining Estimate: 0h > > [~basil] JCLOUDS-1627 upgrading some of the packages to jakarta but not the > important annotations one for Guice. When changing these to Jakarta I could > get Guice 7 working but not 6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (JCLOUDS-1629) Upgrade to Guice 7
Andrew Gaul created JCLOUDS-1629: Summary: Upgrade to Guice 7 Key: JCLOUDS-1629 URL: https://issues.apache.org/jira/browse/JCLOUDS-1629 Project: jclouds Issue Type: Improvement Components: jclouds-core Affects Versions: 2.5.0 Reporter: Andrew Gaul Assignee: Andrew Gaul [~basil] JCLOUDS-1627 upgrading some of the packages to jakarta but not the important annotations one for Guice. When changing these to Jakarta I could get Guice 7 working but not 6. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container
[
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820237#comment-17820237
]
Andrew Gaul commented on JCLOUDS-1371:
--
If you'd like you can submit a PR but I don't consider this part of the public
API. Why are you implementing LocalStorageStrategy? The in-memory transient
implementation should suffice for tests.
> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Assignee: Andrew Gaul
>Priority: Major
> Labels: filesystem
> Fix For: 2.5.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire
> container even when prefix and delimiter set. The File API does not provide
> a way to list a subset of files except for those within a specific directory
> and the underlying filesystem makes no guarantees about enumeration order.
> We can still optimize the case where prefix is set and delimiter is /.
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1371) LocalBlobStore.list enumerates entire container
[
https://issues.apache.org/jira/browse/JCLOUDS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820150#comment-17820150
]
Basil Crow commented on JCLOUDS-1371:
-
The addition of the {{delimiter}} argument in the last commit broke API
compatibility and required me to add the new parameter to my mock
implementation of {{LocalStorageStrategy}} in order for my project to compile
again. Should this API change be documented before the next release?
> LocalBlobStore.list enumerates entire container
> ---
>
> Key: JCLOUDS-1371
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1371
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-blobstore
>Affects Versions: 2.0.3
>Reporter: Andrew Gaul
>Assignee: Andrew Gaul
>Priority: Major
> Labels: filesystem
> Fix For: 2.5.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{LocalBlobStore.list}} with the filesystem blobstore enumerates the entire
> container even when prefix and delimiter set. The File API does not provide
> a way to list a subset of files except for those within a specific directory
> and the underlying filesystem makes no guarantees about enumeration order.
> We can still optimize the case where prefix is set and delimiter is /.
> Reference:
> https://lists.apache.org/thread.html/72e8a101d8a8f99b6f728336633db2cecae1dc443e4c5b195eee8f0d@%3Cuser.jclouds.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1627) Java Jakarta Support
[
https://issues.apache.org/jira/browse/JCLOUDS-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820144#comment-17820144
]
Basil Crow commented on JCLOUDS-1627:
-
[~gaul] I have tested 2.6.0-SNAPSHOT at commit b5e4e1d0fd with [Artifact
Manager on S3|https://plugins.jenkins.io/artifact-manager-s3/] running against
Guice 7. I still get the following exception:
{noformat}
java.lang.NoClassDefFoundError: javax/inject/Provider
at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at
java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3549)
at java.base/java.lang.Class.getDeclaredConstructors(Class.java:2727)
at
com.google.inject.spi.InjectionPoint.forConstructorOf(InjectionPoint.java:299)
at
com.google.inject.internal.ConstructorBindingImpl.create(ConstructorBindingImpl.java:121)
at
com.google.inject.internal.InjectorImpl.createUninitializedBinding(InjectorImpl.java:737)
at
com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:982)
at
com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:902)
at
com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:302)
at
com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:225)
at
com.google.inject.internal.InjectorImpl.createParameterInjector(InjectorImpl.java:1083)
at
com.google.inject.internal.InjectorImpl.getParametersInjectors(InjectorImpl.java:1070)
at
com.google.inject.internal.ProviderMethod.initialize(ProviderMethod.java:164)
at
com.google.inject.internal.InternalProviderInstanceBindingImpl.initialize(InternalProviderInstanceBindingImpl.java:64)
at
com.google.inject.internal.InjectorImpl.initializeBinding(InjectorImpl.java:593)
at
com.google.inject.internal.AbstractBindingProcessor$Processor.initializeBinding(AbstractBindingProcessor.java:176)
at
com.google.inject.internal.AbstractBindingProcessor$Processor.lambda$scheduleInitialization$0(AbstractBindingProcessor.java:163)
at
com.google.inject.internal.ProcessedBindingData.initializeBindings(ProcessedBindingData.java:49)
at
com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:126)
at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
at com.google.inject.Guice.createInjector(Guice.java:87)
at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:405)
at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:328)
at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:615)
at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:595)
at
io.jenkins.plugins.artifact_manager_jclouds.s3.S3BlobStore.getContext(S3BlobStore.java:136)
at
io.jenkins.plugins.artifact_manager_jclouds.s3.CustomBehaviorBlobStoreProvider.getContext(CustomBehaviorBlobStoreProvider.java:68)
at
io.jenkins.plugins.artifact_manager_jclouds.JCloudsArtifactManager.getContext(JCloudsArtifactManager.java:384)
at
io.jenkins.plugins.artifact_manager_jclouds.JCloudsArtifactManager.archive(JCloudsArtifactManager.java:127)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:257)
at
hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at
hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
at
hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
at hudson.model.Build$BuildExecution.post2(Build.java:179)
at
hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
at hudson.model.Run.execute(Run.java:1918)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
{noformat}
> Java Jakarta Support
>
>
> Key: JCLOUDS-1627
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1627
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.5.0
>Reporter: Paolo Bazzi
>Assignee: Andrew Gaul
>Priority: Major
> Fix For: 2.6.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Module jclouds-core (latest version 2.5.0) depends on non-jakarta version of
> javax APIs:
> - javax.annotation » javax.annotation-api
> - javax.ws.rs » java
[jira] [Commented] (JCLOUDS-1626) `FileInputStream` leak when using `filesystem` provider and performing a multipart upload
[
https://issues.apache.org/jira/browse/JCLOUDS-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819995#comment-17819995
]
Andrew Gaul commented on JCLOUDS-1626:
--
I don't quite understand this issue since it appears to be specific to S3Proxy?
Where in jclouds does it not close the {{{}FileInputStream{}}}?
> `FileInputStream` leak when using `filesystem` provider and performing a
> multipart upload
> -
>
> Key: JCLOUDS-1626
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1626
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Reporter: Ji Xinchi
>Priority: Critical
> Labels: filesystem
>
> I'm using gaul/s3proxy built on the latest jclouds. The provider
> configuration is "filesystem", and basedir configuration is a directory where
> a NFS is mounted.
>
> When I performed a multipart upload, I found that files such as
> ".nfs10bf0005" remained in the directory after uploading.
> Each file corresponded to one part and the files could not be deleted. The
> error "Device Busy" was reported. These files will disappear automatically
> after restarting s3proxy.
>
> I found this PR, and tried to restore following snippet and it worked. But
> this is just a temporary fix.
> {code:java}
> InputStream is = blob.getPayload().openStream();
> if (is instanceof FileInputStream) {
> // except for FileInputStream since large MPU can open too many
> fds
> is.close();
> payload = blob.getPayload();
> } else {
> blob.resetPayload(/*release=*/ false);
> payload = new InputStreamPayload(is);
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1626) `FileInputStream` leak when using `filesystem` provider and performing a multipart upload
[
https://issues.apache.org/jira/browse/JCLOUDS-1626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1626:
-
Labels: filesystem (was: )
> `FileInputStream` leak when using `filesystem` provider and performing a
> multipart upload
> -
>
> Key: JCLOUDS-1626
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1626
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Reporter: Ji Xinchi
>Priority: Critical
> Labels: filesystem
>
> I'm using gaul/s3proxy built on the latest jclouds. The provider
> configuration is "filesystem", and basedir configuration is a directory where
> a NFS is mounted.
>
> When I performed a multipart upload, I found that files such as
> ".nfs10bf0005" remained in the directory after uploading.
> Each file corresponded to one part and the files could not be deleted. The
> error "Device Busy" was reported. These files will disappear automatically
> after restarting s3proxy.
>
> I found this PR, and tried to restore following snippet and it worked. But
> this is just a temporary fix.
> {code:java}
> InputStream is = blob.getPayload().openStream();
> if (is instanceof FileInputStream) {
> // except for FileInputStream since large MPU can open too many
> fds
> is.close();
> payload = blob.getPayload();
> } else {
> blob.resetPayload(/*release=*/ false);
> payload = new InputStreamPayload(is);
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-747) Determine level of android support and how to ensure we keep it.
[
https://issues.apache.org/jira/browse/JCLOUDS-747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-747.
-
Resolution: Won't Fix
Android is out of scope.
> Determine level of android support and how to ensure we keep it.
>
>
> Key: JCLOUDS-747
> URL: https://issues.apache.org/jira/browse/JCLOUDS-747
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Reporter: Adrian Cole
>Priority: Major
>
> One of the knock-on effects of moving on is tracking how we deal with
> android. One way is to establish a floor android level we aim to support
> (even if it is best efforts). That's due to the fact that android != java and
> only a subset of features are present, on each version. Here's a handy link
> that begins to discuss this complexity.
> http://stackoverflow.com/questions/20480090/does-android-support-jdk-6-or-7
> Modern android libraries typically use a combination of plugins and
> integration tests to ensure android isn't accidentally broken. Some projects
> just rely on folks to remember the rules.
> Here's an example of a signature-checking plugin
> {code}
>
> org.codehaus.mojo
> animal-sniffer-maven-plugin
> ${animal.sniffer.version}
>
>
> test
>
> check
>
>
>
>
>
> org.codehaus.mojo.signature
> java16
> 1.1
>
>
>
> {code}
> In short, I think we should be careful and consciously decide whether certain
> features that break some level of android support are worthwhile. We should
> also note that entrypoints that aren't used by android callers will not
> affect compatibility. In other words, we are most concerned with the common
> paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1627) Java Jakarta Support
[ https://issues.apache.org/jira/browse/JCLOUDS-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul resolved JCLOUDS-1627. -- Fix Version/s: 2.6.0 Resolution: Fixed Please wait a few hours for mirrors to propagate then test 2.6.0-SNAPSHOT to see if it resolves your symptoms. > Java Jakarta Support > > > Key: JCLOUDS-1627 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1627 > Project: jclouds > Issue Type: Improvement > Components: jclouds-core >Affects Versions: 2.5.0 >Reporter: Paolo Bazzi >Assignee: Andrew Gaul >Priority: Major > Fix For: 2.6.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Module jclouds-core (latest version 2.5.0) depends on non-jakarta version of > javax APIs: > - javax.annotation » javax.annotation-api > - javax.ws.rs » javax.ws.rs-api > > Are there any plans to release a version based on new Jakarta APIs instead of > former javax modules? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1627) Java Jakarta Support
[ https://issues.apache.org/jira/browse/JCLOUDS-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819981#comment-17819981 ] ASF subversion and git services commented on JCLOUDS-1627: -- Commit b5e4e1d0fd466dffcbb0fb7921e52732145cc732 in jclouds's branch refs/heads/master from Andrew Gaul [ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=b5e4e1d0fd ] JCLOUDS-1627: Upgrade to Jakarta packages This resolves an issue with newer Guice versions. > Java Jakarta Support > > > Key: JCLOUDS-1627 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1627 > Project: jclouds > Issue Type: Improvement > Components: jclouds-core >Affects Versions: 2.5.0 >Reporter: Paolo Bazzi >Assignee: Andrew Gaul >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Module jclouds-core (latest version 2.5.0) depends on non-jakarta version of > javax APIs: > - javax.annotation » javax.annotation-api > - javax.ws.rs » javax.ws.rs-api > > Are there any plans to release a version based on new Jakarta APIs instead of > former javax modules? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1131) jclouds-2.0.0 not compatible with gson-2.7
[ https://issues.apache.org/jira/browse/JCLOUDS-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul resolved JCLOUDS-1131. -- Fix Version/s: 2.3.0 Assignee: Andrew Gaul Resolution: Fixed > jclouds-2.0.0 not compatible with gson-2.7 > -- > > Key: JCLOUDS-1131 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1131 > Project: jclouds > Issue Type: Bug > Components: jclouds-core >Affects Versions: 2.0.0 >Reporter: Olivier Voortman >Assignee: Andrew Gaul >Priority: Major > Fix For: 2.3.0 > > > It seems jclouds is using some of gson internal classes, which is wrong. > There is now an incompatibility in jclouds-core-2.0.0-SNAPSHOT.jar. > The class > org.jclouds.json.internal.DeserializationConstructorAndReflectiveTypeAdapterFactory > is importing com.google.gson.internal.bind.ReflectiveTypeAdapterFactory > which is clearly marked as internal. > There is now a 4th parameter on the constructor since gson-2.7. > Is it something that you can fix, maybe by cloning that > ReflectiveTypeAdapterFactory inside your packages ? > Thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1101) JDK 9 compatibility
[
https://issues.apache.org/jira/browse/JCLOUDS-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1101.
--
Fix Version/s: 2.4.0
Resolution: Fixed
> JDK 9 compatibility
> ---
>
> Key: JCLOUDS-1101
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1101
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-core
>Affects Versions: 2.1.0
>Reporter: Andrew Gaul
>Assignee: Andrew Gaul
>Priority: Major
> Fix For: 2.4.0
>
>
> Compiling with JDK 9 build 111 yields:
> {noformat}
> [ERROR] COMPILATION ERROR :
> [ERROR]
> /home/gaul/work/jclouds/core/src/main/java/org/jclouds/json/internal/NamingStrategies.java:[264,36]
> incompatible types: com.google.common.base.Predicate java.lang.Class> cannot be
> converted to com.google.common.base.Predicate extends java.lang.annotation.Annotation>>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project jclouds-core: Compilation failure
> [ERROR]
> /home/gaul/work/jclouds/core/src/main/java/org/jclouds/json/internal/NamingStrategies.java:[264,36]
> incompatible types: com.google.common.base.Predicate java.lang.Class> cannot be
> converted to com.google.common.base.Predicate extends java.lang.annotation.Annotation>>
> {noformat}
> Apparently our code is incorrect:
> https://bugs.openjdk.java.net/browse/JDK-8075793
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1402) openjdk9 - An illegal reflective access operation has occurred
[
https://issues.apache.org/jira/browse/JCLOUDS-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1402.
--
Fix Version/s: 2.5.0
Assignee: Andrew Gaul
Resolution: Fixed
I believe JCLOUDS-1586 addressed this.
> openjdk9 - An illegal reflective access operation has occurred
> --
>
> Key: JCLOUDS-1402
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1402
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-core
>Affects Versions: 2.1.0
> Environment: $ uname -r
> 4.15.14-1-ARCH
> $ archlinux-java status
> Available Java environments:
> java-9-openjdk (default)
>Reporter: Mathieu Tortuyaux
>Assignee: Andrew Gaul
>Priority: Major
> Fix For: 2.5.0
>
>
> Hi !
> While trying to run [this
> example|https://github.com/jclouds/jclouds-examples/blob/master/google-lb/src/main/java/org/jclouds/examples/google/lb/MainApp.java],
> I got the following warning:
> {noformat}
> WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by
> com.google.inject.internal.cglib.core.$ReflectUtils$2
> (file:/home/mathieu/Desktop/lab/jclouds-optel/test/target/cloud-scheduler-1.0-SNAPSHOT-shaded.jar)
> to method
> java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
> WARNING: Please consider reporting this to the maintainers of
> com.google.inject.internal.cglib.core.$ReflectUtils$2
> WARNING: Use --illegal-access=warn to enable warnings of further illegal
> reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> {noformat}
> After a few searches, I found that this is a known issue caused by
> Google/Guice and one one his dependency: cglib. It has been fixed in
> Google/Guice release 4.2. But jclouds-core is using Google/Guice release 3.0
> {noformat}
> $ mvn dependency:tree | grep guice -B 2
> [INFO] | | \- org.apache.jclouds:jclouds-core:jar:2.1.0:compile
> [INFO] | | +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile
> [INFO] | | +-
> com.google.inject.extensions:guice-assistedinject:jar:3.0:compile
> [INFO] | | +- com.google.inject:guice:jar:3.0:compile
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (JCLOUDS-1623) Caused by: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
[
https://issues.apache.org/jira/browse/JCLOUDS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1623:
-
Labels: aws-s3 (was: )
> Caused by: java.io.EOFException: reached end of stream after skipping
> 14933328 bytes; 67108864 bytes expected
> -
>
> Key: JCLOUDS-1623
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1623
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Ivan Dimitrov
>Assignee: Andrew Gaul
>Priority: Major
> Labels: aws-s3
> Attachments: jclouds-wire.log, jclouds.log
>
>
> Hello,
> We are observing an issue with JClouds while transferring a file with an
> input stream only.
> We are opening an input stream to a remote file and we are passing the stream
> to the JClouds library, but after some time it fails with the following error:
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: java.io.EOFException:
> reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
> at com.google.common.base.Throwables.propagate(Throwables.java:241)
> at
> org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
> at
> org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:385)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
> at
> org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
> at org.example.Main.main(Main.java:50)
> Caused by: java.io.EOFException: reached end of stream after skipping
> 14933328 bytes; 67108864 bytes expected
> at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:807)
> at
> org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
> ... 5 more
> {noformat}
> Note: The issue occurs only for larger files (more than 32 mb).
> Java version: sapmachine-jdk-11.0.15.0.1.jdk
> IAAS: AWS-s3
> If we pass a BufferedInputStream the code works, so I guess that there is
> some issue with the processing of the stream. The stream returned by the Java
> Http Client is HttpResponseInputStream.
> Code:
> {code:title=Main.java|borderStyle=solid}
> package org.example;
> import com.google.common.collect.ImmutableSet;
> import com.google.inject.Module;
> import org.jclouds.ContextBuilder;
> import org.jclouds.blobstore.BlobStore;
> import org.jclouds.blobstore.BlobStoreContext;
> import org.jclouds.blobstore.domain.Blob;
> import org.jclouds.blobstore.options.PutOptions;
> import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;
> import java.io.BufferedInputStream;
> import java.io.InputStream;
> import java.net.Authenticator;
> import java.net.PasswordAuthentication;
> import java.net.URI;
> import java.net.http.HttpClient;
> import java.net.http.HttpRequest;
> import java.net.http.HttpResponse;
> import java.time.Duration;
> import java.util.UUID;
> public class Main {
> public static void main(String[] args) throws Exception {
> Iterable modules = ImmutableSet.of(
> new SLF4JLoggingModule());
> ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-s3")
> .credentials("", "")
> .modules(modules);
> BlobStoreContext blobStoreContext =
> contextBuilder.buildView(BlobStoreContext.class);
> BlobStore blobStore = blobStoreContext.getBlobStore();
> String decodedUrl = "";
> HttpClient client = buildHttpClient(decodedUrl);
> HttpResponse response =
> callRemoteEndpointWithRetry(client, decodedUrl);
> long fileSize = response.headers()
> .firstValueAsLong("Content-Length")
> .orElseThrow(() -> new IllegalArgumentException("No
> Content-Length"));
> System.out.println("Bytes: " + fileSize);
> InputStream inputStream = response.body();
> String container = "";
> Blob blob = blobStore.blobBuilder("TEST.zip")
> //.payload(new BufferedInputStream(inputStream, 8 * 1024))
> .payload(inputStream)
> .contentDisposition(UUID.randomUUID().toString())
>
[jira] [Updated] (JCLOUDS-1623) Caused by: java.io.EOFException: reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
[
https://issues.apache.org/jira/browse/JCLOUDS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul updated JCLOUDS-1623:
-
Component/s: jclouds-blobstore
> Caused by: java.io.EOFException: reached end of stream after skipping
> 14933328 bytes; 67108864 bytes expected
> -
>
> Key: JCLOUDS-1623
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1623
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Ivan Dimitrov
>Assignee: Andrew Gaul
>Priority: Major
> Attachments: jclouds-wire.log, jclouds.log
>
>
> Hello,
> We are observing an issue with JClouds while transferring a file with an
> input stream only.
> We are opening an input stream to a remote file and we are passing the stream
> to the JClouds library, but after some time it fails with the following error:
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: java.io.EOFException:
> reached end of stream after skipping 14933328 bytes; 67108864 bytes expected
> at com.google.common.base.Throwables.propagate(Throwables.java:241)
> at
> org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:253)
> at
> org.jclouds.io.internal.BasePayloadSlicer.slice(BasePayloadSlicer.java:228)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:385)
> at
> org.jclouds.blobstore.internal.BaseBlobStore.putMultipartBlob(BaseBlobStore.java:349)
> at
> org.jclouds.aws.s3.blobstore.AWSS3BlobStore.putBlob(AWSS3BlobStore.java:79)
> at org.example.Main.main(Main.java:50)
> Caused by: java.io.EOFException: reached end of stream after skipping
> 14933328 bytes; 67108864 bytes expected
> at com.google.common.io.ByteStreams.skipFully(ByteStreams.java:807)
> at
> org.jclouds.io.internal.BasePayloadSlicer.doSlice(BasePayloadSlicer.java:251)
> ... 5 more
> {noformat}
> Note: The issue occurs only for larger files (more than 32 mb).
> Java version: sapmachine-jdk-11.0.15.0.1.jdk
> IAAS: AWS-s3
> If we pass a BufferedInputStream the code works, so I guess that there is
> some issue with the processing of the stream. The stream returned by the Java
> Http Client is HttpResponseInputStream.
> Code:
> {code:title=Main.java|borderStyle=solid}
> package org.example;
> import com.google.common.collect.ImmutableSet;
> import com.google.inject.Module;
> import org.jclouds.ContextBuilder;
> import org.jclouds.blobstore.BlobStore;
> import org.jclouds.blobstore.BlobStoreContext;
> import org.jclouds.blobstore.domain.Blob;
> import org.jclouds.blobstore.options.PutOptions;
> import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;
> import java.io.BufferedInputStream;
> import java.io.InputStream;
> import java.net.Authenticator;
> import java.net.PasswordAuthentication;
> import java.net.URI;
> import java.net.http.HttpClient;
> import java.net.http.HttpRequest;
> import java.net.http.HttpResponse;
> import java.time.Duration;
> import java.util.UUID;
> public class Main {
> public static void main(String[] args) throws Exception {
> Iterable modules = ImmutableSet.of(
> new SLF4JLoggingModule());
> ContextBuilder contextBuilder = ContextBuilder.newBuilder("aws-s3")
> .credentials("", "")
> .modules(modules);
> BlobStoreContext blobStoreContext =
> contextBuilder.buildView(BlobStoreContext.class);
> BlobStore blobStore = blobStoreContext.getBlobStore();
> String decodedUrl = "";
> HttpClient client = buildHttpClient(decodedUrl);
> HttpResponse response =
> callRemoteEndpointWithRetry(client, decodedUrl);
> long fileSize = response.headers()
> .firstValueAsLong("Content-Length")
> .orElseThrow(() -> new IllegalArgumentException("No
> Content-Length"));
> System.out.println("Bytes: " + fileSize);
> InputStream inputStream = response.body();
> String container = "";
> Blob blob = blobStore.blobBuilder("TEST.zip")
> //.payload(new BufferedInputStream(inputStream, 8 * 1024))
> .payload(inputStream)
> .contentDisposition(UUID.randomUUID().toString())
>
[jira] [Resolved] (JCLOUDS-1408) Consider relying on jaxb-api
[
https://issues.apache.org/jira/browse/JCLOUDS-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1408.
--
Fix Version/s: 2.4.0
Assignee: Andrew Gaul
Resolution: Fixed
> Consider relying on jaxb-api
>
>
> Key: JCLOUDS-1408
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1408
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.1.0
>Reporter: Andrew Gaul
>Assignee: Andrew Gaul
>Priority: Major
> Labels: dependency
> Fix For: 2.4.0
>
>
> Java 9 removes many EE features which breaks jclouds and we recommend that
> users add {{--add-modules java.xml.bind}} to their flags. However, we can
> eliminate these flags via an explicit dependency:
> {code:xml}
>
> javax.xml.bind
> jaxb-api
> 2.3.0
>
> {code}
> Ideally we would add this to 2.1.1, ignoring our policy of no new
> dependencies in minor releases, since this is not really a new dependency.
> Comments?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1408) Consider relying on jaxb-api
[
https://issues.apache.org/jira/browse/JCLOUDS-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819965#comment-17819965
]
Andrew Gaul commented on JCLOUDS-1408:
--
Fixed in d861768d495257b0c16b2f97c45e3124f28bac34.
> Consider relying on jaxb-api
>
>
> Key: JCLOUDS-1408
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1408
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.1.0
>Reporter: Andrew Gaul
>Priority: Major
> Labels: dependency
>
> Java 9 removes many EE features which breaks jclouds and we recommend that
> users add {{--add-modules java.xml.bind}} to their flags. However, we can
> eliminate these flags via an explicit dependency:
> {code:xml}
>
> javax.xml.bind
> jaxb-api
> 2.3.0
>
> {code}
> Ideally we would add this to 2.1.1, ignoring our policy of no new
> dependencies in minor releases, since this is not really a new dependency.
> Comments?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1625) getSize() from blob metadata returns full size when sending range request to JClouds running on GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819962#comment-17819962
]
Andrew Gaul commented on JCLOUDS-1625:
--
Please submit a PR if you are able.
> getSize() from blob metadata returns full size when sending range request to
> JClouds running on GCS
> ---
>
> Key: JCLOUDS-1625
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1625
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Amine RACHYD
>Priority: Major
> Labels: blobstore, gcs
>
> Hello
> We noticed an error when requesting a blob via a range request when *running
> on GCS mode*
> The payload requested is correct, however the size announced by its metadata
> points to the full blob size.
> This was noticed on JClouds 2.5.0, running on Java 8
> For some reason I couldn't package a reproducer project but I'm putting a
> code snippet that reproduces this error:
>
> {code:java}
> // REDACTED: Creating a blobStore context from GCS provider -
> // Create blob, payload size is 12 bytes
> byte[] payload = "Hello World!".getBytes();
> assertEquals(12, payload.length);
> Blob blob = blobStore.blobBuilder(blobName).payload(payload).build();
> // Put blob to GCS
> blobStore.putBlob(bucketName, blob);
> // - 1. Retrieve blob -
> Blob resultBlob = blobStore.getBlob(bucketName, blobName);
> // - 2. Retrieve blob with range query -
> GetOptions getOptions = new GetOptions().range(0,4);
> Blob resultRangeBlob = blobStore.getBlob(bucketName, blobName, getOptions);
> // - 3. Read content of blob and assert size (12 for the first one, 5 for
> the second) -
> byte[] buffer = new byte[15];
> int readBytesFromBlob = resultBlob.getPayload().openStream().read(buffer);
> int readBytesFromRangeBlob =
> resultRangeBlob.getPayload().openStream().read(buffer);Assert.assertEquals(12,
> readBytesFromBlob);
> assertEquals(12, readBytesFromBlob);
> assertEquals(5, readBytesFromRangeBlob);
> // - 4. Get blobs sizes from respective metadata -
> Long blobSize = resultBlob.getMetadata().getSize();
> Long rangeBlobSize = resultRangeBlob.getMetadata().getSize();
> assertEquals(Long.valueOf(12), blobSize);
> // The following assertion fails
> // The rangeBlobSize equals 12, the full blob length
> // Whereas it should reflect only the size that has been read (5 in this case)
> assertEquals(Long.valueOf(5), rangeBlobSize);
> {code}
> For info, I think it's this
> [line|https://github.com/apache/jclouds/blob/611b4c4a1107501b705191495fb034e2b8ac1bcc/providers/google-cloud-storage/src/main/java/org/jclouds/googlecloudstorage/blobstore/GoogleCloudStorageBlobStore.java#L255]
> that causes the error, as the payload metadata is being set from the initial
> metadata of the full blob (object retrieved on L246).
> I'm opening this Jira ticket before proceeding by creating a PR to fix this
> bug.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (JCLOUDS-1606) Cannot upload more than 32 parts to GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lari Hotari reopened JCLOUDS-1606:
--
Reopening since GCS documentation still states that a single request has a 32
object limit.
https://cloud.google.com/storage/docs/composite-objects
> Cannot upload more than 32 parts to GCS
> ---
>
> Key: JCLOUDS-1606
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1606
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Lari Hotari
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> There's currently a limitation in JClouds that it cannot upload more than 32
> parts to GCS.
> {code:java}
> org.jclouds.http.HttpResponseException: command: POST
> https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose
> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
> "error": {
> "code": 400,
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "errors": [
> {
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "domain": "global",
> "reason": "invalid"
> }
> ]
> }
> } {code}
> The limitation of 32 parts is per API call to the compose endpoint.
>
> When there are more than 32 parts, the endpoint should be called multiple
> times. The total limit is 1 parts in GCS.
> [https://cloud.google.com/storage/docs/composite-objects]
>
> {quote}When you perform a composition:
> * The source objects are unaffected.
> * You can use between {*}1 and 32 source objects{*}.
> * {*}Source objects can themselves be composite objects{*}.{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1606) Cannot upload more than 32 parts to GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819242#comment-17819242
]
Lari Hotari commented on JCLOUDS-1606:
--
[~gaul] Is there a test that proves that this has been resolved. The last time
I checked, if there are more than 32 objects, you must use composite objects.
When did this change?
> Cannot upload more than 32 parts to GCS
> ---
>
> Key: JCLOUDS-1606
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1606
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Lari Hotari
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> There's currently a limitation in JClouds that it cannot upload more than 32
> parts to GCS.
> {code:java}
> org.jclouds.http.HttpResponseException: command: POST
> https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose
> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
> "error": {
> "code": 400,
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "errors": [
> {
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "domain": "global",
> "reason": "invalid"
> }
> ]
> }
> } {code}
> The limitation of 32 parts is per API call to the compose endpoint.
>
> When there are more than 32 parts, the endpoint should be called multiple
> times. The total limit is 1 parts in GCS.
> [https://cloud.google.com/storage/docs/composite-objects]
>
> {quote}When you perform a composition:
> * The source objects are unaffected.
> * You can use between {*}1 and 32 source objects{*}.
> * {*}Source objects can themselves be composite objects{*}.{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (JCLOUDS-1628) Netty 3.10.* has multiple security vulnerabilities
Andrew Gaul created JCLOUDS-1628: Summary: Netty 3.10.* has multiple security vulnerabilities Key: JCLOUDS-1628 URL: https://issues.apache.org/jira/browse/JCLOUDS-1628 Project: jclouds Issue Type: Improvement Components: jclouds-drivers Affects Versions: 2.5.0 Reporter: Andrew Gaul Need to upgrade to 4.x to resolve this but this has multiple API changes. Propose removing the driver entirely. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1608) Slicing of large files can lead to exceed the 32 parts limit of GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819224#comment-17819224
]
Andrew Gaul commented on JCLOUDS-1608:
--
Thank you for your contribution [~jan.vermeu...@isencia.com] ! Next time
please submit a GitHub PR so this is easier to test and merge.
> Slicing of large files can lead to exceed the 32 parts limit of GCS
> ---
>
> Key: JCLOUDS-1608
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1608
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.2.1, 2.5.0
>Reporter: Jan Vermeulen
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> MultipartUploadSlicingAlgorithm calculates slices for a large file by first
> using the defaultPartSize. If the results of that slicing gives parts that
> don't exceed the min/maxPartSizes (5MB and 5GB for
> GoogleCloudStorageBlobStore) but that do exceed the maxNumberOfParts (32 for
> GoogleCloudStorageBlobStore), the algoritm sets the number of parts to 32 and
> recalculates the size of the parts. If there is any remainder after that,
> JClouds ends up uploading 33 parts in total to GCS, causing the process to
> fail in completeMultipartUpload() when recomposing the original content from
> the parts.
> The following simple unitTest proves the case:
> {{public class AlgoritmTest extends TestCase {}}
> {{ public void testSlicing() {}}
> {{ MultipartUploadSlicingAlgorithm algorithm = new
> MultipartUploadSlicingAlgorithm(1024*1024*5,1024*1024*1024*5,32);}}
> {{ algorithm.calculateChunkSize(1024*1024*1200+33);}}
> {{ assertTrue(algorithm.getParts()+((algorithm.getRemaining() >
> 0)?(1):(0)) <= 32);}}
> {{}}}
> It simulates the slicing of a file of 1.2GB+33 bytes (to make sure there is a
> remainder).
> The following patch fixes the issue:
> {{ ...}}
> {{ long remainder = length % unitPartSize; }}
> {{ // SHB patch}}
> {{ // remainder should be distributed over parts if we are at the
> maximumNumberOfParts}}
> {{ // (if not, an additional part is uploaded to GCS thus exceeding the
> maximum allowed parts)}}
> {{ // if (remainder == 0 && parts > 0) {}}
> {{ // parts -= 1;}}
> {{ if (remainder > 0 && parts == maximumNumberOfParts) {}}
> {{ parts -= 1;}}
> {{ partSize = length/parts;}}
> {{// end of SHB patch}}
> {{ ...}}
> I also commented the code that reduces the number of parts when there is no
> remainder, since that ends up creating a remaining part that is the same size
> as the others.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1606) Cannot upload more than 32 parts to GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1606.
--
Fix Version/s: 2.6.0
Assignee: Andrew Gaul
Resolution: Fixed
> Cannot upload more than 32 parts to GCS
> ---
>
> Key: JCLOUDS-1606
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1606
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Lari Hotari
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> There's currently a limitation in JClouds that it cannot upload more than 32
> parts to GCS.
> {code:java}
> org.jclouds.http.HttpResponseException: command: POST
> https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose
> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
> "error": {
> "code": 400,
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "errors": [
> {
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "domain": "global",
> "reason": "invalid"
> }
> ]
> }
> } {code}
> The limitation of 32 parts is per API call to the compose endpoint.
>
> When there are more than 32 parts, the endpoint should be called multiple
> times. The total limit is 1 parts in GCS.
> [https://cloud.google.com/storage/docs/composite-objects]
>
> {quote}When you perform a composition:
> * The source objects are unaffected.
> * You can use between {*}1 and 32 source objects{*}.
> * {*}Source objects can themselves be composite objects{*}.{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1608) Slicing of large files can lead to exceed the 32 parts limit of GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Gaul resolved JCLOUDS-1608.
--
Fix Version/s: 2.6.0
Assignee: Andrew Gaul
Resolution: Fixed
> Slicing of large files can lead to exceed the 32 parts limit of GCS
> ---
>
> Key: JCLOUDS-1608
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1608
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.2.1, 2.5.0
>Reporter: Jan Vermeulen
>Assignee: Andrew Gaul
>Priority: Major
> Labels: google-cloud-storage
> Fix For: 2.6.0
>
>
> MultipartUploadSlicingAlgorithm calculates slices for a large file by first
> using the defaultPartSize. If the results of that slicing gives parts that
> don't exceed the min/maxPartSizes (5MB and 5GB for
> GoogleCloudStorageBlobStore) but that do exceed the maxNumberOfParts (32 for
> GoogleCloudStorageBlobStore), the algoritm sets the number of parts to 32 and
> recalculates the size of the parts. If there is any remainder after that,
> JClouds ends up uploading 33 parts in total to GCS, causing the process to
> fail in completeMultipartUpload() when recomposing the original content from
> the parts.
> The following simple unitTest proves the case:
> {{public class AlgoritmTest extends TestCase {}}
> {{ public void testSlicing() {}}
> {{ MultipartUploadSlicingAlgorithm algorithm = new
> MultipartUploadSlicingAlgorithm(1024*1024*5,1024*1024*1024*5,32);}}
> {{ algorithm.calculateChunkSize(1024*1024*1200+33);}}
> {{ assertTrue(algorithm.getParts()+((algorithm.getRemaining() >
> 0)?(1):(0)) <= 32);}}
> {{}}}
> It simulates the slicing of a file of 1.2GB+33 bytes (to make sure there is a
> remainder).
> The following patch fixes the issue:
> {{ ...}}
> {{ long remainder = length % unitPartSize; }}
> {{ // SHB patch}}
> {{ // remainder should be distributed over parts if we are at the
> maximumNumberOfParts}}
> {{ // (if not, an additional part is uploaded to GCS thus exceeding the
> maximum allowed parts)}}
> {{ // if (remainder == 0 && parts > 0) {}}
> {{ // parts -= 1;}}
> {{ if (remainder > 0 && parts == maximumNumberOfParts) {}}
> {{ parts -= 1;}}
> {{ partSize = length/parts;}}
> {{// end of SHB patch}}
> {{ ...}}
> I also commented the code that reduces the number of parts when there is no
> remainder, since that ends up creating a remaining part that is the same size
> as the others.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1606) Cannot upload more than 32 parts to GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819222#comment-17819222
]
ASF subversion and git services commented on JCLOUDS-1606:
--
Commit 47f34770c9daaa5fe0224ce5bc1adbb7332d9aea in jclouds's branch
refs/heads/master from Jan Vermeulen
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=47f34770c9 ]
JCLOUDS-1606: JCLOUDS-1608: Fix MPU off-by-one
Previously GCS could not upload large objects due to its 32 part
limit.
> Cannot upload more than 32 parts to GCS
> ---
>
> Key: JCLOUDS-1606
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1606
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.5.0
>Reporter: Lari Hotari
>Priority: Major
> Labels: google-cloud-storage
>
> There's currently a limitation in JClouds that it cannot upload more than 32
> parts to GCS.
> {code:java}
> org.jclouds.http.HttpResponseException: command: POST
> https://www.googleapis.com/storage/v1/b/somebucket/o/ff553922-1fa3-4ceb-abcd-60106603b5c8-object-123456/compose
> HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
> "error": {
> "code": 400,
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "errors": [
> {
> "message": "The number of source components provided (35) exceeds the
> maximum (32)",
> "domain": "global",
> "reason": "invalid"
> }
> ]
> }
> } {code}
> The limitation of 32 parts is per API call to the compose endpoint.
>
> When there are more than 32 parts, the endpoint should be called multiple
> times. The total limit is 1 parts in GCS.
> [https://cloud.google.com/storage/docs/composite-objects]
>
> {quote}When you perform a composition:
> * The source objects are unaffected.
> * You can use between {*}1 and 32 source objects{*}.
> * {*}Source objects can themselves be composite objects{*}.{quote}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1608) Slicing of large files can lead to exceed the 32 parts limit of GCS
[
https://issues.apache.org/jira/browse/JCLOUDS-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819223#comment-17819223
]
ASF subversion and git services commented on JCLOUDS-1608:
--
Commit 47f34770c9daaa5fe0224ce5bc1adbb7332d9aea in jclouds's branch
refs/heads/master from Jan Vermeulen
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=47f34770c9 ]
JCLOUDS-1606: JCLOUDS-1608: Fix MPU off-by-one
Previously GCS could not upload large objects due to its 32 part
limit.
> Slicing of large files can lead to exceed the 32 parts limit of GCS
> ---
>
> Key: JCLOUDS-1608
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1608
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-blobstore
>Affects Versions: 2.2.1, 2.5.0
>Reporter: Jan Vermeulen
>Priority: Major
> Labels: google-cloud-storage
>
> MultipartUploadSlicingAlgorithm calculates slices for a large file by first
> using the defaultPartSize. If the results of that slicing gives parts that
> don't exceed the min/maxPartSizes (5MB and 5GB for
> GoogleCloudStorageBlobStore) but that do exceed the maxNumberOfParts (32 for
> GoogleCloudStorageBlobStore), the algoritm sets the number of parts to 32 and
> recalculates the size of the parts. If there is any remainder after that,
> JClouds ends up uploading 33 parts in total to GCS, causing the process to
> fail in completeMultipartUpload() when recomposing the original content from
> the parts.
> The following simple unitTest proves the case:
> {{public class AlgoritmTest extends TestCase {}}
> {{ public void testSlicing() {}}
> {{ MultipartUploadSlicingAlgorithm algorithm = new
> MultipartUploadSlicingAlgorithm(1024*1024*5,1024*1024*1024*5,32);}}
> {{ algorithm.calculateChunkSize(1024*1024*1200+33);}}
> {{ assertTrue(algorithm.getParts()+((algorithm.getRemaining() >
> 0)?(1):(0)) <= 32);}}
> {{}}}
> It simulates the slicing of a file of 1.2GB+33 bytes (to make sure there is a
> remainder).
> The following patch fixes the issue:
> {{ ...}}
> {{ long remainder = length % unitPartSize; }}
> {{ // SHB patch}}
> {{ // remainder should be distributed over parts if we are at the
> maximumNumberOfParts}}
> {{ // (if not, an additional part is uploaded to GCS thus exceeding the
> maximum allowed parts)}}
> {{ // if (remainder == 0 && parts > 0) {}}
> {{ // parts -= 1;}}
> {{ if (remainder > 0 && parts == maximumNumberOfParts) {}}
> {{ parts -= 1;}}
> {{ partSize = length/parts;}}
> {{// end of SHB patch}}
> {{ ...}}
> I also commented the code that reduces the number of parts when there is no
> remainder, since that ends up creating a remaining part that is the same size
> as the others.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1519) Support for Backblaze application keys
[ https://issues.apache.org/jira/browse/JCLOUDS-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818764#comment-17818764 ] ASF subversion and git services commented on JCLOUDS-1519: -- Commit 41d842d449e125411709613e27ca01c8fc5c4a4f in jclouds's branch refs/heads/dependabot/maven/project/org.testng-testng-7.5.1 from davidsenk [ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=41d842d449 ] JCLOUDS-1519: Fix the authorization error with b2 application keys > Support for Backblaze application keys > -- > > Key: JCLOUDS-1519 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1519 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.1.2 >Reporter: Elton Carneiro >Assignee: Andrew Gaul >Priority: Major > Fix For: 2.6.0 > > > The current implementation of the B2 API in JClouds only supports Master Keys > and not application keys. > When using the Master Key, the Key ID and the Account ID are the same. > However, when using Application Keys, the Key ID of the application key is > not the same as the account ID. The account ID is what is needed to make > certain calls such as bucket operations. > The response of the > [b2_authorize_account|https://www.backblaze.com/b2/docs/b2_authorize_account.html] > contains the account ID. > If you look at the > [b2_list_buckets|https://www.backblaze.com/b2/docs/b2_list_buckets.html] > endpoint, it takes in an account ID. This Account ID is returned in the > authorize call. > Looking at the source code, It seems that the account ID is being pulled from > the credentials that the user entered versus the response from the authorize > call. > This issue prevents users from using Application keys when using JClouds. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (JCLOUDS-1519) Support for Backblaze application keys
[ https://issues.apache.org/jira/browse/JCLOUDS-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Gaul resolved JCLOUDS-1519. -- Fix Version/s: 2.6.0 Assignee: Andrew Gaul Resolution: Fixed > Support for Backblaze application keys > -- > > Key: JCLOUDS-1519 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1519 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.1.2 >Reporter: Elton Carneiro >Assignee: Andrew Gaul >Priority: Major > Fix For: 2.6.0 > > > The current implementation of the B2 API in JClouds only supports Master Keys > and not application keys. > When using the Master Key, the Key ID and the Account ID are the same. > However, when using Application Keys, the Key ID of the application key is > not the same as the account ID. The account ID is what is needed to make > certain calls such as bucket operations. > The response of the > [b2_authorize_account|https://www.backblaze.com/b2/docs/b2_authorize_account.html] > contains the account ID. > If you look at the > [b2_list_buckets|https://www.backblaze.com/b2/docs/b2_list_buckets.html] > endpoint, it takes in an account ID. This Account ID is returned in the > authorize call. > Looking at the source code, It seems that the account ID is being pulled from > the credentials that the user entered versus the response from the authorize > call. > This issue prevents users from using Application keys when using JClouds. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1519) Support for Backblaze application keys
[ https://issues.apache.org/jira/browse/JCLOUDS-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818322#comment-17818322 ] ASF subversion and git services commented on JCLOUDS-1519: -- Commit 41d842d449e125411709613e27ca01c8fc5c4a4f in jclouds's branch refs/heads/master from davidsenk [ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=41d842d449 ] JCLOUDS-1519: Fix the authorization error with b2 application keys > Support for Backblaze application keys > -- > > Key: JCLOUDS-1519 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1519 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.1.2 >Reporter: Elton Carneiro >Priority: Major > > The current implementation of the B2 API in JClouds only supports Master Keys > and not application keys. > When using the Master Key, the Key ID and the Account ID are the same. > However, when using Application Keys, the Key ID of the application key is > not the same as the account ID. The account ID is what is needed to make > certain calls such as bucket operations. > The response of the > [b2_authorize_account|https://www.backblaze.com/b2/docs/b2_authorize_account.html] > contains the account ID. > If you look at the > [b2_list_buckets|https://www.backblaze.com/b2/docs/b2_list_buckets.html] > endpoint, it takes in an account ID. This Account ID is returned in the > authorize call. > Looking at the source code, It seems that the account ID is being pulled from > the credentials that the user entered versus the response from the authorize > call. > This issue prevents users from using Application keys when using JClouds. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1618) Jclouds 2.3.0 and above is incompatible with SpringBoot 2.7.*
[
https://issues.apache.org/jira/browse/JCLOUDS-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818285#comment-17818285
]
ASF subversion and git services commented on JCLOUDS-1618:
--
Commit 8075bbe50aced7c75ee089fe76ba64e6b8e6afd2 in jclouds's branch
refs/heads/dependabot/maven/project/org.testng-testng-7.5.1 from Andrew Gaul
[ https://gitbox.apache.org/repos/asf?p=jclouds.git;h=8075bbe50a ]
JCLOUDS-1618: Upgrade to gson 2.10.1
> Jclouds 2.3.0 and above is incompatible with SpringBoot 2.7.*
> -
>
> Key: JCLOUDS-1618
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1618
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-core
>Affects Versions: 2.3.0, 2.4.0, 2.5.0
>Reporter: Biswa Ranjan Ray
>Assignee: Andrew Gaul
>Priority: Major
> Labels: gson
> Fix For: 2.6.0
>
> Attachments: JClouds_Gson_Error.txt
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Hi,
> We developed a multi-cloud application using jclouds and springboot to
> support operations on storage services e.g s3, gcs and azure storage.
> We're trying to migrate our application from springboot 2.5.* to latest
> (springboot 2.7.*). However we're facing issues with jclouds. Its observed
> that errors are thrown during the creation of BlobStoreContext. A full
> stacktrace is attached.
> Here is a code snippet of BlobStoreContext:
> public BlobStoreContext getBlobStoreContext()
> {
>return ContextBuilder.newBuilder(CloudProviders.PROVIDER_AWS.toString())
>
>.credentials(this.getAccessKeyId(), this.getSecretAccessKey())
>.buildView(BlobStoreContext.class);
> }
> We also tried jclouds 2.4.0 & 2.5.0 but no luck. However its found jclouds
> 2.3.0 is compatible with springboot 2.6.10 and currently we're using the
> same, but our goal is to upgrade to springboot 2.7.
> Any suggestions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1519) Support for Backblaze application keys
[ https://issues.apache.org/jira/browse/JCLOUDS-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814373#comment-17814373 ] David Senk commented on JCLOUDS-1519: - This is fixed with PR # 190 [https://github.com/apache/jclouds/pull/190] > Support for Backblaze application keys > -- > > Key: JCLOUDS-1519 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1519 > Project: jclouds > Issue Type: Bug > Components: jclouds-blobstore >Affects Versions: 2.1.2 >Reporter: Elton Carneiro >Priority: Major > > The current implementation of the B2 API in JClouds only supports Master Keys > and not application keys. > When using the Master Key, the Key ID and the Account ID are the same. > However, when using Application Keys, the Key ID of the application key is > not the same as the account ID. The account ID is what is needed to make > certain calls such as bucket operations. > The response of the > [b2_authorize_account|https://www.backblaze.com/b2/docs/b2_authorize_account.html] > contains the account ID. > If you look at the > [b2_list_buckets|https://www.backblaze.com/b2/docs/b2_list_buckets.html] > endpoint, it takes in an account ID. This Account ID is returned in the > authorize call. > Looking at the source code, It seems that the account ID is being pulled from > the credentials that the user entered versus the response from the authorize > call. > This issue prevents users from using Application keys when using JClouds. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1618) Jclouds 2.3.0 and above is incompatible with SpringBoot 2.7.*
[
https://issues.apache.org/jira/browse/JCLOUDS-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810151#comment-17810151
]
Basil Crow commented on JCLOUDS-1618:
-
Are there any plans for a release?
> Jclouds 2.3.0 and above is incompatible with SpringBoot 2.7.*
> -
>
> Key: JCLOUDS-1618
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1618
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-core
>Affects Versions: 2.3.0, 2.4.0, 2.5.0
>Reporter: Biswa Ranjan Ray
>Assignee: Andrew Gaul
>Priority: Major
> Labels: gson
> Fix For: 2.6.0
>
> Attachments: JClouds_Gson_Error.txt
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Hi,
> We developed a multi-cloud application using jclouds and springboot to
> support operations on storage services e.g s3, gcs and azure storage.
> We're trying to migrate our application from springboot 2.5.* to latest
> (springboot 2.7.*). However we're facing issues with jclouds. Its observed
> that errors are thrown during the creation of BlobStoreContext. A full
> stacktrace is attached.
> Here is a code snippet of BlobStoreContext:
> public BlobStoreContext getBlobStoreContext()
> {
>return ContextBuilder.newBuilder(CloudProviders.PROVIDER_AWS.toString())
>
>.credentials(this.getAccessKeyId(), this.getSecretAccessKey())
>.buildView(BlobStoreContext.class);
> }
> We also tried jclouds 2.4.0 & 2.5.0 but no luck. However its found jclouds
> 2.3.0 is compatible with springboot 2.6.10 and currently we're using the
> same, but our goal is to upgrade to springboot 2.7.
> Any suggestions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (JCLOUDS-1627) Java Jakarta Support
[
https://issues.apache.org/jira/browse/JCLOUDS-1627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810150#comment-17810150
]
Basil Crow commented on JCLOUDS-1627:
-
I suggest upgrading to Guice 6.x and migrating from {{javax.inject}} to
{{{}jakarta.inject{}}}. Guice 6 supports both, while Guice 7 only supports
{{{}jakarta.inject{}}}. So following this suggestion will allow JClouds to be
placed on the classpath of an application running Guice 6 and using
{{javax.inject}} outside of JClouds as well as the classpath of an application
running Guice 7 and using {{jakarta.inject}} outside of JClouds.
> Java Jakarta Support
>
>
> Key: JCLOUDS-1627
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1627
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-core
>Affects Versions: 2.5.0
>Reporter: Paolo Bazzi
>Assignee: Andrew Gaul
>Priority: Major
>
> Module jclouds-core (latest version 2.5.0) depends on non-jakarta version of
> javax APIs:
> - javax.annotation » javax.annotation-api
> - javax.ws.rs » javax.ws.rs-api
>
> Are there any plans to release a version based on new Jakarta APIs instead of
> former javax modules?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
