Re: Problem with camel-aws S3 when multiPartUpload is set to true

2018-01-05 Thread Andrea Cosentino 
If you are passing the client as bean from registry, it makes no sense to 
define the region, you can set the region directly on the client you're passing.

--
Andrea Cosentino 
--
Apache Camel PMC Member
Apache Karaf Committer
Apache Servicemix PMC Member
Email: ancosen1...@yahoo.com
Twitter: @oscerd2
Github: oscerd






On Friday, January 5, 2018, 2:24:53 PM GMT+1, kretin  wrote: 





I tried adding a region like this and it still fails with the same error: 



> On Jan 5, 2018, at 4:23 AM, Artur Jablonski  wrote:
> 
> Try setting region on your AmazonS3Client bean.
> 
> On Fri, Jan 5, 2018 at 3:19 AM, kretin  wrote:
> 
>> I created a simple camel route to poll for files in a local directory and
>> upload them to a Ceph (S3) server at my University. I am using apache camel
>> 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false
>> (the default) in the uri, everything works fine, but if I change to
>> multiPartUpload=true, it fails.
>> 
>> I know there is nothing wrong with my s3 secret or s3 access key because
>> when I set multiPartUpload=false, everything works (there are no crazy plus
>> (+) characters that need to be escaped in the keys).
>> 
>> Here is the stack trace:
>> 
>> com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon
>> S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
>> tx002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
>> Request ID: 2213a2-uky-campus-1-uky)
>>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> handleErrorResponse(AmazonHttpClient.java:1592)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> executeOneRequest(AmazonHttpClient.java:1257)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>    at 
>>com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>    at 
>>com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186.
>> jar:?]
>>    at 
>>com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>    at 
>>com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.
>> execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?]
>>    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>    at 
>>com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>>    at 
>>com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>>    at 
>>com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>>    at org.apache.camel.component.aws.s3.S3Producer.
>> processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0]
>>    at 
>>org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84)
>> ~[camel-aws-2.20.0.jar:2.20.0]
>>    at ...
>> 
>> My camel-context.xml looks like:
>> 
>> 
>> http://www.springframework.org/schema/beans;
>>    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>>    xsi:schemaLocation="
>>      http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans.xsd
>>      http://camel.apache.org/schema/spring http://camel.apache.org/
>> schema/spring/camel-spring.xsd">
>> 
>>    >        class="org.apache.camel.component.properties.PropertiesComponent">
>>        
>>    
>> 
>>    >        class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi
>> gurer">
>>        > value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
>>        
>>    
>> 
>>    
>>        
>>            
>>                
>>                
>>            
>>        
>>        
>>            
>>                
>>            
>>        
>>        
>>    
>> 
>>    http://camel.apache.org/schema/spring;>
>>        
>>            
>> 
>>            
>> 
>>            
>>                testMultiPart/${in.header.CamelFileName}
>>                
>>            
>> 
>>            
>>                ${in.header.CamelFileLength}
>>                
>>            
>> 
>>            
>> 
>>            >uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClientmultiPartUpload=true"
>> />
>> 
>>            
>> 
>>        
>>    
>> 
>> 
>> Again, everything works fine if I set multiPartUpload=false in the above
>> camel-context.xml
>> 
>> I have tried a lot of things like:
>> 
>>        •

Re: Problem with camel-aws S3 when multiPartUpload is set to true

2018-01-05 Thread kretin 
I tried adding a region like this and it still fails with the same error: 



> On Jan 5, 2018, at 4:23 AM, Artur Jablonski  wrote:
> 
> Try setting region on your AmazonS3Client bean.
> 
> On Fri, Jan 5, 2018 at 3:19 AM, kretin  wrote:
> 
>> I created a simple camel route to poll for files in a local directory and
>> upload them to a Ceph (S3) server at my University. I am using apache camel
>> 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false
>> (the default) in the uri, everything works fine, but if I change to
>> multiPartUpload=true, it fails.
>> 
>> I know there is nothing wrong with my s3 secret or s3 access key because
>> when I set multiPartUpload=false, everything works (there are no crazy plus
>> (+) characters that need to be escaped in the keys).
>> 
>> Here is the stack trace:
>> 
>> com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon
>> S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
>> tx002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
>> Request ID: 2213a2-uky-campus-1-uky)
>>at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> handleErrorResponse(AmazonHttpClient.java:1592)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> executeOneRequest(AmazonHttpClient.java:1257)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>at 
>> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>at 
>> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
>> executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186.
>> jar:?]
>>at 
>> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>at 
>> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.
>> execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?]
>>at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
>> ~[aws-java-sdk-core-1.11.186.jar:?]
>>at 
>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>>at 
>> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>>at 
>> com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928)
>> ~[aws-java-sdk-s3-1.11.186.jar:?]
>>at org.apache.camel.component.aws.s3.S3Producer.
>> processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0]
>>at 
>> org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84)
>> ~[camel-aws-2.20.0.jar:2.20.0]
>>at ...
>> 
>> My camel-context.xml looks like:
>> 
>> 
>> http://www.springframework.org/schema/beans;
>>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>>xsi:schemaLocation="
>>   http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans.xsd
>>   http://camel.apache.org/schema/spring http://camel.apache.org/
>> schema/spring/camel-spring.xsd">
>> 
>>>class="org.apache.camel.component.properties.PropertiesComponent">
>>
>>
>> 
>>>class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi
>> gurer">
>>> value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
>>
>>
>> 
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 
>>http://camel.apache.org/schema/spring;>
>>
>>
>> 
>>
>> 
>>
>>testMultiPart/${in.header.CamelFileName}
>>
>>
>> 
>>
>>${in.header.CamelFileLength}
>>
>>
>> 
>>
>> 
>>> uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClientmultiPartUpload=true"
>> />
>> 
>>
>> 
>>
>>
>> 
>> 
>> Again, everything works fine if I set multiPartUpload=false in the above
>> camel-context.xml
>> 
>> I have tried a lot of things like:
>> 
>>• setting the CamelAwsS3ContentMD5 header to the MD5 hash of the
>> file (which doesn't make sense for multi-part files)
>>• various settings for the partSize parameter
>>• different sized files from very large to very small
>>• setting the system parameter: System.setProperty("com.
>> amazonaws.services.s3.disablePutObjectMD5Validation", "true");
>> 
>> If I turn on trace debugging for camel, it doesnt

Re: Problem with camel-aws S3 when multiPartUpload is set to true

2018-01-05 Thread Artur Jablonski 
Try setting region on your AmazonS3Client bean.

On Fri, Jan 5, 2018 at 3:19 AM, kretin  wrote:

> I created a simple camel route to poll for files in a local directory and
> upload them to a Ceph (S3) server at my University. I am using apache camel
> 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false
> (the default) in the uri, everything works fine, but if I change to
> multiPartUpload=true, it fails.
>
> I know there is nothing wrong with my s3 secret or s3 access key because
> when I set multiPartUpload=false, everything works (there are no crazy plus
> (+) characters that need to be escaped in the keys).
>
> Here is the stack trace:
>
> com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon
> S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
> tx002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended
> Request ID: 2213a2-uky-campus-1-uky)
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> handleErrorResponse(AmazonHttpClient.java:1592)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> executeOneRequest(AmazonHttpClient.java:1257)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutor.
> executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186.
> jar:?]
> at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at 
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.
> execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?]
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
> ~[aws-java-sdk-core-1.11.186.jar:?]
> at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
> ~[aws-java-sdk-s3-1.11.186.jar:?]
> at 
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
> ~[aws-java-sdk-s3-1.11.186.jar:?]
> at 
> com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928)
> ~[aws-java-sdk-s3-1.11.186.jar:?]
> at org.apache.camel.component.aws.s3.S3Producer.
> processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0]
> at 
> org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84)
> ~[camel-aws-2.20.0.jar:2.20.0]
> at ...
>
> My camel-context.xml looks like:
>
> 
> http://www.springframework.org/schema/beans;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="
>http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
>http://camel.apache.org/schema/spring http://camel.apache.org/
> schema/spring/camel-spring.xsd">
>
>  class="org.apache.camel.component.properties.PropertiesComponent">
> 
> 
>
>  class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi
> gurer">
>  value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
> 
> 
>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
> http://camel.apache.org/schema/spring;>
> 
> 
>
> 
>
> 
> testMultiPart/${in.header.CamelFileName}
> 
> 
>
> 
> ${in.header.CamelFileLength}
> 
> 
>
> 
>
>  uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClientmultiPartUpload=true"
> />
>
> 
>
> 
> 
> 
>
> Again, everything works fine if I set multiPartUpload=false in the above
> camel-context.xml
>
> I have tried a lot of things like:
>
> • setting the CamelAwsS3ContentMD5 header to the MD5 hash of the
> file (which doesn't make sense for multi-part files)
> • various settings for the partSize parameter
> • different sized files from very large to very small
> • setting the system parameter: System.setProperty("com.
> amazonaws.services.s3.disablePutObjectMD5Validation", "true");
>
> If I turn on trace debugging for camel, it doesnt help much:
>
> [d #2 - file://target/sendToS3/] S3Producer TRACE
> Initiating multipart upload [com.amazonaws.services.s3.model.
> InitiateMultipartUploadRequest@3731147a] from exchange
> [Exchange[ID-Toucan-local-1515115111374-0-1]]...
> [d #2 -

Problem with camel-aws S3 when multiPartUpload is set to true

2018-01-04 Thread kretin 
I created a simple camel route to poll for files in a local directory and 
upload them to a Ceph (S3) server at my University. I am using apache camel 
2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false (the 
default) in the uri, everything works fine, but if I change to 
multiPartUpload=true, it fails. 

I know there is nothing wrong with my s3 secret or s3 access key because when I 
set multiPartUpload=false, everything works (there are no crazy plus (+) 
characters that need to be escaped in the keys). 

Here is the stack trace:

com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; 
Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 
tx002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID: 
2213a2-uky-campus-1-uky)
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1592)
 ~[aws-java-sdk-core-1.11.186.jar:?]
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257)
 ~[aws-java-sdk-core-1.11.186.jar:?]
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
 ~[aws-java-sdk-core-1.11.186.jar:?]
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
 ~[aws-java-sdk-core-1.11.186.jar:?]
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
 ~[aws-java-sdk-core-1.11.186.jar:?]
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
 ~[aws-java-sdk-core-1.11.186.jar:?]
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
 ~[aws-java-sdk-core-1.11.186.jar:?]
at 
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
 ~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) 
~[aws-java-sdk-core-1.11.186.jar:?]
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) 
~[aws-java-sdk-s3-1.11.186.jar:?]
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) 
~[aws-java-sdk-s3-1.11.186.jar:?]
at 
com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928)
 ~[aws-java-sdk-s3-1.11.186.jar:?]
at 
org.apache.camel.component.aws.s3.S3Producer.processMultiPart(S3Producer.java:181)
 ~[camel-aws-2.20.0.jar:2.20.0]
at org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84) 
~[camel-aws-2.20.0.jar:2.20.0]
at ...

My camel-context.xml looks like:


http://www.springframework.org/schema/beans;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="
   http://www.springframework.org/schema/beans 
http://www.springframework.org/schema/beans/spring-beans.xsd
   http://camel.apache.org/schema/spring 
http://camel.apache.org/schema/spring/camel-spring.xsd;>

























http://camel.apache.org/schema/spring;>






testMultiPart/${in.header.CamelFileName}




${in.header.CamelFileLength}













Again, everything works fine if I set multiPartUpload=false in the above 
camel-context.xml

I have tried a lot of things like:

• setting the CamelAwsS3ContentMD5 header to the MD5 hash of the file 
(which doesn't make sense for multi-part files)
• various settings for the partSize parameter
• different sized files from very large to very small
• setting the system parameter: 
System.setProperty("com.amazonaws.services.s3.disablePutObjectMD5Validation", 
"true");

If I turn on trace debugging for camel, it doesnt help much:

[d #2 - file://target/sendToS3/] S3Producer TRACE 
Initiating multipart upload 
[com.amazonaws.services.s3.model.InitiateMultipartUploadRequest@3731147a] from 
exchange [Exchange[ID-Toucan-local-1515115111374-0-1]]...
[d #2 - file://target/sendToS3/] S3Producer TRACE Uploading 
part [1] for testMultiPart/testfile.zip
[d #2 - file://target/sendToS3/] DefaultErrorHandlerTRACE Is 
exchangeId: ID-Toucan-local-1515115111374-0-1 interrupted? false
[d #2 - file://target/sendToS3/] DefaultErrorHandlerTRACE Is 
exchangeId: ID-Toucan-local-1515115111374-0-1 done? false
[d #2 - file://target/sendToS3/] DefaultErrorHandlerTRACE 
isRunAllowed() -> true (Run allowed if we are not stopped/stopping)
[d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE Finding 
best suited