Re: Problem with camel-aws S3 when multiPartUpload is set to true
If you are passing the client as bean from registry, it makes no sense to define the region, you can set the region directly on the client you're passing. -- Andrea Cosentino -- Apache Camel PMC Member Apache Karaf Committer Apache Servicemix PMC Member Email: ancosen1...@yahoo.com Twitter: @oscerd2 Github: oscerd On Friday, January 5, 2018, 2:24:53 PM GMT+1, kretin wrote: I tried adding a region like this and it still fails with the same error: > On Jan 5, 2018, at 4:23 AM, Artur Jablonski wrote: > > Try setting region on your AmazonS3Client bean. > > On Fri, Jan 5, 2018 at 3:19 AM, kretin wrote: > >> I created a simple camel route to poll for files in a local directory and >> upload them to a Ceph (S3) server at my University. I am using apache camel >> 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false >> (the default) in the uri, everything works fine, but if I change to >> multiPartUpload=true, it fails. >> >> I know there is nothing wrong with my s3 secret or s3 access key because >> when I set multiPartUpload=false, everything works (there are no crazy plus >> (+) characters that need to be escaped in the keys). >> >> Here is the stack trace: >> >> com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon >> S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: >> tx002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended >> Request ID: 2213a2-uky-campus-1-uky) >> at com.amazonaws.http.AmazonHttpClient$RequestExecutor. >> handleErrorResponse(AmazonHttpClient.java:1592) >> ~[aws-java-sdk-core-1.11.186.jar:?] >> at com.amazonaws.http.AmazonHttpClient$RequestExecutor. >> executeOneRequest(AmazonHttpClient.java:1257) >> ~[aws-java-sdk-core-1.11.186.jar:?] >> at >>com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) >> ~[aws-java-sdk-core-1.11.186.jar:?] >> at >>com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) >> ~[aws-java-sdk-core-1.11.186.jar:?] >> at com.amazonaws.http.AmazonHttpClient$RequestExecutor. >> executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186. >> jar:?] >> at >>com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) >> ~[aws-java-sdk-core-1.11.186.jar:?] >> at >>com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) >> ~[aws-java-sdk-core-1.11.186.jar:?] >> at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl. >> execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?] >> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) >> ~[aws-java-sdk-core-1.11.186.jar:?] >> at >>com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) >> ~[aws-java-sdk-s3-1.11.186.jar:?] >> at >>com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) >> ~[aws-java-sdk-s3-1.11.186.jar:?] >> at >>com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928) >> ~[aws-java-sdk-s3-1.11.186.jar:?] >> at org.apache.camel.component.aws.s3.S3Producer. >> processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0] >> at >>org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84) >> ~[camel-aws-2.20.0.jar:2.20.0] >> at ... >> >> My camel-context.xml looks like: >> >> >> http://www.springframework.org/schema/beans"; >> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> xsi:schemaLocation=" >> http://www.springframework.org/schema/beans >> http://www.springframework.org/schema/beans/spring-beans.xsd >> http://camel.apache.org/schema/spring http://camel.apache.org/ >> schema/spring/camel-spring.xsd"> >> >> > class="org.apache.camel.component.properties.PropertiesComponent"> >> >> >> >> > class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi >> gurer"> >> > value="SYSTEM_PROPERTIES_MODE_OVERRIDE" /> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> http://camel.apache.org/schema/spring";> >> >> >> >> >> >> >> testMultiPart/${in.header.CamelFileName} >> >> >> >> >> ${in.header.CamelFileLength} >> >> >> >> >> >> >uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true" >> /> >> >> >> >> >> >> >> >> Again, everything works fine if I set multiPartUpload=false in the above >> camel-context.xml >> >> I have tried a lot of things like: >> >> • setting the CamelAwsS3ContentMD5 header to the MD5 hash of
Re: Problem with camel-aws S3 when multiPartUpload is set to true
I tried adding a region like this and it still fails with the same error: > On Jan 5, 2018, at 4:23 AM, Artur Jablonski wrote: > > Try setting region on your AmazonS3Client bean. > > On Fri, Jan 5, 2018 at 3:19 AM, kretin wrote: > >> I created a simple camel route to poll for files in a local directory and >> upload them to a Ceph (S3) server at my University. I am using apache camel >> 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false >> (the default) in the uri, everything works fine, but if I change to >> multiPartUpload=true, it fails. >> >> I know there is nothing wrong with my s3 secret or s3 access key because >> when I set multiPartUpload=false, everything works (there are no crazy plus >> (+) characters that need to be escaped in the keys). >> >> Here is the stack trace: >> >> com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon >> S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: >> tx002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended >> Request ID: 2213a2-uky-campus-1-uky) >>at com.amazonaws.http.AmazonHttpClient$RequestExecutor. >> handleErrorResponse(AmazonHttpClient.java:1592) >> ~[aws-java-sdk-core-1.11.186.jar:?] >>at com.amazonaws.http.AmazonHttpClient$RequestExecutor. >> executeOneRequest(AmazonHttpClient.java:1257) >> ~[aws-java-sdk-core-1.11.186.jar:?] >>at >> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) >> ~[aws-java-sdk-core-1.11.186.jar:?] >>at >> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) >> ~[aws-java-sdk-core-1.11.186.jar:?] >>at com.amazonaws.http.AmazonHttpClient$RequestExecutor. >> executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186. >> jar:?] >>at >> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) >> ~[aws-java-sdk-core-1.11.186.jar:?] >>at >> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) >> ~[aws-java-sdk-core-1.11.186.jar:?] >>at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl. >> execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?] >>at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) >> ~[aws-java-sdk-core-1.11.186.jar:?] >>at >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) >> ~[aws-java-sdk-s3-1.11.186.jar:?] >>at >> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) >> ~[aws-java-sdk-s3-1.11.186.jar:?] >>at >> com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928) >> ~[aws-java-sdk-s3-1.11.186.jar:?] >>at org.apache.camel.component.aws.s3.S3Producer. >> processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0] >>at >> org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84) >> ~[camel-aws-2.20.0.jar:2.20.0] >>at ... >> >> My camel-context.xml looks like: >> >> >> http://www.springframework.org/schema/beans"; >>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>xsi:schemaLocation=" >> http://www.springframework.org/schema/beans >> http://www.springframework.org/schema/beans/spring-beans.xsd >> http://camel.apache.org/schema/spring http://camel.apache.org/ >> schema/spring/camel-spring.xsd"> >> >>>class="org.apache.camel.component.properties.PropertiesComponent"> >> >> >> >>>class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi >> gurer"> >>> value="SYSTEM_PROPERTIES_MODE_OVERRIDE" /> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>http://camel.apache.org/schema/spring";> >> >> >> >> >> >> >>testMultiPart/${in.header.CamelFileName} >> >> >> >> >>${in.header.CamelFileLength} >> >> >> >> >> >>> uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true" >> /> >> >> >> >> >> >> >> >> Again, everything works fine if I set multiPartUpload=false in the above >> camel-context.xml >> >> I have tried a lot of things like: >> >>• setting the CamelAwsS3ContentMD5 header to the MD5 hash of the >> file (which doesn't make sense for multi-part files) >>• various settings for the partSize parameter >>• different sized files from very large to very small >>• setting the system parameter: System.setProperty("com. >> amazonaws.services.s3.disablePutObjectMD5Validation", "true"); >> >> If I turn on trace debugging for camel, it doesnt help much: >> >> [d #2 - file://target/se
Re: Problem with camel-aws S3 when multiPartUpload is set to true
Try setting region on your AmazonS3Client bean. On Fri, Jan 5, 2018 at 3:19 AM, kretin wrote: > I created a simple camel route to poll for files in a local directory and > upload them to a Ceph (S3) server at my University. I am using apache camel > 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false > (the default) in the uri, everything works fine, but if I change to > multiPartUpload=true, it fails. > > I know there is nothing wrong with my s3 secret or s3 access key because > when I set multiPartUpload=false, everything works (there are no crazy plus > (+) characters that need to be escaped in the keys). > > Here is the stack trace: > > com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon > S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: > tx002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended > Request ID: 2213a2-uky-campus-1-uky) > at com.amazonaws.http.AmazonHttpClient$RequestExecutor. > handleErrorResponse(AmazonHttpClient.java:1592) > ~[aws-java-sdk-core-1.11.186.jar:?] > at com.amazonaws.http.AmazonHttpClient$RequestExecutor. > executeOneRequest(AmazonHttpClient.java:1257) > ~[aws-java-sdk-core-1.11.186.jar:?] > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) > ~[aws-java-sdk-core-1.11.186.jar:?] > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) > ~[aws-java-sdk-core-1.11.186.jar:?] > at com.amazonaws.http.AmazonHttpClient$RequestExecutor. > executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186. > jar:?] > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) > ~[aws-java-sdk-core-1.11.186.jar:?] > at > com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) > ~[aws-java-sdk-core-1.11.186.jar:?] > at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl. > execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?] > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) > ~[aws-java-sdk-core-1.11.186.jar:?] > at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) > ~[aws-java-sdk-s3-1.11.186.jar:?] > at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) > ~[aws-java-sdk-s3-1.11.186.jar:?] > at > com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928) > ~[aws-java-sdk-s3-1.11.186.jar:?] > at org.apache.camel.component.aws.s3.S3Producer. > processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0] > at > org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84) > ~[camel-aws-2.20.0.jar:2.20.0] > at ... > > My camel-context.xml looks like: > > > http://www.springframework.org/schema/beans"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; > xsi:schemaLocation=" >http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd >http://camel.apache.org/schema/spring http://camel.apache.org/ > schema/spring/camel-spring.xsd"> > > class="org.apache.camel.component.properties.PropertiesComponent"> > > > > class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfi > gurer"> > value="SYSTEM_PROPERTIES_MODE_OVERRIDE" /> > > > > > > > > > > > > > > > > > > > http://camel.apache.org/schema/spring";> > > > > > > > testMultiPart/${in.header.CamelFileName} > > > > > ${in.header.CamelFileLength} > > > > > > uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true" > /> > > > > > > > > Again, everything works fine if I set multiPartUpload=false in the above > camel-context.xml > > I have tried a lot of things like: > > • setting the CamelAwsS3ContentMD5 header to the MD5 hash of the > file (which doesn't make sense for multi-part files) > • various settings for the partSize parameter > • different sized files from very large to very small > • setting the system parameter: System.setProperty("com. > amazonaws.services.s3.disablePutObjectMD5Validation", "true"); > > If I turn on trace debugging for camel, it doesnt help much: > > [d #2 - file://target/sendToS3/] S3Producer TRACE > Initiating multipart upload [com.amazonaws.services.s3.model. > InitiateMultipartUploadRequest@3731147a] from exchange > [Exchange[ID-Toucan-local-1515115111374-0-1]]... > [d #2 - file://targe
Problem with camel-aws S3 when multiPartUpload is set to true
I created a simple camel route to poll for files in a local directory and upload them to a Ceph (S3) server at my University. I am using apache camel 2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false (the default) in the uri, everything works fine, but if I change to multiPartUpload=true, it fails. I know there is nothing wrong with my s3 secret or s3 access key because when I set multiPartUpload=false, everything works (there are no crazy plus (+) characters that need to be escaped in the keys). Here is the stack trace: com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: tx002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID: 2213a2-uky-campus-1-uky) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1592) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511) ~[aws-java-sdk-core-1.11.186.jar:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227) ~[aws-java-sdk-s3-1.11.186.jar:?] at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174) ~[aws-java-sdk-s3-1.11.186.jar:?] at com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928) ~[aws-java-sdk-s3-1.11.186.jar:?] at org.apache.camel.component.aws.s3.S3Producer.processMultiPart(S3Producer.java:181) ~[camel-aws-2.20.0.jar:2.20.0] at org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84) ~[camel-aws-2.20.0.jar:2.20.0] at ... My camel-context.xml looks like: http://www.springframework.org/schema/beans"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd";> http://camel.apache.org/schema/spring";> testMultiPart/${in.header.CamelFileName} ${in.header.CamelFileLength} Again, everything works fine if I set multiPartUpload=false in the above camel-context.xml I have tried a lot of things like: • setting the CamelAwsS3ContentMD5 header to the MD5 hash of the file (which doesn't make sense for multi-part files) • various settings for the partSize parameter • different sized files from very large to very small • setting the system parameter: System.setProperty("com.amazonaws.services.s3.disablePutObjectMD5Validation", "true"); If I turn on trace debugging for camel, it doesnt help much: [d #2 - file://target/sendToS3/] S3Producer TRACE Initiating multipart upload [com.amazonaws.services.s3.model.InitiateMultipartUploadRequest@3731147a] from exchange [Exchange[ID-Toucan-local-1515115111374-0-1]]... [d #2 - file://target/sendToS3/] S3Producer TRACE Uploading part [1] for testMultiPart/testfile.zip [d #2 - file://target/sendToS3/] DefaultErrorHandlerTRACE Is exchangeId: ID-Toucan-local-1515115111374-0-1 interrupted? false [d #2 - file://target/sendToS3/] DefaultErrorHandlerTRACE Is exchangeId: ID-Toucan-local-1515115111374-0-1 done? false [d #2 - file://target/sendToS3/] DefaultErrorHandlerTRACE isRunAllowed() -> true (Run allowed if we are not stopped/stopping) [d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE Finding best suited exc