I created a simple camel route to poll for files in a local directory and
upload them to a Ceph (S3) server at my University. I am using apache camel
2.20.0 with the camel-aws S3 component, when I set multiPartUpload=false (the
default) in the uri, everything works fine, but if I change to
multiPartUpload=true, it fails.
I know there is nothing wrong with my s3 secret or s3 access key because when I
set multiPartUpload=false, everything works (there are no crazy plus (+)
characters that need to be escaped in the keys).
Here is the stack trace:
com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3;
Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID:
2213a2-uky-campus-1-uky)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1592)
~[aws-java-sdk-core-1.11.186.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1257)
~[aws-java-sdk-core-1.11.186.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1029)
~[aws-java-sdk-core-1.11.186.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:741)
~[aws-java-sdk-core-1.11.186.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:715)
~[aws-java-sdk-core-1.11.186.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:697)
~[aws-java-sdk-core-1.11.186.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:665)
~[aws-java-sdk-core-1.11.186.jar:?]
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:647)
~[aws-java-sdk-core-1.11.186.jar:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:511)
~[aws-java-sdk-core-1.11.186.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4227)
~[aws-java-sdk-s3-1.11.186.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4174)
~[aws-java-sdk-s3-1.11.186.jar:?]
at
com.amazonaws.services.s3.AmazonS3Client.abortMultipartUpload(AmazonS3Client.java:2928)
~[aws-java-sdk-s3-1.11.186.jar:?]
at
org.apache.camel.component.aws.s3.S3Producer.processMultiPart(S3Producer.java:181)
~[camel-aws-2.20.0.jar:2.20.0]
at org.apache.camel.component.aws.s3.S3Producer.process(S3Producer.java:84)
~[camel-aws-2.20.0.jar:2.20.0]
at ...
My camel-context.xml looks like:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://camel.apache.org/schema/spring
http://camel.apache.org/schema/spring/camel-spring.xsd";>
<bean id="properties"
class="org.apache.camel.component.properties.PropertiesComponent">
<property name="location" value="config.properties" />
</bean>
<bean id="bridgePropertyPlaceholder"
class="org.apache.camel.spring.spi.BridgePropertyPlaceholderConfigurer">
<property name="systemPropertiesModeName"
value="SYSTEM_PROPERTIES_MODE_OVERRIDE" />
<property name="location" value="config.properties" />
</bean>
<bean id="amazonClient" class="com.amazonaws.services.s3.AmazonS3Client">
<constructor-arg>
<bean class="com.amazonaws.auth.BasicAWSCredentials">
<constructor-arg name="accessKey" value="${s3AccessKey}" />
<constructor-arg name="secretKey" value="${s3SecretKey}" />
</bean>
</constructor-arg>
<property name="s3ClientOptions">
<bean class="com.amazonaws.services.s3.S3ClientOptions">
<property name="pathStyleAccess" value="true" />
</bean>
</property>
<property name="endpoint" value="${s3Endpoint}" />
</bean>
<camelContext xmlns="http://camel.apache.org/schema/spring";>
<route>
<from uri="file:target/sendToS3/?antInclude=*.*" />
<log message="Found file: ${in.header.CamelFileName}" />
<setHeader headerName="CamelAwsS3Key">
<simple>testMultiPart/${in.header.CamelFileName}
</simple>
</setHeader>
<setHeader headerName="CamelAwsS3ContentLength">
<simple>${in.header.CamelFileLength}
</simple>
</setHeader>
<log message="Send file to S3: ${properties:s3Endpoint}" />
<to
uri="aws-s3://{{s3Bucket}}?amazonS3Client=#amazonClient&multiPartUpload=true"
/>
<log message="Done sending file." />
</route>
</camelContext>
</beans>
Again, everything works fine if I set multiPartUpload=false in the above
camel-context.xml
I have tried a lot of things like:
• setting the CamelAwsS3ContentMD5 header to the MD5 hash of the file
(which doesn't make sense for multi-part files)
• various settings for the partSize parameter
• different sized files from very large to very small
• setting the system parameter:
System.setProperty("com.amazonaws.services.s3.disablePutObjectMD5Validation",
"true");
If I turn on trace debugging for camel, it doesnt help much:
[d #2 - file://target/sendToS3/] S3Producer TRACE
Initiating multipart upload
[com.amazonaws.services.s3.model.InitiateMultipartUploadRequest@3731147a] from
exchange [Exchange[ID-Toucan-local-1515115111374-0-1]]...
[d #2 - file://target/sendToS3/] S3Producer TRACE Uploading
part [1] for testMultiPart/testfile.zip
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is
exchangeId: ID-Toucan-local-1515115111374-0-1 interrupted? false
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE Is
exchangeId: ID-Toucan-local-1515115111374-0-1 done? false
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE
isRunAllowed() -> true (Run allowed if we are not stopped/stopping)
[d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE Finding
best suited exception policy for thrown exception
com.amazonaws.services.s3.model.AmazonS3Exception
[d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE Finding
best suited exception policy for thrown exception
com.amazonaws.services.s3.model.AmazonS3Exception
[d #2 - file://target/sendToS3/] DefaultExceptionPolicyStrategy TRACE Found 0
candidates
[d #2 - file://target/sendToS3/] DefaultErrorHandler DEBUG Failed
delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on ExchangeId:
ID-Toucan-local-1515115111374-0-1). On delivery attempt: 0 caught:
com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3;
Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID:
2213a2-uky-campus-1-uky), S3 Extended Request ID: 2213a2-uky-campus-1-uky
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE
isRedeliveryAllowed() -> true (we are not stopping/stopped)
[d #2 - file://target/sendToS3/] DefaultErrorHandler TRACE This
exchange is not handled or continued so its marked as failed:
Exchange[ID-Toucan-local-1515115111374-0-1]
[d #2 - file://target/sendToS3/] InstrumentationProcessor TRACE to:
Recording duration: 356930 millis for exchange:
Exchange[ID-Toucan-local-1515115111374-0-1]
[d #2 - file://target/sendToS3/] DefaultErrorHandler ERROR Failed
delivery for (MessageId: ID-Toucan-local-1515115111374-0-2 on ExchangeId:
ID-Toucan-local-1515115111374-0-1). Exhausted after delivery attempt: 1 caught:
com.amazonaws.services.s3.model.AmazonS3Exception: null (Service: Amazon S3;
Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
tx000000000000002e9edee-005a4ed3d2-2213a2-uky-campus-1; S3 Extended Request ID:
2213a2-uky-campus-1-uky), S3 Extended Request ID: 2213a2-uky-campus-1-uky
The reason I need multi-part uploads to work is for very large files, but I
can't get it to work on any file size.
(I posted this on StackOverflow since I am not sure how active this mailing
list is:
https://stackoverflow.com/questions/48106180/apache-camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu
<https://stackoverflow.com/questions/48106180/apache-camel-s3-upload-fails-with-status-code-403-signaturedoesnotmatch-when-mu>)
