Marcell Hegedus created FLUME-3174:
--------------------------------------
Summary: HdfsSink AWS S3A authentication does not work on JDK 8
Key: FLUME-3174
URL: https://issues.apache.org/jira/browse/FLUME-3174
Project: Flume
Issue Type: Bug
Reporter: Marcell Hegedus
Priority: Minor
Flume writing to S3A with the following Hdfs Sink configuration fails with
AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code:
403 Forbidden...
{code}
a1.sinks = k1
a1.sinks.k1.channel = c1
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = s3a://testflume/logs
{code}
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are provided either in flume-env or
in core-site.xml and running "hdfs dfs -ls s3a://testflume/logs" works properly.
The cause and the fix is documented in
[hadoop-aws/index.md|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md#authentication-failures-when-running-on-java-8u60]
{quote}A change in the Java 8 JVM broke some of the toString() string
generation of Joda Time 2.8.0, which stopped the Amazon S3 client from being
able to generate authentication headers suitable for validation by S3.
Fix: Make sure that the version of Joda Time is 2.8.1 or later, or use a new
version of Java 8.{quote}
Tested that authentication is successful with
* JDK 7
* JDK 8 + joda-time updated to v2.9.6.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
