t oo created HADOOP-15553: ----------------------------- Summary: DOCO - not clear how to integrate hadoop-aws and aws-sdk jars with S3A Key: HADOOP-15553 URL: https://issues.apache.org/jira/browse/HADOOP-15553 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.8.3 Reporter: t oo
Documentation update, a compatibility matrix would be nice, especially since upgrading to new aws java sdk does that work : On a hadoop common 2.7.5 cluster, I take AWS Java SDK [1.7.5|https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.7.5] and Hadoop-aws [2.7.5|https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.7.5] and use them in my Spark app writing to S3a, this is fine. On another cluster (this time hadoop common 2.8.3 cluster) I take AWS Java SDK 1.10.6 and Hadoop-aws 2.8.3 and try to use them in my Spark app writing to S3a, but get errors about missing classes. Turns out this version of aws java sdk is tiny with basically nothing inside. Couldn't find any documentation helping users on this incompatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org