[
https://issues.apache.org/jira/browse/SPARK-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rafal Kwasny closed SPARK-5348.
-------------------------------
Resolution: Not a Problem
> s3a:// protocol and hadoop-aws dependency
> -----------------------------------------
>
> Key: SPARK-5348
> URL: https://issues.apache.org/jira/browse/SPARK-5348
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 1.2.0
> Reporter: Rafal Kwasny
> Priority: Minor
>
> Hadoop 2.6.0+ includes new s3a library which is a lot better than the old one.
> https://issues.apache.org/jira/browse/HADOOP-10400
> -Unfortunately this library is linking with amazon sdk which I think is not
> compatible with apache licensing. For this reason it is not a dependency for
> hadoop-client.-
> What I did is just add a profile for maven build, not sure if there is a
> better option to integrate this, we could also add a new hadoop-2.6 profile.
> {noformat}
> diff --git a/pom.xml b/pom.xml
> index a9e7c9d..71ee776 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -1214,6 +1214,27 @@
> </profile>
> <profile>
> + <id>hadoop-aws</id>
> + <dependencies>
> + <dependency>
> + <groupId>org.apache.hadoop</groupId>
> + <artifactId>hadoop-aws</artifactId>
> + <version>${hadoop.version}</version>
> + <exclusions>
> + <exclusion>
> + <groupId>com.fasterxml.jackson.core</groupId>
> + <artifactId>jackson-annotations</artifactId>
> + </exclusion>
> + <exclusion>
> + <groupId>com.fasterxml.jackson.core</groupId>
> + <artifactId>jackson-databind</artifactId>
> + </exclusion>
> + </exclusions>
> + </dependency>
> + </dependencies>
> + </profile>
> +
> + <profile>
> <id>java8-tests</id>
> <build>
> <plugins>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
