[
https://issues.apache.org/jira/browse/SPARK-1121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400167#comment-15400167
]
Apache Spark commented on SPARK-1121:
-------------------------------------
User 'pwendell' has created a pull request for this issue:
https://github.com/apache/spark/pull/37
> Only add avro if the build is for Hadoop 0.23.X and SPARK_YARN is set
> ---------------------------------------------------------------------
>
> Key: SPARK-1121
> URL: https://issues.apache.org/jira/browse/SPARK-1121
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Patrick Cogan
> Assignee: prashant
> Fix For: 1.0.0
>
>
> The reason why this is needed is that in the 0.23.X versions of hadoop-client
> the avro dependency is fully excluded:
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/0.23.10/hadoop-client-0.23.10.pom
> In later versions 2.2.X the avro dependency is correctly inherited from
> hadoop-common:
> http://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.2.0/hadoop-client-2.2.0.pom
> So as a workaround Spark currently depends on Avro directly in the sbt and
> scala builds. This is a bit ugly so I'd like to propose the following:
> 1. In the Maven build just remove avro and make a note on the
> building-with-maven page that they will need to manually add avro for this
> build.
> 2. On sbt only add the avro dependency if the version is 0.23.X and
> SPARK_YARN is true. Also we only need to add avro not both {avro, avro-ipc}
> like is there now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
