[
https://issues.apache.org/jira/browse/SPARK-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102472#comment-14102472
]
Sean Owen commented on SPARK-3122:
----------------------------------
[~gq] You do not need to depend on hadoop-client for Spark's sake. You may need
to depend on it because your code uses Hadoop directly. Again if you include
hadoop-client, it should be "provided". Your example shows a compile time
dependency, and duplicates a single dependency.
[~ran.l...@hp.com] spark-yarn is really the 'server side' implementation. You
depend on spark-core. It should be a "provided" dependency. The Spark artifacts
published to Maven happens to reference Hadoop 1.x but it will not matter for
your build, because you would not inherit its dependencies. All you care about
is Spark's API, which does not vary with Hadoop version.
> hadoop-yarn dependencies cannot be resolved
> -------------------------------------------
>
> Key: SPARK-3122
> URL: https://issues.apache.org/jira/browse/SPARK-3122
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0, 1.0.2
> Environment: spark 1.0.1 + YARN + hadoop 2.4.0 cluster on linux
> machines
> client on windows 7
> maven 3.0.4
> maven repository http://repo1.maven.org/maven2/org/apache/hadoop/
> Reporter: Ran Levi
> Labels: build, easyfix, hadoop, maven
>
> When adding spark-yarn_2.10:1.0.2 dependency to java project, other
> hadoop-yarn-XXX dependencies are needed. Those dependencies are downloaded
> using version 1.0.4 which does not exist, resulting in build error.
> Version 1.0.4 is taken from hadoop.version variable in spark-parent-1.0.2.pom.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
