[
https://issues.apache.org/jira/browse/SPARK-23151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906218#comment-16906218
]
Thomas Graves commented on SPARK-23151:
---------------------------------------
Hey so for the hadoop-3.2 profile, the dependencies are quite a bit different
since we are using the hive version of orc. The things it pulls in are quite
different then the nohive version of orc. It would be really nice if we had a
different classifier or something so users can actually tell the difference in
what is published on mvn. Right now all we know is its a spark jar with version
X.x and scala version y.y but don't know which hadoop profile and thus
dependencies got pulled in. Something to think about anyway
> Provide a distribution of Spark with Hadoop 3.0
> -----------------------------------------------
>
> Key: SPARK-23151
> URL: https://issues.apache.org/jira/browse/SPARK-23151
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Louis Burke
> Priority: Major
>
> Provide a Spark package that supports Hadoop 3.0.0. Currently the Spark
> package
> only supports Hadoop 2.7 i.e. spark-2.2.1-bin-hadoop2.7.tgz. The implication
> is
> that using up to date Kinesis libraries alongside s3 causes a clash w.r.t
> aws-java-sdk.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
