Right, it only works for Maven
On Tue, Apr 22, 2014 at 6:23 PM, Gordon Wang <gw...@gopivotal.com> wrote: > Hi Sandy, > > Thanks for your reply ! > > Does this work for sbt ? > > I checked the commit, looks like only maven build has such option. > > > > On Wed, Apr 23, 2014 at 12:38 AM, Sandy Ryza <sandy.r...@cloudera.com>wrote: > >> Hi Gordon, >> >> We recently handled this in SPARK-1064. As of 1.0.0, you'll be able to >> pass -Phadoop-provided to Maven and avoid including Hadoop and its >> dependencies in the assembly jar. >> >> -Sandy >> >> >> On Tue, Apr 22, 2014 at 2:43 AM, Gordon Wang <gw...@gopivotal.com> wrote: >> >>> In this page http://spark.apache.org/docs/0.9.0/running-on-yarn.html >>> >>> We have to use spark assembly to submit spark apps to yarn cluster. >>> And I checked the assembly jars of spark. It contains some yarn classes >>> which are added during compile time. The yarn classes are not what I want. >>> >>> My question is that is it possible to use other jars to submit spark app >>> to yarn cluster. >>> I do not want to use the assembly jar because it has yarn classes which >>> may overwrite the yarn class in HADOOP_CLASSPATH. If the yarn cluster is >>> upgraded, even if the YARN apis are same, spark has to be recompiled >>> against to the new version of yarn. >>> >>> Any help is appreciated ! Thanks. >>> >>> -- >>> Regards >>> Gordon Wang >>> >> >> > > > -- > Regards > Gordon Wang >
