In this page http://spark.apache.org/docs/0.9.0/running-on-yarn.html
We have to use spark assembly to submit spark apps to yarn cluster.
And I checked the assembly jars of spark. It contains some yarn classes
which are added during compile time. The yarn classes are not what I want.
My question i
Hi Gordon,
We recently handled this in SPARK-1064. As of 1.0.0, you'll be able to
pass -Phadoop-provided to Maven and avoid including Hadoop and its
dependencies in the assembly jar.
-Sandy
On Tue, Apr 22, 2014 at 2:43 AM, Gordon Wang wrote:
> In this page http://spark.apache.org/docs/0.9.0/
Hi Sandy,
Thanks for your reply !
Does this work for sbt ?
I checked the commit, looks like only maven build has such option.
On Wed, Apr 23, 2014 at 12:38 AM, Sandy Ryza wrote:
> Hi Gordon,
>
> We recently handled this in SPARK-1064. As of 1.0.0, you'll be able to
> pass -Phadoop-provided
Right, it only works for Maven
On Tue, Apr 22, 2014 at 6:23 PM, Gordon Wang wrote:
> Hi Sandy,
>
> Thanks for your reply !
>
> Does this work for sbt ?
>
> I checked the commit, looks like only maven build has such option.
>
>
>
> On Wed, Apr 23, 2014 at 12:38 AM, Sandy Ryza wrote:
>
>> Hi Gord
Thanks I see. Do you guys have plan to port this to sbt?
On Wed, Apr 23, 2014 at 10:24 AM, Sandy Ryza wrote:
> Right, it only works for Maven
>
>
> On Tue, Apr 22, 2014 at 6:23 PM, Gordon Wang wrote:
>
>> Hi Sandy,
>>
>> Thanks for your reply !
>>
>> Does this work for sbt ?
>>
>> I checked the
I currently don't have plans to work on that.
-Sandy
> On Apr 22, 2014, at 8:06 PM, Gordon Wang wrote:
>
> Thanks I see. Do you guys have plan to port this to sbt?
>
>
>> On Wed, Apr 23, 2014 at 10:24 AM, Sandy Ryza wrote:
>> Right, it only works for Maven
>>
>>
>>> On Tue, Apr 22, 2014 at
