Re: Possible bug in ClientBase.scala?

2014-07-16 Thread Sandy Ryza 
Hi Ron,

I just checked and this bug is fixed in recent releases of Spark.

-Sandy


On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen  wrote:

> Ron,
> Which distribution and Version of Hadoop are you using ?
>
>  I just looked at CDH5 (  hadoop-mapreduce-client-core-
> 2.3.0-cdh5.0.0),
>
> MRJobConfig does have the field :
>
> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>
> Chester
>
>
>
> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez 
> wrote:
>
>> Hi,
>>   I was doing programmatic submission of Spark yarn jobs and I saw code
>> in ClientBase.getDefaultYarnApplicationClasspath():
>>
>> val field =
>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>> MRJobConfig doesn't have this field so the created launch env is
>> incomplete. Workaround is to set yarn.application.classpath with the value
>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>
>> This results in having the spark job hang if the submission config is
>> different from the default config. For example, if my resource manager port
>> is 8050 instead of 8030, then the spark app is not able to register itself
>> and stays in ACCEPTED state.
>>
>> I can easily fix this by changing this to YarnConfiguration instead of
>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>
>> Thanks,
>> Ron
>>
>> Sent from my iPhone
>
>
>

Re: Possible bug in ClientBase.scala?

2014-07-13 Thread Chester Chen 
Ron,
Which distribution and Version of Hadoop are you using ?

 I just looked at CDH5 (  hadoop-mapreduce-client-core-
2.3.0-cdh5.0.0),

MRJobConfig does have the field :

java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;

Chester



On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez  wrote:

> Hi,
>   I was doing programmatic submission of Spark yarn jobs and I saw code in
> ClientBase.getDefaultYarnApplicationClasspath():
>
> val field =
> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
> MRJobConfig doesn't have this field so the created launch env is
> incomplete. Workaround is to set yarn.application.classpath with the value
> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>
> This results in having the spark job hang if the submission config is
> different from the default config. For example, if my resource manager port
> is 8050 instead of 8030, then the spark app is not able to register itself
> and stays in ACCEPTED state.
>
> I can easily fix this by changing this to YarnConfiguration instead of
> MRJobConfig but was wondering what the steps are for submitting a fix.
>
> Thanks,
> Ron
>
> Sent from my iPhone

Re: Possible bug in ClientBase.scala?

2014-07-13 Thread Nicholas Chammas 
On Sun, Jul 13, 2014 at 9:49 PM, Ron Gonzalez  wrote:

> I can easily fix this by changing this to YarnConfiguration instead of
> MRJobConfig but was wondering what the steps are for submitting a fix.
>

Relevant links:

   - https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
   - https://github.com/apache/spark/pulls

Nick

Possible bug in ClientBase.scala?

2014-07-13 Thread Ron Gonzalez 
Hi,
  I was doing programmatic submission of Spark yarn jobs and I saw code in 
ClientBase.getDefaultYarnApplicationClasspath():

val field = classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
MRJobConfig doesn't have this field so the created launch env is incomplete. 
Workaround is to set yarn.application.classpath with the value from 
YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.

This results in having the spark job hang if the submission config is different 
from the default config. For example, if my resource manager port is 8050 
instead of 8030, then the spark app is not able to register itself and stays in 
ACCEPTED state.

I can easily fix this by changing this to YarnConfiguration instead of 
MRJobConfig but was wondering what the steps are for submitting a fix.

Thanks,
Ron

Sent from my iPhone