[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-08 Thread sarutak
GitHub user sarutak opened a pull request:

https://github.com/apache/spark/pull/2728

[SPARK-3657] yarn alpha YarnRMClientImpl throws NPE 
appMasterRequest.setTrackingUrl starting spark-shell

@tgraves reported this issue.

Following is quoted from @tgraves' report.

YarnRMClientImpl.registerApplicationMaster can throw null pointer exception 
when setting the trackingurl if its empty:

appMasterRequest.setTrackingUrl(new URI(uiAddress).getAuthority())


I hit this just start spark-shell without the tracking url set.

14/09/23 16:18:34 INFO yarn.YarnRMClientImpl: Connecting to ResourceManager 
at kryptonitered-jt1.red.ygrid.yahoo.com/98.139.154.99:8030
Exception in thread "main" java.lang.NullPointerException
at 
org.apache.hadoop.yarn.proto.YarnServiceProtos$RegisterApplicationMasterRequestProto$Builder.setTrackingUrl(YarnServiceProtos.java:710)
at 
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.RegisterApplicationMasterRequestPBImpl.setTrackingUrl(RegisterApplicationMasterRequestPBImpl.java:132)
at 
org.apache.spark.deploy.yarn.YarnRMClientImpl.registerApplicationMaster(YarnRMClientImpl.scala:102)
at 
org.apache.spark.deploy.yarn.YarnRMClientImpl.register(YarnRMClientImpl.scala:55)
at 
org.apache.spark.deploy.yarn.YarnRMClientImpl.register(YarnRMClientImpl.scala:38)
at 
org.apache.spark.deploy.yarn.ApplicationMaster.registerAM(ApplicationMaster.scala:168)
at 
org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:206)
at 
org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:120)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sarutak/spark SPARK-3657

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2728.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2728


commit 8b5a96e9ed312ac3936147ce0aea6b37347eedfc
Author: Kousuke Saruta 
Date:   2014-10-09T05:40:14Z

Fixed NPE thrown when null is passed to setTrackingUrl




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-08 Thread sarutak
Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58465727
  
CC @tgravescs 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-08 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58466046
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21524/consoleFull)
 for   PR 2728 at commit 
[`8b5a96e`](https://github.com/apache/spark/commit/8b5a96e9ed312ac3936147ce0aea6b37347eedfc).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58469995
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21524/Test 
PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-08 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58469990
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21524/consoleFull)
 for   PR 2728 at commit 
[`8b5a96e`](https://github.com/apache/spark/commit/8b5a96e9ed312ac3936147ce0aea6b37347eedfc).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-09 Thread tgravescs
Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58504345
  
@sarutak thanks for working on this.  Did you by chance test this to see if 
the tracking url is really not being set in the RM UI for all the modes 
(including using spark-shell)?  If its not there might be a bug somewhere else 
as that used to work.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58572778
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-10 Thread sarutak
Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58680922
  
@tgravescs I didn't really understand the root cause.
Now I resolved that.
When we run YARN cluster mode, uiAddress starts with scheme but in YARN 
client mode, uiAddress is only authority, not with scheme so 
URI(uiAddress).getAuthority returns null right?

I attached this patch and run client/cluster mode including Spark Shell.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-10 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58680913
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21597/consoleFull)
 for   PR 2728 at commit 
[`592b5d7`](https://github.com/apache/spark/commit/592b5d7735687416e708acdb6b71a879fd1efe7a).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-10 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58689483
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21597/consoleFull)
 for   PR 2728 at commit 
[`592b5d7`](https://github.com/apache/spark/commit/592b5d7735687416e708acdb6b71a879fd1efe7a).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58689495
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21597/Test 
PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58776274
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21663/consoleFull)
 for   PR 2728 at commit 
[`2d43c64`](https://github.com/apache/spark/commit/2d43c64c83b4fbd293dec5a8cdd8a96fdad2c125).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58776270
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21663/consoleFull)
 for   PR 2728 at commit 
[`2d43c64`](https://github.com/apache/spark/commit/2d43c64c83b4fbd293dec5a8cdd8a96fdad2c125).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58776276
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21663/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread sarutak
Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58776416
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58776475
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21666/consoleFull)
 for   PR 2728 at commit 
[`2d43c64`](https://github.com/apache/spark/commit/2d43c64c83b4fbd293dec5a8cdd8a96fdad2c125).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58783062
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21666/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58783054
  
**[Tests timed 
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21666/consoleFull)**
 for PR 2728 at commit 
[`2d43c64`](https://github.com/apache/spark/commit/2d43c64c83b4fbd293dec5a8cdd8a96fdad2c125)
 after a configured wait of `120m`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread sarutak
Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58798685
  
retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58798823
  
  [QA tests have 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21668/consoleFull)
 for   PR 2728 at commit 
[`2d43c64`](https://github.com/apache/spark/commit/2d43c64c83b4fbd293dec5a8cdd8a96fdad2c125).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58802491
  
  [QA tests have 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21668/consoleFull)
 for   PR 2728 at commit 
[`2d43c64`](https://github.com/apache/spark/commit/2d43c64c83b4fbd293dec5a8cdd8a96fdad2c125).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-58802496
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21668/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-17 Thread sarutak
Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-59483505
  
test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-20 Thread tgravescs
Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-59771521
  
So one issue is that the scheme was added to properly handle when yarn 
using https (SPARK-3286).  If client mode isn't passing the scheme then that is 
probably broken.  If it was passing the scheme that you wouldn't hit this 
issue. I think changing the YarnClientSchedulerBackend.start routine where it 
sets the spark.driver.appUIAddress would be the equivalent change.  And then we 
would need to test.

With the above change it would have the scheme included and wouldn't hit 
the null.  If we want to add the check in anyway for handling the case where it 
is null just in case something else comes up, thats fine, but I'm not real fond 
of pattern matching here.  How about just checking the URI.getScheme and if 
null we pass it in as is, otherwise we do the getAuthority()?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread tgravescs
Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-60761644
  
@sarutak could you submit a different pr for this so jenkins picks it up?  
Then close this one


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread sarutak
GitHub user sarutak opened a pull request:

https://github.com/apache/spark/pull/2981

[SPARK-3657] yarn alpha YarnRMClientImpl throws NPE 
appMasterRequest.setTrackingUrl starting spark-shell

@tgravescs reported this issue.

Following is quoted from @tgravescs' report.

YarnRMClientImpl.registerApplicationMaster can throw null pointer exception 
when setting the trackingurl if its empty:

appMasterRequest.setTrackingUrl(new URI(uiAddress).getAuthority())


I hit this just start spark-shell without the tracking url set.

14/09/23 16:18:34 INFO yarn.YarnRMClientImpl: Connecting to ResourceManager 
at kryptonitered-jt1.red.ygrid.yahoo.com/98.139.154.99:8030
Exception in thread "main" java.lang.NullPointerException
at 
org.apache.hadoop.yarn.proto.YarnServiceProtos$RegisterApplicationMasterRequestProto$Builder.setTrackingUrl(YarnServiceProtos.java:710)
at 
org.apache.hadoop.yarn.api.protocolrecords.impl.pb.RegisterApplicationMasterRequestPBImpl.setTrackingUrl(RegisterApplicationMasterRequestPBImpl.java:132)
at 
org.apache.spark.deploy.yarn.YarnRMClientImpl.registerApplicationMaster(YarnRMClientImpl.scala:102)
at 
org.apache.spark.deploy.yarn.YarnRMClientImpl.register(YarnRMClientImpl.scala:55)
at 
org.apache.spark.deploy.yarn.YarnRMClientImpl.register(YarnRMClientImpl.scala:38)
at 
org.apache.spark.deploy.yarn.ApplicationMaster.registerAM(ApplicationMaster.scala:168)
at 
org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:206)
at 
org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:120)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sarutak/spark SPARK-3657-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/2981.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2981


commit 70b8882fe48bba6c90f3f0ba15d4d9659c04b1f9
Author: Kousuke Saruta 
Date:   2014-10-23T10:36:21Z

Fixed NPE thrown

commit e2fd6bc5964407cdf09bd86b1c46aba3bb2d1e62
Author: Kousuke Saruta 
Date:   2014-10-28T16:41:24Z

Merge branch 'master' of git://git.apache.org/spark into SPARK-3657




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2981#issuecomment-60789490
  
  [Test build #22363 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22363/consoleFull)
 for   PR 2981 at commit 
[`e2fd6bc`](https://github.com/apache/spark/commit/e2fd6bc5964407cdf09bd86b1c46aba3bb2d1e62).
 * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread sarutak
Github user sarutak commented on the pull request:

https://github.com/apache/spark/pull/2728#issuecomment-60789634
  
@tgravescs  I opened new PR and Jenkins worked in #2981. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread sarutak
Github user sarutak closed the pull request at:

https://github.com/apache/spark/pull/2728


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread SparkQA
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/2981#issuecomment-60801632
  
  [Test build #22363 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22363/consoleFull)
 for   PR 2981 at commit 
[`e2fd6bc`](https://github.com/apache/spark/commit/e2fd6bc5964407cdf09bd86b1c46aba3bb2d1e62).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/2981#issuecomment-60801645
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22363/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread vanzin
Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/2981#issuecomment-60802597
  
hmmm... wonder if #2958 has anything to do with this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread andrewor14
Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/2981#issuecomment-60812921
  
Hm yeah though it looks like this JIRA has been open for a while


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread tgravescs
Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/2981#issuecomment-60813861
  
I asked him to file new pr since old pr wasn't being picked up by Jenkins.  
Yes pr2958 should have fixed the root cause of this.  It doesn't hurt to have 
the check here too.  Its going to end up being deprecated here soon anyway.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread andrewor14
Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/2981#issuecomment-60817683
  
I see. Alright I'll merge this. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-28 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/2981


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org