[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145932373
  
Merging this, master and branch-1.5.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/8959


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8959#discussion_r41173657
  
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1183,11 +1194,22 @@ object Client extends Logging {
   private def getUserClasspath(
--- End diff --

(I see the other one calls this; both could be merged into a single method, 
though.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145609327
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145606571
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145619279
  
sorry typo, running scalastyle manually and will post updated patch shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145620685
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145620682
  
  [Test build #43245 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43245/console)
 for   PR 8959 at commit 
[`e627adc`](https://github.com/apache/spark/commit/e627adc4c5e239ff52ec9c1e33fe57dfe8294b0d).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145624942
  
  [Test build #43246 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43246/consoleFull)
 for   PR 8959 at commit 
[`ac3ffcf`](https://github.com/apache/spark/commit/ac3ffcfca047804a2cbdea47064a6234ae6f9fac).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145606547
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145609903
  
LGTM pending tests and a minor cleanup.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145619563
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145619542
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145620687
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43245/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145609320
  
  [Test build #43242 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43242/console)
 for   PR 8959 at commit 
[`b860226`](https://github.com/apache/spark/commit/b860226c563fc86a84d2ed80353c6502852829f9).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/8959#discussion_r41173518
  
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1183,11 +1194,22 @@ object Client extends Logging {
   private def getUserClasspath(
--- End diff --

This method is not needed anymore, is it? Only the one that takes a 
`SparkConf` directly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145609329
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43242/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145620136
  
  [Test build #43245 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43245/consoleFull)
 for   PR 8959 at commit 
[`e627adc`](https://github.com/apache/spark/commit/e627adc4c5e239ff52ec9c1e33fe57dfe8294b0d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145608874
  
  [Test build #43242 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43242/consoleFull)
 for   PR 8959 at commit 
[`b860226`](https://github.com/apache/spark/commit/b860226c563fc86a84d2ed80353c6502852829f9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145623250
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145623218
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145671670
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145671588
  
  [Test build #43246 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43246/console)
 for   PR 8959 at commit 
[`ac3ffcf`](https://github.com/apache/spark/commit/ac3ffcfca047804a2cbdea47064a6234ae6f9fac).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145671673
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43246/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145674662
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145674686
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145676066
  
  [Test build #43251 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43251/consoleFull)
 for   PR 8959 at commit 
[`ac3ffcf`](https://github.com/apache/spark/commit/ac3ffcfca047804a2cbdea47064a6234ae6f9fac).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145673897
  
Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145706494
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43251/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145706396
  
  [Test build #43251 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43251/console)
 for   PR 8959 at commit 
[`ac3ffcf`](https://github.com/apache/spark/commit/ac3ffcfca047804a2cbdea47064a6234ae6f9fac).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145706493
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145125393
  
Hmm, I think I see what you mean. Maybe there should be separate handling 
for app jar vs. other jars when `spark.yarn.user.classpath.first` is set.

May this code is overdue for a cleanup / sanity check...

If it starts to get too nasty it might just be easier to add `APP_JAR` to 
the classpath like you're doing, and hope the bug doesn't affect anything else.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-145123502
  
At first glance that sounds good, but on further looking it doesn't work 
because of the way addFileToClasspath is called. It passes them as the uri and 
not filename.

That is why originally I did this patch as just passing it in.  I'll look 
to see if I can clean it up more. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144818093
  
Hi @tgravescs ,

This fixes the problem but I think it's actually just masking a subtle bug 
elsewhere. In `getUserClasspath`, there's this code:

val mainUri = mainJar.orElse(Some(APP_JAR)).map(new URI(_))

That is not actually doing the right thing in certain cases. When invoked 
from a `Client` instance, the `mainJar` argument comes from 
`ClientArguments.userJar`, so it's never going to be `None` (and thus always 
return the name of the original jar instead of `APP_JAR`).

The "cleanest" thing would be to have just a single version of 
`getUserClasspath` that gets things from `SparkConf`, but that runs into the 
problem that the conf has not yet been updated when `populateClasspath` is 
called.

I think changing that `map` call to something like the following would fix 
the source of the problem:

.map { path =>
  val uri = new URI(path)
  if (uri.getScheme == LOCAL_SCHEME) new URI(uri.getPath()) else uri
}

(Not tested.) Does that make sense? Could you try that out?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144810614
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144810576
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[tgravescs]

Github user tgravescs commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144821230
  

where are you suggesting putting this .map so its clear? in getUserPath on 
mainUri?  I'm not seeing how your map call fixes anything so I"m guessing I'm 
missing the context.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144812185
  
  [Test build #43157 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43157/consoleFull)
 for   PR 8959 at commit 
[`08f9382`](https://github.com/apache/spark/commit/08f93822de809946e565793adab7c30c8c8c3430).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[tgravescs]

GitHub user tgravescs opened a pull request:

https://github.com/apache/spark/pull/8959

[SPARK-10901] [YARN] spark.yarn.user.classpath.first doesn't work

This should go into 1.5.2 also.

The issue is we were no longer adding the __app__.jar to the system 
classpath.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgravescs/spark SPARK-10901

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8959.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8959


commit 08f93822de809946e565793adab7c30c8c8c3430
Author: Tom Graves 
Date:   2015-10-01T18:29:47Z

[SPARK-10901] spark.yarn.user.classpath.first doesn't work




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144825265
  
Sorry, my suggestion wasn't very clear and maybe not completely correct. 

Here's the possible values for `mainJar` when that method is called:

- `Some("local:/path")`: in `Client` instance when 
`spark.yarn.user.classpath.first` is true
- `Some("/path")`: in `Client` instance when 
`spark.yarn.user.classpath.first` is true
- `None`: in cluster when user jar is not `local:`
- `Some("local:/path")`: in cluster when user jar is `local:`

The return value for that method for these cases should be URIs pointing to:

- `"/path"`
- `APP_JAR`
- `APP_JAR`
- `"/path"`

So I think the following code should handle that:

val mainUri = mainJar.flatMap { path =>
val uri = new URI(path)
if (uri.getScheme == LOCAL_SCHEME) Some(new URI(uri.getPath())) 
else None
  }.getOrElse(new URI(APP_JAR))

Hope that makes better sense now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144852722
  
  [Test build #43157 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43157/console)
 for   PR 8959 at commit 
[`08f9382`](https://github.com/apache/spark/commit/08f93822de809946e565793adab7c30c8c8c3430).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144852880
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43157/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10901] [YARN] spark.yarn.user.classpath...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8959#issuecomment-144852878
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org