[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8649 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138983239 LGTM, merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138855538 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138855539 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42194/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138855250 [Test build #42194 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42194/console) for PR 8649 at commit [`08f7c9a`](https://github.com/apache/spark/commit/08f7c9a588cd88837b48ce107b6e36311d4ff64d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138808603 [Test build #42194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42194/consoleFull) for PR 8649 at commit [`08f7c9a`](https://github.com/apache/spark/commit/08f7c9a588cd88837b48ce107b6e36311d4ff64d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138808192 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138808209 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user zjffdu commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138808084 Thanks @vanzin update the patch with re-word message. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138739307 @zjffdu could you re-word the message? I'd suggest something like: Could not find jar containing Spark classes. The jar can be defined using the spark.yarn.jar configuration option. If testing Spark, either set that option or make sure SPARK_PREPEND_CLASSES is not set. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138738866 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138738867 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42155/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138738797 [Test build #42155 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42155/console) for PR 8649 at commit [`39fe23a`](https://github.com/apache/spark/commit/39fe23a646f5c269adf37dc70ac661adda009c38). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138716140 @vanzin makes sense if its only going to impact spark developers (thought it might also be a user facing exception). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138715312 [Test build #42155 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42155/consoleFull) for PR 8649 at commit [`39fe23a`](https://github.com/apache/spark/commit/39fe23a646f5c269adf37dc70ac661adda009c38). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138714778 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138714751 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138714001 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138708571 > Wouldn't it make more sense to check for this in SparkSubmitArguments or maybe the yarn specific validation code? I'm not sure there's a common enough place where you could call this code, at least without reflection. You could try to copy it, but duplicated code is bad. In any case, this should only really affect Spark developers, so while it's nice to have a better message, we shouldn't need to be too fancy here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8649#discussion_r38982863
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1045,7 +1045,8 @@ object Client extends Logging {
s"in favor of the $CONF_SPARK_JAR configuration variable.")
System.getenv(ENV_SPARK_JAR)
} else {
- SparkContext.jarOfClass(this.getClass).head
+ SparkContext.jarOfClass(this.getClass).getOrElse(throw new
SparkException("Could not " +
+"find jar with spark-yarn related code, make sure
SPARK_PREPEND_CLASSES is not set"))
--- End diff --
If you want to be more helpful, you can also mention that the user can set
`spark.yarn.jar` to force Spark to use a specific Spark archive instead of
trying to figure it out from the classpath.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138706578 Wouldn't it make more sense to check for this in SparkSubmitArguments or maybe the yarn specific validation code? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/8649#discussion_r38887638
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1045,7 +1045,8 @@ object Client extends Logging {
s"in favor of the $CONF_SPARK_JAR configuration variable.")
System.getenv(ENV_SPARK_JAR)
} else {
- SparkContext.jarOfClass(this.getClass).head
+ SparkContext.jarOfClass(this.getClass).getOrElse(throw new
SparkException("Could not find jar with spark-yarn related code, " +
--- End diff --
Nit: this line is too long (100 chars).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user zjffdu commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138411720 Paste the new exception after this patch. ``` 15/09/08 10:05:52 ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Could not find jar with spark-yarn related code,Make sure SPARK_PREPEND_CLASSES is not set at org.apache.spark.deploy.yarn.Client$$anonfun$org$apache$spark$deploy$yarn$Client$$sparkJar$2.apply(Client.scala:1048) at org.apache.spark.deploy.yarn.Client$$anonfun$org$apache$spark$deploy$yarn$Client$$sparkJar$2.apply(Client.scala:1048) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.deploy.yarn.Client$.org$apache$spark$deploy$yarn$Client$$sparkJar(Client.scala:1048) at org.apache.spark.deploy.yarn.Client$.populateClasspath(Client.scala:1160) at org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:534) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:645) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144) at org.apache.spark.SparkContext.(SparkContext.scala:514) at com.zjffdu.tutorial.spark.WordCount$.main(WordCount.scala:24) at com.zjffdu.tutorial.spark.WordCount.main(WordCount.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:680) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8649#issuecomment-138410759 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10481 SPARK_PREPEND_CLASSES make spark-y...
GitHub user zjffdu opened a pull request: https://github.com/apache/spark/pull/8649 SPARK-10481 SPARK_PREPEND_CLASSES make spark-yarn related jar could n⦠Throw a more readable exception. Please help review. Thanks You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/spark SPARK-10481 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8649.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8649 commit 354c067e477a6101cf458003022346b61609be8f Author: Jeff Zhang Date: 2015-09-08T02:14:21Z SPARK-10481 SPARK_PREPEND_CLASSES make spark-yarn related jar could not be found --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
