[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45550412 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45551414 Tested in PivotalHD 1.1 Yarn 4 node cluster. With --addjars file:///somePath/to/jar, launching spark application works. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45527937 @dbtsai could you update the title of this PR to make it more clear what issue this relates to? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45543808 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45451701 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
GitHub user dbtsai opened a pull request: https://github.com/apache/spark/pull/1013 [SPARK-1870] Ported from 1.0 branch to 0.9 branch. Made deployment with --jars work in yarn-standalone mode. Sent secondary jars to distributed cache of all containers and add the cached jars to classpath before executors start. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dbtsai/spark branch-0.9 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1013.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1013 commit 0956af95e24bc37303525fde6f85e0b3aeebd946 Author: DB Tsai dbt...@dbtsai.com Date: 2014-06-08T23:16:53Z Ported SPARK-1870 from 1.0 branch to 0.9 branch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45451705 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45451719 CC: @mengxr and @sryza --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45452356 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15549/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45452355 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1013#discussion_r13529033
--- Diff:
yarn/alpha/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -491,11 +494,18 @@ object Client {
Apps.addToEnvironment(env, Environment.CLASSPATH.name,
Environment.PWD.$() +
Path.SEPARATOR + LOG4J_PROP)
}
+
+val cachedSecondaryJarLinks =
+
sparkConf.getOption(CONF_SPARK_YARN_SECONDARY_JARS).getOrElse().split(,)
+
// Normally the users app.jar is last in case conflicts with spark jars
val userClasspathFirst =
sparkConf.get(spark.yarn.user.classpath.first, false).toBoolean
if (userClasspathFirst) {
Apps.addToEnvironment(env, Environment.CLASSPATH.name,
Environment.PWD.$() +
Path.SEPARATOR + APP_JAR)
+ cachedSecondaryJarLinks.foreach(jarLink =
+Apps.addToEnvironment(env, Environment.CLASSPATH.name,
Environment.PWD.$() +
+Path.SEPARATOR + jarLink))
--- End diff --
minor: add two more spaces.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45459644 @dbtsai Thanks for fixing it in branch-0.9! Did you verify the patch on a YARN cluster? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-1870] Ported from 1.0 branch to 0.9 bra...
Github user dbtsai commented on the pull request: https://github.com/apache/spark/pull/1013#issuecomment-45459920 Work in my local VM. Should work in real yarn cluster. Will test it tomorrow in the office. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
