[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-55744803 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20387/consoleFull) for PR 1113 at commit [`ab61734`](https://github.com/apache/spark/commit/ab61734844c983cafbbf6eb519f2be319aff5d3f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-55744971 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20387/consoleFull) for PR 1113 at commit [`ab61734`](https://github.com/apache/spark/commit/ab61734844c983cafbbf6eb519f2be319aff5d3f). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` class ArrayConstructor extends net.razorvine.pickle.objects.ArrayConstructor ` * `class SCCallSiteSync(object):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1113#discussion_r17454306
--- Diff:
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala ---
@@ -139,28 +139,32 @@ private[spark] class DriverRunner(
* Download the user jar into the supplied directory and return its
local path.
* Will throw an exception if there are errors downloading the jar.
*/
- private def downloadUserJar(driverDir: File): String = {
-
-val jarPath = new Path(driverDesc.jarUrl)
-
+ private def downloadUserJar(driverDir: File): ArrayBuffer[String] = {
--- End diff --
This should be `downloadUserJars` now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1113#discussion_r17454349
--- Diff:
core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala ---
@@ -139,28 +139,32 @@ private[spark] class DriverRunner(
* Download the user jar into the supplied directory and return its
local path.
* Will throw an exception if there are errors downloading the jar.
*/
- private def downloadUserJar(driverDir: File): String = {
-
-val jarPath = new Path(driverDesc.jarUrl)
-
+ private def downloadUserJar(driverDir: File): ArrayBuffer[String] = {
--- End diff --
Also, can't you just keep the existing `downloadUserJar`, and introduce a
new method called `downloadUserJars` that calls the old one after splitting
`driverDesc.jarUrl`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-55340219 @lianhuiwang Yes, have you looked at SparkSubmit? In particular, does the following do what you want: ``` bin/spark-submit --master standalone --deploy-mode cluster --jars jar1,jar2,jar3 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-54457835 @JoshRosen @andrewor14 I have update comment. has any question for it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-54562091 Could you add `[SPARK-2301]` to the title? This will help us organize PRs and link them to JIRAs properly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-54379494 Thanks for opening a JIRA, but do you mind editing it to actually describe the motivation for this feature? The current JIRA just duplicates the title of this PR, which isn't very helpful. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-54380213 @lianhuiwang Is the motivation for this such that all additional jars (that would be passed through `--jars` through spark-submit, for instance) need not be visible to the driver node, but only to the node submitting the application? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-54380405 My understanding is that if we launch something in `standalone-cluster` mode and our driver uses additional jars, then we need to make sure all the slave nodes on the cluster have access to these jars, because the driver could be launched on either one of these nodes. Is this correct? As Josh suggested, maybe you can add some more description in both the JIRA and the PR description. Also, could you add the JIRA to the title so it's easier for us to track, and up merge this to resolve any conflicts with master? Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-50126195 @andrewor14 can you take a look at this? thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-47295220 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-47295238 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-47297725 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16176/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-47297724 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user xiajunluan commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-46537162 you could open a JIRA issue to describe you request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
GitHub user lianhuiwang opened a pull request: https://github.com/apache/spark/pull/1113 add ability to submit multiple jars for Driver add ability to submit multiple jars for Driver You can merge this pull request into a Git repository by running: $ git pull https://github.com/lianhuiwang/spark driver-multiple-jars Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1113.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1113 commit 7918434b3ae2dbd708caa0ecc940287d0e068d87 Author: unknown administra...@taguswang-pc1.tencent.com Date: 2014-06-18T08:51:24Z driver can submit multiple jars commit 9166e8d8cb2f863dbe1ae59c6290fb0004b432eb Author: lianhuiwang lianhuiwan...@gmail.com Date: 2014-06-18T09:07:17Z fixbug driver can submit multiple jars commit 21b286d2b2f60311ceb827562bf2bf4334a84e14 Author: lianhuiwang lianhuiwan...@gmail.com Date: 2014-06-18T09:10:03Z fixbug format driver can submit multiple jars --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-46413148 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-46413162 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-46416660 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15871/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: add ability to submit multiple jars for Driver
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1113#issuecomment-46416659 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
