[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-172890735 I opened jayv/spark#2 with two more filters: `spark.submit.deployMode` and `spark.master`. I think it's ok to remove `[WIP]` from the title, and if you agree with my

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-172549567 This PR might help with #8872 testing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-172569833 Ok, it seems you need to filter out a few more things, particularly `spark.submit.deployMode="cluster"`. I see errors[1] in the driver `stderr`, since the driver tries

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on a diff in the pull request: https://github.com/apache/spark/pull/10370#discussion_r48311266 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -440,6 +444,9 @@ private[spark] class

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-166797736 **[Test build #48229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48229/consoleFull)** for PR 10370 at commit

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-166811314 **[Test build #48229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48229/consoleFull)** for PR 10370 at commit

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-166811371 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-166811373 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on a diff in the pull request: https://github.com/apache/spark/pull/10370#discussion_r48090568 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -440,6 +444,9 @@ private[spark] class

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on a diff in the pull request: https://github.com/apache/spark/pull/10370#discussion_r48090616 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -440,6 +444,9 @@ private[spark] class

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165961841 thanks @jayv, I left one note, otherwise this looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165893840 @tnachen I've tested that patch against branch-1.5 and I'm now able to specify options such as ```--conf spark.executor.userClassPathFirst=true --conf

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165916177 @jayv Looks like you need to fix the line length > 100 style rule. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165704664 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165706471 **[Test build #47989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47989/consoleFull)** for PR 10370 at commit

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165706192 **[Test build #47989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47989/consoleFull)** for PR 10370 at commit

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165706476 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165706473 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165704814 At first I was wondering if we just copy all scheduler properties will it clash with the ones we used above (i.e --total-executor-cores), but looks like the command

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

GitHub user jayv opened a pull request: https://github.com/apache/spark/pull/10370 [WIP] [SPARK-11327] [MESOS] Dispatcher does not respect all args from the Submit request Supersedes https://github.com/apache/spark/pull/9752 You can merge this pull request into a Git repository by

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv closed the pull request at: https://github.com/apache/spark/pull/9752 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-165655382 New PR against master: https://github.com/apache/spark/pull/10370 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10370#issuecomment-165655626 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-165317051 I wasn't able to make time for this, but I should have time tomorrow. Sorry for the delay. - Jo Voordeckers On Mon, Dec 14, 2015 at 4:51 PM,

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-164606612 @jayv have you had the chance to work on this patch? If not, shall one of us take it over? --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158715518 I will get to it on Monday. - Jo Voordeckers On Sat, Nov 21, 2015 at 2:31 PM, Iulian Dragos wrote: > @jayv

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158687388 @jayv will you have time to update this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158149585 Yes, @jayv would you mind opening this against the master branch? Then committers will decide which branches to put in when we merge. --- If your project is set up

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158180964 I needed a patch for our version, so branched off of that. I'll make a new PR. @dragos no docker, puppet installs our spark build on all our mesos slaves in

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158184439 @jayv I see. I wonder if forwarding all `spark.*` properties will fix it for me. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158185915 I would assume so. When I used `spark.*.extraJavaOptions` to specify `-Dfoo=bar` it got applied to both my driver and tasks which it didn't before my patch. --- If your

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158060398 I noticed this targets branch-1.5. Usually fixes go to master first and then they are backported if needed. I don't think there are plans for another 1.5 release at this

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-158064094 How did you test this? I'm trying to run this using a `spark.executor.uri` pointing to an FTP server, but tasks fail (they don't retrieve the URI). The driver launches

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user dragos commented on a diff in the pull request: https://github.com/apache/spark/pull/9752#discussion_r45224067 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -422,6 +422,37 @@ private[spark] class

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/9752#discussion_r45243548 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -422,6 +422,37 @@ private[spark] class

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on a diff in the pull request: https://github.com/apache/spark/pull/9752#discussion_r45244546 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -422,6 +422,37 @@ private[spark] class

[GitHub] spark pull request: [WIP] [SPARK-11327] Mesos Dispatcher does not ...

Github user jayv commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-157464149 This problem is discussed here: https://issues.apache.org/jira/browse/SPARK-11327 --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [WIP] [SPARK-11327] Mesos Dispatcher does not ...

Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/9752#issuecomment-157547860 Please add [MESOS] and the jira ticket [SPARK-11327] on the title so it gets picked up the Spark pr tool. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/9752#discussion_r45141605 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -422,6 +422,37 @@ private[spark] class

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user jayv commented on a diff in the pull request: https://github.com/apache/spark/pull/9752#discussion_r45144080 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -422,6 +422,37 @@ private[spark] class

[GitHub] spark pull request: [WIP] [SPARK-11327] [MESOS] Dispatcher does no...

Github user tnachen commented on a diff in the pull request: https://github.com/apache/spark/pull/9752#discussion_r45144395 --- Diff: core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala --- @@ -422,6 +422,37 @@ private[spark] class