[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20641 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20641 yap, that sounds like a good idea --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user krcz commented on the issue: https://github.com/apache/spark/pull/20641 @felixcheung It fixes the same problem so in terms of implementation it is not very different. But when I created this one 21014 didn't exist. The only difference is that this PR adds an unit test, which 21014 had not. That's why I'm asking if I should modify this PR to just add the test. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20641 @krcz how is your PR different from 21014? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user felixb commented on the issue: https://github.com/apache/spark/pull/20641 @krcz @felixcheung <- different felix ;) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user krcz commented on the issue: https://github.com/apache/spark/pull/20641 @felixb @skonto I've triend rebasing, but it looks that the issue has been meanwhile fixed independently in [other pull request](https://github.com/apache/spark/pull/21014) while someone was working on [SPARK-23941 issue](https://issues.apache.org/jira/browse/SPARK-23941). It looks that there is no unit test there though, so I can modify his PR to incorporate just the test. Do you want that or should I just close the pull request? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20641 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91820/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20641 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20641 **[Test build #91820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91820/testReport)** for PR 20641 at commit [`22c6739`](https://github.com/apache/spark/commit/22c67392b9f920af8fae9de1f7c7f5fb5a25abb8). * This patch **fails due to an unknown error code, -9**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20641 **[Test build #91820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91820/testReport)** for PR 20641 at commit [`22c6739`](https://github.com/apache/spark/commit/22c67392b9f920af8fae9de1f7c7f5fb5a25abb8). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20641 @mgummelt too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20641 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20641 ok, could you rebase this PR --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user susanxhuynh commented on the issue: https://github.com/apache/spark/pull/20641 @skonto Thanks for testing it. Tests results look good. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/20641 @krcz yes it should be ok. So I verified it: Dispatcher passes the following command to mesos: ```18/06/13 10:30:45 INFO MesosClusterScheduler: Command for the driver (test):./bin/spark-submit --name test --master mesos://zk://master.mesos:2181/mesos --driver-cores 1.0 --driver-memory 1000M --class org.apache.spark.examples.SparkPi --conf spark.executor.extraJavaOptions="-Dexecutor.test.param1=\"value 1\" -Dexecutor.test.param2=value\\ 2 -Dexecutor.test.path=\$PATH" --conf spark.mesos.containerizer=mesos --conf spark.driver.supervise=false --conf spark.app.name=test --conf spark.driver.memory=1000M --conf ``` spark-class will run this at the executor side: ``` exec /usr/lib/jvm/jre1.8.0_152/bin/java -cp '/opt/spark/dist/conf/:/opt/spark/dist/jars/*:/etc/hadoop/' '-Dexecutor.test.param1=value 1' '-Dexecutor.test.param2=value 2' '-Dexecutor.test.path=$PATH' -Xmx1024m org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://coarsegrainedschedu...@ip-10-0-0-77.us-west-1.compute.internal:33887 --executor-id 9 --hostname 10.0.0.77 --cores 3 --app-id bddf96ce-45f6-45f7-ac83-5a3622cafc41-0032-driver-20180613103044-0001 ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user krcz commented on the issue: https://github.com/apache/spark/pull/20641 @skonto The one directly below "At the driver side in DC/OS:" in your comment. But if it's output of ps that's ok, as `ps` wouldn't be able to discern between `command "foo bar"` and `command foo bar`. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/20641 @krcz which one? Btw both job run successfully although might not have been escaped properly. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user krcz commented on the issue: https://github.com/apache/spark/pull/20641 @skonto The command you have pasted as "At the driver side in DC/OS:" concerns me. It doesn't seem escaped properly. Is it the command to be run by shell or is it just space-joined arguments list (as in Spark Dispatcher web UI), which would explain why it is not escaped? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user skonto commented on the issue: https://github.com/apache/spark/pull/20641 @susanxhuynh I tested this. Here is what you get: ``` ./bin/spark-submit --deploy-mode cluster --name test --master mesos://spark.marathon.mesos:13830 --driver-cores 1.0 --driver-memory 1000M --class org.apache.spark.examples.SparkPi --conf spark.executor.extraJavaOptions="-Dparam1=\"value 1\" -Dparam2=value\\ 2 -Dpath=\$PATH" --conf spark.mesos.containerizer=mesos --conf spark.mesos.executor.docker.image=skonto/spark-test:test https://...jar 1 ``` At the driver side in DC/OS: ``` /usr/lib/jvm/jre1.8.0_152/bin/java -cp /opt/spark/dist/conf/:/opt/spark/dist/jars/*:/etc/hadoop/ -Dspark.mesos.driver.frameworkId=bddf96ce-45f6-45f7-ac83-5a3622cafc41-0030-driver-20180612154509-0007 -Xmx1000M org.apache.spark.deploy.SparkSubmit --master mesos://zk://master.mesos:2181/mesos --conf spark.driver.memory=1000M --conf spark.driver.cores=1.0 --conf spark.mesos.executor.docker.image=skonto/spark-test:test --conf spark.app.name=test --conf spark.mesos.containerizer=mesos --conf spark.executor.extraJavaOptions=-Dparam1="value 1" -Dparam2=value\ 2 -Dpath=$PATH --conf spark.driver.supervise=false --class org.apache.spark.examples.SparkPi --name test --driver-cores 1.0 /mnt/mesos/sandbox/jar 1 ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20641 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user krcz commented on the issue: https://github.com/apache/spark/pull/20641 @susanxhuynh Thanks for the example! I have added similar test, covering more cases. After you or someone else reviews it, I'll rebase the pull request. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user susanxhuynh commented on the issue: https://github.com/apache/spark/pull/20641 Thanks for the PR! It seems that the previous attempt to fix this (SPARK-18114) was wrong -- I'm not sure why we didn't catch the problem before, maybe lack of testing? @krcz My suggestion for this patch is to add a test, in order to prevent another regression in the future. I've written a unit test for this -- you could do something similar: https://github.com/mesosphere/spark/commit/4812ba3d10264f6d22ec654fa16b5810d70c27a9 I will also do more testing with my own integration tests. cc @skonto --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20641 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20641: [SPARK-23464][MESOS] Fix mesos cluster scheduler options...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20641 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
