[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165061942 I'm not sure it's that. It's that `SPARK_HOME` is copied by the mesos infrastructure to other machines. I think this in inherently local. Still, not-exporting `SPARK_HOME

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165050181 Ahhh, I see. The problem is that we expose `SPARK_HOME` in all the shell scripts. --- If your project is set up for it, you can reply to this email and have your repl

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread skonto
Github user skonto commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165048947 Its because Mesos Dispatcher passed that env variable to the driver at line: builder.setEnvironment(envBuilder.build()) In 1.6 the spark-submit script has chan

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165046374 Yes, agree with @skonto. Why still honor SPARK_HOME, a little strange. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread skonto
Github user skonto commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165046198 If you set executorUri.isDefined then it wont reach (this is what we have as well): val executorSparkHome = conf.getOption("spark.mesos.executor.home")

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165044532 I admit it is very surprising. Here's what I see Driver page on the Mesos dispatcher (note that `spark.executor.uri` is set, but also `SPARK_HOME` -- which is wro

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165043119 Anyway I think we should not export `SPARK_HOME` in all the scripts, what do you think @srowen ? --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165038060 It is a little weird. I assume if `spark.mesos.executor.home` is correctly set, from the code Mesos should be able to find the right path of executables: ```

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165034309 @jerryshao Yes, it is. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165034162 I see, can you please confirm Spark is installed in this path `spark.mesos.executor.home`? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165033648 @jerryshao I observed the same failure using `spark.mesos.executor.home` --- If your project is set up for it, you can reply to this email and have your reply appear on G

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165030984 I believe I tried both.. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have thi

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165030861 One more question @dragos , did you specify `spark.executor.uri` or `SPARK_EXECUTOR_URI`? --- If your project is set up for it, you can reply to this email and have y

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165029743 I'll check that. I don't have Spark installed on every slave node in my test setup. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-16 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165026987 @dragos , if `spark.mesos.executor.home` is specified to the directory where Spark is installed in the slave node, does your problem still exist? Also from my understa

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-15 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165025882 Ohh, I see. That possibly will be the problem. One fix is to remove `export` in all the scripts, only using `SPARK_HOME` as a shell variable. --- If your project is s

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-15 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165025023 @jerryshao though the scripts will now _set_ `SPARK_HOME` for you if not already set. Should the Mesos code ship this variable across machines though? I'd have thoug

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-15 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165024672 @dragos Did you specify `SPARH_HOME` manually in your environment? By default no one will specify this `SPARK_HOME`. --- If your project is set up for it, you can re

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-15 Thread dragos
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165024221 The MesosDispatchScheduler ships a number of environment variables. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-15 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-165023375 It's a good lead; I'm looking too. The effect of this change should be to respect `SPARK_HOME` if it's set to find Spark scripts and files. It's not clear why this would

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-15 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-164964677 Thanks @marmbrus for your notice, I will take a look at the issues. --- If your project is set up for it, you can reply to this email and have your reply appear on Git

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-12-15 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-164916308 It has been pointed out that this commit might be the cause of a regression that is possibly holding up the 1.6 release (https://issues.apache.org/jira/browse/SPARK-123

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-04 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8669 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-04 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-153681509 LGTM. There's a lot going on here but it's all really standardizing on one approach to defining the Spark base directory. I scanned all the lines and it looks correct and

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-04 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43865494 --- Diff: sbin/spark-config.sh --- @@ -19,21 +19,12 @@ # should not be executable directly # also should not be passed any arguments, since we need

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-04 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43865181 --- Diff: sbin/spark-config.sh --- @@ -19,21 +19,12 @@ # should not be executable directly # also should not be passed any arguments, since we need or

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-153572262 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-153572261 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-153572150 **[Test build #44988 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44988/consoleFull)** for PR 8669 at commit [`16ea884`](https://git

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-153548710 **[Test build #44988 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44988/consoleFull)** for PR 8669 at commit [`16ea884`](https://gith

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-153547820 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-153547836 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-03 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43827693 --- Diff: sbin/start-slaves.sh --- @@ -52,11 +51,11 @@ if [ "$SPARK_MASTER_IP" = "" ]; then fi if [ "$START_TACHYON" == "true" ]; then -

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-11-03 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43784748 --- Diff: sbin/start-slaves.sh --- @@ -52,11 +51,11 @@ if [ "$SPARK_MASTER_IP" = "" ]; then fi if [ "$START_TACHYON" == "true" ]; then - "$

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-29 Thread patrungel
Github user patrungel commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43403265 --- Diff: bin/run-example --- @@ -44,7 +46,7 @@ JAR_COUNT=0 for f in "${JAR_PATH}"/spark-examples-*hadoop*.jar; do if [[ ! -e "$f" ]]; then

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-29 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43388323 --- Diff: sbin/start-slaves.sh --- @@ -52,11 +51,11 @@ if [ "$SPARK_MASTER_IP" = "" ]; then fi if [ "$START_TACHYON" == "true" ]; then - "$

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151718776 **[Test build #44483 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44483/consoleFull)** for PR 8669 at commit [`5795800`](https://git

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151718830 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151718832 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151696127 **[Test build #44483 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44483/consoleFull)** for PR 8669 at commit [`5795800`](https://gith

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151694896 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151694884 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43208324 --- Diff: bin/run-example --- @@ -44,7 +46,7 @@ JAR_COUNT=0 for f in "${JAR_PATH}"/spark-examples-*hadoop*.jar; do if [[ ! -e "$f" ]]; then

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151554933 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151554936 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151554578 **[Test build #44431 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44431/consoleFull)** for PR 8669 at commit [`28fd58d`](https://git

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread patrungel
Github user patrungel commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43127050 --- Diff: bin/run-example --- @@ -44,7 +46,7 @@ JAR_COUNT=0 for f in "${JAR_PATH}"/spark-examples-*hadoop*.jar; do if [[ ! -e "$f" ]]; then

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151493025 **[Test build #44431 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44431/consoleFull)** for PR 8669 at commit [`28fd58d`](https://gith

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151491326 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151491352 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151490838 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151490254 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151490169 **[Test build #44423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44423/consoleFull)** for PR 8669 at commit [`28fd58d`](https://git

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151490255 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151452749 **[Test build #44423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44423/consoleFull)** for PR 8669 at commit [`28fd58d`](https://gith

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151450453 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151450526 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151405505 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151405504 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151405232 **[Test build #44408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44408/consoleFull)** for PR 8669 at commit [`41511b4`](https://git

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43088776 --- Diff: bin/beeline --- @@ -23,8 +23,10 @@ # Enter posix mode for bash set -o posix -# Figure out where Spark is installed -FWDIR="

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r43088465 --- Diff: bin/beeline --- @@ -23,8 +23,10 @@ # Enter posix mode for bash set -o posix -# Figure out where Spark is installed -FWDIR="$(c

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151393309 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151393307 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151393236 **[Test build #44400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44400/consoleFull)** for PR 8669 at commit [`c796397`](https://git

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151378760 **[Test build #44408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44408/consoleFull)** for PR 8669 at commit [`41511b4`](https://gith

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151378254 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151378261 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151377956 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151370471 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151370470 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151366001 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151366067 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151363932 **[Test build #44400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44400/consoleFull)** for PR 8669 at commit [`c796397`](https://gith

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151363719 Hi @srowen @patrungel @JoshRosen and @vazin, I changed this patch to honor `SPARK_HOME` to solve symlink issue, if users want to run the executables from symlink, they

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151363503 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-26 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151363517 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-22 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-150225081 If `SPARK_HOME` is the standard way, we should update the scripts to honor this, it's not a good idea to make this issue as won't fix, because lots of deployment and p

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-22 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-150223128 Hi @srowen , I admitted currently solution is quite ugly, but like what you mentioned, it is a chicken-and-egg problem, hard to remove the duplication. Another solutio

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-22 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-150210063 Gotcha, is the way forward really to not over-write `SPARK_HOME` if set? Because most of the work here seems to be about discovering the home otherwise. The symlinks them

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-22 Thread patrungel
Github user patrungel commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-150209426 The change aims to make spark binaries launch properly when symlinked, not just to avoid setting `SPARK_HOME`. As of current master, ```SPARK_HOME``` value is overw

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-22 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-150181761 I think this is a whole lot of change, and the purpose really is just to avoid setting `SPARK_HOME` when Spark is symlinked. But that's pretty standard procedure. RIght?

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-18 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r42329927 --- Diff: bin/load-spark-env.sh --- @@ -20,7 +20,29 @@ # This script loads spark-env.sh if it exists, and ensures it is only loaded once. # spark-

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-17 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r42311628 --- Diff: bin/load-spark-env.sh --- @@ -20,7 +20,29 @@ # This script loads spark-env.sh if it exists, and ensures it is only loaded once. # spark-env

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-17 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r42311622 --- Diff: bin/load-spark-env.sh --- @@ -20,7 +20,29 @@ # This script loads spark-env.sh if it exists, and ensures it is only loaded once. # spark-env

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-16 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-148891805 @JoshRosen yes agree. See the discussion above out the small chicken-and-egg problem there about the location of the helper script. Still may be a better solution than th

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-16 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-148833502 Woah, this PR is really big! @jerryshao, do you think that you could move this function into a common "helper library" file then source / include that file into the ot

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-10-08 Thread aniro
Github user aniro commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-146579493 Can the following be fixed as well? ``` --- a/bin/load-spark-env.sh2015-10-08 14:51:20.874478368 + +++ b/bin/load-spark-env.sh2015-10-08 15:04:02.109

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-14 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r39416439 --- Diff: bin/beeline --- @@ -23,8 +23,29 @@ # Enter posix mode for bash set -o posix +realpath () { --- End diff -- Yes, th

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-14 Thread patrungel
Github user patrungel commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-140042834 Hi, @jerryshao I did a quick test with alternatives (against ```sbin/start-history-server.sh```); the fix works to me as expected. Thanks tons. --- If your proje

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-14 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r39372833 --- Diff: bin/beeline --- @@ -23,8 +23,29 @@ # Enter posix mode for bash set -o posix +realpath () { --- End diff -- I mean the

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-139184667 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-139184664 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-10 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-139184558 [Test build #42245 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42245/console) for PR 8669 at commit [`d6183b9`](https://github.

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-10 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-139135267 [Test build #42245 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42245/consoleFull) for PR 8669 at commit [`d6183b9`](https://gith

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-139134889 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-139134907 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-10 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-139134344 Hi @patrungel , I've also fixed this issue for sbin executables, mind taking a look at it? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-09 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/8669#discussion_r39120864 --- Diff: bin/beeline --- @@ -23,8 +23,29 @@ # Enter posix mode for bash set -o posix +realpath () { --- End diff -- Hi @sro

[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...

2015-09-09 Thread patrungel
Github user patrungel commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-139048552 Applying the same fix to sbin executables would be great. In fact, the whole fix would be incomplete without it. --- If your project is set up for it, you can reply t

  1   2   >