[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread zjffdu 
GitHub user zjffdu opened a pull request:

https://github.com/apache/spark/pull/13146

[SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonExec of driver and 
executor through conf…

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

Before this PR, user have to export environment variable to specify the 
python of driver & executor which is not so convenient for users. This PR is 
trying to allow user to specify python through configuration 
"--pyspark-driver-python" & "--pyspark-executor-python"

## How was this patch tested?

Manually test in local & yarn mode for pyspark-shell and pyspark batch 
mode. 

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)


…and executor through configuration

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zjffdu/spark SPARK-13081

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13146.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13146


commit 62db33040a2f3ab9bea942d88c820beabf39977f
Author: Jeff Zhang 
Date:   2016-05-17T08:07:06Z

[SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow set pythonExec of driver and 
executor through configuration




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219671444
  
**[Test build #58681 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58681/consoleFull)**
 for PR 13146 at commit 
[`62db330`](https://github.com/apache/spark/commit/62db33040a2f3ab9bea942d88c820beabf39977f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219690423
  
**[Test build #58681 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58681/consoleFull)**
 for PR 13146 at commit 
[`62db330`](https://github.com/apache/spark/commit/62db33040a2f3ab9bea942d88c820beabf39977f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219690545
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58681/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219690542
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219700414
  
**[Test build #58690 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58690/consoleFull)**
 for PR 13146 at commit 
[`abf4c2e`](https://github.com/apache/spark/commit/abf4c2eb29751a78bed9144ce67f84c9647fe775).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219732299
  
**[Test build #58690 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58690/consoleFull)**
 for PR 13146 at commit 
[`abf4c2e`](https://github.com/apache/spark/commit/abf4c2eb29751a78bed9144ce67f84c9647fe775).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219732719
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58690/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219732714
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread BryanCutler 
Github user BryanCutler commented on a diff in the pull request:

https://github.com/apache/spark/pull/13146#discussion_r63565228
  
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -62,7 +62,8 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable 
with Logging {
 
   private[spark] def loadFromSystemProperties(silent: Boolean): SparkConf 
= {
 // Load any spark.* system properties
-for ((key, value) <- Utils.getSystemProperties if 
key.startsWith("spark.")) {
+for ((key, value) <- Utils.getSystemProperties if 
key.startsWith("spark.")
+  || key.startsWith("pyspark.")) {
--- End diff --

I think you could just call your properties `spark.pyspark.driver.python` 
and `spark.pyspark.python` then not have to change these checks.  All spark 
properties should start with that anyway.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread BryanCutler 
Github user BryanCutler commented on a diff in the pull request:

https://github.com/apache/spark/pull/13146#discussion_r63565781
  
--- Diff: core/src/test/scala/org/apache/spark/SparkConfSuite.scala ---
@@ -281,6 +285,7 @@ class SparkConfSuite extends SparkFunSuite with 
LocalSparkContext with ResetSyst
 assert(conf.contains("spark.io.compression.lz4.blockSize"))
 assert(conf.contains("spark.io.unknown") === false)
   }
+
--- End diff --

nit: remove blank line


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread BryanCutler 
Github user BryanCutler commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219792598
  
@zjffdu I made some comments, however I sort of feel that the Python 
executable is an environment setting and not really a Spark property, so it 
would be better to leave as is.  That is just my opinion though, others may 
think differently.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread zjffdu 
Github user zjffdu commented on a diff in the pull request:

https://github.com/apache/spark/pull/13146#discussion_r63625742
  
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -62,7 +62,8 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable 
with Logging {
 
   private[spark] def loadFromSystemProperties(silent: Boolean): SparkConf 
= {
 // Load any spark.* system properties
-for ((key, value) <- Utils.getSystemProperties if 
key.startsWith("spark.")) {
+for ((key, value) <- Utils.getSystemProperties if 
key.startsWith("spark.")
+  || key.startsWith("pyspark.")) {
--- End diff --

yes, that way also works. Just feel "spark.pyspark.driver.python" is too 
verbose. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread zjffdu 
Github user zjffdu commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219890162
  
@BryanCutler Thanks for review. The reason I create this ticket is that it 
is not convenient for user to specify environment variable especially when user 
want to switch between different python version. And SPARK-13587 requires this 
feature. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219899338
  
**[Test build #58732 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58732/consoleFull)**
 for PR 13146 at commit 
[`305125d`](https://github.com/apache/spark/commit/305125d4fa68fc733b085cf383a5ad11f537f574).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219914744
  
**[Test build #58732 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58732/consoleFull)**
 for PR 13146 at commit 
[`305125d`](https://github.com/apache/spark/commit/305125d4fa68fc733b085cf383a5ad11f537f574).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219914888
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58732/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-17 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-219914887
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-13081][PYSPARK][SPARK_SUBMIT]. Allow se...

2016-05-30 Thread zjffdu 
Github user zjffdu commented on the pull request:

https://github.com/apache/spark/pull/13146#issuecomment-222476521
  
@davies @JoshRosen  Could any of you review this PR ? Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org