[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-21 Thread srowen 
Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-173606818
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-21 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/10491


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-20 Thread srowen 
Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-173170970
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-19 Thread srowen 
Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-172793610
  
Looks reasonable; while we're here, does `running-on-yarn.md` need to 
document `spark.driver.memory` for consistency? I see `spark.driver.cores` 
documented.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-19 Thread felixcheung 
Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-173000121
  
good idea, added spark.driver.memory


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-19 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-173014955
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-19 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-173014960
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49710/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-19 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-173014806
  
**[Test build #49710 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49710/consoleFull)**
 for PR 10491 at commit 
[`94001dd`](https://github.com/apache/spark/commit/94001ddf7eb8a366091b37efab68249a7eed043d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-19 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-173004136
  
**[Test build #49710 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49710/consoleFull)**
 for PR 10491 at commit 
[`94001dd`](https://github.com/apache/spark/commit/94001ddf7eb8a366091b37efab68249a7eed043d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-18 Thread felixcheung 
Github user felixcheung commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-172735041
  
Thank you for the detailed explanation, @srowen.
Agreed it would be valuable to have a dedicated doc page for CLI flags, 
perhaps as a bigger project later. As of now, I have moved references to config 
prop to job-scheduling.md. I hope that looks alright.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-18 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-172735969
  
**[Test build #49662 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49662/consoleFull)**
 for PR 10491 at commit 
[`6983d15`](https://github.com/apache/spark/commit/6983d15524d260ae49574b27e47221b902165f30).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-18 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-172736310
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49661/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-18 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-172736309
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-18 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-172737638
  
**[Test build #49662 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49662/consoleFull)**
 for PR 10491 at commit 
[`6983d15`](https://github.com/apache/spark/commit/6983d15524d260ae49574b27e47221b902165f30).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-18 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-172737697
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2016-01-18 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-172737699
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/49662/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-29 Thread srowen 
Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167759582
  
I guess my point is that these CLI flags are already documented in the CLI, 
as help text. 

I can see an argument for formally, separately documenting them in the 
website, and/or their relation to other Spark properties if applicable. It's 
duplicative and means keeping them in sync, but it's useful. However you'd want 
to do that for all or none, I suppose, rather than just a few. Right now I 
don't see any existing place for them. 

In bits and pieces, the flags are referenced in various parts of the docs. 
My suggestion would be to not do this at all, or else, find where similar 
arguments are already mentioned in the docs and mention these few additional 
ones there. For example `executor-cores` is mentioned a little in 
`job-scheduling.md`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-28 Thread felixcheung 
Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/10491#discussion_r48517225
  
--- Diff: docs/configuration.md ---
@@ -120,7 +120,8 @@ of the most common options to set are:
   spark.driver.cores
   1
   
-Number of cores to use for the driver process, only in cluster mode.
+Number of cores to use for the driver process, only in cluster mode. 
This can be set through
+--driver-cores command line option.
--- End diff --

I moved these to the running-on-yarn doc, would that work?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-28 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167692412
  
**[Test build #48387 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48387/consoleFull)**
 for PR 10491 at commit 
[`27c6976`](https://github.com/apache/spark/commit/27c6976cb33c8a418635a46255301b027db8615c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-28 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167697095
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-28 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167696817
  
**[Test build #48387 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48387/consoleFull)**
 for PR 10491 at commit 
[`27c6976`](https://github.com/apache/spark/commit/27c6976cb33c8a418635a46255301b027db8615c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-28 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167697099
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48387/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-28 Thread srowen 
Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/10491#discussion_r48471720
  
--- Diff: docs/configuration.md ---
@@ -120,7 +120,8 @@ of the most common options to set are:
   spark.driver.cores
   1
   
-Number of cores to use for the driver process, only in cluster mode.
+Number of cores to use for the driver process, only in cluster mode. 
This can be set through
+--driver-cores command line option.
--- End diff --

I don't think the purpose of this file is to document how the CLI works. It 
should stick to documenting the underlying properties.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread felixcheung 
GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/10491

[SPARK-12534][DOC] update documentation to list command line equivalent to 
properties

Several Spark properties equivalent to Spark submit command line options 
are missing.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark sparksubmitdoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/10491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #10491


commit 0fb3a40d8bc6d9186328348881a0bd0a28895124
Author: felixcheung 
Date:   2015-12-28T03:21:42Z

update doc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167473135
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48358/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread AmplabJenkins 
Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167473131
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167472857
  
**[Test build #48358 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48358/consoleFull)**
 for PR 10491 at commit 
[`0fb3a40`](https://github.com/apache/spark/commit/0fb3a40d8bc6d9186328348881a0bd0a28895124).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12534][DOC] update documentation to lis...

2015-12-27 Thread SparkQA 
Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/10491#issuecomment-167469480
  
**[Test build #48358 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48358/consoleFull)**
 for PR 10491 at commit 
[`0fb3a40`](https://github.com/apache/spark/commit/0fb3a40d8bc6d9186328348881a0bd0a28895124).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org