[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145527571 Ah, I thought you had reviewed this and the previous iteration of this PR also? seemed pretty cut and dried to me at this stage. I don't know how long to wait for your explicit ack. In this case, being just docs, the downside of committing early is small -- you can always follow on with another commit if you really had something to add. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145525454 @srowen , what happened here, why was this closed, it looks like it was committed but I don't see anyone +1. I was mostly good with it but it would have been nice to look at the final version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8968 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145186626 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43199/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145186625 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145186579 [Test build #43199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43199/console) for PR 8968 at commit [`e8db144`](https://github.com/apache/spark/commit/e8db144ef75a702f411e7005ba68d890ecc155b2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145160361 [Test build #43199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43199/consoleFull) for PR 8968 at commit [`e8db144`](https://github.com/apache/spark/commit/e8db144ef75a702f411e7005ba68d890ecc155b2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145159254 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145159238 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/8968#discussion_r41068987 --- Diff: docs/submitting-applications.md --- @@ -122,21 +123,23 @@ The master URL passed to Spark can be in one of the following formats: Master URLMeaning - local Run Spark locally with one worker thread (i.e. no parallelism at all). - local[K] Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine). - local[*] Run Spark locally with as many worker threads as logical cores on your machine. - spark://HOST:PORT Connect to the given Spark standalone + local Run Spark locally with one worker thread (i.e. no parallelism at all). + local[K] Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine). + local[*] Run Spark locally with as many worker threads as logical cores on your machine. + spark://HOST:PORT Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default. - mesos://HOST:PORT Connect to the given Mesos cluster. + mesos://HOST:PORT Connect to the given Mesos cluster. The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use mesos://zk:// - yarn-client Connect to a YARN cluster in -client mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. + yarn Connect to a YARN cluster in +client or cluster mode depending on the value of --deploy-mode. +The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. - yarn-cluster Connect to a YARN cluster in -cluster mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. + yarn-client Equivalent to yarn with --deploy-mode client --- End diff -- Although deprecated feels a little strong ... yeah maybe best to sell it that way. I'd feel funny about completely omitting them as they have been used widely to date and still fully work. Oops yes missed an occurrence of yarn-client, will fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145151608 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43196/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145151605 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145151567 [Test build #43196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43196/console) for PR 8968 at commit [`47d9e8a`](https://github.com/apache/spark/commit/47d9e8a5bd5984ebe9582285e0412640d9dc9858). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145147435 [Test build #43196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43196/consoleFull) for PR 8968 at commit [`47d9e8a`](https://github.com/apache/spark/commit/47d9e8a5bd5984ebe9582285e0412640d9dc9858). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user nssalian commented on a diff in the pull request: https://github.com/apache/spark/pull/8968#discussion_r41064621 --- Diff: docs/submitting-applications.md --- @@ -122,21 +123,23 @@ The master URL passed to Spark can be in one of the following formats: Master URLMeaning - local Run Spark locally with one worker thread (i.e. no parallelism at all). - local[K] Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine). - local[*] Run Spark locally with as many worker threads as logical cores on your machine. - spark://HOST:PORT Connect to the given Spark standalone + local Run Spark locally with one worker thread (i.e. no parallelism at all). + local[K] Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine). + local[*] Run Spark locally with as many worker threads as logical cores on your machine. + spark://HOST:PORT Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default. - mesos://HOST:PORT Connect to the given Mesos cluster. + mesos://HOST:PORT Connect to the given Mesos cluster. The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use mesos://zk:// - yarn-client Connect to a YARN cluster in -client mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. + yarn Connect to a YARN cluster in +client or cluster mode depending on the value of --deploy-mode. +The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. - yarn-cluster Connect to a YARN cluster in -cluster mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. + yarn-client Equivalent to yarn with --deploy-mode client --- End diff -- Shouldn't deploy-mode come first here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/8968#discussion_r41064442 --- Diff: docs/submitting-applications.md --- @@ -122,21 +123,23 @@ The master URL passed to Spark can be in one of the following formats: Master URLMeaning - local Run Spark locally with one worker thread (i.e. no parallelism at all). - local[K] Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine). - local[*] Run Spark locally with as many worker threads as logical cores on your machine. - spark://HOST:PORT Connect to the given Spark standalone + local Run Spark locally with one worker thread (i.e. no parallelism at all). + local[K] Run Spark locally with K worker threads (ideally, set this to the number of cores on your machine). + local[*] Run Spark locally with as many worker threads as logical cores on your machine. + spark://HOST:PORT Connect to the given Spark standalone cluster master. The port must be whichever one your master is configured to use, which is 7077 by default. - mesos://HOST:PORT Connect to the given Mesos cluster. + mesos://HOST:PORT Connect to the given Mesos cluster. The port must be whichever one your is configured to use, which is 5050 by default. Or, for a Mesos cluster using ZooKeeper, use mesos://zk:// - yarn-client Connect to a YARN cluster in -client mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. + yarn Connect to a YARN cluster in +client or cluster mode depending on the value of --deploy-mode. +The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. - yarn-cluster Connect to a YARN cluster in -cluster mode. The cluster location will be found based on the HADOOP_CONF_DIR or YARN_CONF_DIR variable. + yarn-client Equivalent to yarn with --deploy-mode client --- End diff -- should we either leave these off or perhaps put deprecated by them? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/8968#discussion_r41064265 --- Diff: docs/running-on-yarn.md --- @@ -388,6 +390,6 @@ If you need a reference to the proper location to put log files in the YARN so t # Important notes - Whether core requests are honored in scheduling decisions depends on which scheduler is in use and how it is configured. -- In `yarn-cluster` mode, the local directories used by the Spark executors and the Spark driver will be the local directories configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies `spark.local.dir`, it will be ignored. In `yarn-client` mode, the Spark executors will use the local directories configured for YARN while the Spark driver will use those defined in `spark.local.dir`. This is because the Spark driver does not run on the YARN cluster in `yarn-client` mode, only the Spark executors do. +- In `cluster` mode, the local directories used by the Spark executors and the Spark driver will be the local directories configured for YARN (Hadoop YARN config `yarn.nodemanager.local-dirs`). If the user specifies `spark.local.dir`, it will be ignored. In `yarn-client` mode, the Spark executors will use the local directories configured for YARN while the Spark driver will use those defined in `spark.local.dir`. This is because the Spark driver does not run on the YARN cluster in `yarn-client` mode, only the Spark executors do. --- End diff -- still using yarn-client here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145146090 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8968#issuecomment-145146077 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...
GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/8968
[SPARK-9570] [DOCS] Consistent recommendation for submitting spark apps to
YARN, -master yarn --deploy-mode x vs -master yarn-x'.
Recommend `--master yarn --deploy-mode {cluster,client}` consistently in
docs.
Follow-on to https://github.com/apache/spark/pull/8385
CC @nssalian
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-9570
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8968.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8968
commit 47d9e8a5bd5984ebe9582285e0412640d9dc9858
Author: Sean Owen
Date: 2015-10-02T20:14:15Z
Recommend --master yarn --deploy-mode {cluster,client} consistently in docs
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
