[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-65767093 @tgravescs @andrewor14 do you feel comfortable merging this now that 1.2 is out the door? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-65767819 [Test build #24180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24180/consoleFull) for PR 3215 at commit [`1c5ac08`](https://github.com/apache/spark/commit/1c5ac0889f387a27650b7f4dd37bb315f96dd201). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-65768623 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24179/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-65776868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24180/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-65776860 [Test build #24180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24180/consoleFull) for PR 3215 at commit [`1c5ac08`](https://github.com/apache/spark/commit/1c5ac0889f387a27650b7f4dd37bb315f96dd201). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-65342878 [Test build #24069 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24069/consoleFull) for PR 3215 at commit [`3128915`](https://github.com/apache/spark/commit/31289152db53522fbd4c0b2e50767acb996699f5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-65350500 [Test build #24069 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24069/consoleFull) for PR 3215 at commit [`3128915`](https://github.com/apache/spark/commit/31289152db53522fbd4c0b2e50767acb996699f5). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-65350508 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24069/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-64164593 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23774/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-64164590 [Test build #23774 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23774/consoleFull) for PR 3215 at commit [`7175ae8`](https://github.com/apache/spark/commit/7175ae8c4680ff9effb07326507d42d75059929f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-64157741 [Test build #23774 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23774/consoleFull) for PR 3215 at commit [`7175ae8`](https://github.com/apache/spark/commit/7175ae8c4680ff9effb07326507d42d75059929f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user ksakellis commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-64061471 LGTM - can't wait for this to merge in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-63275953 The patch does remove the yarn/stable directory. Updated patch includes the doc fix. Currently testing it on a pseudo-distributed cluster. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-63276099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23469/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-63276093 [Test build #23469 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23469/consoleFull) for PR 3215 at commit [`8fbf896`](https://github.com/apache/spark/commit/8fbf8965e89068d7a70c42b069c2dad644703dd0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3215#discussion_r20439527 --- Diff: docs/building-spark.md --- @@ -95,8 +74,11 @@ mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean package # Apache Hadoop 2.4.X mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package +# Apache Hadoop 2.5.0 and greater +mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=YOUR_VERSION -DskipTests clean package --- End diff -- Thanks for adding this. Since someone else had asked for this on the mailing list. I'm wondering if we shouldn't pull this few line doc changes into the 1.2 release also, thoughts? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-63339779 Successfully ran spark-shell in yarn-client mode and an app in yarn-cluster mode. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/3215#discussion_r20454469 --- Diff: docs/building-spark.md --- @@ -95,8 +74,11 @@ mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean package # Apache Hadoop 2.4.X mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package +# Apache Hadoop 2.5.0 and greater +mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=YOUR_VERSION -DskipTests clean package --- End diff -- That seems reasonable to me. Are you able to just push that change to branch 1.2 or should I submit a PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3215#discussion_r20454840 --- Diff: docs/building-spark.md --- @@ -95,8 +74,11 @@ mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean package # Apache Hadoop 2.4.X mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package +# Apache Hadoop 2.5.0 and greater +mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=YOUR_VERSION -DskipTests clean package --- End diff -- a separate PR would be best. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/3215#discussion_r20459377 --- Diff: docs/building-spark.md --- @@ -95,8 +74,11 @@ mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean package # Apache Hadoop 2.4.X mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package +# Apache Hadoop 2.5.0 and greater +mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=YOUR_VERSION -DskipTests clean package --- End diff -- https://github.com/apache/spark/pull/3322 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-63269364 [Test build #23469 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23469/consoleFull) for PR 3215 at commit [`8fbf896`](https://github.com/apache/spark/commit/8fbf8965e89068d7a70c42b069c2dad644703dd0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3215#discussion_r20363809 --- Diff: docs/building-spark.md --- @@ -96,7 +75,7 @@ mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean package mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package --- End diff -- I know you didn't change this here (so we can file separate jira for it if you want) but perhaps we should add example and some text here about hadoop 2.5 and . Since there is no hadoop profile 2.5 and the user should use the 2.4 profile. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-63078085 was the yarn/stable directory removed with this? I couldn't tell from applying the patch and not sure how directory removals show up here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62763548 I have a whole set of simplifying changes that I want to go in (e.g. YARN-1714), but thought it would probably be good to break things up a bit for easier review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62769738 +1; cleanups can come later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62792878 @tgravescs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62793237 [Test build #23279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23279/consoleFull) for PR 3215 at commit [`8427a9b`](https://github.com/apache/spark/commit/8427a9b507a99bfd849b880963983598e138dcc6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/3215#discussion_r20251642
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
@@ -178,21 +178,25 @@ private[spark] class ClientArguments(args:
Array[String], sparkConf: SparkConf)
private def getUsageMessage(unknownParam: List[String] = null): String =
{
val message = if (unknownParam != null) sUnknown/unsupported param
$unknownParam\n else
-message +
- Usage: org.apache.spark.deploy.yarn.Client [options] \n +
- Options:\n +
---jar JAR_PATH Path to your application's JAR file
(required in yarn-cluster mode)\n +
---class CLASS_NAME Name of your application's main class
(required)\n +
---arg ARG Argument to be passed to your
application's main class.\n +
- Multiple invocations are possible,
each will be passed in order.\n +
---num-executors NUMNumber of executors to start (Default:
2)\n +
---executor-cores NUM Number of cores for the executors
(Default: 1).\n +
---driver-memory MEMMemory for driver (e.g. 1000M, 2G)
(Default: 512 Mb)\n +
---executor-memory MEM Memory per executor (e.g. 1000M, 2G)
(Default: 1G)\n +
---name NAMEThe name of your application (Default:
Spark)\n +
---queue QUEUE The hadoop queue to use for allocation
requests (Default: 'default')\n +
---addJars jars Comma separated list of local jars
that want SparkContext.addJar to work with.\n +
---files files Comma separated list of files to be
distributed with the job.\n +
---archives archivesComma separated list of archives to be
distributed with the job.
+message +
+ |Usage: org.apache.spark.deploy.yarn.Client [options]
+ |Options:
+ | --jar JAR_PATH Path to your application's JAR file
(required in yarn-cluster
+ | mode)
+ | --class CLASS_NAME Name of your application's main class
(required)\n +
--- End diff --
is \n still needed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/3215#discussion_r20251737
--- Diff:
yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala ---
@@ -178,21 +178,25 @@ private[spark] class ClientArguments(args:
Array[String], sparkConf: SparkConf)
private def getUsageMessage(unknownParam: List[String] = null): String =
{
val message = if (unknownParam != null) sUnknown/unsupported param
$unknownParam\n else
-message +
- Usage: org.apache.spark.deploy.yarn.Client [options] \n +
- Options:\n +
---jar JAR_PATH Path to your application's JAR file
(required in yarn-cluster mode)\n +
---class CLASS_NAME Name of your application's main class
(required)\n +
---arg ARG Argument to be passed to your
application's main class.\n +
- Multiple invocations are possible,
each will be passed in order.\n +
---num-executors NUMNumber of executors to start (Default:
2)\n +
---executor-cores NUM Number of cores for the executors
(Default: 1).\n +
---driver-memory MEMMemory for driver (e.g. 1000M, 2G)
(Default: 512 Mb)\n +
---executor-memory MEM Memory per executor (e.g. 1000M, 2G)
(Default: 1G)\n +
---name NAMEThe name of your application (Default:
Spark)\n +
---queue QUEUE The hadoop queue to use for allocation
requests (Default: 'default')\n +
---addJars jars Comma separated list of local jars
that want SparkContext.addJar to work with.\n +
---files files Comma separated list of files to be
distributed with the job.\n +
---archives archivesComma separated list of archives to be
distributed with the job.
+message +
+ |Usage: org.apache.spark.deploy.yarn.Client [options]
+ |Options:
+ | --jar JAR_PATH Path to your application's JAR file
(required in yarn-cluster
+ | mode)
+ | --class CLASS_NAME Name of your application's main class
(required)\n +
--- End diff --
Oops sorry missed that one
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62794341 If we don't pull this in right away (wait til closer to 1.2 release) is it going to delay your other prs? It would be nice to hold off a bit to pull this in if not. I took a quick look at the pr, we need to update docs/building-spark.md. I still need to download this and try it out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62800213 +1. I would imagine this opens up a world of opportunities to cleanup, which can come later in separate PRs. In particular we relied on a lot of abstractions that were necessary to avoid duplicate code, but these are no longer needed and it would simplify the code significantly if we get rid of them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62806914 [Test build #23279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23279/consoleFull) for PR 3215 at commit [`8427a9b`](https://github.com/apache/spark/commit/8427a9b507a99bfd849b880963983598e138dcc6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62806929 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23279/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62819777 [Test build #23290 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23290/consoleFull) for PR 3215 at commit [`66e731f`](https://github.com/apache/spark/commit/66e731fba464bbbd0e5fd0f1ff022289d5f85f13). * This patch **does not merge cleanly**. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62822018 @tgravescs it would delay other PRs, but not a huge deal if you think it's too soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62822213 [Test build #23293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23293/consoleFull) for PR 3215 at commit [`ec3d8da`](https://github.com/apache/spark/commit/ec3d8dac9ac1fc544505cf8acf8c152f154993b2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62827744 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23290/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62827739 [Test build #23290 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23290/consoleFull) for PR 3215 at commit [`66e731f`](https://github.com/apache/spark/commit/66e731fba464bbbd0e5fd0f1ff022289d5f85f13). * This patch **passes all tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62829384 [Test build #23293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23293/consoleFull) for PR 3215 at commit [`ec3d8da`](https://github.com/apache/spark/commit/ec3d8dac9ac1fc544505cf8acf8c152f154993b2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62829393 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23293/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
GitHub user sryza opened a pull request: https://github.com/apache/spark/pull/3215 SPARK-4338. Ditch yarn-alpha. Sorry if this is a little premature with 1.2 still not out the door, but it will make other work like SPARK-4136 and SPARK-2089 a lot easier. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sryza/spark sandy-spark-4338 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3215.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3215 commit 52a4a75319e9cb8ae7a06f6396d56f53a22b8017 Author: Sandy Ryza sa...@cloudera.com Date: 2014-11-11T09:38:45Z SPARK-4338. Ditch yarn-alpha. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62653364 [Test build #23234 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23234/consoleFull) for PR 3215 at commit [`b0bbb13`](https://github.com/apache/spark/commit/b0bbb13a16f0f023eb1dca72ebd1e64892ec1a1f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62653482 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23234/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62653479 [Test build #23234 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23234/consoleFull) for PR 3215 at commit [`b0bbb13`](https://github.com/apache/spark/commit/b0bbb13a16f0f023eb1dca72ebd1e64892ec1a1f). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62662749 [Test build #23242 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23242/consoleFull) for PR 3215 at commit [`f8c90f3`](https://github.com/apache/spark/commit/f8c90f3249f6797634e18a0c4d1153208a011920). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62668976 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23242/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62668973 [Test build #23242 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23242/consoleFull) for PR 3215 at commit [`f8c90f3`](https://github.com/apache/spark/commit/f8c90f3249f6797634e18a0c4d1153208a011920). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4338. Ditch yarn-alpha.
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3215#issuecomment-62670914 Yes, if it's targeted for 1.3, it would be nice to get this in not long after 1.2. I think there's more that simplifies, like some reflection-related code to bridge the gap between the two versions. That's worth a review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
