[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user shijinkui commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r46532508
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ // The setApplicationTags method was only introduced in Hadoop
2.4+, so we need to use
+ // reflection to set it, printing a warning if a tag was
specified but the YARN version
+ // doesn't support it.
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+} catch {
+ case e: NoSuchMethodException =>
+logWarning(s"Ignoring $CONF_SPARK_YARN_APPLICATION_TAGS
because this version of " +
+ "YARN does not support it")
+}
+ }
--- End diff --
using reflect make the code reading more difficult. generally not be
recommanded.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8072 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-132321367 Thanks everyone for the reviews! @sryza it sounds like after further clarification with @vanzin we're okay with the code as-is. Is there anything else I should look at, or are you able to merge this in the current state? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r37324265
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
--- End diff --
Ah, I see. `setApplicationTags` takes a `Set[String]`, not a single string,
so you're not calling it once per tag in the collection. My bad. Current code
is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r37323794
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
--- End diff --
Thanks for the suggestion! I'm a bit rusty in Scala, so I could be
misapplying some idioms here, but my intent on this line was to use the
foreach, etc., only to access the "None or single element" of the Option, as
indicated on http://www.scala-lang.org/api/current/index.html#scala.Option , so
the "foreach" here is supposed to only iterate over the "single" element which
happens to itself be a collection, as opposed to iterating over elements of the
inner collection. So there shouldn't be any way to cause multiple log
statements or multiple reflection-based lookups of the method.
I also wanted to err on the side of minimizing behavioral changes for
existing setups, so that if the Option is None, then this chaining as a monad
avoids ever needing to lookup a method, or even invoking any of the
option-processing methods like StringUtils.getTrimmedStringCollection.
I could add a note to make it more clear that the map/filter/foreach is on
the Option, as opposed to the Collection if that'd help.
Anyhow, I'll be happy to apply the reversal to start with the method as you
suggest if you prefer, just wanted to hear your thoughts on this Option usage
first.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r37316986
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
--- End diff --
BTW if you do this you're probably better off doing:
Try(...) match {
case Success =>
case Failure =>
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-132218054 Cool, in that case @dennishuo mind making the change that @vanzin suggested and then I'll merge this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-132217285 thats fine. we can keep it tags and if another config with tags in it comes along be more specific on that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-132214961 @tgravescs my thinking for just spark.yarn.tags was that it's redundant to include application, because configs are by definition per-application. We'd also be consistent with spark.yarn.queue and spark.yarn.jars. If we do include "application", I think it should be spark.yarn.application-tags, because then we're not adding a new "application" namespace. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-132213070 I think the name is ok, but it might be better if we have put application in it: spark.yarn.application.tags. thoughts? My thinking is that way if in the future some other config with tags in the name of it comes along it won't be confusing. It matches the setApplicationTags name too. If both of you prefer the current name I'm ok with it though. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-131904066 Config name LGTM. Code too, just left a minor suggestion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r37214306
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
--- End diff --
Not a big deal, but I'd flip this around and do something like:
Try(appContext.getClass().getMethod("setApplicationTags",
classOf[java.util.Set[String]])).foreach { setTagsMethod =>
...
}
That way you print the log message only once, lookup the method only once,
and skip all the processing of the option if the method is not available.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-131679667 @tgravescs @vanzin @andrewor14 can I get one of you to sign off on my proposed property name `spark.yarn.tags`? Otherwise, this LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36892350
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ // The setApplicationTags method was only introduced in Hadoop
2.4+, so we need to use
+ // reflection to set it, printing a warning if a tag was
specified but the YARN version
+ // doesn't support it.
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+} catch {
+ case e: NoSuchMethodException =>
+logWarning("Ignoring %s='%s' because this version of YARN does
not support it"
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36833771
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ // The setApplicationTags method was only introduced in Hadoop
2.4+, so we need to use
+ // reflection to set it, printing a warning if a tag was
specified but the YARN version
+ // doesn't support it.
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+} catch {
+ case e: NoSuchMethodException =>
+logWarning("Ignoring %s='%s' because this version of YARN does
not support it"
--- End diff --
Nit: use `s"Ignoring $CONF_SPARK_YARN_APPLICATION_TAGS because"`...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-130196650 jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36814288
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala ---
@@ -170,6 +173,39 @@ class ClientSuite extends SparkFunSuite with Matchers
with BeforeAndAfterAll {
cp should contain ("/remotePath/my1.jar")
}
+ test("configuration and args propagate through
createApplicationSubmissionContext") {
--- End diff --
Console output from my test runs here:
https://gist.github.com/dennishuo/79212ba4f8a3bfb71227
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36814036
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala ---
@@ -170,6 +173,39 @@ class ClientSuite extends SparkFunSuite with Matchers
with BeforeAndAfterAll {
cp should contain ("/remotePath/my1.jar")
}
+ test("configuration and args propagate through
createApplicationSubmissionContext") {
--- End diff --
Tests were green with both a 2.3 and 2.4 build. Thanks for the help
reviewing!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36709395
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala ---
@@ -170,6 +173,39 @@ class ClientSuite extends SparkFunSuite with Matchers
with BeforeAndAfterAll {
cp should contain ("/remotePath/my1.jar")
}
+ test("configuration and args propagate through
createApplicationSubmissionContext") {
--- End diff --
Indeed I did before sending out the first PR:
$ sbt/sbt -Pyarn -Phadoop-2.3 -Phive -Phive-thriftserver yarn/test
test-only
...
[info] Compiling 1 Scala source to
/home/dhuo/github/dhuo_spark2/spark/yarn/target/scala-2.10/test-classes...
[info] YarnAllocatorSuite:
[info] - single container allocated (667 milliseconds)
[info] - some containers allocated (221 milliseconds)
[info] - receive more containers than requested (200 milliseconds)
[info] - decrease total requested executors (17 milliseconds)
[info] - decrease total requested executors to less than currently running
(13 milliseconds)
[info] - kill executors (25 milliseconds)
[info] - lost executor removed from backend (26 milliseconds)
[info] - memory exceeded diagnostic regexes (5 milliseconds)
[info] YarnClusterSuite:
[info] - run Spark in yarn-client mode (14 seconds, 642 milliseconds)
[info] - run Spark in yarn-cluster mode (13 seconds, 292 milliseconds)
[info] - run Spark in yarn-cluster mode unsuccessfully (10 seconds, 150
milliseconds)
[info] - run Python application in yarn-client mode (16 seconds, 129
milliseconds)
[info] - run Python application in yarn-cluster mode (17 seconds, 323
milliseconds)
[info] - user class path first in client mode (13 seconds, 628 milliseconds)
[info] - user class path first in cluster mode (13 seconds, 120
milliseconds)
[info] ContainerPlacementStrategySuite:
[info] - allocate locality preferred containers with enough resource and no
matched existed containers (838 milliseconds)
[info] - allocate locality preferred containers with enough resource and
partially matched containers (21 milliseconds)
[info] - allocate locality preferred containers with limited resource and
partially matched containers (16 milliseconds)
[info] - allocate locality preferred containers with fully matched
containers (14 milliseconds)
[info] - allocate containers with no locality preference (16 milliseconds)
[info] YarnSparkHadoopUtilSuite:
[info] - shell script escaping (17 milliseconds)
[info] - Yarn configuration override (53 milliseconds)
[info] - test getApplicationAclsForYarn acls on (10 milliseconds)
[info] - test getApplicationAclsForYarn acls on and specify users (12
milliseconds)
[info] - test expandEnvironment result (4 milliseconds)
[info] - test getClassPathSeparator result (6 milliseconds)
[info] - check access nns empty (20 milliseconds)
[info] - check access nns unset (18 milliseconds)
[info] - check access nns (16 milliseconds)
[info] - check access nns space (17 milliseconds)
[info] - check access two nns (16 milliseconds)
[info] - check token renewer (206 milliseconds)
[info] - check token renewer default (32 milliseconds)
[info] ClientSuite:
[info] - default Yarn application classpath (29 milliseconds)
[info] - default MR application classpath (3 milliseconds)
[info] - resultant classpath for an application that defines a classpath
for YARN (16 milliseconds)
[info] - resultant classpath for an application that defines a classpath
for MR (11 milliseconds)
[info] - resultant classpath for an application that defines both
classpaths, YARN and MR (9 milliseconds)
[info] - Local jar URIs (40 milliseconds)
[info] - Jar path propagation through SparkConf (115 milliseconds)
[info] - Cluster path translation (14 milliseconds)
[info] - configuration and args propagate through
createApplicationSubmissionContext (8 milliseconds)
[info] ClientDistributedCacheManagerSuite:
[info] - test getFileStatus empty (24 milliseconds)
[info] - test getFileStatus cached (1 millisecond)
[info] - test addResource (10 milliseconds)
[info] - test addResource link null (3 milliseconds)
[info] - test addResource appmaster only (1 millisecond)
[info] - test addResource archive (7 milliseconds)
[info] ScalaTest
[info] Run completed in 2 minutes.
[info] Total number of tests run: 48
[info] Suites: completed 6, aborted 0
[info] Tests: succeeded 48, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Passed: Total 48, Failed 0, Errors 0, Passed 48
[success] Total time: 136 s, completed Aug 7, 2015 6:47:43 PM
$ sbt/sbt -Pyarn -Phadoop-2.4 -Phive -Phive-thriftserver yarn/test
...
[info] YarnAllocatorSuite:
[info] - single container al
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36709333
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+ logInfo("Applied setApplicationTags based on %s='%s'"
+.format(CONF_SPARK_YARN_APPLICATION_TAGS, tagCollection))
+
+} catch {
+ case e: NoSuchMethodException =>
+logWarning("Ignoring conf %s='%s'; setApplicationTags missing
in this version of YARN."
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request: https://github.com/apache/spark/pull/8072#discussion_r36709324 --- Diff: docs/running-on-yarn.md --- @@ -320,6 +320,14 @@ If you need a reference to the proper location to put log files in the YARN so t + spark.yarn.application.tags --- End diff -- Done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36709328
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+ logInfo("Applied setApplicationTags based on %s='%s'"
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36709327
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+ logInfo("Applied setApplicationTags based on %s='%s'"
+.format(CONF_SPARK_YARN_APPLICATION_TAGS, tagCollection))
+
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36709312
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ val method = appContext.getClass().getMethod(
--- End diff --
Done.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user dennishuo commented on a diff in the pull request: https://github.com/apache/spark/pull/8072#discussion_r36709305 --- Diff: docs/running-on-yarn.md --- @@ -320,6 +320,14 @@ If you need a reference to the proper location to put log files in the YARN so t + spark.yarn.application.tags + (none) + + Comma-separated list of strings to pass through as YARN application tags appearing + in YARN ApplicationReports, which can be used for filtering when querying YARN. --- End diff -- Done. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36709001
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+ logInfo("Applied setApplicationTags based on %s='%s'"
+.format(CONF_SPARK_YARN_APPLICATION_TAGS, tagCollection))
+
+} catch {
+ case e: NoSuchMethodException =>
+logWarning("Ignoring conf %s='%s'; setApplicationTags missing
in this version of YARN."
--- End diff --
I'd say "Ignoring spark.yarn.tags because this version of YARN does not
support it"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36709027
--- Diff:
yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala ---
@@ -170,6 +173,39 @@ class ClientSuite extends SparkFunSuite with Matchers
with BeforeAndAfterAll {
cp should contain ("/remotePath/my1.jar")
}
+ test("configuration and args propagate through
createApplicationSubmissionContext") {
--- End diff --
Have you verified that this test passes both against versions of YARN that
support app tags and against versions that do not?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36708965
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+ logInfo("Applied setApplicationTags based on %s='%s'"
--- End diff --
I'd omit this log message. I don't think it's more important than other
attributes like queue and app name, which we don't log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36708697
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ val method = appContext.getClass().getMethod(
+"setApplicationTags", classOf[java.util.Set[String]])
+ method.invoke(appContext, new
java.util.HashSet[String](tagCollection))
+ logInfo("Applied setApplicationTags based on %s='%s'"
+.format(CONF_SPARK_YARN_APPLICATION_TAGS, tagCollection))
+
--- End diff --
Extra space unnecessary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/8072#discussion_r36708684 --- Diff: docs/running-on-yarn.md --- @@ -320,6 +320,14 @@ If you need a reference to the proper location to put log files in the YARN so t + spark.yarn.application.tags --- End diff -- I would change this to simply "spark.yarn.tags" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/8072#discussion_r36708654
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -163,6 +163,23 @@ private[spark] class Client(
appContext.setQueue(args.amQueue)
appContext.setAMContainerSpec(containerContext)
appContext.setApplicationType("SPARK")
+sparkConf.getOption(CONF_SPARK_YARN_APPLICATION_TAGS)
+ .map(StringUtils.getTrimmedStringCollection(_))
+ .filter(!_.isEmpty())
+ .foreach { tagCollection =>
+try {
+ val method = appContext.getClass().getMethod(
--- End diff --
Add a comment here mentioning why this needs to be called via reflection
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/8072#discussion_r36708634 --- Diff: docs/running-on-yarn.md --- @@ -320,6 +320,14 @@ If you need a reference to the proper location to put log files in the YARN so t + spark.yarn.application.tags + (none) + + Comma-separated list of strings to pass through as YARN application tags appearing + in YARN ApplicationReports, which can be used for filtering when querying YARN. --- End diff -- "when querying YARN apps" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-129546684 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-129545498 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-129545474 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-129545012 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8072#issuecomment-129543940 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9782] [YARN] Support YARN application t...
GitHub user dennishuo opened a pull request: https://github.com/apache/spark/pull/8072 [SPARK-9782] [YARN] Support YARN application tags via SparkConf Add a new test case in yarn/ClientSuite which checks how the various SparkConf and ClientArguments propagate into the ApplicationSubmissionContext. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dennishuo/spark dhuo-yarn-application-tags Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8072.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8072 commit f35e8ee7487e4171dda10fc076754ee6b495f1d7 Author: Dennis Huo Date: 2015-08-07T23:15:51Z [SPARK-9782] [YARN] Support YARN application tags via SparkConf Add a new test case in yarn/ClientSuite which checks how the various SparkConf and ClientArguments propagate into the ApplicationSubmissionContext. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
