[jira] [Commented] (SPARK-9782) Add support for YARN application tags running Spark on YARN
[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680413#comment-14680413 ] Apache Spark commented on SPARK-9782: - User 'dennishuo' has created a pull request for this issue: https://github.com/apache/spark/pull/8072 Add support for YARN application tags running Spark on YARN --- Key: SPARK-9782 URL: https://issues.apache.org/jira/browse/SPARK-9782 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 1.4.1 Reporter: Dennis Huo https://issues.apache.org/jira/browse/YARN-1390 originally added the new “Application Tags” feature to YARN to help track the sources of applications among many possible YARN clients. https://issues.apache.org/jira/browse/YARN-1399 improved on this to allow a set of tags to be applied, and for comparison, https://issues.apache.org/jira/browse/MAPREDUCE-5699 added support for MapReduce to easily propagate tags through to YARN via Configuration settings. Since the ApplicationSubmissionContext.setApplicationTags method was only added in Hadoop 2.4+, Spark support will invoke the method via reflection the same way other such version-specific methods are called in elsewhere in the YARN client. Since the usage of tags is generally not critical to the functionality of older YARN setups, it should be safe to handle NoSuchMethodException with just a logWarning. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9782) Add support for YARN application tags running Spark on YARN
[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680415#comment-14680415 ] Sean Owen commented on SPARK-9782: -- This is different from https://issues.apache.org/jira/browse/SPARK-7173 ? I also doubt this will go in any time soon if it needs Hadoop 2.x, since even 1.x is still supported, even with reflection -- the complexity may not be worth it. Add support for YARN application tags running Spark on YARN --- Key: SPARK-9782 URL: https://issues.apache.org/jira/browse/SPARK-9782 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 1.4.1 Reporter: Dennis Huo https://issues.apache.org/jira/browse/YARN-1390 originally added the new “Application Tags” feature to YARN to help track the sources of applications among many possible YARN clients. https://issues.apache.org/jira/browse/YARN-1399 improved on this to allow a set of tags to be applied, and for comparison, https://issues.apache.org/jira/browse/MAPREDUCE-5699 added support for MapReduce to easily propagate tags through to YARN via Configuration settings. Since the ApplicationSubmissionContext.setApplicationTags method was only added in Hadoop 2.4+, Spark support will invoke the method via reflection the same way other such version-specific methods are called in elsewhere in the YARN client. Since the usage of tags is generally not critical to the functionality of older YARN setups, it should be safe to handle NoSuchMethodException with just a logWarning. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9782) Add support for YARN application tags running Spark on YARN
[ https://issues.apache.org/jira/browse/SPARK-9782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14680431#comment-14680431 ] Dennis Huo commented on SPARK-9782: --- Correct, from what I understand, the node labels JIRA is a more heavyweight behavioral-change feature, for being able to control packing of requested containers onto machines based on node labels. YARN application tags are distinct from node labels, and are only used by workflow orchestrators on top of YARN, without affecting how YARN does packing at all. Add support for YARN application tags running Spark on YARN --- Key: SPARK-9782 URL: https://issues.apache.org/jira/browse/SPARK-9782 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 1.4.1 Reporter: Dennis Huo https://issues.apache.org/jira/browse/YARN-1390 originally added the new “Application Tags” feature to YARN to help track the sources of applications among many possible YARN clients. https://issues.apache.org/jira/browse/YARN-1399 improved on this to allow a set of tags to be applied, and for comparison, https://issues.apache.org/jira/browse/MAPREDUCE-5699 added support for MapReduce to easily propagate tags through to YARN via Configuration settings. Since the ApplicationSubmissionContext.setApplicationTags method was only added in Hadoop 2.4+, Spark support will invoke the method via reflection the same way other such version-specific methods are called in elsewhere in the YARN client. Since the usage of tags is generally not critical to the functionality of older YARN setups, it should be safe to handle NoSuchMethodException with just a logWarning. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
