[jira] [Commented] (SPARK-14628) Remove all the Options in TaskMetrics
[ https://issues.apache.org/jira/browse/SPARK-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245079#comment-15245079 ] Apache Spark commented on SPARK-14628: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/12462 > Remove all the Options in TaskMetrics > - > > Key: SPARK-14628 > URL: https://issues.apache.org/jira/browse/SPARK-14628 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Reporter: Reynold Xin >Assignee: Wenchen Fan > Fix For: 2.0.0 > > > Part of the reason why TaskMetrics and its callers are complicated are due to > the optional metrics we collect, including input, output, shuffle read, and > shuffle write. Given their values are zero, I think we can always track them. > It is usually very obvious whether a task is supposed to read any data or > not. By always tracking them, we can remove a lot of map, foreach, flatMap, > getOrElse(0L) calls throughout Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14628) Remove all the Options in TaskMetrics
[ https://issues.apache.org/jira/browse/SPARK-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242924#comment-15242924 ] Apache Spark commented on SPARK-14628: -- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/12417 > Remove all the Options in TaskMetrics > - > > Key: SPARK-14628 > URL: https://issues.apache.org/jira/browse/SPARK-14628 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Reporter: Reynold Xin > > Part of the reason why TaskMetrics and its callers are complicated are due to > the optional metrics we collect, including input, output, shuffle read, and > shuffle write. Given their values are zero, I think we can always track them. > It is usually very obvious whether a task is supposed to read any data or > not. By always tracking them, we can remove a lot of map, foreach, flatMap, > getOrElse(0L) calls throughout Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14628) Remove all the Options in TaskMetrics
[ https://issues.apache.org/jira/browse/SPARK-14628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240772#comment-15240772 ] Apache Spark commented on SPARK-14628: -- User 'rxin' has created a pull request for this issue: https://github.com/apache/spark/pull/12388 > Remove all the Options in TaskMetrics > - > > Key: SPARK-14628 > URL: https://issues.apache.org/jira/browse/SPARK-14628 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Reporter: Reynold Xin > > Part of the reason why TaskMetrics and its callers are complicated are due to > the optional metrics we collect, including input, output, shuffle read, and > shuffle write. Given their values are zero, I think we can always track them. > It is usually very obvious whether a task is supposed to read any data or > not. By always tracking them, we can remove a lot of map, foreach, flatMap, > getOrElse(0L) calls throughout Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
