[
https://issues.apache.org/jira/browse/SPARK-42206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17681238#comment-17681238
]
Apache Spark commented on SPARK-42206:
--------------------------------------
User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/39770
> Omit "Task Executor Metrics" field in JsonProtocol output if values are all
> zero
> --------------------------------------------------------------------------------
>
> Key: SPARK-42206
> URL: https://issues.apache.org/jira/browse/SPARK-42206
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
>
> SPARK-26329 added "Task Executor Metrics" to JsonProtocol
> SparkListenerTaskEnd JSON. With the default
> `spark.executor.metrics.pollingInterval = 0` configuration these metric
> values are only updated when heartbeats occur. If a task launches and
> finishes between executor heartbeats then all of these metric values will be
> zero. For jobs with large numbers of short tasks, this contributes to
> significant event log bloat.
> JsonProtocol already knows how to handle the absence of the "Task Executor
> Metrics" field, so I think it's safe for us to omit this field when all
> values are zero.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
