Josh Rosen created SPARK-42206:
----------------------------------
Summary: Omit "Task Executor Metrics" field in JsonProtocol output
if values are all zero
Key: SPARK-42206
URL: https://issues.apache.org/jira/browse/SPARK-42206
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Josh Rosen
SPARK-26329 added "Task Executor Metrics" to JsonProtocol SparkListenerTaskEnd
JSON. With the default `spark.executor.metrics.pollingInterval = 0`
configuration these metric values are only updated when heartbeats occur. If a
task launches and finishes between executor heartbeats then all of these metric
values will be zero. For jobs with large numbers of short tasks, this
contributes to significant event log bloat.
JsonProtocol already knows how to handle the absence of the "Task Executor
Metrics" field, so I think it's safe for us to omit this field when all values
are zero.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
