[
https://issues.apache.org/jira/browse/SPARK-42203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-42203:
-------------------------------
Description:
This is a followup to SPARK-36620:
When push-based shuffle is disabled (the default), I think that we should skip
the logging of the new push-based shuffle read metrics. Because these metrics
are logged for every task, they will add significant additional size to Spark
event logs. It would be great to avoid this cost in cases where it's not
necessary.
was:
This is a followup to SPARK-36620:
When push-based shuffle is disabled (the default), I think that we should skip
the logging of the new push-based shuffle read metrics. Because these metrics
are logged for every task, they will add significant additional size to Spark
event logs.
> JsonProtocol should skip logging of push-based shuffle read metrics when
> push-based shuffle is disabled
> -------------------------------------------------------------------------------------------------------
>
> Key: SPARK-42203
> URL: https://issues.apache.org/jira/browse/SPARK-42203
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle
> Affects Versions: 3.4.0
> Reporter: Josh Rosen
> Priority: Major
>
> This is a followup to SPARK-36620:
> When push-based shuffle is disabled (the default), I think that we should
> skip the logging of the new push-based shuffle read metrics. Because these
> metrics are logged for every task, they will add significant additional size
> to Spark event logs. It would be great to avoid this cost in cases where it's
> not necessary.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
