[ https://issues.apache.org/jira/browse/SPARK-48628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-48628: ----------------------------------- Labels: pull-request-available (was: ) > Add task peak on/off heap execution memory metrics > -------------------------------------------------- > > Key: SPARK-48628 > URL: https://issues.apache.org/jira/browse/SPARK-48628 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 4.0.0 > Reporter: Ziqi Liu > Priority: Major > Labels: pull-request-available > > Currently there is no task on/off heap execution memory metrics. There is a > [peakExecutionMemory|https://github.com/apache/spark/blob/3cd35f8cb6462051c621cf49de54b9c5692aae1d/core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala#L114] > metrics, however, the semantic is a confusing: it only cover the execution > memory used by shuffle/join/aggregate/sort, which is accumulated in specific > operators. > > We can easily maintain the whole task-level peak memory in TaskMemoryManager, > assuming *acquireExecutionMemory* is the only one narrow waist for acquiring > execution memory. > > Also it's nice to cleanup/deprecate that poorly-named `peakExecutionMemory`. > > Creating two followup sub tickets: > * https://issues.apache.org/jira/browse/SPARK-48788 :accumulate task metrics > in stage, and display in Spark UI > * https://issues.apache.org/jira/browse/SPARK-48789 : deprecate > `peakExecutionMemory` once we have replacement for it. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org