liuzqt commented on code in PR #36162:
URL: https://github.com/apache/spark/pull/36162#discussion_r2134049538
##########
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:
##########
@@ -863,6 +872,39 @@ private[spark] class TaskSchedulerImpl(
executorUpdates)
}
+ private def getTaskAccumulableInfosAndProcessRate(
+ updates: Seq[AccumulatorV2[_, _]]): (Seq[AccumulableInfo], Double) = {
+ var recordsRead = 0L
+ var executorRunTime = 0L
+ val accInfos = updates.map { acc =>
+ if (efficientTaskCalcualtionEnabled && acc.name.isDefined) {
+ val name = acc.name.get
+ if (name == shuffleRead.RECORDS_READ || name == input.RECORDS_READ) {
Review Comment:
Hi @weixiuli , I'm reading through this part of code, and I think it will
never work because task metric `EXECUTOR_RUN_TIME` is only set at the end of
task. When task is still running, this metric in heartbeat is always zero. (I
might be wrong...please feel free to correct me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
