pan3793 commented on code in PR #57332:
URL: https://github.com/apache/spark/pull/57332#discussion_r3613604278
##########
core/src/main/scala/org/apache/spark/executor/Executor.scala:
##########
@@ -950,10 +954,15 @@ private[spark] class Executor(
(taskStartTimeNs - deserializeStartTimeNs) +
task.executorDeserializeTimeNs))
task.metrics.setExecutorDeserializeCpuTime(
(taskStartCpu - deserializeStartCpuTime) +
task.executorDeserializeCpuTime)
- // We need to subtract Task.run()'s deserialization time to avoid
double-counting
- task.metrics.setExecutorRunTime(TimeUnit.NANOSECONDS.toMillis(
- (taskFinishNs - taskStartTimeNs) * taskDescription.cpus
- - task.executorDeserializeTimeNs))
+ // We need to subtract Task.run()'s deserialization time to avoid
double-counting.
+ // With a fractional cpus below 1 the scaled elapsed time can be
smaller than the in-run
+ // deserialization time, so clamp at zero rather than reporting a
negative run time.
+ task.metrics.setExecutorRunTime(math.max(0L,
TimeUnit.NANOSECONDS.toMillis(
+ // Strip trailing zeros from the scale-9 cpus so this product stays
in BigDecimal's
+ // compact (long-backed) form instead of inflating a long duration
into a BigInteger.
+ (BigDecimal(taskFinishNs - taskStartTimeNs) *
+ CpuAmount.stripTrailingZeros(taskDescription.cpus)).toLong
+ - task.executorDeserializeTimeNs)))
Review Comment:
the success path now computes `cpus * (elapsed - in-run deserialization)`
instead of `cpus * elapsed - deserialization`, fixing both the fractional-cpus
negative case and the pre-existing over-count for integer cpus > 1. The zero
clamp stays as a guard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
