Spenserrrr commented on code in PR #59246:
URL: https://github.com/apache/spark/pull/59246#discussion_r4251958177
##########
core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:
##########
@@ -213,21 +213,12 @@ private[spark] object BasePythonRunner extends Logging {
}
}
- // Values used to update the existing Python SQL metrics and task spill
metrics.
- private[python] case class WorkerMetrics(
- bootTimestampMs: Long,
- initTimestampMs: Long,
- finishTimestampMs: Long,
- pythonExecutionDurationMs: Long,
- memoryBytesSpilled: Long,
- diskBytesSpilled: Long)
-
private lazy val workerMetricsMapper = new ObjectMapper()
.enable(JsonParser.Feature.STRICT_DUPLICATE_DETECTION)
.enable(DeserializationFeature.FAIL_ON_TRAILING_TOKENS)
- /** Read and validate worker metrics after METRICS_DATA; ignore additional
fields. */
- private[python] def readWorkerMetrics(stream: DataInputStream):
WorkerMetrics = {
+ /** Decode all numeric fields after METRICS_DATA without knowing their
metric names. */
+ private[python] def readWorkerMetrics(stream: DataInputStream): Map[String,
Long] = {
Review Comment:
Currently, all the metrics are numeric because the values are fed into
SQLMetrics, and that accumulates only Long values. The downstream QPL operator
map also uses string metric names with int64 values. Thus, I let
readWorkerMetrics only accept Long. A string value would require a separately
designed path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
