Yicong-Huang commented on code in PR #59246:
URL: https://github.com/apache/spark/pull/59246#discussion_r4202203085
##########
python/pyspark/eval_handlers/_base.py:
##########
@@ -99,11 +99,8 @@ def __init__(
self._udfs = udfs
self._runner_conf = runner_conf
self._eval_conf = eval_conf
- self._worker_metrics: Optional["WorkerMetrics"] = None
-
- def set_worker_metrics(self, metrics: "WorkerMetrics") -> None:
- """Attach the collector owned by the current worker task."""
- self._worker_metrics = metrics
+ # A handler is constructed for each task, including tasks in a reused
worker.
+ self.metrics = WorkerMetrics()
Review Comment:
this change looks good to me! now a new question:
shall we rename this to `HanlderMetrics` since it is indeed collecting time
spent on the handler, i.e., the time to serializer/deserialize data, time to
invoke/process udf?
Also It might be better to move the definition of metrics in this file,
instead of keeping it in worker.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
