LucaCanali commented on a change in pull request #26953:
URL: https://github.com/apache/spark/pull/26953#discussion_r565433850
##########
File path: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala
##########
@@ -434,7 +443,10 @@ private[spark] abstract class BasePythonRunner[IN, OUT](
override def hasNext: Boolean = nextObj != null || {
if (!eos) {
+ val startTime = System.nanoTime()
nextObj = read()
+ val deltaTime = System.nanoTime()-startTime
+ PythonMetrics.incFromWorkerReadTime(deltaTime)
Review comment:
@HyukjinKwon I have finally managed to work on your suggestion to
instrument Python execution using SQL Metrics, so that users can see the
metrics via the WebUI. See [SPARK-34265]. I imagine that I could later refactor
the work on this PR based on that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
