Haotian Sun created SPARK-60663:
-----------------------------------
Summary: Collect detailed timing metrics for mapInPandas
Key: SPARK-60663
URL: https://issues.apache.org/jira/browse/SPARK-60663
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.4.0
Reporter: Haotian Sun
Extend detailed Python worker timing to mapInPandas: read, input preparation,
UDF execution, output preparation, and write durations, with
consumed-input-batch counts.
Reuse a shared eligibility gate and Arrow stream timers, measure eager and
iterator UDF execution, and subtract nested input/read intervals for sequential
phase attribution. Timing remains disabled by default.
Pipelined phase attribution is deferred; reports marked by
pythonNumPipelinedTasks must be excluded from phase analysis.
Implementation: https://github.com/apache/spark/pull/59892
Generated-by: Codex (GPT-6.1).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
