Haotian Sun created SPARK-62067:
-----------------------------------
Summary: Report Python worker timing metrics in microseconds
Key: SPARK-62067
URL: https://issues.apache.org/jira/browse/SPARK-62067
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 5.0.0
Reporter: Haotian Sun
Python worker timing metrics are floored to whole milliseconds per task, so
sub-millisecond work reports 0 and sums over many tasks undercount. Report the
worker's timestamps and durations in microseconds, and expose the Python timing
SQL metrics as microsecond sums whose keys and names include the unit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]