Karuppayya created SPARK-58206:
----------------------------------
Summary: Support custom write metrics for shuffle storage plugins
Key: SPARK-58206
URL: https://issues.apache.org/jira/browse/SPARK-58206
Project: Spark
Issue Type: Improvement
Components: Spark Core, SQL
Affects Versions: 4.3.0
Reporter: Karuppayya
Spark supports pluggable shuffle storage (ShuffleDataIO), so shuffle data can
be written to substrates like a remote/external shuffle service, a distributed
filesystem (HDFS), or the default local disk.
However, plugins have no way to expose substrate-specific write metrics to
users — e.g. bytes pushed to a remote shuffle service, or spill-file counts and
fsync counts for local disk — so these remain invisible in the SQL UI.
(Somewhat similar to the custom-metrics support that DSV2 sources already have.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
