This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new be6eef97a81c [SPARK-47726][DOC] Document push-based shuffle metrics
be6eef97a81c is described below
commit be6eef97a81c147272d5bee09afc5d423586762f
Author: Luca Canali <luca.can...@cern.ch>
AuthorDate: Wed Apr 17 09:35:05 2024 -0700
[SPARK-47726][DOC] Document push-based shuffle metrics
### What changes were proposed in this pull request?
This adds documentation for the push-based shuffle metrics
### Why are the changes needed?
The push-based shuffle metrics are currently not documented
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
N/A
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #45872 from LucaCanali/documentPushBasedShuffle.
Authored-by: Luca Canali <luca.can...@cern.ch>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
docs/monitoring.md | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/docs/monitoring.md b/docs/monitoring.md
index 5e11d5aef81e..a008b71c3fe9 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -1301,6 +1301,17 @@ These metrics are exposed by Spark executors.
- shuffleRemoteBytesReadToDisk.count
- shuffleTotalBytesRead.count
- shuffleWriteTime.count
+ - Metrics related to push-based shuffle:
+ - shuffleCorruptMergedBlockChunks
+ - shuffleMergedFetchFallbackCount
+ - shuffleMergedRemoteBlocksFetched
+ - shuffleMergedLocalBlocksFetched
+ - shuffleMergedRemoteChunksFetched
+ - shuffleMergedLocalChunksFetched
+ - shuffleMergedRemoteBytesRead
+ - shuffleMergedLocalBytesRead
+ - shuffleRemoteReqsDuration
+ - shuffleMergedRemoteReqsDuration
- succeededTasks.count
- threadpool.activeTasks
- threadpool.completeTasks
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
