This is an automated email from the ASF dual-hosted git repository.
zaynt4606 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new c277ec1911 [CELEBORN-2506] Add user documentation for Celeborn Spark
UI extension
c277ec1911 is described below
commit c277ec1911b2719dbc43f9471b7298b6934cc23d
Author: yew1eb <[email protected]>
AuthorDate: Sun Nov 1 16:40:43 2026 +0800
[CELEBORN-2506] Add user documentation for Celeborn Spark UI extension
### What changes were proposed in this pull request?
Add a user guide (`docs/developers/sparkui.md`) for the Celeborn Spark UI
extension and register it in the mkdocs navigation. It covers how to enable the
plugin via `spark.plugins`, what the Celeborn tab shows, and Spark History
Server replay support.
### Why are the changes needed?
The Spark UI extension is available in the shaded client jars but has no
user documentation, so users have no way to discover or use it.
### Does this PR resolve a correctness bug?
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
- [x] Yes
Users who read the documentation.
### How was this patch tested?
Rendered the docs site with `mkdocs build --strict` and reviewed the page
content against the current Spark UI implementation.
Closes #3902 from yew1eb/spark-ui-doc.
Authored-by: yew1eb <[email protected]>
Signed-off-by: zhengtao <[email protected]>
AI-Contributed/Feature: 0/46
AI-Contributed/UT: 0/0
---
docs/developers/sparkui.md | 45 +++++++++++++++++++++++++++++++++++++++++++++
mkdocs.yml | 1 +
2 files changed, 46 insertions(+)
diff --git a/docs/developers/sparkui.md b/docs/developers/sparkui.md
new file mode 100644
index 0000000000..35813058d0
--- /dev/null
+++ b/docs/developers/sparkui.md
@@ -0,0 +1,45 @@
+---
+license: |
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ https://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+---
+
+# Celeborn Spark UI
+
+Celeborn ships a Spark plugin that adds a **Celeborn** tab to the Spark Web UI
of applications using Celeborn for shuffle. The tab aggregates Celeborn shuffle
metrics from task metrics, making it easy to check shuffle write/read
throughput and the impact of shuffle on task duration at a glance.
+
+The plugin classes are packaged in the shaded Celeborn client jars
(`celeborn-client-spark-3-shaded` / `celeborn-client-spark-4-shaded`), so no
extra jar is required — the shaded jar already on the classpath is enough.
+
+## Enabling the Plugin
+
+The UI extension is disabled by default. To enable it, set `spark.plugins` in
the Spark configuration on top of the regular Celeborn shuffle settings (see
[Deployment](../../deploy)):
+
+```properties
+spark.shuffle.manager org.apache.spark.shuffle.celeborn.SparkShuffleManager
+...
+spark.plugins org.apache.spark.shuffle.celeborn.CelebornPlugin
+```
+
+**Note**: `spark.plugins` accepts a comma-separated list; append
`org.apache.spark.shuffle.celeborn.CelebornPlugin` if other Spark plugins are
already configured.
+
+## Using the Celeborn Tab
+
+Once enabled, the Spark Web UI shows a **Celeborn** tab with a shuffle summary
page. The summary covers overall shuffle write/read throughput and the share of
task time spent on shuffle, followed by collapsible tables with details such as
the effective `spark.celeborn.*` configuration (sensitive values are
automatically redacted), shuffle slot assignments per shuffle and per worker,
compression ratio, and the Celeborn client build information.
+
+## Spark History Server
+
+The plugin also works with the Spark History Server.
`CelebornHistoryServerPlugin` is registered through Spark's
`AppHistoryServerPlugin` SPI, so finished applications are replayed with the
same Celeborn tab. Make sure the shaded Celeborn client jar is on the History
Server classpath.
+
+**Note**: the application must have enabled `CelebornPlugin` and set
`spark.eventLog.enabled=true` while it was running, so that the Celeborn UI
data is recorded in the event log. Adding the jar only to the History Server
will not add the Celeborn tab to applications that did not enable the plugin.
diff --git a/mkdocs.yml b/mkdocs.yml
index 1c89271650..70a8e4a7a3 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -99,6 +99,7 @@ nav:
- LifecycleManager: developers/lifecyclemanager.md
- ShuffleClient: developers/shuffleclient.md
- JavaColumnarShuffle: developers/java-columnar-shuffle.md
+ - Spark UI: developers/sparkui.md
- Configuration: developers/configuration.md
- Fault Tolerant: developers/faulttolerant.md
- Worker Exclusion: developers/workerexclusion.md
