(celeborn) branch main updated: [CELEBORN-2463] Add ManuallyExcludedWorkerCount metric to distinguish manually excluded workers

[zaynt]

This is an automated email from the ASF dual-hosted git repository.

zaynt4606 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new 931e730cde [CELEBORN-2463] Add ManuallyExcludedWorkerCount metric to 
distinguish manually excluded workers
931e730cde is described below

commit 931e730cde445cfd1a1bc185e360dd50f8f32453
Author: yew1eb <yew...@gmail.com>
AuthorDate: Tue Oct 13 12:23:36 2026 +0800

    [CELEBORN-2463] Add ManuallyExcludedWorkerCount metric to distinguish 
manually excluded workers
    
    ### What changes were proposed in this pull request?
    
    Add a new master gauge metric `ManuallyExcludedWorkerCount` that reports 
the count of workers in the manually excluded list 
(`statusSystem.manuallyExcludedWorkers.size`).
    
    The existing `ExcludedWorkerCount` metric is kept unchanged (still the sum 
of both excluded and manually excluded workers) for backward compatibility.
    
    ### Why are the changes needed?
    
    `ExcludedWorkerCount` counts both workers excluded due to failure reports / 
unhealthy heartbeats, and workers excluded manually via the `/exclude` API 
during maintenance. As a result, alerts based on `ExcludedWorkerCount` fire 
during routine maintenance even though the cluster is healthy.
    
    The two sets are already tracked separately internally (`excludedWorkers` 
vs `manuallyExcludedWorkers`) and exposed separately in the `/workers` API.
    
    With this PR, users can alert on `ExcludedWorkerCount - 
ManuallyExcludedWorkerCount` to monitor only failure-driven exclusions.
    
    ### Does this PR resolve a correctness bug?
    
    - [ ] Yes
    
    ### Does this PR introduce _any_ user-facing change?
    
    - [x] Yes
    
    Add a new master metric `ManuallyExcludedWorkerCount`.
    
    ### How was this patch tested?
    
    Compiled the master module locally. The new gauge follows the exact same 
pattern as the existing `ExcludedWorkerCount` gauge.
    
    Closes #3848 from yew1eb/CELEBORN-2463.
    
    Authored-by: yew1eb <yew...@gmail.com>
    Signed-off-by: zhengtao <shuaizhentao....@alibaba-inc.com>
    
    AI-Contributed/Feature: 0/6
    AI-Contributed/UT: 0/0
---
 docs/monitoring.md                                                     | 1 +
 .../main/scala/org/apache/celeborn/service/deploy/master/Master.scala  | 3 +++
 .../scala/org/apache/celeborn/service/deploy/master/MasterSource.scala | 2 ++
 3 files changed, 6 insertions(+)

diff --git a/docs/monitoring.md b/docs/monitoring.md
index 1aa4b3749a..58c4d7ad32 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -115,6 +115,7 @@ These metrics are exposed by Celeborn master.
     | WorkerCount                  | The count of active workers.              
                                                |
     | LostWorkerCount              | The count of workers in lost list.        
                                                |
     | ExcludedWorkerCount          | The count of workers in excluded list.    
                                                |
+    | ManuallyExcludedWorkerCount  | The count of workers in manually excluded 
list.                                           |
     | AvailableWorkerCount         | The count of workers in available list.   
                                                |
     | ShutdownWorkerCount          | The count of workers in shutdown list.    
                                                |
     | DecommissionWorkerCount      | The count of workers in decommission 
list.                                                |
diff --git 
a/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala 
b/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala
index 81a5027384..61a444b7ca 100644
--- 
a/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala
+++ 
b/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala
@@ -244,6 +244,9 @@ private[celeborn] class Master(
   masterSource.addGauge(MasterSource.EXCLUDED_WORKER_COUNT) { () =>
     statusSystem.excludedWorkers.size + 
statusSystem.manuallyExcludedWorkers.size
   }
+  masterSource.addGauge(MasterSource.MANUALLY_EXCLUDED_WORKER_COUNT) { () =>
+    statusSystem.manuallyExcludedWorkers.size
+  }
   masterSource.addGauge(MasterSource.AVAILABLE_WORKER_COUNT) { () =>
     statusSystem.availableWorkers.size()
   }
diff --git 
a/master/src/main/scala/org/apache/celeborn/service/deploy/master/MasterSource.scala
 
b/master/src/main/scala/org/apache/celeborn/service/deploy/master/MasterSource.scala
index 3e22411ae1..b1f376148b 100644
--- 
a/master/src/main/scala/org/apache/celeborn/service/deploy/master/MasterSource.scala
+++ 
b/master/src/main/scala/org/apache/celeborn/service/deploy/master/MasterSource.scala
@@ -50,6 +50,8 @@ object MasterSource {
 
   val EXCLUDED_WORKER_COUNT = "ExcludedWorkerCount"
 
+  val MANUALLY_EXCLUDED_WORKER_COUNT = "ManuallyExcludedWorkerCount"
+
   val SHUTDOWN_WORKER_COUNT = "ShutdownWorkerCount"
 
   val AVAILABLE_WORKER_COUNT = "AvailableWorkerCount"