Hi Venkata,

Thanks for the proposal. I support adding an opt-in, worker-side extension 
point that reuses the existing decommission path. Keeping cloud-specific 
implementations and shuffle-data migration outside this CIP seems reasonable.

After reviewing the updated proposal and PR #3939, I have a few comments:
1. Checker lifecycle and bounded execution. Could we extend the 
bounded-execution contract to getReason() and close()? Currently, getReason() 
runs before the decommission call, so a blocked diagnostic method can prevent 
an already-detected signal from taking effect. Also, the five-second shutdown 
wait only bounds the in-flight poll; the subsequent synchronous close() can 
still block worker cleanup. Please specify how implementations should safely 
handle close() overlapping an unfinished check.
2. Timing assumptions. Could the timing section distinguish entering 
InDecommission from actually rejecting new writes? In the existing path, 
sendWorkerDecommissionToMaster() runs synchronously before shutdown is set, so 
master RPC delays and retries can consume additional time. The 80s/105s 
examples therefore assume fast checks and a responsive master. Documenting 
those assumptions and testing an unavailable master would make the operational 
guidance more reliable.
3. Concurrent exit requests. Thanks for explicitly identifying the 
shutdown-hook timeout race. Handling it in a separate fix is reasonable, but 
could we link a tracking issue and add regression coverage? “The earlier 
decision wins” should cover both the exit mode and its timeout; a rejected 
decommission request should not change an accepted graceful shutdown’s timeout.
4. End-to-end validation. The proposed cluster test is useful. Please also 
cover SIGTERM arriving during polling, an interruption-ignoring checker, and 
blocked getReason()/`close()` callbacks. The current WorkerSuite test replaces 
System.exit, so a forked-JVM or cluster test is needed to validate the actual 
shutdown-hook and cleanup behavior.
Regarding the per-check timeout question, I am comfortable starting with 
plugin-managed I/O timeouts, provided the full lifecycle contract is explicit. 
A generic executor timeout alone cannot stop non-interruptible plugin code.

Regards,
Nicholas Jiang

On 2026-11-02 06:56 UTC Venkatakrishnan Sowrirajan wrote:
> Hi Celeborn Devs,
> 
> I would like to propose adding support for a pluggable worker
> self-decommission mechanism based on an external signal which is especially
> useful in the context of cloud environments with spot instances, K8s
> environments with planned maintenance etc.
> 
> Here is the proposal doc/design of CIP-22:
> https://docs.google.com/document/d/1cfNhyz4ABzxFSNpY8ISND1nefKHlLC9X8BNHv7Mki-E/edit?usp=sharing
> 
> Please share your review comments or questions in the CIP-22 doc.
> 
> Regards
> Venkata krishnan
> 

Reply via email to