Gentle ping @Nicholas Jiang <[email protected]> and @Keyong Zhou <[email protected]> for further reviews! If you think it looks good, I can start a VOTE thread. Thanks!
Regards Venkata krishnan On Mon, Nov 2, 2026 at 11:07 AM Venkatakrishnan Sowrirajan <[email protected]> wrote: > Hi Nicholas, Keyong, > > Thanks for taking a look. Keyong, sounds great. With both of you aligned, > we'll keep the contract and plugins to own their I/O timeouts. > > Nicholas, agreed! Here's my plan. > > 1. Lifecycle contract. You're right, getReason() runs before the trigger > today, so a blocked getReason() can hold back a check that already returned > true. I'll trigger first and log the reason after. The CIP-22 now clarifies > the contract for all three methods. shouldDecommission(), getReason() and > close() must return quickly and put their own timeouts on I/O. close() can > overlap a check that ignored the interrupt and outlived the 5s wait, so it > has to be thread-safe against that check and must not wait on it. > > 2. Timing. I hadn't separated entering InDecommission from rejecting > writes, and they're a few steps apart. The hook sends > ReportWorkerDecommission synchronously and only then sets shutdown. With > the defaults (askTimeout 60s, maxRetries 15), a hung master can stretch > that to minutes. The CIP now clarifies each step, says the 80s/105s > numbers assume a fast check and a responsive master, and adds an > unavailable-master test. REST and master-driven decommission share this > ordering, so I've filed CELEBORN-2538 to fix it for all triggers rather > than inside CIP-22. Does that work? > > 3. Exit race. Agreed, the first accepted exit should decide both the mode > and the timeout. I filed CELEBORN-2537 and linked it from the CIP. Will > add regression test (graceful accepted first, a later decommission > rejected, timeout unchanged) along with the fix. > > 4. Validation. I'll add unit tests for a checker that ignores interrupts > and for blocked getReason() and close(). And you're right that WorkerSuite > can't prove the shutdown path, since it stubs out System.exit. The test > plan now has cluster cases with real worker processes: SIGTERM during > polling, the hook and file cleanup, and an unavailable master. > > CIP doc is now updated, and will address the changes in the PR shortly. > > Regards > Venkata krishnan > > > On Sun, Nov 1, 2026 at 11:59 PM Keyong Zhou via dev < > [email protected]> wrote: > >> Hi Venkata, >> >> Thanks for the proposal! The design looks good to me. For the open >> question >> in the document, I'm also inclined to keep the contract. >> >> Regards, >> Keyong Zhou. >> >> Venkatakrishnan Sowrirajan <[email protected]> 于2026年11月2日周一 14:56写道: >> >> > Hi Celeborn Devs, >> > >> > I would like to propose adding support for a pluggable worker >> > self-decommission mechanism based on an external signal which is >> especially >> > useful in the context of cloud environments with spot instances, K8s >> > environments with planned maintenance etc. >> > >> > Here is the proposal doc/design of CIP-22: >> > >> > >> https://docs.google.com/document/d/1cfNhyz4ABzxFSNpY8ISND1nefKHlLC9X8BNHv7Mki-E/edit?usp=sharing >> > >> > Please share your review comments or questions in the CIP-22 doc. >> > >> > Regards >> > Venkata krishnan >> > >> >
