Hi Venkata, Yes, that works for me. Thanks for the detailed follow-up and for updating CIP-22.
Keeping I/O timeouts within the plugins sounds reasonable, with the prompt-return requirements and the possible overlap between close() and an outstanding check explicitly documented. Triggering decommission before retrieving the reason also addresses that concern. Separating the shared shutdown-ordering issue and the exit race into CELEBORN-2538 and CELEBORN-2537 makes sense, since both affect existing triggers as well. Thanks for keeping them linked from the CIP and making the timing assumptions explicit. The additional blocked-method tests and real-worker-process cases cover the validation gaps we discussed. In particular, the unavailable-master case should help verify the distinction between accepting decommission and actually rejecting writes. This addresses my concerns at the design level. I’ll take another look once the corresponding PR changes and tests are ready. Best regards, Nicholas Jiang On 2026/11/04 18:39:03 Venkatakrishnan Sowrirajan wrote: > Gentle ping @Nicholas Jiang <[email protected]> and @Keyong Zhou > <[email protected]> for further reviews! If you think it looks good, I can > start a VOTE thread. Thanks! > > Regards > Venkata krishnan > > > On Mon, Nov 2, 2026 at 11:07 AM Venkatakrishnan Sowrirajan <[email protected]> > wrote: > > > Hi Nicholas, Keyong, > > > > Thanks for taking a look. Keyong, sounds great. With both of you aligned, > > we'll keep the contract and plugins to own their I/O timeouts. > > > > Nicholas, agreed! Here's my plan. > > > > 1. Lifecycle contract. You're right, getReason() runs before the trigger > > today, so a blocked getReason() can hold back a check that already returned > > true. I'll trigger first and log the reason after. The CIP-22 now clarifies > > the contract for all three methods. shouldDecommission(), getReason() and > > close() must return quickly and put their own timeouts on I/O. close() can > > overlap a check that ignored the interrupt and outlived the 5s wait, so it > > has to be thread-safe against that check and must not wait on it. > > > > 2. Timing. I hadn't separated entering InDecommission from rejecting > > writes, and they're a few steps apart. The hook sends > > ReportWorkerDecommission synchronously and only then sets shutdown. With > > the defaults (askTimeout 60s, maxRetries 15), a hung master can stretch > > that to minutes. The CIP now clarifies each step, says the 80s/105s > > numbers assume a fast check and a responsive master, and adds an > > unavailable-master test. REST and master-driven decommission share this > > ordering, so I've filed CELEBORN-2538 to fix it for all triggers rather > > than inside CIP-22. Does that work? > > > > 3. Exit race. Agreed, the first accepted exit should decide both the mode > > and the timeout. I filed CELEBORN-2537 and linked it from the CIP. Will > > add regression test (graceful accepted first, a later decommission > > rejected, timeout unchanged) along with the fix. > > > > 4. Validation. I'll add unit tests for a checker that ignores interrupts > > and for blocked getReason() and close(). And you're right that WorkerSuite > > can't prove the shutdown path, since it stubs out System.exit. The test > > plan now has cluster cases with real worker processes: SIGTERM during > > polling, the hook and file cleanup, and an unavailable master. > > > > CIP doc is now updated, and will address the changes in the PR shortly. > > > > Regards > > Venkata krishnan > > > > > > On Sun, Nov 1, 2026 at 11:59 PM Keyong Zhou via dev < > > [email protected]> wrote: > > > >> Hi Venkata, > >> > >> Thanks for the proposal! The design looks good to me. For the open > >> question > >> in the document, I'm also inclined to keep the contract. > >> > >> Regards, > >> Keyong Zhou. > >> > >> Venkatakrishnan Sowrirajan <[email protected]> 于2026年11月2日周一 14:56写道: > >> > >> > Hi Celeborn Devs, > >> > > >> > I would like to propose adding support for a pluggable worker > >> > self-decommission mechanism based on an external signal which is > >> especially > >> > useful in the context of cloud environments with spot instances, K8s > >> > environments with planned maintenance etc. > >> > > >> > Here is the proposal doc/design of CIP-22: > >> > > >> > > >> https://docs.google.com/document/d/1cfNhyz4ABzxFSNpY8ISND1nefKHlLC9X8BNHv7Mki-E/edit?usp=sharing > >> > > >> > Please share your review comments or questions in the CIP-22 doc. > >> > > >> > Regards > >> > Venkata krishnan > >> > > >> > > >
