Gentle ping @Nicholas Jiang <[email protected]> and @Keyong Zhou
<[email protected]> for further reviews! If you think it looks good, I can
start a VOTE thread. Thanks!

Regards
Venkata krishnan


On Mon, Nov 2, 2026 at 11:07 AM Venkatakrishnan Sowrirajan <[email protected]>
wrote:

> Hi Nicholas, Keyong,
>
> Thanks for taking a look. Keyong, sounds great. With both of you aligned,
> we'll keep the contract and plugins to own their I/O timeouts.
>
> Nicholas, agreed! Here's my plan.
>
> 1. Lifecycle contract. You're right, getReason() runs before the trigger
> today, so a blocked getReason() can hold back a check that already returned
> true. I'll trigger first and log the reason after. The CIP-22 now clarifies
> the contract for all three methods. shouldDecommission(), getReason() and
> close() must return quickly and put their own timeouts on I/O. close() can
> overlap a check that ignored the interrupt and outlived the 5s wait, so it
> has to be thread-safe against that check and must not wait on it.
>
> 2. Timing. I hadn't separated entering InDecommission from rejecting
> writes, and they're a few steps apart. The hook sends
> ReportWorkerDecommission synchronously and only then sets shutdown. With
> the defaults (askTimeout 60s, maxRetries 15), a hung master can stretch
> that to minutes. The CIP now clarifies each step, says the 80s/105s
> numbers assume a fast check and a responsive master, and adds an
> unavailable-master test. REST and master-driven decommission share this
> ordering, so I've filed CELEBORN-2538 to fix it for all triggers rather
> than inside CIP-22. Does that work?
>
> 3. Exit race. Agreed, the first accepted exit should decide both the mode
> and the timeout. I filed CELEBORN-2537 and linked it from the CIP. Will
> add regression test (graceful accepted first, a later decommission
> rejected, timeout unchanged) along with the fix.
>
> 4. Validation. I'll add unit tests for a checker that ignores interrupts
> and for blocked getReason() and close(). And you're right that WorkerSuite
> can't prove the shutdown path, since it stubs out System.exit. The test
> plan now has cluster cases with real worker processes: SIGTERM during
> polling, the hook and file cleanup, and an unavailable master.
>
> CIP doc is now updated, and will address the changes in the PR shortly.
>
> Regards
> Venkata krishnan
>
>
> On Sun, Nov 1, 2026 at 11:59 PM Keyong Zhou via dev <
> [email protected]> wrote:
>
>> Hi Venkata,
>>
>> Thanks for the proposal! The design looks good to me. For the open
>> question
>> in the document, I'm also inclined to keep the contract.
>>
>> Regards,
>> Keyong Zhou.
>>
>> Venkatakrishnan Sowrirajan <[email protected]> 于2026年11月2日周一 14:56写道:
>>
>> > Hi Celeborn Devs,
>> >
>> > I would like to propose adding support for a pluggable worker
>> > self-decommission mechanism based on an external signal which is
>> especially
>> > useful in the context of cloud environments with spot instances, K8s
>> > environments with planned maintenance etc.
>> >
>> > Here is the proposal doc/design of CIP-22:
>> >
>> >
>> https://docs.google.com/document/d/1cfNhyz4ABzxFSNpY8ISND1nefKHlLC9X8BNHv7Mki-E/edit?usp=sharing
>> >
>> > Please share your review comments or questions in the CIP-22 doc.
>> >
>> > Regards
>> > Venkata krishnan
>> >
>>
>

Reply via email to