Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-09-08 Thread Michael Marshall
Thanks for your response, Matteo. The benefits of my proposed alternative are dependent on the amount of time between when a broker stops accepting messages and when the client learns of the new host for the topic. The main areas this will matter are failure scenarios where that delta or that "mea

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-09-05 Thread Matteo Merli
-- Matteo Merli On Tue, Aug 8, 2023 at 9:56 PM Michael Marshall wrote: > Thanks for your proposal, Heesung. Sorry for my late response. Just > want to make a few points to hopefully improve the implementation. > Overall, I think this feature is absolutely the right direction for > Pulsar and w

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-24 Thread PengHui Li
> The previous source broker information is persisted in the system topic with its state info. Upon receiving the Owned state via ownership transfer or recovery commands, the source broker will close the topics in the bundle(disconnect clients), if any. Oh, got it. It looks good. Thanks for the ex

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-22 Thread Heesung Sohn
> How long will it take to collect sufficient broker load data and bundle load data? It would be normally bounded by `loadBalancerReportUpdateMinIntervalMillis,` but in the worst case, `loadBalancerReportUpdateMaxIntervalMinutes`. > And the new leader might only know the new owner of that topic?

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-21 Thread PengHui Li
> Actually, this is incorrect. Currently, the leader broker globally triggers unloading. When the leader crahshes, the new leader broker will delay unloading until it collects sufficient broker load data and bundle load data. Ok, good to know. How long will it take to collect sufficient broker loa

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-21 Thread Heesung Sohn
Hi Penghui, I added one comment below. On Mon, Aug 21, 2023 at 6:12 AM PengHui Li wrote: > > However, with this optional field approach, we don't need to introduce a > new protocol version, which could be arguably beneficial. > I will leave the decision to the community. Let me know if I have t

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-21 Thread PengHui Li
> However, with this optional field approach, we don't need to introduce a new protocol version, which could be arguably beneficial. I will leave the decision to the community. Let me know if I have to use new commands and increase the protocolVersion for this change. I don't see any drawbacks of

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-17 Thread Michael Marshall
> However, with this optional field approach, we don't need to introduce a > new protocol version, which could be arguably beneficial. I don't see versioning the protocol as a negative as long as there is sufficient benefit. In this case, the new command serves an explicit purpose to make the prot

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-10 Thread Heesung Sohn
On Tue, Aug 8, 2023 at 9:56 PM Michael Marshall wrote: > Thanks for your proposal, Heesung. Sorry for my late response. Just > want to make a few points to hopefully improve the implementation. > Overall, I think this feature is absolutely the right direction for > Pulsar and will be a great impr

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-08 Thread Michael Marshall
Thanks for your proposal, Heesung. Sorry for my late response. Just want to make a few points to hopefully improve the implementation. Overall, I think this feature is absolutely the right direction for Pulsar and will be a great improvement. I remember discussing this at some community meetings la

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-04 Thread Heesung Sohn
Hi, I updated the PIP to add more details about the recent discussion. Thanks, Heesung On Thu, Aug 3, 2023 at 6:46 PM Heesung Sohn wrote: > > > On Wed, Aug 2, 2023 at 8:02 PM PengHui Li wrote: > >> > One of the advantages of this proposal (add these optional dstBrokerUrl >> fields in the exis

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-03 Thread Heesung Sohn
On Wed, Aug 2, 2023 at 8:02 PM PengHui Li wrote: > > One of the advantages of this proposal (add these optional dstBrokerUrl > fields in the existing CommandCloseProducer and CommandCloseConsumer) is > the backward compatibility > > Ah, yes. I missed this point. It's a good convince. > > > After

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-02 Thread PengHui Li
> One of the advantages of this proposal (add these optional dstBrokerUrl fields in the existing CommandCloseProducer and CommandCloseConsumer) is the backward compatibility Ah, yes. I missed this point. It's a good convince. > After the client connects to the destination broker, the next command

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-02 Thread Heesung Sohn
Hi PengHui, Thank you for your questions. Please find my answers inline. On Wed, Aug 2, 2023 at 3:47 AM PengHui Li wrote: > Hi Heesung, > > I'm sorry for not getting back to you sooner. > The motivation and the solution look good to me. > > I just want to leave some comments to make the proposal

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-02 Thread PengHui Li
Hi Heesung, I'm sorry for not getting back to you sooner. The motivation and the solution look good to me. I just want to leave some comments to make the proposal clear for users and developers.

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-08-01 Thread Heesung Sohn
Hi, It has been over three weeks since this discussion started, and I replied to the questions in the PIP. I will send out a vote email tomorrow if looking good. Thanks, Heesung On Tue, Jul 25, 2023 at 1:12 PM Heesung Sohn wrote: > Hi, > > I replied to the comments from Kai and Ran in the PIP.

Re: [DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-07-25 Thread Heesung Sohn
Hi, I replied to the comments from Kai and Ran in the PIP. Please review the PIP by any chance. Thank you, Heesung On Thu, Jul 6, 2023 at 4:32 PM Heesung Sohn wrote: > Hi dev, > > I proposed this PIP, https://github.com/apache/pulsar/pull/20748, to make > unloaded clients directly and gracefu

[DISCUSS] PIP-281: Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager

2023-07-06 Thread Heesung Sohn
Hi dev, I proposed this PIP, https://github.com/apache/pulsar/pull/20748, to make unloaded clients directly and gracefully connect to new owner brokers without lookups. Please take a look and let me know what you think. Regards, Heesung