Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2024-04-03 Thread Asaf Mesika
Hi Rajan, Thanks a lot for contributing to this feature. I think it is super helpful. Can you add documentation for it? I tried searching for it and couldn't find it in the docs. Once you do, I can add it to the list of features pulsar has in this new page we have:

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-14 Thread Rajan Dhabalia
On Thu, Jul 14, 2022 at 9:59 AM Asaf Mesika wrote: > > > > No, as it's mentioned in PIP: this API will terminate the topic so, it > will > > not allow any new write and producers on that topic, and then it flags > the > > topic as migrated and finally it sends migrated-topic response to > >

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-14 Thread Asaf Mesika
> > No, as it's mentioned in PIP: this API will terminate the topic so, it will > not allow any new write and producers on that topic, and then it flags the > topic as migrated and finally it sends migrated-topic response to > producers/caught-up consumers(with 0 backlog) for further redirection.

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-14 Thread Rajan Dhabalia
On Thu, Jul 14, 2022 at 8:29 AM Dave Maughan wrote: > Hi Rajan, > > Just a note that 184 was already taken by > https://github.com/apache/pulsar/issues/16481 > >>> Sure, I have renamed it as PIP-188 and updated the wiki: https://github.com/apache/pulsar/wiki - Dave > > > On Tue, Jul 12, 2022

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-14 Thread Dave Maughan
Hi Rajan, Just a note that 184 was already taken by https://github.com/apache/pulsar/issues/16481 - Dave On Tue, Jul 12, 2022 at 6:23 PM Rajan Dhabalia wrote: > Hi, > > We have created PIP-184 which helps users to perform cluster migration with > Apache Pulsar. Cluster migration or

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-13 Thread Rajan Dhabalia
On Wed, Jul 13, 2022 at 10:57 AM Joe F wrote: > Hi Rajan, > > > > > For consumers, the message will be sent when there are no more messages > > to > > > read? > > > > > Yes. > > > I think it would be helpful to add a bit more detail on the consumer > switch (sp. Shared consumers). There is a

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-13 Thread Joe F
Hi Rajan, > > > For consumers, the message will be sent when there are no more messages > to > > read? > > > Yes. I think it would be helpful to add a bit more detail on the consumer switch (sp. Shared consumers). There is a window of transition where "dispatched-but-not-acked messages" might

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-13 Thread Rajan Dhabalia
On Wed, Jul 13, 2022 at 2:30 AM Enrico Olivelli wrote: > Very interesting. Overall I support this proposal > > A couple of questions: > - What about monitoring the status of the migration? > There will be a cluster level state. Once all topics are migrated and deleted, the broker will change

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-13 Thread Rajan Dhabalia
On Wed, Jul 13, 2022 at 1:55 AM Asaf Mesika wrote: > Few questions > > "CompletableFuture asyncMigrate();" > Does this method only change the status of the managed ledger? > No, as it's mentioned in PIP: this API will terminate the topic so, it will not allow any new write and producers on that

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-13 Thread Enrico Olivelli
Very interesting. Overall I support this proposal A couple of questions: - What about monitoring the status of the migration? - Should we block all maintenance operations on the "blue" cluster ? like deleting/creating stuff - Should we stop ledger trimming and offloading ? - What about

Re: [DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-13 Thread Asaf Mesika
Few questions "CompletableFuture asyncMigrate();" Does this method only change the status of the managed ledger? "message ManagedLedgerInfo { // Flag to check if topic is terminated and migrated to different cluster optional bool migrated = 4; }" This flag then is only changed to true

[DISCUSS] PIP-184: Cluster migration or Blue-Green cluster deployment support in Pulsar

2022-07-12 Thread Rajan Dhabalia
Hi, We have created PIP-184 which helps users to perform cluster migration with Apache Pulsar. Cluster migration or Blue-Green cluster deployment is one of the proven solutions to migrate live traffic from one cluster to another. One of the examples is applications running on Kubernetes sometimes