Hi Rajan, Thanks a lot for contributing to this feature. I think it is super helpful.
Can you add documentation for it? I tried searching for it and couldn't find it in the docs. Once you do, I can add it to the list of features pulsar has in this new page we have: https://pulsar.apache.org/features/ On Thu, Jul 14, 2022 at 8:13 PM Rajan Dhabalia <rdhaba...@apache.org> wrote: > On Thu, Jul 14, 2022 at 9:59 AM Asaf Mesika <asaf.mes...@gmail.com> wrote: > > > > > > > No, as it's mentioned in PIP: this API will terminate the topic so, it > > will > > > not allow any new write and producers on that topic, and then it flags > > the > > > topic as migrated and finally it sends migrated-topic response to > > > producers/caught-up consumers(with 0 backlog) for further redirection. > > > > So, this method will complete once "termination" of topic is complete: > wait > > for all producers to disconnect, wait for consumers to finish backlog and > > then all disconnect? > > > >> Producer can be immediately disconnected and allowed to redirect to > green cluster as soon as topic is terminated and marked as part of > termination. (considering the topic doesn't have geo-replication enabled. > For geo-replicated topics , you can read "Replicator and message ordering > handling" section in PIP). > > > > > This flag is to persist the state of managed-ledger that it's considered > > > for migration so, if a broker crashes then it knows about the > > > managed-ledger state. > > > > > > So perhaps the flag means "migrationInProgress" or "migrationStarted"? > > > >> It's just a state to mark that broker has considered it for migration > similar to the Terminated state. we don't need extra state for completion > because the broker is going to delete the topic once all subscribers reach > the end of the topic. > > > > > On Wed, Jul 13, 2022 at 7:23 PM Rajan Dhabalia <rdhaba...@apache.org> > > wrote: > > > > > On Wed, Jul 13, 2022 at 1:55 AM Asaf Mesika <asaf.mes...@gmail.com> > > wrote: > > > > > > > Few questions > > > > > > > > "CompletableFuture<Position> asyncMigrate();" > > > > Does this method only change the status of the managed ledger? > > > > > > > No, as it's mentioned in PIP: this API will terminate the topic so, it > > will > > > not allow any new write and producers on that topic, and then it flags > > the > > > topic as migrated and finally it sends migrated-topic response to > > > producers/caught-up consumers(with 0 backlog) for further redirection. > > > > > > > > > > > "message ManagedLedgerInfo { > > > > > > > > // Flag to check if topic is terminated and migrated to different > > > > cluster > > > > optional bool migrated = 4; > > > > > > > > }" > > > > > > > > This flag then is only changed to true when it has finished > migration: > > > i.e. > > > > no new messages were written, all existing consumers finished reading > > all > > > > messages and disconnected and the topic can now be deleted? > > > > > > > This flag is to persist the state of managed-ledger that it's > considered > > > for migration so, if a broker crashes then it knows about the > > > managed-ledger state. > > > > > > > > > > "Broker sends topic migration message to client so, producer/consumer > > at > > > > client side can handle redirection accordingly" > > > > > > > > For producers, the message will be sent the moment the status of the > > > topic > > > > has changed, so all messages from there on will be written to the new > > > > cluster? > > > > > > > Yes. > > > > > > > > > > For consumers, the message will be sent when there are no more > messages > > > to > > > > read? > > > > > > > Yes. > > > > > > > > > > > > > > > > > > > On Tue, Jul 12, 2022 at 8:23 PM Rajan Dhabalia <rdhaba...@apache.org > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > We have created PIP-184 which helps users to perform cluster > > migration > > > > with > > > > > Apache Pulsar. Cluster migration or Blue-Green cluster deployment > is > > > one > > > > of > > > > > the proven solutions to migrate live traffic from one cluster to > > > another. > > > > > One of the examples is applications running on Kubernetes sometimes > > > > require > > > > > a Kubernetes cluster upgrade which can cause downtime for the > entire > > > > > application during a Kubernetes cluster upgrade. Blue-green > > deployment > > > is > > > > > an application release model that gradually transfers user traffic > > > from a > > > > > previous version of an app or microservice to a nearly identical > new > > > > > release—both of which are running in production. > > > > > > > > > > The old version can be called the blue environment while the new > > > version > > > > > can be known as the green environment. Once production traffic is > > fully > > > > > transferred from blue to green, blue can standby in case of > rollback > > or > > > > be > > > > > pulled from production and updated to become the template upon > which > > > the > > > > > next update is made. We need such capability in Apache pulsar to > > > migrate > > > > > live traffic from the blue cluster to the green cluster so, > > eventually, > > > > the > > > > > entire traffic moves from the blue cluster to the green cluster > > without > > > > > causing downtime for the topics. > > > > > > > > > > PIP: https://github.com/apache/pulsar/issues/16551 > > > > > > > > > > Thanks, > > > > > Rajan > > > > > > > > > > > > > > >
