Hi Sijie, Thanks for detailed procedure to achieve rollback. Just for an info, our product comprises of many Open Source components like Pulsar, Hadoop ecosystem and our own Java (stateless) application, rollback can be triggered if there is issue in any of the components.
Having said that, my queries/thoughts on both the approaches you have mentioned: * But a general approach I mentioned in my previous is doing rollback via re-replicating the data. Basically you can start a brand new bookie server running with the old version that is configured using old disk format. Then you can tear down the bookie server that runs a newer version that is configured to use a new disk format. The auto-recovery process will re-replicate the data to the bookies run old version. This is the way you achieve "rollback". – I’m not sure how the re-replication works, may be I’ve to go through the relevant document to understand it better. But from the outset, are you advising to run two instance of broker in a machine (old and new) or similar to below we can have both old and new version in the cluster and after performing canary test we can decide to rollback or not and if rollback is decided, then it can be done using re-replication? * Rollback after performing canary test with both old and new version are active in a cluster for some period of time – This option will help us only if there is issue with Pulsar/Bookkeeper. As I mentioned above, if there are issue is any of the Open source component / application we will be forced to rollback, in that case we won’t be able pick this option. For now (with my limited knowledge on Pulsar & Bookkeeper and for our product), the best possible option that I see is: * After upgrade, the new disk format version will be updated in bookkeeper configuration. In case of rollback, the disk format version has be reverted to an older version. We have to ensure that all data are re-written in older format before rolling back the binaries. Regards, Subash Kunjupillai From: Sijie Guo <[email protected]> Sent: Monday, April 20, 2020 9:56 PM To: [email protected] Cc: Bookkeeper-Dev <[email protected]> Subject: Re: Rollback of Apache Bookeeper This is the one of the approaches that you can downgrade. But a general approach I mentioned in my previous is doing rollback via re-replicating the data. Basically you can start a brand new bookie server running with the old version that is configured using old disk format. Then you can tear down the bookie server that runs a newer version that is configured to use a new disk format. The auto-recovery process will re-replicate the data to the bookies run old version. This is the way you achieve "rollback". If you are managing a very large bookkeeper cluster, you typically configure a rack-aware placement policy. When you want to roll out a newer version of bookkeeper, you typically deploy the newer version of bookkeeper to a set of bookies within one single rack. So you make sure an ensemble has both old version and newer version. You canary the newer version for a while to ensure this version is running well. Then you roll out the newer version to the whole cluster. If there is any problem observed during the canary period, you can stop the bookies that already rolled out the newer version. The auto-recovery process will re-replicate the data to the bookies running an old version. In this way, you "rollback" the whole cluster to an old version. This is a generally safe approach that applies to all versions. So you don't need to take special care for individual versions. - Sijie On Mon, Apr 20, 2020 at 3:13 AM Subash K <[email protected]<mailto:[email protected]>> wrote: Okay, now I got. If we upgrade and we are using the latest disk format version, before rollback we can downgrade the disk format version (bookkeeper conf) as per the Bookkeeper version we are rolling back to and once all the files are written in expected format we can rollback the binaries to the older version. Is there any API or tool through which we can check the disk format version of all data before rollback? Regards, Subash Kunjupillai From: Enrico Olivelli <[email protected]<mailto:[email protected]>> Sent: Monday, April 20, 2020 3:24 PM To: [email protected]<mailto:[email protected]> Cc: Bookkeeper-Dev <[email protected]<mailto:[email protected]>> Subject: Re: Rollback of Apache Bookeeper Il giorno lun 20 apr 2020 alle ore 11:49 Subash K <[email protected]<mailto:[email protected]>> ha scritto: Thanks Enrico and Sijie, so I see the only way to rollback is not to use the new data format after upgrade. And once we change the data format, then we lose the option of rollback. And regarding “For the second question, it depends on the changes in a given release. In certain releases, the disk format change supports rollback. In other releases, it doesn't. But you usually can still do rollback via re-replicating the data. So in general, you can assume you can do rollback..” -- I believe you are pointing out that, at times the new data format version is compatible with the older version. But if it is not compatible, then I hope we can’t re-replicate the data (to older version) as the entire Bookkeeper cluster data will be in new version of data format. I believe in that case, we might need a dedicated tool to rollback the data to previous version and then only we can rollback our binaries. Please let me know if my understanding is wrong. Also, after upgrade if we step up the data format version, does compaction (minor/major) of Bookkeeper re-write the existing data according to the new data format version? The bookie uses the configured disk format version. It is always able to read the old format, and it will write during regular writes and during compaction only the format configured in bk_server.conf Enrico Regards, Subash Kunjupillai From: Sijie Guo <[email protected]<mailto:[email protected]>> Sent: Monday, April 20, 2020 12:47 PM To: Bookkeeper-Dev <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: Rollback of Apache Bookeeper + users@ back to the discussion. Sorry for being late to the discussion here. When talking about "rollback" behavior, it usually involves two questions: 1) does wire protocol support rollback? 2) does data(metadata) format support rollback? For the first question, the bookkeeper provides very good wire protocol compatibility between versions. So if there are no data format changes, it is generally safe to assume that you can rollback bookies to an old version. For the second question, it depends on the changes in a given release. In certain releases, the disk format change supports rollback. In other releases, it doesn't. But you usually can still do rollback via re-replicating the data. So in general, you can assume you can do rollback.. Hope this clarifies things. - Sijie On Fri, Apr 17, 2020 at 11:48 AM Enrico Olivelli <[email protected]<mailto:[email protected]>> wrote: I am forwarding this request from user@ We should state somehow how we support rollbacks. In general we do not have much support for it. We only try to not enable new format of data on disk by default. This way you can update the binaries and then use the new format only if you need to use a new feature that requires such format. If you enable the new format there is no way to rollback. We could have some tool that tries to convert to the previous format. We should also talk about this topic on the website, even if we do not still provide any tool for the rollback. Thoughts? ---------- Forwarded message --------- Da: Subash K <[email protected]<mailto:[email protected]>> Date: Ven 17 Apr 2020, 16:32 Subject: RE: Rollback of Apache Bookeeper To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Thanks a lot Enrico 😊 Regards, Subash Kunjupillai *From:* Enrico Olivelli <[email protected]<mailto:[email protected]>> *Sent:* Friday, April 17, 2020 7:10 PM *To:* user <[email protected]<mailto:[email protected]>> *Subject:* Re: Rollback of Apache Bookeeper Subash I will discuss about this topic on [email protected]<mailto:[email protected]> list best regards Enrico Il giorno ven 17 apr 2020 alle ore 12:47 Subash K <[email protected]<mailto:[email protected]>> ha scritto: If rollback is technically supported by Bookkeeper, may be updating the website will help us better for now. Regards, Subash Kunjupillai *From:* Enrico Olivelli <[email protected]<mailto:[email protected]>> *Sent:* Friday, April 17, 2020 3:44 PM *To:* user <[email protected]<mailto:[email protected]>> *Subject:* Re: Rollback of Apache Bookeeper Il Ven 17 Apr 2020, 11:59 Subash K <[email protected]<mailto:[email protected]>> ha scritto: Sure. But as I’m just evaluating Apache Pulsar along with Bookkeeper, we are trying to understand various factors like Deployment, Life Cycle Management, etc.. Because these factors will be critical as it would create an impact on our existing product strategy. For now, we will take in that Rollback is not supported out of box. I feel that, rollback support should be brought in at the earliest to support organizations like us. In my opinion current releases of Bookkeeper support rollback. We do not have automated testing but we have this principle of being backward compatible with the previous version. Maybe we just lack an official statement on the website about this rule. Enrico Regards, Subash Kunjupillai *From:* Enrico Olivelli <[email protected]<mailto:[email protected]>> *Sent:* Friday, April 17, 2020 3:16 PM *To:* user <[email protected]<mailto:[email protected]>> *Subject:* Re: Rollback of Apache Bookeeper Il Ven 17 Apr 2020, 10:59 Subash K <[email protected]<mailto:[email protected]>> ha scritto: Thanks for your answers Enrico. On a high level, I understand that rollback is not supported and there is no plan to test and support it. But technically it is possible to do a rollback if new feature is not enabled after upgrade which is likely to change the file format. From the document, I see that there is option to rollback the file format after upgrade. Will this help us if we have to perform the rollback of data format as well? bookkeeper-server/bin/bookkeeper upgrade --rollback Actually that command was needed for very old versions of BK. I have been using it since 4.4 and never needed that command. In my opinion it is because since 4.4 BK payed more attention in making changes more safely A good option to you may be to check the version you are using and the one you want to upgrade to and if you have questions ask on this ML. We will be happy to support Enrico Regards, Subash Kunjupillai *From:* Enrico Olivelli <[email protected]<mailto:[email protected]>> *Sent:* 16 April 2020 19:47 *To:* user <[email protected]<mailto:[email protected]>> *Subject:* Re: Rollback of Apache Bookeeper Il giorno gio 16 apr 2020 alle ore 15:01 Subash K <[email protected]<mailto:[email protected]>> ha scritto: Hi Enrico, Thanks for your response. I’ve following queries on top of your response: - In general is community planning to test and support issues in rollback of Bookkeeper down the lane? Unfortunately no. We are only testing regular upgrades, not rollback. It should be doable with current docker based integration testing tools. - - We might not frequently upgrade Bookkeeper unless it is demanded by Pulsar. In that case, we might skip few intermediate versions and upgrade directly to an higher version. In that case, are we foreseeing any compatibility change that will cause hinderance during rollback? Usually this is not a problem. As far as you continue to upgrade from a supported version to a supported version. If I remember correctly currently we are officially supporting only BK 4.8 to 4.10. - - I’ve not fully gone through the internal working of Bookkeeper. But I would like to understand, can old version of bookie(after rollback) access the new data stored by upgraded bookie? Usually yes, as far as you do not activate new features that need a new format of data on disk Enrico - Regards, Subash Kunjupillai *From:* Enrico Olivelli <[email protected]<mailto:[email protected]>> *Sent:* 16 April 2020 16:05 *To:* user <[email protected]<mailto:[email protected]>> *Subject:* Re: Rollback of Apache Bookeeper Hi Subash, usually by default there is no problem in rolling back to the previous version. For instance if you are running 4.9 and you upgrade to 4.10 your bookies you can downgrade to 4.9 just by using the old code. This is because we introduce new features but they are disabled by default at least for one release. But I suggest you to perform a test about the rollback before going to production Enrico Il giorno gio 16 apr 2020 alle ore 12:23 Subash K <[email protected]<mailto:[email protected]>> ha scritto: Hi, We are planning to introduce Apache Pulsar in our product. In general we always support upgrade and rollback of our product. So I'm trying to understand is there any defined steps provided to support rollback of Apache Bookeeper to its earlier version after completing full cluster upgrade? I was not able to find those information in https://bookkeeper.apache.org/docs/latest/admin/upgrade/ <https://slack-redir.net/link?url=https%3A%2F%2Fbookkeeper.apache.org%2Fdocs%2Flatest%2Fadmin%2Fupgrade%2F&v=3> Regards, Subash Kunjupillai
