taking BK dev off this.. Pulsar storage has to be rock stable at all times. I will make a few suggestions to that end.
Note that there is a direct conflict of interests here. For new installations, they have no baggage and would like the latest/greatest features. But for users who are already running it in production, stability to existing applications is almost always not welcome. No bleeding edge changes for the latter case. - Storage upgrades should be less frequent. They are not cheap operationally. Rollback is a mitigation action only. Even rollback comes with a cost, of degradation and disruption to service. And loss/corruption of data is not an operationally recoverable event. For any user that runs Pulsar in production, a BK upgrade (any storage upgrade fro that matter) has to be planned and executed to cover business risks. - Decoupling BK and Pulsar. There was a hard-wried dependency in Pulsar to a specific version of BK till now. A byproduct of that was a kind of forced storage stability across many Pulsar releases. Going forward, stability is not automatic. My suggestion is that any release of Pulsar must be able to run on a range (min version - max version) of compatible BK releases and not just one version. Now that Pulsar can run on the Apache BK, maybe relax the dependency to some extent. That would make storage upgrade optional, till say some major release that mandated it. So while I can understand the need for a storage upgrade in this case now, the Pulsar community has to make a release plan and set a pace for storage upgrades. Storage system releases are quite different from stateless serving software. So far, Pulsar upgrades have been easy, because more or less the storage was the same and stable. I don't want to be in a place where I cannot deploy a Pulsar release because of the risks in a storage upgrade. --- Is every release of Pulsar going to have a BK upgrade? -- How many times will storage upgrade occur in a year? --- Is the storage upgrade optional? These are important questions that the community should discuss. Joe On Mon, Mar 5, 2018 at 1:41 AM, Ivan Kelly <iv...@apache.org> wrote: > > Keep in mind the next Pulsar release needs to be based on a particular > BookKeeper release. An Apache project cannot release against another > project's SNAPSHOT since that has not been validated by their release > process. > > The bookkeeper 4.7.0 release process should be starting soon, so it > should be out before the pulsar 2.0 release process kicks off. > > -Ivan >
