Il giorno mer 12 ott 2022 alle ore 00:40 Matteo Merli <matteo.me...@gmail.com> ha scritto: > > Agree, though let's make separate discussions. Putting all random > ideas into the same cauldron is a good recipe for making no one able > to follow or see a common line. > > That's what I meant when I started the proposal of having 3.0 > completely detached from "features". > > If you start making a big container, you're going to fill it up with > all the "breaking changes" that you want to include, because "hey, > that's the only window of opportunity". That in turn is the surest way > to not ship anything for the next 24 months, as all the changes are > unavoidably going to get delayed and will take a long time to > stabilize. > > Going back to API breakages: > 1. We never break wire protocol compatibility > 2. We try to never break client API > 3. We need a very good & compelling reason in order to break client API > 4. When we do so, we need to provide a clear path for users (eg: > pulsar-client-1.x compatibly drop-in).
I totally agree with Matteo and Joe, We cannot break compatibility. and if we need to introduce some new form of inter-broker communication protocol we must support the current protocol and provide a smooth upgrade path. Now there are many Pulsar clusters around the world that cannot tolerate stop-the-world upgrades and we MUST also allow some sort of rollback in case of problems. Enrico > > > > > > > -- > Matteo Merli > <matteo.me...@gmail.com> > > On Tue, Oct 11, 2022 at 11:58 AM Dave Fisher <w...@apache.org> wrote: > > > > Let’s discuss any and all ideas for improvement. As each is discussed we > > can figure out how to make them non-breaking, We all want Pulsar to improve. > > > > We should encourage an open discussion where no idea is automatically bad > > or wrong. They can just be discussed without fear. > > > > Thanks, > > Dave > > > > > On Oct 10, 2022, at 3:05 PM, Joe F <joefranc...@gmail.com> wrote: > > > > > > I would prefer that we avoid using the term “breaking changes”, which is > > > too vague to convey any specific meaning. So let me try to bring some > > > clarity. > > > > > > > > > There have been many changes to implementations, APIs and data storage > > > formats in Pulsar (and book keeper also). I have deployed many of these > > > changes to production. And I know that Matteo and Rajan (and others too, > > > about whom I’m not up to date on) have implemented and deployed many > > > such > > > changes. But none of those changes ever required taking the system > > > offline. NONE. > > > > > > > > > Pulsar was developed as a 24x7x365 system, and rolling upgrades and > > > rollbacks were a given. Like “this is water”, there was no special > > > callout > > > needed for declaring this reality. No change, including enhancements to > > > wire protocols, broke client compatibility. Existing clients continued to > > > work; they may not be able to use all the new features. Use of new > > > features > > > would require the app to be rebuilt anyway. (Checksums, e2e encryption > > > are > > > examples) > > > > > > > > > We have even succeeded in getting Pulsar adopted for some use cases, just > > > because the complexity of upgrading from K’s old clients to new ones were > > > costly enough to allow consideration of an alternative like Pulsar. The > > > business cost of forcing a client upgrade can be significant, to the > > > point > > > of this being unviable for business. That just cannot be hand-waved over > > > > > > > > > There have also been changes in storage formats(the ZK metadata change > > > from > > > text to binary is an example). But through all such changes, compatibility > > > and upgradeability has been a given. There has never been a situation > > > where > > > a live Pulsar upgrade was not possible, and a coordinated client > > > upgrade > > > was mandatory. > > > > > > > > > So the question should not be about whether “signifcant” changes should > > > be made or not. Changes can be made and released in a way that breaks > > > *business*, or they can be made in a way that lets businesses sail > > > smoothly through that change. So the question is about how such changes > > > gets rolled out. > > > > > > > > > And to that question, my strong opinion is that any change that does not > > > allow a live/rolling upgrade or rollback, or anything that forces a client > > > to upgrade just to continue functioning, is a non-starter. All changes > > > can be made in a compatible, phased manner, and in a way that does not > > > penalise older versions ( older versions doing worse on new releases is > > > also not an acceptable way of making changes) Changes can be made in a > > > manner that make l A/B testing possible by the user, with limited risk, > > > and > > > then choosing to a not go back. It has all been done in Pulsar before. > > > > > > > > > Would that be harder than just breaking stuff? Yes. But that is far more > > > preferable than forcing users to take a hit. > > > > > > > > > -joe > > > > > > On Sat, Oct 8, 2022 at 1:25 PM Rajan Dhabalia <rdhaba...@apache.org> > > > wrote: > > > > > >> I would say first we should gather a list of changes which we want to > > >> target and find out which improvements really need major version release. > > >> We can take the Pulsar-1.0 to Pulsar-2.0 upgrade example to avoid major > > >> interruption and impact on existing systems and still achieve our goal. > > >> So, > > >> the first step is discovery of such features and then we can discuss how > > >> to > > >> introduce them in Pulsar with minimum impact on existing systems. > > >> > > >> Thanks, > > >> Rajan > > >> > > >> On Sat, Oct 8, 2022 at 1:05 PM Devin Bost <devin.b...@gmail.com> wrote: > > >> > > >>> I'm noticing some pushback on the idea of pre-emptively proposing any > > >> kind > > >>> of breaking upgrade that would necessitate cutting a 3.0 release. > > >>> I do understand the concern about introducing a breaking change... For a > > >>> distributed messaging application like Pulsar, if clients needed to be > > >>> simultaneously upgraded with brokers, that could be extremely difficult > > >> or > > >>> infeasible for companies to coordinate without treating it like a > > >> migration > > >>> to a new technology. > > >>> > > >>> At the same time, do we want to be completely closed to the possibility > > >>> that a breaking change could be required at some point in the future? If > > >> a > > >>> circumstance like that appears, those are the kinds of situations that > > >> can > > >>> lead to a fork. Are there certain kinds of breaking changes that are > > >>> more > > >>> acceptable than others? > > >>> > > >>> Also, if the forward looking plan is to never introduce breaking > > >>> changes, > > >>> when *would* we ever cut a Pulsar 3.x release? Do we have any criteria > > >> on > > >>> what kinds of changes would necessitate cutting a new major release but > > >>> would still be considered acceptable by the community? > > >>> > > >>> -- > > >>> Devin Bost > > >>> Sent from mobile > > >>> Cell: 801-400-4602 > > >>> > > >>> On Sat, Oct 8, 2022, 2:14 PM Rajan Dhabalia <rdhaba...@apache.org> > > >> wrote: > > >>> > > >>>> This sounds like the current state of Apache Pulsar has a lot of issues > > >>> and > > >>>> it requires fundamental design changes to make it promising which is > > >>>> definitely not true and I disagree with it. And I would be careful > > >>>> comparing with Kafka as I still don't think the Kafka release has > > >>> anything > > >>>> to do with Pulsar's improvement. I would still recommend to list down > > >> all > > >>>> the changes at one place so we can bring everyone on the same page. > > >>> discuss > > >>>> as a community and we make sure existing usecases continue using Pulsar > > >>> and > > >>>> not try to find Pulsar alternatives with incorrect disruption > > >> impression > > >>>> and efforts they might have to put to upgrade or maintain pulsar. > > >>>> > > >>>> Thanks, > > >>>> Rajan > > >>>> > > >>>> On Fri, Oct 7, 2022 at 7:49 PM Lari Hotari <lhot...@apache.org> wrote: > > >>>> > > >>>>> We could all have our own favorite names for this work. :) > > >>>>> > > >>>>> There's advice that you should disrupt yourself before someone > > >> disrupts > > >>>>> you. > > >>>>> Shouldn't we follow that advice for Apache Pulsar? We can disrupt > > >>> Pulsar > > >>>>> together with our Apache hats on. The catch is that since we are > > >> doing > > >>>>> this, we will be able to learn and improve Pulsar so that we stay > > >> ahead > > >>>> of > > >>>>> competition. Pulsar was long ways ahead of competition for so many > > >>> years, > > >>>>> but Kafka is finally catching up. Did Kafka surpass Pulsar in some > > >>>> aspects > > >>>>> with the recent 3.3 release, where Kraft became GA? That's a question > > >>>> that > > >>>>> many might be asking. Why wouldn't we rev up Pulsar's engine and show > > >>> the > > >>>>> tail lights to Kafka? > > >>>>> > > >>>>> We don't have to have deadlines or any restrictions like that right > > >>> now. > > >>>>> The sky's the limit. > > >>>>> Linus Torvalds has written a book called "Just for fun". I got my > > >> copy > > >>> of > > >>>>> this book signed by Linus himself in year 2000 at an event that the > > >>> book > > >>>>> publisher had organized in Finland. > > >>>>> > > >>>>> What if we did this "just for fun"? The intention could also be to > > >> beat > > >>>>> Kafka, but that could be a boring goal for many. What if we could > > >>> unleash > > >>>>> some talent that is among us and hasn't had a chance to show its full > > >>>>> potential? Opensource is about joy. It is about welcoming everyone to > > >>>> join. > > >>>>> Opensource should be egoless, although we must all admit that we > > >> don't > > >>>>> succeed in that aspect. We must fight our biases. > > >>>>> > > >>>>> Jarek Potiuk explains the importance of being welcoming for success > > >> at > > >>>>> Apache, in a 3-minute YouTube interview: > > >>>>> https://www.youtube.com/watch?v=Dx5kQnVFo7E > > >>>>> This interview is about Jarek's blog post "Success at Apache: > > >> Welcoming > > >>>>> communities strengthens the Apache way": > > >>>>> > > >>>>> > > >>>> > > >>> > > >> https://news.apache.org/foundation/entry/success-at-apache-welcoming-communities > > >>>>> I was pleased to meet Jarek at ApacheCon among so many other > > >> welcoming > > >>>>> personalities of the Apache community and the Apache Pulsar > > >> community. > > >>>>> > > >>>>> Goals have to be ambitious. What if we set the bar really high? > > >>>>> Apache Pulsar with 10 million topics in a cluster? > > >>>>> Why not go up to 100 million topics? > > >>>>> Just for fun. :) > > >>>>> > > >>>>> -Lari > > >>>>> > > >>>>> On 2022/10/07 22:53:59 Matteo Merli wrote: > > >>>>>> I actually disagree with the term "Pulsar Next Gen", because I > > >>> haven't > > >>>>>> seen any proposal for which that would make sense to me to be > > >> called > > >>>>>> so. > > >>>>>> > > >>>>>> Rajan: That's the whole point of breaking it down. If you > > >> accumulate > > >>>>>> many "big" changes it introduces a lot of risk for instabilities > > >> and > > >>>>>> incompatibilities. Breaking it down in multiple steps helps to see > > >>> the > > >>>>>> incremental changes and introduced them in a phased manner. > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> Matteo Merli > > >>>>>> <matteo.me...@gmail.com> > > >>>>>> > > >>>>>> On Fri, Oct 7, 2022 at 3:37 PM Rajan Dhabalia < > > >> rdhaba...@apache.org> > > >>>>> wrote: > > >>>>>>> > > >>>>>>> Hi, > > >>>>>>> > > >>>>>>> Can we get the list of changes at one place which we are planning > > >>> to > > >>>>> get as > > >>>>>>> part of 3.0. One thing I would like to see as a part of a major > > >>>>> release, it > > >>>>>>> CAN NOT impact existing usecases and users in any way which can > > >>> force > > >>>>> them > > >>>>>>> to upgrade the client library. Applications using < 3.0 version > > >>>> should > > >>>>>>> continue getting all the client and server side enhancements and > > >>> bug > > >>>>> fixes. > > >>>>>>> Failing to provide bug-fixes and features to client < 3.0 means > > >> we > > >>>> are > > >>>>>>> forcing them to upgrade client version by putting efforts to > > >> handle > > >>>> all > > >>>>>>> incompatibility. and that's something we should definitely > > >> prevent > > >>>>> because > > >>>>>>> Apache Pulsar is used by many large scale business usecases and > > >> we > > >>>>> should > > >>>>>>> accommodate and motivate them to continue using Apache Pulsar. > > >>>>>>> I understand as a Pulsar community we should always try to > > >> progress > > >>>> and > > >>>>>>> build better but not at the cost of losing or reducing the Apache > > >>>>> Pulsar > > >>>>>>> community. > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> Rajan > > >>>>>>> > > >>>>>>> > > >>>>>>> On Fri, Oct 7, 2022 at 12:41 PM Lari Hotari <lhot...@apache.org> > > >>>>> wrote: > > >>>>>>> > > >>>>>>>> Thank you, Matteo. I agree that features should be delivered > > >>>>> continuously > > >>>>>>>> when that is possible. In this case, that might not apply. > > >>>>>>>> > > >>>>>>>> I also agree that calling this Pulsar 3.0 isn't necessarily > > >>> aligned > > >>>>> with > > >>>>>>>> PIP-175 since an LTS release is when the major version is > > >> bumped. > > >>>>> I'm fine > > >>>>>>>> in calling this "Pulsar Next Gen" or something that calls out > > >>> that > > >>>>> this is > > >>>>>>>> planning for making a major leap in Pulsar. > > >>>>>>>> > > >>>>>>>> There are several unresolved issues with PIP-45 and the Pulsar > > >>> Load > > >>>>>>>> balancer. The previously referred email threads contain a lot > > >> of > > >>>>> context to > > >>>>>>>> this. Resolving the issues efficiently will most likely result > > >> in > > >>>>> breaking > > >>>>>>>> changes, which will be the reason why it deserves a major > > >> version > > >>>>> upgrade. > > >>>>>>>> > > >>>>>>>> We have discussed it before that it's crucial to have a path to > > >>>>> migrate > > >>>>>>>> users when there are breaking changes. This should be covered > > >> in > > >>>> any > > >>>>> of the > > >>>>>>>> solutions that are introduced. Optimally, users of Pulsar would > > >>> be > > >>>>> able to > > >>>>>>>> upgrade seamlessly to Pulsar Next Gen / Pulsar 3.0, but rolling > > >>>> back > > >>>>> might > > >>>>>>>> not be directly supported. > > >>>>>>>> > > >>>>>>>> I am welcoming everyone to join this planning for the Apache > > >>> Pulsar > > >>>>> Next > > >>>>>>>> Gen architecture. Please check the first email in this thread > > >> for > > >>>>> details > > >>>>>>>> of context, and start participating and contributing today. The > > >>>> best > > >>>>> way to > > >>>>>>>> contribute is to participate in the email threads, since they > > >>>> contain > > >>>>>>>> details with better context. > > >>>>>>>> > > >>>>>>>> -Lari > > >>>>>>>> > > >>>>>>>> On 2022/10/07 18:03:00 Matteo Merli wrote: > > >>>>>>>>> Given the past experiences and the discussions that already > > >>>>> happened > > >>>>>>>>> around "PIP-175: Extend time based release process", the idea > > >>> is > > >>>> to > > >>>>>>>>> detach the 3.0 from "big-features" items or "incompatible > > >>>> changes". > > >>>>>>>>> > > >>>>>>>>> The changes are going to get included as they are ready, > > >> within > > >>>>>>>>> feature releases, and in a fully compatible way. We don't > > >> need > > >>> to > > >>>>>>>>> group them together and create unnecessary risk for the > > >> release > > >>>>>>>>> schedule and the users. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> -- > > >>>>>>>>> Matteo Merli > > >>>>>>>>> <matteo.me...@gmail.com> > > >>>>>>>>> > > >>>>>>>>> On Fri, Oct 7, 2022 at 10:47 AM Lari Hotari < > > >>> lhot...@apache.org> > > >>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>> Hi all, > > >>>>>>>>>> > > >>>>>>>>>> Greeting from ApacheCon North America 2022 from New > > >> Orleans! > > >>>>>>>>>> We had a great conference with a dedicated Pulsar track. > > >>> Thanks > > >>>>> to all > > >>>>>>>> presenters and everyone who attended. The talks weren't > > >> recorded, > > >>>>> but the > > >>>>>>>> slides will be later on posted on the conference website [1]. > > >>>>>>>>>> > > >>>>>>>>>> At ApacheCon there were several presentations about "the > > >>> Apache > > >>>>> way" > > >>>>>>>> and what that means in practice. Based on that, we all know > > >> that > > >>> no > > >>>>> person > > >>>>>>>> is nominated as the CTO of Apache Pulsar who decides on Pulsar > > >>> 3.0 > > >>>>> and when > > >>>>>>>> that happens. It's us, the community, that serve that role > > >>>> together. > > >>>>> We > > >>>>>>>> come together as individuals with the Apache hat on. Everyone > > >> is > > >>>>> equal in > > >>>>>>>> the community, regardless of whether they are contributors, > > >>>>> committers or > > >>>>>>>> PMC members. > > >>>>>>>>>> We welcome everyone to participate. The small detail about > > >>>> voting > > >>>>>>>> shouldn't stop anyone from participating in any aspects of the > > >>>>> planning for > > >>>>>>>> the roadmap. > > >>>>>>>>>> > > >>>>>>>>>> I'll like to get the discussions going for Pulsar 3.0. We > > >>> don't > > >>>>> need a > > >>>>>>>> separate decision to start planning that. Please correct me if > > >>> I'm > > >>>>> wrong or > > >>>>>>>> if you have a different opinion. > > >>>>>>>>>> > > >>>>>>>>>> There are a few previous discussion threads that are > > >> related > > >>> to > > >>>>> Pulsar > > >>>>>>>> 3.0 planning. > > >>>>>>>>>> If you are interested in getting involved with Apache > > >> Pulsar > > >>>> 3.0 > > >>>>>>>> planning, I think that it makes sense for you to read these > > >>> threads > > >>>>>>>> carefully and reply to them. Please also suggest what you think > > >>>>> makes sense. > > >>>>>>>>>> > > >>>>>>>>>> PIP-45 related: > > >>>>>>>> > > >> https://lists.apache.org/thread/tvco1orf0hsyt59pjtfbwoq0vf6hfrcj > > >>>>>>>>>> Pulsar Load balancer / namespace bundle related: > > >>>>>>>>>> > > >>>> https://lists.apache.org/thread/roohoc9h2gthvmd7t81do4hfjs2gphpk > > >>>>>>>>>> renaming topics: > > >>>>>>>>>> > > >>>> https://lists.apache.org/thread/vrr75rrh4trqlp14objh3snlfvmzdrp2 > > >>>>>>>>>> backpressure: > > >>>>>>>> > > >> https://lists.apache.org/thread/v7xy57qfzbhopoqbm75s6ng8xlhbr2q6 > > >>>>>>>>>> > > >>>>>>>>>> Long list of Metadata inconsistency issues by Zac Bentley: > > >>>>>>>>>> https://github.com/apache/pulsar/issues/12555 > > >>>>>>>>>> That would be a good starting point to understanding the > > >> data > > >>>>>>>> inconsistency issues related to current PIP-45 design. Perhaps > > >>>> those > > >>>>> could > > >>>>>>>> be addressed already before Pulsar 3.0? > > >>>>>>>>>> > > >>>>>>>>>> I'm looking forward to everyone's participation in the > > >> Apache > > >>>>> Pulsar > > >>>>>>>> 3.0 planning discussions. > > >>>>>>>>>> > > >>>>>>>>>> Best Regards, > > >>>>>>>>>> > > >>>>>>>>>> -Lari > > >>>>>>>>>> > > >>>>>>>>>> 1 - https://www.apachecon.com/acna2022/schedule.html > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > >
