> Companies want to be cloud agnostic since it reduces their risk, and the idea of multi-cloud integration is very attractive to companies that want the best of both worlds or need to integrate data across platforms. If Pulsar is not part of this wave, I'm concerned we will be left behind.
That sounds a lot of marketish. If you want to work on cloud-events integration, please do. I still don't see how's that related to Pulsar 3.0 discussion. -- Matteo Merli <matteo.me...@gmail.com> On Tue, Oct 11, 2022 at 9:54 AM Devin Bost <devin.b...@gmail.com> wrote: > > How about the gap > <https://lists.apache.org/thread/jfwv1wkbk4nsd9lhpnh5o2r0pgq39qz8> between > SchemaInfo and interoperability with CloudEvents? We're seeing a > convergence between cloud providers on CloudEvents. I just learned that SAP > Event Mesh now is working on integration with Azure EventGrid via > CloudEvents. GCP is already there. AWS has an adapter from EventBridge. > There is work taking place on solutions for discovery, like a search engine > for event sources. Kafka has a binding for CloudEvents. Pretty soon, I > suspect we're going to see increasing interest in the integration of events > across organizational and technological boundaries, and technologies will > emerge with direct support for events that conform to the CloudEvents > spec. Companies want to be cloud agnostic since it reduces their risk, and > the idea of multi-cloud integration is very attractive to companies that > want the best of both worlds or need to integrate data across platforms. If > Pulsar is not part of this wave, I'm concerned we will be left behind. > > Devin G. Bost > > > On Mon, Oct 10, 2022 at 5:06 PM Joe F <joefranc...@gmail.com> wrote: > > > I would prefer that we avoid using the term “breaking changes”, which is > > too vague to convey any specific meaning. So let me try to bring some > > clarity. > > > > > > There have been many changes to implementations, APIs and data storage > > formats in Pulsar (and book keeper also). I have deployed many of these > > changes to production. And I know that Matteo and Rajan (and others too, > > about whom I’m not up to date on) have implemented and deployed many such > > changes. But none of those changes ever required taking the system > > offline. NONE. > > > > > > Pulsar was developed as a 24x7x365 system, and rolling upgrades and > > rollbacks were a given. Like “this is water”, there was no special callout > > needed for declaring this reality. No change, including enhancements to > > wire protocols, broke client compatibility. Existing clients continued to > > work; they may not be able to use all the new features. Use of new features > > would require the app to be rebuilt anyway. (Checksums, e2e encryption are > > examples) > > > > > > We have even succeeded in getting Pulsar adopted for some use cases, just > > because the complexity of upgrading from K’s old clients to new ones were > > costly enough to allow consideration of an alternative like Pulsar. The > > business cost of forcing a client upgrade can be significant, to the point > > of this being unviable for business. That just cannot be hand-waved over > > > > > > There have also been changes in storage formats(the ZK metadata change from > > text to binary is an example). But through all such changes, compatibility > > and upgradeability has been a given. There has never been a situation where > > a live Pulsar upgrade was not possible, and a coordinated client upgrade > > was mandatory. > > > > > > So the question should not be about whether “signifcant” changes should > > be made or not. Changes can be made and released in a way that breaks > > *business*, or they can be made in a way that lets businesses sail > > smoothly through that change. So the question is about how such changes > > gets rolled out. > > > > > > And to that question, my strong opinion is that any change that does not > > allow a live/rolling upgrade or rollback, or anything that forces a client > > to upgrade just to continue functioning, is a non-starter. All changes > > can be made in a compatible, phased manner, and in a way that does not > > penalise older versions ( older versions doing worse on new releases is > > also not an acceptable way of making changes) Changes can be made in a > > manner that make l A/B testing possible by the user, with limited risk, and > > then choosing to a not go back. It has all been done in Pulsar before. > > > > > > Would that be harder than just breaking stuff? Yes. But that is far more > > preferable than forcing users to take a hit. > > > > > > -joe > > > > On Sat, Oct 8, 2022 at 1:25 PM Rajan Dhabalia <rdhaba...@apache.org> > > wrote: > > > > > I would say first we should gather a list of changes which we want to > > > target and find out which improvements really need major version release. > > > We can take the Pulsar-1.0 to Pulsar-2.0 upgrade example to avoid major > > > interruption and impact on existing systems and still achieve our goal. > > So, > > > the first step is discovery of such features and then we can discuss how > > to > > > introduce them in Pulsar with minimum impact on existing systems. > > > > > > Thanks, > > > Rajan > > > > > > On Sat, Oct 8, 2022 at 1:05 PM Devin Bost <devin.b...@gmail.com> wrote: > > > > > > > I'm noticing some pushback on the idea of pre-emptively proposing any > > > kind > > > > of breaking upgrade that would necessitate cutting a 3.0 release. > > > > I do understand the concern about introducing a breaking change... For > > a > > > > distributed messaging application like Pulsar, if clients needed to be > > > > simultaneously upgraded with brokers, that could be extremely difficult > > > or > > > > infeasible for companies to coordinate without treating it like a > > > migration > > > > to a new technology. > > > > > > > > At the same time, do we want to be completely closed to the possibility > > > > that a breaking change could be required at some point in the future? > > If > > > a > > > > circumstance like that appears, those are the kinds of situations that > > > can > > > > lead to a fork. Are there certain kinds of breaking changes that are > > more > > > > acceptable than others? > > > > > > > > Also, if the forward looking plan is to never introduce breaking > > changes, > > > > when *would* we ever cut a Pulsar 3.x release? Do we have any criteria > > > on > > > > what kinds of changes would necessitate cutting a new major release but > > > > would still be considered acceptable by the community? > > > > > > > > -- > > > > Devin Bost > > > > Sent from mobile > > > > Cell: 801-400-4602 > > > > > > > > On Sat, Oct 8, 2022, 2:14 PM Rajan Dhabalia <rdhaba...@apache.org> > > > wrote: > > > > > > > > > This sounds like the current state of Apache Pulsar has a lot of > > issues > > > > and > > > > > it requires fundamental design changes to make it promising which is > > > > > definitely not true and I disagree with it. And I would be careful > > > > > comparing with Kafka as I still don't think the Kafka release has > > > > anything > > > > > to do with Pulsar's improvement. I would still recommend to list down > > > all > > > > > the changes at one place so we can bring everyone on the same page. > > > > discuss > > > > > as a community and we make sure existing usecases continue using > > Pulsar > > > > and > > > > > not try to find Pulsar alternatives with incorrect disruption > > > impression > > > > > and efforts they might have to put to upgrade or maintain pulsar. > > > > > > > > > > Thanks, > > > > > Rajan > > > > > > > > > > On Fri, Oct 7, 2022 at 7:49 PM Lari Hotari <lhot...@apache.org> > > wrote: > > > > > > > > > > > We could all have our own favorite names for this work. :) > > > > > > > > > > > > There's advice that you should disrupt yourself before someone > > > disrupts > > > > > > you. > > > > > > Shouldn't we follow that advice for Apache Pulsar? We can disrupt > > > > Pulsar > > > > > > together with our Apache hats on. The catch is that since we are > > > doing > > > > > > this, we will be able to learn and improve Pulsar so that we stay > > > ahead > > > > > of > > > > > > competition. Pulsar was long ways ahead of competition for so many > > > > years, > > > > > > but Kafka is finally catching up. Did Kafka surpass Pulsar in some > > > > > aspects > > > > > > with the recent 3.3 release, where Kraft became GA? That's a > > question > > > > > that > > > > > > many might be asking. Why wouldn't we rev up Pulsar's engine and > > show > > > > the > > > > > > tail lights to Kafka? > > > > > > > > > > > > We don't have to have deadlines or any restrictions like that right > > > > now. > > > > > > The sky's the limit. > > > > > > Linus Torvalds has written a book called "Just for fun". I got my > > > copy > > > > of > > > > > > this book signed by Linus himself in year 2000 at an event that the > > > > book > > > > > > publisher had organized in Finland. > > > > > > > > > > > > What if we did this "just for fun"? The intention could also be to > > > beat > > > > > > Kafka, but that could be a boring goal for many. What if we could > > > > unleash > > > > > > some talent that is among us and hasn't had a chance to show its > > full > > > > > > potential? Opensource is about joy. It is about welcoming everyone > > to > > > > > join. > > > > > > Opensource should be egoless, although we must all admit that we > > > don't > > > > > > succeed in that aspect. We must fight our biases. > > > > > > > > > > > > Jarek Potiuk explains the importance of being welcoming for success > > > at > > > > > > Apache, in a 3-minute YouTube interview: > > > > > > https://www.youtube.com/watch?v=Dx5kQnVFo7E > > > > > > This interview is about Jarek's blog post "Success at Apache: > > > Welcoming > > > > > > communities strengthens the Apache way": > > > > > > > > > > > > > > > > > > > > > > > > > > https://news.apache.org/foundation/entry/success-at-apache-welcoming-communities > > > > > > I was pleased to meet Jarek at ApacheCon among so many other > > > welcoming > > > > > > personalities of the Apache community and the Apache Pulsar > > > community. > > > > > > > > > > > > Goals have to be ambitious. What if we set the bar really high? > > > > > > Apache Pulsar with 10 million topics in a cluster? > > > > > > Why not go up to 100 million topics? > > > > > > Just for fun. :) > > > > > > > > > > > > -Lari > > > > > > > > > > > > On 2022/10/07 22:53:59 Matteo Merli wrote: > > > > > > > I actually disagree with the term "Pulsar Next Gen", because I > > > > haven't > > > > > > > seen any proposal for which that would make sense to me to be > > > called > > > > > > > so. > > > > > > > > > > > > > > Rajan: That's the whole point of breaking it down. If you > > > accumulate > > > > > > > many "big" changes it introduces a lot of risk for instabilities > > > and > > > > > > > incompatibilities. Breaking it down in multiple steps helps to > > see > > > > the > > > > > > > incremental changes and introduced them in a phased manner. > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Matteo Merli > > > > > > > <matteo.me...@gmail.com> > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 3:37 PM Rajan Dhabalia < > > > rdhaba...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Can we get the list of changes at one place which we are > > planning > > > > to > > > > > > get as > > > > > > > > part of 3.0. One thing I would like to see as a part of a major > > > > > > release, it > > > > > > > > CAN NOT impact existing usecases and users in any way which can > > > > force > > > > > > them > > > > > > > > to upgrade the client library. Applications using < 3.0 version > > > > > should > > > > > > > > continue getting all the client and server side enhancements > > and > > > > bug > > > > > > fixes. > > > > > > > > Failing to provide bug-fixes and features to client < 3.0 means > > > we > > > > > are > > > > > > > > forcing them to upgrade client version by putting efforts to > > > handle > > > > > all > > > > > > > > incompatibility. and that's something we should definitely > > > prevent > > > > > > because > > > > > > > > Apache Pulsar is used by many large scale business usecases and > > > we > > > > > > should > > > > > > > > accommodate and motivate them to continue using Apache Pulsar. > > > > > > > > I understand as a Pulsar community we should always try to > > > progress > > > > > and > > > > > > > > build better but not at the cost of losing or reducing the > > Apache > > > > > > Pulsar > > > > > > > > community. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Rajan > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 12:41 PM Lari Hotari < > > lhot...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > > > > > Thank you, Matteo. I agree that features should be delivered > > > > > > continuously > > > > > > > > > when that is possible. In this case, that might not apply. > > > > > > > > > > > > > > > > > > I also agree that calling this Pulsar 3.0 isn't necessarily > > > > aligned > > > > > > with > > > > > > > > > PIP-175 since an LTS release is when the major version is > > > bumped. > > > > > > I'm fine > > > > > > > > > in calling this "Pulsar Next Gen" or something that calls out > > > > that > > > > > > this is > > > > > > > > > planning for making a major leap in Pulsar. > > > > > > > > > > > > > > > > > > There are several unresolved issues with PIP-45 and the > > Pulsar > > > > Load > > > > > > > > > balancer. The previously referred email threads contain a lot > > > of > > > > > > context to > > > > > > > > > this. Resolving the issues efficiently will most likely > > result > > > in > > > > > > breaking > > > > > > > > > changes, which will be the reason why it deserves a major > > > version > > > > > > upgrade. > > > > > > > > > > > > > > > > > > We have discussed it before that it's crucial to have a path > > to > > > > > > migrate > > > > > > > > > users when there are breaking changes. This should be covered > > > in > > > > > any > > > > > > of the > > > > > > > > > solutions that are introduced. Optimally, users of Pulsar > > would > > > > be > > > > > > able to > > > > > > > > > upgrade seamlessly to Pulsar Next Gen / Pulsar 3.0, but > > rolling > > > > > back > > > > > > might > > > > > > > > > not be directly supported. > > > > > > > > > > > > > > > > > > I am welcoming everyone to join this planning for the Apache > > > > Pulsar > > > > > > Next > > > > > > > > > Gen architecture. Please check the first email in this thread > > > for > > > > > > details > > > > > > > > > of context, and start participating and contributing today. > > The > > > > > best > > > > > > way to > > > > > > > > > contribute is to participate in the email threads, since they > > > > > contain > > > > > > > > > details with better context. > > > > > > > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > > > > > On 2022/10/07 18:03:00 Matteo Merli wrote: > > > > > > > > > > Given the past experiences and the discussions that already > > > > > > happened > > > > > > > > > > around "PIP-175: Extend time based release process", the > > idea > > > > is > > > > > to > > > > > > > > > > detach the 3.0 from "big-features" items or "incompatible > > > > > changes". > > > > > > > > > > > > > > > > > > > > The changes are going to get included as they are ready, > > > within > > > > > > > > > > feature releases, and in a fully compatible way. We don't > > > need > > > > to > > > > > > > > > > group them together and create unnecessary risk for the > > > release > > > > > > > > > > schedule and the users. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Matteo Merli > > > > > > > > > > <matteo.me...@gmail.com> > > > > > > > > > > > > > > > > > > > > On Fri, Oct 7, 2022 at 10:47 AM Lari Hotari < > > > > lhot...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > Greeting from ApacheCon North America 2022 from New > > > Orleans! > > > > > > > > > > > We had a great conference with a dedicated Pulsar track. > > > > Thanks > > > > > > to all > > > > > > > > > presenters and everyone who attended. The talks weren't > > > recorded, > > > > > > but the > > > > > > > > > slides will be later on posted on the conference website [1]. > > > > > > > > > > > > > > > > > > > > > > At ApacheCon there were several presentations about "the > > > > Apache > > > > > > way" > > > > > > > > > and what that means in practice. Based on that, we all know > > > that > > > > no > > > > > > person > > > > > > > > > is nominated as the CTO of Apache Pulsar who decides on > > Pulsar > > > > 3.0 > > > > > > and when > > > > > > > > > that happens. It's us, the community, that serve that role > > > > > together. > > > > > > We > > > > > > > > > come together as individuals with the Apache hat on. Everyone > > > is > > > > > > equal in > > > > > > > > > the community, regardless of whether they are contributors, > > > > > > committers or > > > > > > > > > PMC members. > > > > > > > > > > > We welcome everyone to participate. The small detail > > about > > > > > voting > > > > > > > > > shouldn't stop anyone from participating in any aspects of > > the > > > > > > planning for > > > > > > > > > the roadmap. > > > > > > > > > > > > > > > > > > > > > > I'll like to get the discussions going for Pulsar 3.0. We > > > > don't > > > > > > need a > > > > > > > > > separate decision to start planning that. Please correct me > > if > > > > I'm > > > > > > wrong or > > > > > > > > > if you have a different opinion. > > > > > > > > > > > > > > > > > > > > > > There are a few previous discussion threads that are > > > related > > > > to > > > > > > Pulsar > > > > > > > > > 3.0 planning. > > > > > > > > > > > If you are interested in getting involved with Apache > > > Pulsar > > > > > 3.0 > > > > > > > > > planning, I think that it makes sense for you to read these > > > > threads > > > > > > > > > carefully and reply to them. Please also suggest what you > > think > > > > > > makes sense. > > > > > > > > > > > > > > > > > > > > > > PIP-45 related: > > > > > > > > > > > > https://lists.apache.org/thread/tvco1orf0hsyt59pjtfbwoq0vf6hfrcj > > > > > > > > > > > Pulsar Load balancer / namespace bundle related: > > > > > > > > > > > > > > > > https://lists.apache.org/thread/roohoc9h2gthvmd7t81do4hfjs2gphpk > > > > > > > > > > > renaming topics: > > > > > > > > > > > > > > > > https://lists.apache.org/thread/vrr75rrh4trqlp14objh3snlfvmzdrp2 > > > > > > > > > > > backpressure: > > > > > > > > > > > > https://lists.apache.org/thread/v7xy57qfzbhopoqbm75s6ng8xlhbr2q6 > > > > > > > > > > > > > > > > > > > > > > Long list of Metadata inconsistency issues by Zac > > Bentley: > > > > > > > > > > > https://github.com/apache/pulsar/issues/12555 > > > > > > > > > > > That would be a good starting point to understanding the > > > data > > > > > > > > > inconsistency issues related to current PIP-45 design. > > Perhaps > > > > > those > > > > > > could > > > > > > > > > be addressed already before Pulsar 3.0? > > > > > > > > > > > > > > > > > > > > > > I'm looking forward to everyone's participation in the > > > Apache > > > > > > Pulsar > > > > > > > > > 3.0 planning discussions. > > > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > > > > > > > > > > > > > -Lari > > > > > > > > > > > > > > > > > > > > > > 1 - https://www.apachecon.com/acna2022/schedule.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
