PIP-371 documentation: https://github.com/apache/pulsar/pull/23143
PIP-371 implementation code: https://github.com/apache/pulsar/pull/23194 This is a native implementation of the request-reply model in pulsar. Although at present, there is no support from many people in the email discussion. But I think things still have to have a result. I am sorry that I am just a personal developer. I have not done pulsar-related work in the company, and I have no business scenarios to test. The implementation code is as follows, if you are interested in this request-reply synchronization model, or if you have the usage scenario of this function. Can help with internal testing. This is just a simple implementation, there are many exception handling or the metrics is not perfect. You are still welcome to discuss your questions on the mailing list or pr. Thanks, sinan SiNan Liu <liusinan1...@gmail.com> 于2024年8月9日周五 15:00写道： > > We all love good performance, however that might not be the most > important matter in all cases. "Such a call would have a long link and lose > performance." Have you measured the performance of the existing solution? > What is your target performance? > > I understand that it's possible to optimize the Pulsar broker for RPC, > but I don't think that there are many that would support this type of > proposal in the Apache Pulsar project. Instead of optimizing the broker for > RPC, I'd recommend optimizing the client programming model that you are > looking for and building that on top of Pulsar, assuming that you would > like to have a RPC programming framework on top of Pulsar. > > I believe that this can be achieved purely in a client side library > without modifying the Pulsar broker. > > I don't want to limit your opportunity to contribute in any way. However > in Apache projects, you would need support from the community to reach > consensus to change the course of the project to something else than what > it currently is. There's really not much value in trying to convince the > project to add a RPC library to Apache Pulsar project. > > What I'd recommend instead is to show how this could be done in a > separate library. If the Pulsar community users find this valuable for > their needs, they will adopt it. There will be plenty of ways to make the > Pulsar users aware of this new framework when it becomes available. Perhaps > you could find other community members who would like to contribute to such > library? > > Yes, if you use the existing message send function to implement the > request-reply synchronization model. It's like > https://github.com/streamnative/pulsar-recipes/tree/main/rpc. The only > solution is to use two topics to receive and process request message and > reply result. This is also the original intention of my proposal. > > Therefore, the proposal makes another design for the only solution in the > industry at present. > > I hope more people will read and pay attention to this proposal. If there > is such a demand or interest in this proposal, it can be discussed. > > > Thanks, > sinan > > > Lari Hotari <lhot...@apache.org>于2024年8月9日 周五13:28写道： > >> On 2024/08/09 01:10:11 SiNan Liu wrote: >> > Thank you for your comments. >> > >> > I took a look at this. There are a few questions. >> > >> > 1. Compared with my proposal to directly add new functions in >> > pulsar-client, the implementation in this repo is through two topics, >> one >> > request topic and one reply topic. The producer sends the message to the >> > request topic, the consumer processes it, the consumer sends it to the >> > reply topic, and the producer subscribes to the reply topic. Such a call >> > would have a long link and lose performance. This is also the downside >> of >> > the external repo wrapping the pulsar-client api directly, which it can >> > only do. >> >> We all love good performance, however that might not be the most >> important matter in all cases. "Such a call would have a long link and lose >> performance." Have you measured the performance of the existing solution? >> What is your target performance? >> >> > And that's not in my proposal. After the consumer processes the request >> > Message, it passes it to the broker through a new ack implementation, >> and >> > the broker passes the reply result directly to the client (producer). >> >> I understand that it's possible to optimize the Pulsar broker for RPC, >> but I don't think that there are many that would support this type of >> proposal in the Apache Pulsar project. Instead of optimizing the broker for >> RPC, I'd recommend optimizing the client programming model that you are >> looking for and building that on top of Pulsar, assuming that you would >> like to have a RPC programming framework on top of Pulsar. >> >> > 2. In my proposal, request and the existing send function of pulsar can >> be >> > mixed. This means that the user can choose, and the call to the server >> side >> > (consumer) can be asynchronous or synchronous, which is controlled by >> the >> > user flexibly. >> >> I believe that this can be achieved purely in a client side library >> without modifying the Pulsar broker. >> >> > >> > >> > 3. The use examples of the new api are provided in the detailed design >> of >> > my proposal. Users use it just like pulsar's existing send >> functionality, >> > without the need to introduce additional encapsulation implementations. >> >> I don't support this statement, "without the need to introduce >> additional encapsulation implementations". This library doesn't directly >> have to be part of Pulsar core or Pulsar Java client, but it could be an >> additional library. >> >> > 4. Of course, the changes made in this proposal do not actually change >> the >> > existing pulsar api, but rather wrap it internally. >> >> Yes, this is the way to go. >> >> > For example, request api. Call the existing send function directly, get >> the >> > request MessageID, and wait for the reply response within the timeout >> > period. The new ack api simply sends a reply result to the broker, which >> > then sends it to the client, after which the existing ack api is called >> > directly. >> >> I don't want to limit your opportunity to contribute in any way. However >> in Apache projects, you would need support from the community to reach >> consensus to change the course of the project to something else than what >> it currently is. There's really not much value in trying to convince the >> project to add a RPC library to Apache Pulsar project. >> >> What I'd recommend instead is to show how this could be done in a >> separate library. If the Pulsar community users find this valuable for >> their needs, they will adopt it. There will be plenty of ways to make the >> Pulsar users aware of this new framework when it becomes available. Perhaps >> you could find other community members who would like to contribute to such >> library? >> >> -Lari >> >> > >> > >> > Thanks, >> > sinan >> > >> > >> > Lari Hotari <lhot...@apache.org>于2024年8月8日 周四23:42写道： >> > >> > > Thanks for the proposal! >> > > >> > > Instead of modifying the existing Pulsar client library, I think it's >> > > better to create a separate library. The request-reply / rpc model >> can be >> > > implemented on top of the Pulsar client instead of making the support >> for >> > > request-reply a first class citizen of the client API. >> > > There's an example of a Pulsar based RPC framework at >> > > https://github.com/streamnative/pulsar-recipes/tree/main/rpc . >> > > >> > > We'd like to keep the Pulsar client library API as stable as possible >> and >> > > without increasing the size of it. The RPC library could be >> considered as a >> > > plugin or an add-on if we were to make it available in the Apache >> Pulsar >> > > project. I'll be happy to support this type of design. >> > > >> > > -Lari >> > > >> > > >> > > On 2024/08/08 15:09:17 SiNan Liu wrote: >> > > > Hi all. >> > > > >> > > > I drafted a proposal to support for request-reply model that >> implements >> > > RPC >> > > > calls in pulsar. >> > > > >> > > > https://github.com/apache/pulsar/pull/23143 >> > > > I'm looking forward to hearing from you. >> > > > >> > > > >> > > > Thanks, >> > > > sinan >> > > > >> > > >> > >> >
