Hi Erik, Thanks for the KIP, again. I am also very much interested in the idea of this KIP, and I want to let you know that we are rewriting the kafka consumer using an event-driven approach, so I think the new impl would make this KIP much easier to implement. In a nutshell, the network IO will become completely asynchronous to the application thread, so that the blocking APIs won't stale the network send/receive. In the new impl, the main role of poll are 1. check if there are any background events such as error or callback invocation, 2. notify the background that the user is polling, and 3. check if there is any data to return to the user. Because the background thread and application thread are inherently working in an async fashion, it is possible to continue to process and commit during the revocation period; however, we have to be very careful with the state of partition ownership as you mentioned in the KIP.
Please keep an eye out on the new consumer implementation, in case if you are interested in digging in, it is the PrototypeKafkaConsumer module. It is not fully functional but we are working full speed to get this to a good state. Thanks - I am still reading to KIP and your previous KIP to see if I can make more constructive suggestions here. P On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten <e.vanoos...@grons.nl.invalid> wrote: > Hello David, > > Thanks, I am happy to hear we agree on the problem. All the tiny details > of an implementation are less important. > > I will read KIP-848 first to answer you question about its relation with > KIP-983. But for sure it makes sense to complete the implementation of > KIP-848 first. > > Kind regards, > Erik. > > > Op 13-10-2023 om 21:00 schreef David Jacot: > > Hi Erik, > > > > Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the > > weaknesses that you point out in it. I will continue to read it. > > > > For your information, we are working full speed on implementing KIP-848 > > while also changing the internal threading model of consumer. Those > changes > > are already extremely large so I would rather prefer to complete them > > before adding more on top of them. Moreover, I think that this KIP should > > build on top of KIP-848 now. Would you agree with this? > > > > > > Best, > > David > > > > Le ven. 13 oct. 2023 à 20:44, Erik van Oosten<e.vanoos...@grons.nl > .invalid> > > a écrit : > > > >> Thanks Philip, > >> > >> No worries, I am not in a hurry. Knowing this is not forgotten is enough > >> for me. If there is anything I can do to help the process please let me > >> know. > >> > >> Kind regards, > >> Erik. > >> > >> > >> Op 13-10-2023 om 20:29 schreef Philip Nee: > >>> Hi Erik, > >>> > >>> Sorry for the delay, I have not finished reviewing the KIP, but I also > >> have > >>> not forgotten about it! > >>> > >>> In general, KIP review process can be lengthy, so I think mailing list > is > >>> the best bet to get the committer's attention. > >>> > >>> P > >>> > >>> On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten > >>> <e.vanoos...@grons.nl.invalid> wrote: > >>> > >>>> Hi client developers, > >>>> > >>>> The text is updated so that it is more clear that you can only use > >>>> auto-commit when doing synchronous processing (approach 1). I am > >>>> assuming that auto-commit commits whatever was consumed in the > previous > >>>> poll. > >>>> > >>>> I am wondering why this KIP doesn't get more attention. Is async > >>>> processing not something that the kafka client wants to support? > >>>> > >>>> Kind regards, > >>>> Erik. > >>>> > >>>> > >>>> Op 25-09-2023 om 18:17 schreef Erik van Oosten: > >>>>> Hi Viktor, > >>>>> > >>>>> Good questions! > >>>>> > >>>>> 1. Auto-commits would only work with approach 1 in the KIP. Any async > >>>>> solution is incompatible with auto-commits. Do you think the text > will > >>>>> improve when this is mentioned? > >>>>> > >>>>> 2. That is entirely correct. If you use async commits you can await > >>>>> completion by doing a single sync commit with an empty offsets Map > >>>>> (this will work as of Kafka 3.6.0). > >>>>> > >>>>> Is there anything I can do to make the text clearer? > >>>>> > >>>>> Kind regards, > >>>>> Erik. > >>>>> > >>>>> > >>>>> Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: > >>>>>> Hi Erik, > >>>>>> > >>>>>> I'm still trying to wrap my head around the KIP, however I have a > few > >>>>>> questions that weren't clear to me regarding offset commits: > >>>>>> 1. Would auto-commits interfere with the behavior defined in your > KIP > >> or > >>>>>> would it work the same as manual commits? > >>>>>> 2. As I see you don't separate offset commits by whether they're > sync > >> or > >>>>>> async. For sync commits timing isn't really a problem but how would > >> you > >>>>>> change work in case of async offset commits? There can be a few > >> caveats > >>>>>> there as you may not know whether a commit is finished or not until > >> your > >>>>>> callback is called. > >>>>>> > >>>>>> Thanks, > >>>>>> Viktor > >>>>>> > >>>>>> On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten > >>>>>> <e.vanoos...@grons.nl.invalid> wrote: > >>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> I would like to start the discussion on KIP-983: Full speed async > >>>>>>> processing during rebalance [1]. > >>>>>>> > >>>>>>> The idea is that we can prevent the drop in throughput during a > >>>>>>> cooperative rebalance. > >>>>>>> > >>>>>>> I am curious to your ideas and comments. > >>>>>>> > >>>>>>> Kind regards, > >>>>>>> Erik. > >>>>>>> > >>>>>>> [1] > >>>>>>> > >>>>>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance > > -- > Erik van Oosten > e.vanoos...@grons.nl > https://day-to-day-stuff.blogspot.com >
