Hi Erik, Our current target is to have a preview in 3.7. This is subject to change though.
Best, David Le dim. 15 oct. 2023 à 13:02, Erik van Oosten <e.vanoos...@grons.nl.invalid> a écrit : > Thanks Philip, > > That sounds pretty good. Meanwhile I'll continue to study KIP-848. It is > a bit too much to digest in 1 go. > > Do you have a rough timeline for when the new consumer implementation > can be tried out in non-production environments? > > Kind regards, > Erik. > > > Op 14-10-2023 om 20:48 schreef Philip Nee: > > Hi Erik, > > > > Thanks for the KIP, again. I am also very much interested in the idea of > > this KIP, and I want to let you know that we are rewriting the kafka > > consumer using an event-driven approach, so I think the new impl would > make > > this KIP much easier to implement. In a nutshell, the network IO will > > become completely asynchronous to the application thread, so that the > > blocking APIs won't stale the network send/receive. In the new impl, the > > main role of poll are 1. check if there are any background events such as > > error or callback invocation, 2. notify the background that the user is > > polling, and 3. check if there is any data to return to the user. > > Because the background thread and application thread are inherently > working > > in an async fashion, it is possible to continue to process and commit > > during the revocation period; however, we have to be very careful with > the > > state of partition ownership as you mentioned in the KIP. > > > > Please keep an eye out on the new consumer implementation, in case if you > > are interested in digging in, it is the PrototypeKafkaConsumer module. > It > > is not fully functional but we are working full speed to get this to a > good > > state. > > > > Thanks - I am still reading to KIP and your previous KIP to see if I can > > make more constructive suggestions here. > > P > > > > On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten > > <e.vanoos...@grons.nl.invalid> wrote: > > > >> Hello David, > >> > >> Thanks, I am happy to hear we agree on the problem. All the tiny details > >> of an implementation are less important. > >> > >> I will read KIP-848 first to answer you question about its relation with > >> KIP-983. But for sure it makes sense to complete the implementation of > >> KIP-848 first. > >> > >> Kind regards, > >> Erik. > >> > >> > >> Op 13-10-2023 om 21:00 schreef David Jacot: > >>> Hi Erik, > >>> > >>> Thanks for the KIP. I haven’t fully read the KIP yet but I agree with > the > >>> weaknesses that you point out in it. I will continue to read it. > >>> > >>> For your information, we are working full speed on implementing KIP-848 > >>> while also changing the internal threading model of consumer. Those > >> changes > >>> are already extremely large so I would rather prefer to complete them > >>> before adding more on top of them. Moreover, I think that this KIP > should > >>> build on top of KIP-848 now. Would you agree with this? > >>> > >>> > >>> Best, > >>> David > >>> > >>> Le ven. 13 oct. 2023 à 20:44, Erik van Oosten<e.vanoos...@grons.nl > >> .invalid> > >>> a écrit : > >>> > >>>> Thanks Philip, > >>>> > >>>> No worries, I am not in a hurry. Knowing this is not forgotten is > enough > >>>> for me. If there is anything I can do to help the process please let > me > >>>> know. > >>>> > >>>> Kind regards, > >>>> Erik. > >>>> > >>>> > >>>> Op 13-10-2023 om 20:29 schreef Philip Nee: > >>>>> Hi Erik, > >>>>> > >>>>> Sorry for the delay, I have not finished reviewing the KIP, but I > also > >>>> have > >>>>> not forgotten about it! > >>>>> > >>>>> In general, KIP review process can be lengthy, so I think mailing > list > >> is > >>>>> the best bet to get the committer's attention. > >>>>> > >>>>> P > >>>>> > >>>>> On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten > >>>>> <e.vanoos...@grons.nl.invalid> wrote: > >>>>> > >>>>>> Hi client developers, > >>>>>> > >>>>>> The text is updated so that it is more clear that you can only use > >>>>>> auto-commit when doing synchronous processing (approach 1). I am > >>>>>> assuming that auto-commit commits whatever was consumed in the > >> previous > >>>>>> poll. > >>>>>> > >>>>>> I am wondering why this KIP doesn't get more attention. Is async > >>>>>> processing not something that the kafka client wants to support? > >>>>>> > >>>>>> Kind regards, > >>>>>> Erik. > >>>>>> > >>>>>> > >>>>>> Op 25-09-2023 om 18:17 schreef Erik van Oosten: > >>>>>>> Hi Viktor, > >>>>>>> > >>>>>>> Good questions! > >>>>>>> > >>>>>>> 1. Auto-commits would only work with approach 1 in the KIP. Any > async > >>>>>>> solution is incompatible with auto-commits. Do you think the text > >> will > >>>>>>> improve when this is mentioned? > >>>>>>> > >>>>>>> 2. That is entirely correct. If you use async commits you can await > >>>>>>> completion by doing a single sync commit with an empty offsets Map > >>>>>>> (this will work as of Kafka 3.6.0). > >>>>>>> > >>>>>>> Is there anything I can do to make the text clearer? > >>>>>>> > >>>>>>> Kind regards, > >>>>>>> Erik. > >>>>>>> > >>>>>>> > >>>>>>> Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: > >>>>>>>> Hi Erik, > >>>>>>>> > >>>>>>>> I'm still trying to wrap my head around the KIP, however I have a > >> few > >>>>>>>> questions that weren't clear to me regarding offset commits: > >>>>>>>> 1. Would auto-commits interfere with the behavior defined in your > >> KIP > >>>> or > >>>>>>>> would it work the same as manual commits? > >>>>>>>> 2. As I see you don't separate offset commits by whether they're > >> sync > >>>> or > >>>>>>>> async. For sync commits timing isn't really a problem but how > would > >>>> you > >>>>>>>> change work in case of async offset commits? There can be a few > >>>> caveats > >>>>>>>> there as you may not know whether a commit is finished or not > until > >>>> your > >>>>>>>> callback is called. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Viktor > >>>>>>>> > >>>>>>>> On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten > >>>>>>>> <e.vanoos...@grons.nl.invalid> wrote: > >>>>>>>> > >>>>>>>>> Hi all, > >>>>>>>>> > >>>>>>>>> I would like to start the discussion on KIP-983: Full speed async > >>>>>>>>> processing during rebalance [1]. > >>>>>>>>> > >>>>>>>>> The idea is that we can prevent the drop in throughput during a > >>>>>>>>> cooperative rebalance. > >>>>>>>>> > >>>>>>>>> I am curious to your ideas and comments. > >>>>>>>>> > >>>>>>>>> Kind regards, > >>>>>>>>> Erik. > >>>>>>>>> > >>>>>>>>> [1] > >>>>>>>>> > >>>>>>>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance > >> > -- > Erik van Oosten > e.vanoos...@grons.nl > https://day-to-day-stuff.blogspot.com > >
