Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-15 Thread David Jacot
Hi Erik, Our current target is to have a preview in 3.7. This is subject to change though. Best, David Le dim. 15 oct. 2023 à 13:02, Erik van Oosten a écrit : > Thanks Philip, > > That sounds pretty good. Meanwhile I'll continue to study KIP-848. It is > a bit too much to digest in 1 go. > > D

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-15 Thread Erik van Oosten
Thanks Philip, That sounds pretty good. Meanwhile I'll continue to study KIP-848. It is a bit too much to digest in 1 go. Do you have a rough timeline for when the new consumer implementation can be tried out in non-production environments? Kind regards,     Erik. Op 14-10-2023 om 20:48 s

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-14 Thread Philip Nee
Hi Erik, Thanks for the KIP, again. I am also very much interested in the idea of this KIP, and I want to let you know that we are rewriting the kafka consumer using an event-driven approach, so I think the new impl would make this KIP much easier to implement. In a nutshell, the network IO will

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread Erik van Oosten
Hello David, Thanks, I am happy to hear we agree on the problem. All the tiny details of an implementation are less important. I will read KIP-848 first to answer you question about its relation with KIP-983. But for sure it makes sense to complete the implementation of KIP-848 first. Kind

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread David Jacot
Hi Erik, Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the weaknesses that you point out in it. I will continue to read it. For your information, we are working full speed on implementing KIP-848 while also changing the internal threading model of consumer. Those changes a

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread Erik van Oosten
Thanks Philip, No worries, I am not in a hurry. Knowing this is not forgotten is enough for me. If there is anything I can do to help the process please let me know. Kind regards,     Erik. Op 13-10-2023 om 20:29 schreef Philip Nee: Hi Erik, Sorry for the delay, I have not finished review

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread Philip Nee
Hi Erik, Sorry for the delay, I have not finished reviewing the KIP, but I also have not forgotten about it! In general, KIP review process can be lengthy, so I think mailing list is the best bet to get the committer's attention. P On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten wrote: > Hi

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread Erik van Oosten
Hi client developers, The text is updated so that it is more clear that you can only use auto-commit when doing synchronous processing (approach 1). I am assuming that auto-commit commits whatever was consumed in the previous poll. I am wondering why this KIP doesn't get more attention. Is a

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-09-25 Thread Erik van Oosten
Hi Viktor, Good questions! 1. Auto-commits would only work with approach 1 in the KIP. Any async solution is incompatible with auto-commits. Do you think the text will improve when this is mentioned? 2. That is entirely correct. If you use async commits you can await completion by doing a s

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-09-25 Thread Viktor Somogyi-Vass
Hi Erik, I'm still trying to wrap my head around the KIP, however I have a few questions that weren't clear to me regarding offset commits: 1. Would auto-commits interfere with the behavior defined in your KIP or would it work the same as manual commits? 2. As I see you don't separate offset commi

[DISCUSS] KIP-983: Full speed async processing during rebalance

2023-09-23 Thread Erik van Oosten
Hi all, I would like to start the discussion on KIP-983: Full speed async processing during rebalance [1]. The idea is that we can prevent the drop in throughput during a cooperative rebalance. I am curious to your ideas and comments. Kind regards,     Erik. [1] https://cwiki.apache.org/