Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Hi Erik, Our current target is to have a preview in 3.7. This is subject to change though. Best, David Le dim. 15 oct. 2023 à 13:02, Erik van Oosten a écrit : > Thanks Philip, > > That sounds pretty good. Meanwhile I'll continue to study KIP-848. It is > a bit too much to digest in 1 go. > > Do you have a rough timeline for when the new consumer implementation > can be tried out in non-production environments? > > Kind regards, > Erik. > > > Op 14-10-2023 om 20:48 schreef Philip Nee: > > Hi Erik, > > > > Thanks for the KIP, again. I am also very much interested in the idea of > > this KIP, and I want to let you know that we are rewriting the kafka > > consumer using an event-driven approach, so I think the new impl would > make > > this KIP much easier to implement. In a nutshell, the network IO will > > become completely asynchronous to the application thread, so that the > > blocking APIs won't stale the network send/receive. In the new impl, the > > main role of poll are 1. check if there are any background events such as > > error or callback invocation, 2. notify the background that the user is > > polling, and 3. check if there is any data to return to the user. > > Because the background thread and application thread are inherently > working > > in an async fashion, it is possible to continue to process and commit > > during the revocation period; however, we have to be very careful with > the > > state of partition ownership as you mentioned in the KIP. > > > > Please keep an eye out on the new consumer implementation, in case if you > > are interested in digging in, it is the PrototypeKafkaConsumer module. > It > > is not fully functional but we are working full speed to get this to a > good > > state. > > > > Thanks - I am still reading to KIP and your previous KIP to see if I can > > make more constructive suggestions here. > > P > > > > On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten > > wrote: > > > >> Hello David, > >> > >> Thanks, I am happy to hear we agree on the problem. All the tiny details > >> of an implementation are less important. > >> > >> I will read KIP-848 first to answer you question about its relation with > >> KIP-983. But for sure it makes sense to complete the implementation of > >> KIP-848 first. > >> > >> Kind regards, > >> Erik. > >> > >> > >> Op 13-10-2023 om 21:00 schreef David Jacot: > >>> Hi Erik, > >>> > >>> Thanks for the KIP. I haven’t fully read the KIP yet but I agree with > the > >>> weaknesses that you point out in it. I will continue to read it. > >>> > >>> For your information, we are working full speed on implementing KIP-848 > >>> while also changing the internal threading model of consumer. Those > >> changes > >>> are already extremely large so I would rather prefer to complete them > >>> before adding more on top of them. Moreover, I think that this KIP > should > >>> build on top of KIP-848 now. Would you agree with this? > >>> > >>> > >>> Best, > >>> David > >>> > >>> Le ven. 13 oct. 2023 à 20:44, Erik van Oosten >> .invalid> > >>> a écrit : > >>> > Thanks Philip, > > No worries, I am not in a hurry. Knowing this is not forgotten is > enough > for me. If there is anything I can do to help the process please let > me > know. > > Kind regards, > Erik. > > > Op 13-10-2023 om 20:29 schreef Philip Nee: > > Hi Erik, > > > > Sorry for the delay, I have not finished reviewing the KIP, but I > also > have > > not forgotten about it! > > > > In general, KIP review process can be lengthy, so I think mailing > list > >> is > > the best bet to get the committer's attention. > > > > P > > > > On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten > > wrote: > > > >> Hi client developers, > >> > >> The text is updated so that it is more clear that you can only use > >> auto-commit when doing synchronous processing (approach 1). I am > >> assuming that auto-commit commits whatever was consumed in the > >> previous > >> poll. > >> > >> I am wondering why this KIP doesn't get more attention. Is async > >> processing not something that the kafka client wants to support? > >> > >> Kind regards, > >> Erik. > >> > >> > >> Op 25-09-2023 om 18:17 schreef Erik van Oosten: > >>> Hi Viktor, > >>> > >>> Good questions! > >>> > >>> 1. Auto-commits would only work with approach 1 in the KIP. Any > async > >>> solution is incompatible with auto-commits. Do you think the text > >> will > >>> improve when this is mentioned? > >>> > >>> 2. That is entirely correct. If you use async commits you can await > >>> completion by doing a single sync commit with an empty offsets Map > >>> (this will work as of Kafka 3.6.0). > >>> > >>> Is there anything I can do to make the text clearer? > >>> > >>> Kind regards, > >>>
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Thanks Philip, That sounds pretty good. Meanwhile I'll continue to study KIP-848. It is a bit too much to digest in 1 go. Do you have a rough timeline for when the new consumer implementation can be tried out in non-production environments? Kind regards, Erik. Op 14-10-2023 om 20:48 schreef Philip Nee: Hi Erik, Thanks for the KIP, again. I am also very much interested in the idea of this KIP, and I want to let you know that we are rewriting the kafka consumer using an event-driven approach, so I think the new impl would make this KIP much easier to implement. In a nutshell, the network IO will become completely asynchronous to the application thread, so that the blocking APIs won't stale the network send/receive. In the new impl, the main role of poll are 1. check if there are any background events such as error or callback invocation, 2. notify the background that the user is polling, and 3. check if there is any data to return to the user. Because the background thread and application thread are inherently working in an async fashion, it is possible to continue to process and commit during the revocation period; however, we have to be very careful with the state of partition ownership as you mentioned in the KIP. Please keep an eye out on the new consumer implementation, in case if you are interested in digging in, it is the PrototypeKafkaConsumer module. It is not fully functional but we are working full speed to get this to a good state. Thanks - I am still reading to KIP and your previous KIP to see if I can make more constructive suggestions here. P On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten wrote: Hello David, Thanks, I am happy to hear we agree on the problem. All the tiny details of an implementation are less important. I will read KIP-848 first to answer you question about its relation with KIP-983. But for sure it makes sense to complete the implementation of KIP-848 first. Kind regards, Erik. Op 13-10-2023 om 21:00 schreef David Jacot: Hi Erik, Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the weaknesses that you point out in it. I will continue to read it. For your information, we are working full speed on implementing KIP-848 while also changing the internal threading model of consumer. Those changes are already extremely large so I would rather prefer to complete them before adding more on top of them. Moreover, I think that this KIP should build on top of KIP-848 now. Would you agree with this? Best, David Le ven. 13 oct. 2023 à 20:44, Erik van Oosten .invalid> a écrit : Thanks Philip, No worries, I am not in a hurry. Knowing this is not forgotten is enough for me. If there is anything I can do to help the process please let me know. Kind regards, Erik. Op 13-10-2023 om 20:29 schreef Philip Nee: Hi Erik, Sorry for the delay, I have not finished reviewing the KIP, but I also have not forgotten about it! In general, KIP review process can be lengthy, so I think mailing list is the best bet to get the committer's attention. P On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten wrote: Hi client developers, The text is updated so that it is more clear that you can only use auto-commit when doing synchronous processing (approach 1). I am assuming that auto-commit commits whatever was consumed in the previous poll. I am wondering why this KIP doesn't get more attention. Is async processing not something that the kafka client wants to support? Kind regards, Erik. Op 25-09-2023 om 18:17 schreef Erik van Oosten: Hi Viktor, Good questions! 1. Auto-commits would only work with approach 1 in the KIP. Any async solution is incompatible with auto-commits. Do you think the text will improve when this is mentioned? 2. That is entirely correct. If you use async commits you can await completion by doing a single sync commit with an empty offsets Map (this will work as of Kafka 3.6.0). Is there anything I can do to make the text clearer? Kind regards, Erik. Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: Hi Erik, I'm still trying to wrap my head around the KIP, however I have a few questions that weren't clear to me regarding offset commits: 1. Would auto-commits interfere with the behavior defined in your KIP or would it work the same as manual commits? 2. As I see you don't separate offset commits by whether they're sync or async. For sync commits timing isn't really a problem but how would you change work in case of async offset commits? There can be a few caveats there as you may not know whether a commit is finished or not until your callback is called. Thanks, Viktor On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten wrote: Hi all, I would like to start the discussion on KIP-983: Full speed async processing during rebalance [1]. The idea is that we can prevent the drop in throughput during a cooperative rebalance. I am curious to your ideas and
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Hi Erik, Thanks for the KIP, again. I am also very much interested in the idea of this KIP, and I want to let you know that we are rewriting the kafka consumer using an event-driven approach, so I think the new impl would make this KIP much easier to implement. In a nutshell, the network IO will become completely asynchronous to the application thread, so that the blocking APIs won't stale the network send/receive. In the new impl, the main role of poll are 1. check if there are any background events such as error or callback invocation, 2. notify the background that the user is polling, and 3. check if there is any data to return to the user. Because the background thread and application thread are inherently working in an async fashion, it is possible to continue to process and commit during the revocation period; however, we have to be very careful with the state of partition ownership as you mentioned in the KIP. Please keep an eye out on the new consumer implementation, in case if you are interested in digging in, it is the PrototypeKafkaConsumer module. It is not fully functional but we are working full speed to get this to a good state. Thanks - I am still reading to KIP and your previous KIP to see if I can make more constructive suggestions here. P On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten wrote: > Hello David, > > Thanks, I am happy to hear we agree on the problem. All the tiny details > of an implementation are less important. > > I will read KIP-848 first to answer you question about its relation with > KIP-983. But for sure it makes sense to complete the implementation of > KIP-848 first. > > Kind regards, > Erik. > > > Op 13-10-2023 om 21:00 schreef David Jacot: > > Hi Erik, > > > > Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the > > weaknesses that you point out in it. I will continue to read it. > > > > For your information, we are working full speed on implementing KIP-848 > > while also changing the internal threading model of consumer. Those > changes > > are already extremely large so I would rather prefer to complete them > > before adding more on top of them. Moreover, I think that this KIP should > > build on top of KIP-848 now. Would you agree with this? > > > > > > Best, > > David > > > > Le ven. 13 oct. 2023 à 20:44, Erik van Oosten .invalid> > > a écrit : > > > >> Thanks Philip, > >> > >> No worries, I am not in a hurry. Knowing this is not forgotten is enough > >> for me. If there is anything I can do to help the process please let me > >> know. > >> > >> Kind regards, > >> Erik. > >> > >> > >> Op 13-10-2023 om 20:29 schreef Philip Nee: > >>> Hi Erik, > >>> > >>> Sorry for the delay, I have not finished reviewing the KIP, but I also > >> have > >>> not forgotten about it! > >>> > >>> In general, KIP review process can be lengthy, so I think mailing list > is > >>> the best bet to get the committer's attention. > >>> > >>> P > >>> > >>> On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten > >>> wrote: > >>> > Hi client developers, > > The text is updated so that it is more clear that you can only use > auto-commit when doing synchronous processing (approach 1). I am > assuming that auto-commit commits whatever was consumed in the > previous > poll. > > I am wondering why this KIP doesn't get more attention. Is async > processing not something that the kafka client wants to support? > > Kind regards, > Erik. > > > Op 25-09-2023 om 18:17 schreef Erik van Oosten: > > Hi Viktor, > > > > Good questions! > > > > 1. Auto-commits would only work with approach 1 in the KIP. Any async > > solution is incompatible with auto-commits. Do you think the text > will > > improve when this is mentioned? > > > > 2. That is entirely correct. If you use async commits you can await > > completion by doing a single sync commit with an empty offsets Map > > (this will work as of Kafka 3.6.0). > > > > Is there anything I can do to make the text clearer? > > > > Kind regards, > > Erik. > > > > > > Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: > >> Hi Erik, > >> > >> I'm still trying to wrap my head around the KIP, however I have a > few > >> questions that weren't clear to me regarding offset commits: > >> 1. Would auto-commits interfere with the behavior defined in your > KIP > >> or > >> would it work the same as manual commits? > >> 2. As I see you don't separate offset commits by whether they're > sync > >> or > >> async. For sync commits timing isn't really a problem but how would > >> you > >> change work in case of async offset commits? There can be a few > >> caveats > >> there as you may not know whether a commit is finished or not until > >> your > >> callback is called. > >> > >> Thanks, > >> Viktor > >> > >> On
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Hello David, Thanks, I am happy to hear we agree on the problem. All the tiny details of an implementation are less important. I will read KIP-848 first to answer you question about its relation with KIP-983. But for sure it makes sense to complete the implementation of KIP-848 first. Kind regards, Erik. Op 13-10-2023 om 21:00 schreef David Jacot: Hi Erik, Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the weaknesses that you point out in it. I will continue to read it. For your information, we are working full speed on implementing KIP-848 while also changing the internal threading model of consumer. Those changes are already extremely large so I would rather prefer to complete them before adding more on top of them. Moreover, I think that this KIP should build on top of KIP-848 now. Would you agree with this? Best, David Le ven. 13 oct. 2023 à 20:44, Erik van Oosten a écrit : Thanks Philip, No worries, I am not in a hurry. Knowing this is not forgotten is enough for me. If there is anything I can do to help the process please let me know. Kind regards, Erik. Op 13-10-2023 om 20:29 schreef Philip Nee: Hi Erik, Sorry for the delay, I have not finished reviewing the KIP, but I also have not forgotten about it! In general, KIP review process can be lengthy, so I think mailing list is the best bet to get the committer's attention. P On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten wrote: Hi client developers, The text is updated so that it is more clear that you can only use auto-commit when doing synchronous processing (approach 1). I am assuming that auto-commit commits whatever was consumed in the previous poll. I am wondering why this KIP doesn't get more attention. Is async processing not something that the kafka client wants to support? Kind regards, Erik. Op 25-09-2023 om 18:17 schreef Erik van Oosten: Hi Viktor, Good questions! 1. Auto-commits would only work with approach 1 in the KIP. Any async solution is incompatible with auto-commits. Do you think the text will improve when this is mentioned? 2. That is entirely correct. If you use async commits you can await completion by doing a single sync commit with an empty offsets Map (this will work as of Kafka 3.6.0). Is there anything I can do to make the text clearer? Kind regards, Erik. Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: Hi Erik, I'm still trying to wrap my head around the KIP, however I have a few questions that weren't clear to me regarding offset commits: 1. Would auto-commits interfere with the behavior defined in your KIP or would it work the same as manual commits? 2. As I see you don't separate offset commits by whether they're sync or async. For sync commits timing isn't really a problem but how would you change work in case of async offset commits? There can be a few caveats there as you may not know whether a commit is finished or not until your callback is called. Thanks, Viktor On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten wrote: Hi all, I would like to start the discussion on KIP-983: Full speed async processing during rebalance [1]. The idea is that we can prevent the drop in throughput during a cooperative rebalance. I am curious to your ideas and comments. Kind regards, Erik. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Hi Erik, Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the weaknesses that you point out in it. I will continue to read it. For your information, we are working full speed on implementing KIP-848 while also changing the internal threading model of consumer. Those changes are already extremely large so I would rather prefer to complete them before adding more on top of them. Moreover, I think that this KIP should build on top of KIP-848 now. Would you agree with this? Best, David Le ven. 13 oct. 2023 à 20:44, Erik van Oosten a écrit : > Thanks Philip, > > No worries, I am not in a hurry. Knowing this is not forgotten is enough > for me. If there is anything I can do to help the process please let me > know. > > Kind regards, > Erik. > > > Op 13-10-2023 om 20:29 schreef Philip Nee: > > Hi Erik, > > > > Sorry for the delay, I have not finished reviewing the KIP, but I also > have > > not forgotten about it! > > > > In general, KIP review process can be lengthy, so I think mailing list is > > the best bet to get the committer's attention. > > > > P > > > > On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten > > wrote: > > > >> Hi client developers, > >> > >> The text is updated so that it is more clear that you can only use > >> auto-commit when doing synchronous processing (approach 1). I am > >> assuming that auto-commit commits whatever was consumed in the previous > >> poll. > >> > >> I am wondering why this KIP doesn't get more attention. Is async > >> processing not something that the kafka client wants to support? > >> > >> Kind regards, > >> Erik. > >> > >> > >> Op 25-09-2023 om 18:17 schreef Erik van Oosten: > >>> Hi Viktor, > >>> > >>> Good questions! > >>> > >>> 1. Auto-commits would only work with approach 1 in the KIP. Any async > >>> solution is incompatible with auto-commits. Do you think the text will > >>> improve when this is mentioned? > >>> > >>> 2. That is entirely correct. If you use async commits you can await > >>> completion by doing a single sync commit with an empty offsets Map > >>> (this will work as of Kafka 3.6.0). > >>> > >>> Is there anything I can do to make the text clearer? > >>> > >>> Kind regards, > >>> Erik. > >>> > >>> > >>> Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: > Hi Erik, > > I'm still trying to wrap my head around the KIP, however I have a few > questions that weren't clear to me regarding offset commits: > 1. Would auto-commits interfere with the behavior defined in your KIP > or > would it work the same as manual commits? > 2. As I see you don't separate offset commits by whether they're sync > or > async. For sync commits timing isn't really a problem but how would > you > change work in case of async offset commits? There can be a few > caveats > there as you may not know whether a commit is finished or not until > your > callback is called. > > Thanks, > Viktor > > On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten > wrote: > > > Hi all, > > > > I would like to start the discussion on KIP-983: Full speed async > > processing during rebalance [1]. > > > > The idea is that we can prevent the drop in throughput during a > > cooperative rebalance. > > > > I am curious to your ideas and comments. > > > > Kind regards, > >Erik. > > > > [1] > > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance > >> -- > >> Erik van Oosten > >> e.vanoos...@grons.nl > >> https://day-to-day-stuff.blogspot.com > >> > >> > -- > Erik van Oosten > e.vanoos...@grons.nl > https://day-to-day-stuff.blogspot.com > >
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Thanks Philip, No worries, I am not in a hurry. Knowing this is not forgotten is enough for me. If there is anything I can do to help the process please let me know. Kind regards, Erik. Op 13-10-2023 om 20:29 schreef Philip Nee: Hi Erik, Sorry for the delay, I have not finished reviewing the KIP, but I also have not forgotten about it! In general, KIP review process can be lengthy, so I think mailing list is the best bet to get the committer's attention. P On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten wrote: Hi client developers, The text is updated so that it is more clear that you can only use auto-commit when doing synchronous processing (approach 1). I am assuming that auto-commit commits whatever was consumed in the previous poll. I am wondering why this KIP doesn't get more attention. Is async processing not something that the kafka client wants to support? Kind regards, Erik. Op 25-09-2023 om 18:17 schreef Erik van Oosten: Hi Viktor, Good questions! 1. Auto-commits would only work with approach 1 in the KIP. Any async solution is incompatible with auto-commits. Do you think the text will improve when this is mentioned? 2. That is entirely correct. If you use async commits you can await completion by doing a single sync commit with an empty offsets Map (this will work as of Kafka 3.6.0). Is there anything I can do to make the text clearer? Kind regards, Erik. Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: Hi Erik, I'm still trying to wrap my head around the KIP, however I have a few questions that weren't clear to me regarding offset commits: 1. Would auto-commits interfere with the behavior defined in your KIP or would it work the same as manual commits? 2. As I see you don't separate offset commits by whether they're sync or async. For sync commits timing isn't really a problem but how would you change work in case of async offset commits? There can be a few caveats there as you may not know whether a commit is finished or not until your callback is called. Thanks, Viktor On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten wrote: Hi all, I would like to start the discussion on KIP-983: Full speed async processing during rebalance [1]. The idea is that we can prevent the drop in throughput during a cooperative rebalance. I am curious to your ideas and comments. Kind regards, Erik. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Hi Erik, Sorry for the delay, I have not finished reviewing the KIP, but I also have not forgotten about it! In general, KIP review process can be lengthy, so I think mailing list is the best bet to get the committer's attention. P On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten wrote: > Hi client developers, > > The text is updated so that it is more clear that you can only use > auto-commit when doing synchronous processing (approach 1). I am > assuming that auto-commit commits whatever was consumed in the previous > poll. > > I am wondering why this KIP doesn't get more attention. Is async > processing not something that the kafka client wants to support? > > Kind regards, > Erik. > > > Op 25-09-2023 om 18:17 schreef Erik van Oosten: > > Hi Viktor, > > > > Good questions! > > > > 1. Auto-commits would only work with approach 1 in the KIP. Any async > > solution is incompatible with auto-commits. Do you think the text will > > improve when this is mentioned? > > > > 2. That is entirely correct. If you use async commits you can await > > completion by doing a single sync commit with an empty offsets Map > > (this will work as of Kafka 3.6.0). > > > > Is there anything I can do to make the text clearer? > > > > Kind regards, > > Erik. > > > > > > Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: > >> Hi Erik, > >> > >> I'm still trying to wrap my head around the KIP, however I have a few > >> questions that weren't clear to me regarding offset commits: > >> 1. Would auto-commits interfere with the behavior defined in your KIP or > >> would it work the same as manual commits? > >> 2. As I see you don't separate offset commits by whether they're sync or > >> async. For sync commits timing isn't really a problem but how would you > >> change work in case of async offset commits? There can be a few caveats > >> there as you may not know whether a commit is finished or not until your > >> callback is called. > >> > >> Thanks, > >> Viktor > >> > >> On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten > >> wrote: > >> > >>> Hi all, > >>> > >>> I would like to start the discussion on KIP-983: Full speed async > >>> processing during rebalance [1]. > >>> > >>> The idea is that we can prevent the drop in throughput during a > >>> cooperative rebalance. > >>> > >>> I am curious to your ideas and comments. > >>> > >>> Kind regards, > >>> Erik. > >>> > >>> [1] > >>> > >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance > >>> > > > -- > Erik van Oosten > e.vanoos...@grons.nl > https://day-to-day-stuff.blogspot.com > >
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Hi client developers, The text is updated so that it is more clear that you can only use auto-commit when doing synchronous processing (approach 1). I am assuming that auto-commit commits whatever was consumed in the previous poll. I am wondering why this KIP doesn't get more attention. Is async processing not something that the kafka client wants to support? Kind regards, Erik. Op 25-09-2023 om 18:17 schreef Erik van Oosten: Hi Viktor, Good questions! 1. Auto-commits would only work with approach 1 in the KIP. Any async solution is incompatible with auto-commits. Do you think the text will improve when this is mentioned? 2. That is entirely correct. If you use async commits you can await completion by doing a single sync commit with an empty offsets Map (this will work as of Kafka 3.6.0). Is there anything I can do to make the text clearer? Kind regards, Erik. Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: Hi Erik, I'm still trying to wrap my head around the KIP, however I have a few questions that weren't clear to me regarding offset commits: 1. Would auto-commits interfere with the behavior defined in your KIP or would it work the same as manual commits? 2. As I see you don't separate offset commits by whether they're sync or async. For sync commits timing isn't really a problem but how would you change work in case of async offset commits? There can be a few caveats there as you may not know whether a commit is finished or not until your callback is called. Thanks, Viktor On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten wrote: Hi all, I would like to start the discussion on KIP-983: Full speed async processing during rebalance [1]. The idea is that we can prevent the drop in throughput during a cooperative rebalance. I am curious to your ideas and comments. Kind regards, Erik. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Hi Viktor, Good questions! 1. Auto-commits would only work with approach 1 in the KIP. Any async solution is incompatible with auto-commits. Do you think the text will improve when this is mentioned? 2. That is entirely correct. If you use async commits you can await completion by doing a single sync commit with an empty offsets Map (this will work as of Kafka 3.6.0). Is there anything I can do to make the text clearer? Kind regards, Erik. Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass: Hi Erik, I'm still trying to wrap my head around the KIP, however I have a few questions that weren't clear to me regarding offset commits: 1. Would auto-commits interfere with the behavior defined in your KIP or would it work the same as manual commits? 2. As I see you don't separate offset commits by whether they're sync or async. For sync commits timing isn't really a problem but how would you change work in case of async offset commits? There can be a few caveats there as you may not know whether a commit is finished or not until your callback is called. Thanks, Viktor On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten wrote: Hi all, I would like to start the discussion on KIP-983: Full speed async processing during rebalance [1]. The idea is that we can prevent the drop in throughput during a cooperative rebalance. I am curious to your ideas and comments. Kind regards, Erik. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com
Re: [DISCUSS] KIP-983: Full speed async processing during rebalance
Hi Erik, I'm still trying to wrap my head around the KIP, however I have a few questions that weren't clear to me regarding offset commits: 1. Would auto-commits interfere with the behavior defined in your KIP or would it work the same as manual commits? 2. As I see you don't separate offset commits by whether they're sync or async. For sync commits timing isn't really a problem but how would you change work in case of async offset commits? There can be a few caveats there as you may not know whether a commit is finished or not until your callback is called. Thanks, Viktor On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten wrote: > Hi all, > > I would like to start the discussion on KIP-983: Full speed async > processing during rebalance [1]. > > The idea is that we can prevent the drop in throughput during a > cooperative rebalance. > > I am curious to your ideas and comments. > > Kind regards, > Erik. > > [1] > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance > > > -- > Erik van Oosten > e.vanoos...@grons.nl > https://day-to-day-stuff.blogspot.com > >
