Hi client developers,
The text is updated so that it is more clear that you can only use
auto-commit when doing synchronous processing (approach 1). I am
assuming that auto-commit commits whatever was consumed in the previous
poll.
I am wondering why this KIP doesn't get more attention. Is async
processing not something that the kafka client wants to support?
Kind regards,
Erik.
Op 25-09-2023 om 18:17 schreef Erik van Oosten:
Hi Viktor,
Good questions!
1. Auto-commits would only work with approach 1 in the KIP. Any async
solution is incompatible with auto-commits. Do you think the text will
improve when this is mentioned?
2. That is entirely correct. If you use async commits you can await
completion by doing a single sync commit with an empty offsets Map
(this will work as of Kafka 3.6.0).
Is there anything I can do to make the text clearer?
Kind regards,
Erik.
Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass:
Hi Erik,
I'm still trying to wrap my head around the KIP, however I have a few
questions that weren't clear to me regarding offset commits:
1. Would auto-commits interfere with the behavior defined in your KIP or
would it work the same as manual commits?
2. As I see you don't separate offset commits by whether they're sync or
async. For sync commits timing isn't really a problem but how would you
change work in case of async offset commits? There can be a few caveats
there as you may not know whether a commit is finished or not until your
callback is called.
Thanks,
Viktor
On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten
<e.vanoos...@grons.nl.invalid> wrote:
Hi all,
I would like to start the discussion on KIP-983: Full speed async
processing during rebalance [1].
The idea is that we can prevent the drop in throughput during a
cooperative rebalance.
I am curious to your ideas and comments.
Kind regards,
Erik.
[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance
--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com
