Hello David,
Thanks, I am happy to hear we agree on the problem. All the tiny details
of an implementation are less important.
I will read KIP-848 first to answer you question about its relation with
KIP-983. But for sure it makes sense to complete the implementation of
KIP-848 first.
Kind regards,
Erik.
Op 13-10-2023 om 21:00 schreef David Jacot:
Hi Erik,
Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the
weaknesses that you point out in it. I will continue to read it.
For your information, we are working full speed on implementing KIP-848
while also changing the internal threading model of consumer. Those changes
are already extremely large so I would rather prefer to complete them
before adding more on top of them. Moreover, I think that this KIP should
build on top of KIP-848 now. Would you agree with this?
Best,
David
Le ven. 13 oct. 2023 à 20:44, Erik van Oosten<e.vanoos...@grons.nl.invalid>
a écrit :
Thanks Philip,
No worries, I am not in a hurry. Knowing this is not forgotten is enough
for me. If there is anything I can do to help the process please let me
know.
Kind regards,
Erik.
Op 13-10-2023 om 20:29 schreef Philip Nee:
Hi Erik,
Sorry for the delay, I have not finished reviewing the KIP, but I also
have
not forgotten about it!
In general, KIP review process can be lengthy, so I think mailing list is
the best bet to get the committer's attention.
P
On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten
<e.vanoos...@grons.nl.invalid> wrote:
Hi client developers,
The text is updated so that it is more clear that you can only use
auto-commit when doing synchronous processing (approach 1). I am
assuming that auto-commit commits whatever was consumed in the previous
poll.
I am wondering why this KIP doesn't get more attention. Is async
processing not something that the kafka client wants to support?
Kind regards,
Erik.
Op 25-09-2023 om 18:17 schreef Erik van Oosten:
Hi Viktor,
Good questions!
1. Auto-commits would only work with approach 1 in the KIP. Any async
solution is incompatible with auto-commits. Do you think the text will
improve when this is mentioned?
2. That is entirely correct. If you use async commits you can await
completion by doing a single sync commit with an empty offsets Map
(this will work as of Kafka 3.6.0).
Is there anything I can do to make the text clearer?
Kind regards,
Erik.
Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass:
Hi Erik,
I'm still trying to wrap my head around the KIP, however I have a few
questions that weren't clear to me regarding offset commits:
1. Would auto-commits interfere with the behavior defined in your KIP
or
would it work the same as manual commits?
2. As I see you don't separate offset commits by whether they're sync
or
async. For sync commits timing isn't really a problem but how would
you
change work in case of async offset commits? There can be a few
caveats
there as you may not know whether a commit is finished or not until
your
callback is called.
Thanks,
Viktor
On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten
<e.vanoos...@grons.nl.invalid> wrote:
Hi all,
I would like to start the discussion on KIP-983: Full speed async
processing during rebalance [1].
The idea is that we can prevent the drop in throughput during a
cooperative rebalance.
I am curious to your ideas and comments.
Kind regards,
Erik.
[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance
--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com
