Thanks Philip,
That sounds pretty good. Meanwhile I'll continue to study KIP-848. It is
a bit too much to digest in 1 go.
Do you have a rough timeline for when the new consumer implementation
can be tried out in non-production environments?
Kind regards,
Erik.
Op 14-10-2023 om 20:48 schreef Philip Nee:
Hi Erik,
Thanks for the KIP, again. I am also very much interested in the idea of
this KIP, and I want to let you know that we are rewriting the kafka
consumer using an event-driven approach, so I think the new impl would make
this KIP much easier to implement. In a nutshell, the network IO will
become completely asynchronous to the application thread, so that the
blocking APIs won't stale the network send/receive. In the new impl, the
main role of poll are 1. check if there are any background events such as
error or callback invocation, 2. notify the background that the user is
polling, and 3. check if there is any data to return to the user.
Because the background thread and application thread are inherently working
in an async fashion, it is possible to continue to process and commit
during the revocation period; however, we have to be very careful with the
state of partition ownership as you mentioned in the KIP.
Please keep an eye out on the new consumer implementation, in case if you
are interested in digging in, it is the PrototypeKafkaConsumer module. It
is not fully functional but we are working full speed to get this to a good
state.
Thanks - I am still reading to KIP and your previous KIP to see if I can
make more constructive suggestions here.
P
On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten
<e.vanoos...@grons.nl.invalid> wrote:
Hello David,
Thanks, I am happy to hear we agree on the problem. All the tiny details
of an implementation are less important.
I will read KIP-848 first to answer you question about its relation with
KIP-983. But for sure it makes sense to complete the implementation of
KIP-848 first.
Kind regards,
Erik.
Op 13-10-2023 om 21:00 schreef David Jacot:
Hi Erik,
Thanks for the KIP. I haven’t fully read the KIP yet but I agree with the
weaknesses that you point out in it. I will continue to read it.
For your information, we are working full speed on implementing KIP-848
while also changing the internal threading model of consumer. Those
changes
are already extremely large so I would rather prefer to complete them
before adding more on top of them. Moreover, I think that this KIP should
build on top of KIP-848 now. Would you agree with this?
Best,
David
Le ven. 13 oct. 2023 à 20:44, Erik van Oosten<e.vanoos...@grons.nl
.invalid>
a écrit :
Thanks Philip,
No worries, I am not in a hurry. Knowing this is not forgotten is enough
for me. If there is anything I can do to help the process please let me
know.
Kind regards,
Erik.
Op 13-10-2023 om 20:29 schreef Philip Nee:
Hi Erik,
Sorry for the delay, I have not finished reviewing the KIP, but I also
have
not forgotten about it!
In general, KIP review process can be lengthy, so I think mailing list
is
the best bet to get the committer's attention.
P
On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten
<e.vanoos...@grons.nl.invalid> wrote:
Hi client developers,
The text is updated so that it is more clear that you can only use
auto-commit when doing synchronous processing (approach 1). I am
assuming that auto-commit commits whatever was consumed in the
previous
poll.
I am wondering why this KIP doesn't get more attention. Is async
processing not something that the kafka client wants to support?
Kind regards,
Erik.
Op 25-09-2023 om 18:17 schreef Erik van Oosten:
Hi Viktor,
Good questions!
1. Auto-commits would only work with approach 1 in the KIP. Any async
solution is incompatible with auto-commits. Do you think the text
will
improve when this is mentioned?
2. That is entirely correct. If you use async commits you can await
completion by doing a single sync commit with an empty offsets Map
(this will work as of Kafka 3.6.0).
Is there anything I can do to make the text clearer?
Kind regards,
Erik.
Op 25-09-2023 om 17:04 schreef Viktor Somogyi-Vass:
Hi Erik,
I'm still trying to wrap my head around the KIP, however I have a
few
questions that weren't clear to me regarding offset commits:
1. Would auto-commits interfere with the behavior defined in your
KIP
or
would it work the same as manual commits?
2. As I see you don't separate offset commits by whether they're
sync
or
async. For sync commits timing isn't really a problem but how would
you
change work in case of async offset commits? There can be a few
caveats
there as you may not know whether a commit is finished or not until
your
callback is called.
Thanks,
Viktor
On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten
<e.vanoos...@grons.nl.invalid> wrote:
Hi all,
I would like to start the discussion on KIP-983: Full speed async
processing during rebalance [1].
The idea is that we can prevent the drop in throughput during a
cooperative rebalance.
I am curious to your ideas and comments.
Kind regards,
Erik.
[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance
--
Erik van Oosten
e.vanoos...@grons.nl
https://day-to-day-stuff.blogspot.com
