Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-15 Thread Erik van Oosten
am still reading to KIP and your previous KIP to see if I can make more constructive suggestions here. P On Fri, Oct 13, 2023 at 11:54 PM Erik van Oosten wrote: Hello David, Thanks, I am happy to hear we agree on the problem. All the tiny details of an implementation are less important. I

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-14 Thread Erik van Oosten
. 2023 à 20:44, Erik van Oosten a écrit : Thanks Philip, No worries, I am not in a hurry. Knowing this is not forgotten is enough for me. If there is anything I can do to help the process please let me know. Kind regards, Erik. Op 13-10-2023 om 20:29 schreef Philip Nee: Hi Erik, Sorry

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread Erik van Oosten
reviewing the KIP, but I also have not forgotten about it! In general, KIP review process can be lengthy, so I think mailing list is the best bet to get the committer's attention. P On Fri, Oct 13, 2023 at 10:55 AM Erik van Oosten wrote: Hi client developers, The text is updated so that it is more

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-10-13 Thread Erik van Oosten
. Is async processing not something that the kafka client wants to support? Kind regards,     Erik. Op 25-09-2023 om 18:17 schreef Erik van Oosten: Hi Viktor, Good questions! 1. Auto-commits would only work with approach 1 in the KIP. Any async solution is incompatible with auto-commits. Do you

Re: [DISCUSS] KIP-983: Full speed async processing during rebalance

2023-09-25 Thread Erik van Oosten
but how would you change work in case of async offset commits? There can be a few caveats there as you may not know whether a commit is finished or not until your callback is called. Thanks, Viktor On Sat, Sep 23, 2023 at 4:00 PM Erik van Oosten wrote: Hi all, I would like to start

[DISCUSS] KIP-983: Full speed async processing during rebalance

2023-09-23 Thread Erik van Oosten
/confluence/display/KAFKA/KIP-983%3A+Full+speed+async+processing+during+rebalance -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com

[jira] [Resolved] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-08-05 Thread Erik van Oosten (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik van Oosten resolved KAFKA-14972. - Resolution: Won't Fix > Make KafkaConsumer usable in async runti

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-08-03 Thread Erik van Oosten
my energy elsewhere. Kind regards,     Erik. Op 24-07-2023 om 18:12 schreef Erik van Oosten: Hello Xiangyuan LI, I am not familiar with coroutines, nor with Kotlin. You will have to work with the documentation: https://kotlinlang.org/docs/coroutines-overview.html However, I am familiar

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-24 Thread Erik van Oosten
aybe you need a more elaborate description to demonstrate how coroutine code run. Erik van Oosten 于2023年7月23日周日 17:47写道: Hi David, > Could you elaborate a bit more on why the callbacks must be ran in another thread vs in the invoker thread? I have been thinking on how to explai

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-23 Thread Erik van Oosten
KIP based on the discussion. Le sam. 22 juil. 2023 à 18:34, Erik van Oosten a écrit : Colin, Matthias, Chris, I have expanded the use case description in KIP-944. I hope it is more clear what we're trying to achieve. https://cwiki.apache.org/confluence/x/chw0Dw Kind regards, Erik. Op 22

[DISCUSS] KIP-957 Support async runtimes in consumer

2023-07-22 Thread Erik van Oosten
. The wiki page for KIP-957 contains more details https://cwiki.apache.org/confluence/x/lY6zDw This is a call for discussion. If possible I would like to include this change in Kafka 3.6. Any questions, comments, ideas and other additions are welcome! Kind regards,     Erik. -- Erik van Oosten

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-22 Thread Erik van Oosten
Colin, Matthias, Chris, I have expanded the use case description in KIP-944. I hope it is more clear what we're trying to achieve. https://cwiki.apache.org/confluence/x/chw0Dw Kind regards,     Erik. Op 22-07-2023 om 17:23 schreef Erik van Oosten: Hello Chris, Thanks for elaborating

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-22 Thread Erik van Oosten
the visibility of these methods from private to protected, it would be possible for subclasses to implement the proposal in KIP-944, without any KIPs or other changes to the official Java clients library. Best, Chris On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten wrote: Hi Matthias, I am getting a bit

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-22 Thread Erik van Oosten
sal in KIP-944, without any KIPs or other changes to the official Java clients library. Best, Chris On Sat, Jul 22, 2023 at 4:24 AM Erik van Oosten wrote: Hi Matthias, I am getting a bit frustrated here. All the concerns and questions I have seen so far are addressed in KIP-944. Please let me k

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-22 Thread Erik van Oosten
the radio silence. It gets like that sometimes, and I apologize for my lack of feedback. I’d personally like to see this lively exchange move over to the DISCUSS thread you’d created before. Thanks, Kirk On Jul 14, 2023, at 1:33 AM, Erik van Oosten wrote: Hi Colin, The way I understood

Re: [DISCUSS] KIP-944 Support async runtimes in consumer

2023-07-16 Thread Erik van Oosten
that are described in the KIP. Kind regards,     Erik. Op 30-06-2023 om 07:56 schreef Erik van Oosten: [This is a resend with the correct KIP number.] Hello developers of the Java based consumer, I submitted https://github.com/apache/kafka/pull/13914 to fix a long standing problem that the Kafka

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-14 Thread Erik van Oosten
any unsafe access. I think this can be demonstrated via diagrams and some code in the KIP. Thanks, P On Sat, Jul 8, 2023 at 12:28 PM Erik van Oosten wrote: Hello Colin, >> In KIP-944, the callback thread can only delegate to another thread after reading from and writing to a thre

Re: KafkaConsumer refactor proposal

2023-07-13 Thread Erik van Oosten
-944) because, concurrent invocations are simply no longer needed. (Of course, if backward compatibility is a goal, not all of these things can be done.) Curious to your thoughts and kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-12 Thread Erik van Oosten
-Threadsafety Kind regards,     Erik. Op 12-07-2023 om 13:24 schreef Erik van Oosten: Thanks Philip, > I think this can be demonstrated via diagrams and some code in the KIP. There are some diagrams in KIP-944. How can they be improved? I will add some code to address the concerns around mem

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-12 Thread Erik van Oosten
we are running inside the callback of the rebalance listener. The only way to execute something and also have a timeout on it is to run the something on another thread. Kind regards, Erik. Op 08-07-2023 om 19:17 schreef Colin McCabe: On Sat, Jul 8, 2023, at 02:41, Erik van Oosten wrote: Hi C

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-08 Thread Erik van Oosten
t we are running inside the callback of the rebalance listener. The only way to execute something and also have a timeout on it is to run the something on another thread. Kind regards,     Erik. Op 08-07-2023 om 19:17 schreef Colin McCabe: On Sat, Jul 8, 2023, at 02:41, Erik van Oosten wrote: H

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-08 Thread Erik van Oosten
not get a quick response. However, overall I think this one needs a lot more discussion before getting anywhere near a vote. I will leave a -1 just as a procedural step. Maybe some of the people working in the client area can also chime in. best, Colin On Thu, Jul 6, 2023, at 12:02, Erik v

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

2023-07-06 Thread Erik van Oosten
-2023 om 11:38 schreef Erik van Oosten: Hello all, I'd like to call a vote on KIP-944 Support async runtimes in consumer. It has has been 'under discussion' for 7 days now. 'Under discussion' between quotes, because there were 0 comments so far. I hope the KIP is clear! KIP description: https

[VOTE] KIP-944 Support async runtimes in consumer

2023-07-05 Thread Erik van Oosten
Hello all, I'd like to call a vote on KIP-944 Support async runtimes in consumer. It has has been 'under discussion' for 7 days now. 'Under discussion' between quotes, because there were 0 comments so far. I hope the KIP is clear! KIP description:

[DISCUSS] KIP-944 Support async runtimes in consumer

2023-06-29 Thread Erik van Oosten
section. As far as I am aware there are no tests for the current behavior. Any help in this area would be appreciated. Kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com

Re: [DISCUSS] KIP-944 Support async runtimes in consumer

2023-06-28 Thread Erik van Oosten
It seems KIP-941 was already taken. Updated to: KIP-944. Op 28-06-2023 om 10:11 schreef Erik van Oosten: Hello developers of the Java based consumer, I submitted https://github.com/apache/kafka/pull/13914 to fix a long standing problem that the Kafka consumer on the JVM is not usable from

[DISCUSS] KIP-941 Support async runtimes in consumer

2023-06-28 Thread Erik van Oosten
for the current behavior. Any help in this area would be appreciated. Kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day-to-day-stuff.blogspot.com

Permissions to contribute to Apache Kafka

2023-06-26 Thread Erik van Oosten
Dear reader, I would like to create a KIP and understand I need to request permissions for that. my wiki username: e.vanoos...@chello.nl  (note, this is /not/ my email address) my Jira username: erikvanoosten Kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https://day

Review request for Java Kafka consumer

2023-06-26 Thread Erik van Oosten
Dear Kafka developers, I submitted https://github.com/apache/kafka/pull/13914 to fix a long standing problem that the Kafka consumer on the JVM is not usable from asynchronous runtimes such as Kotlin co-routines and ZIO. Your review is much appreciated. Kind regards,     Erik. -- Erik van

[jira] [Created] (KAFKA-14972) Make KafkaConsumer usable in async runtimes

2023-05-07 Thread Erik van Oosten (Jira)
Erik van Oosten created KAFKA-14972: --- Summary: Make KafkaConsumer usable in async runtimes Key: KAFKA-14972 URL: https://issues.apache.org/jira/browse/KAFKA-14972 Project: Kafka Issue Type

Re: Kafka client needs KAFKA-10337 to cover async commit use case

2023-05-06 Thread Erik van Oosten
`. Is that an acceptable change? Kind regards,     Erik. Op 05-05-2023 om 20:20 schreef Philip Nee: Hey Eric, Maybe its more straightforward to open a new PR. Thanks! P On Fri, May 5, 2023 at 9:36 AM Erik van Oosten wrote: If I were to rebase the old pull request and re-open KAFKA-10337, would

Re: Kafka client needs KAFKA-10337 to cover async commit use case

2023-05-05 Thread Erik van Oosten
If I were to rebase the old pull request and re-open KAFKA-10337, would it be considered for merging? Kind regards,     Erik. Op 03-05-2023 om 09:21 schreef Erik van Oosten: Hi Philip, Firstly, could you explain the situation in that you would prefer to invoke commitAsync over commitSync

Re: Kafka client needs KAFKA-10337 to cover async commit use case

2023-05-03 Thread Erik van Oosten
, which leads to my second comment/question. is it your concern that we currently don't have a way to invoke the callback, and the user won't be to correctly handle these failed/successful async commits? Thanks, P On Tue, May 2, 2023 at 12:22 PM Erik van Oosten wrote: Dear developers of the Kafka

Kafka client needs KAFKA-10337 to cover async commit use case

2023-05-02 Thread Erik van Oosten
be very happy to hear about that because it means I do not have to wait for a release cycle. If you agree these arguments are sound, I would be happy to make the pull request mergable again. Curious to your thoughts and kind regards,     Erik. -- Erik van Oosten e.vanoos...@grons.nl https

[jira] [Commented] (KAFKA-960) Upgrade Metrics to 3.x

2014-11-03 Thread Erik van Oosten (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14194990#comment-14194990 ] Erik van Oosten commented on KAFKA-960: --- If 2.20 and 2.1.5 are indeed binary