Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1998

See Changes: -- [...truncated 2594 lines...] > Task :connect:transforms:compileTestJava > Task :connect:transforms:testClasses > Task :connect:transforms:spotbugsTest

[jira] [Resolved] (KAFKA-15185) Consumers using the latest strategy may lose data after the topic adds partitions

[ https://issues.apache.org/jira/browse/KAFKA-15185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-15185. --- Resolution: Duplicate > Consumers using the latest strategy may lose data after the topic adds >

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Hi Mayank, See my answer below. On Thu, Jul 13, 2023 at 10:24 AM Mayank Shekhar Narula < mayanks.nar...@gmail.com> wrote: > re. 2 On some busy clusters a single metadata call has been observed to > take order of ~100 milliseconds(I think it's mentioned somewhere in this > motivation). So

Re: [VOTE] KIP-944 Support async runtimes in consumer, votes needed!

HI Philip & Erik, Hmm... if we agree that KIP-945 addresses this use case, I think it would be better to just focus on that KIP. Fundamentally it's a better and cleaner model than a complex scheme involving thread-local variables. I really don't want to be debugging complex interactions

[jira] [Created] (KAFKA-15188) Implement more of the remaining PrototypeAsyncConsumer APIs

Kirk True created KAFKA-15188: - Summary: Implement more of the remaining PrototypeAsyncConsumer APIs Key: KAFKA-15188 URL: https://issues.apache.org/jira/browse/KAFKA-15188 Project: Kafka Issue

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1997

See Changes: -- [...truncated 2641 lines...] * Try: > Run with --stacktrace option to get the stack trace. > Run with --info or --debug option to get more log output. *

Re: [DISCUSS] KIP-943: Add independent "offset.storage.segment.bytes" for connect-distributed.properties

Hey hudeqi, Thanks for the KIP! I did not know about the existing segment.bytes default value, and it does seem rather high in the context of the Connect internal topics. If I think about the segment.size as a "minimum per-partition data transfer on startup", 1GB is certainly not appropriate for

Re: [VOTE] 3.5.1 RC0

Hi Divij, thanks for running the release. I found a couple of issues with licenses. We include plexus-utils-3.3.1, but the LICENSE file has plexus-utils-3.3.0. The other issue is that we have classgraph-4.8.138, but we don't include this library for a few releases (this is non blocking but it

Re: KafkaConsumer refactor proposal

Hi Philip, I have been scanning through https://cwiki.apache.org/confluence/display/KAFKA/Consumer+threading+refactor+design and KIP-848 and from this I understand that the kafka consumer API will not change. Perhaps the refactoring and/or KIP-848 is a good opportunity to improve the API

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

Hi Omnia, Yes, I think we ought to state the backport plan in the KIP, since it's highly unusual for KIP changes or new configuration properties to be backported and we should get the approval of the community (including binding and non-binding voters) before implementing it. Cheers, Chris On

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Hi Mayank, If we bump the version, the broker can tell whether it’s worth providing the leader endpoint information to the client when the leader has changed. That’s my reasoning. Thanks, Andrew > On 13 Jul 2023, at 18:02, Mayank Shekhar Narula > wrote: > > Thanks both for looking into

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Ismael Thanks for the feedback. 1/3 are in the pipeline. re. 2 On some busy clusters a single metadata call has been observed to take order of ~100 milliseconds(I think it's mentioned somewhere in this motivation). So retrying immediately on the Produce path won't make sense, as metadata would

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Thanks both for looking into this. Jose, 1/2 & 4(changes for PRODUCE) & 5 makes sense, will follow 3. If I understood this correctly, certain replicas "aren't" brokers, what are they then? Also how about replacing "Replica" with "Leader", this is more readable on the client. so, how about

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Thanks for the KIP. A couple of high level points and a more specific one: 1. Given the motivation, the performance results are essential for proper evaluation, so I am looking forward to those. 2. The reasoning given for the rejected alternative seems weak to me. It would be useful to have

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1996

See

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Hi José, Thanks. Sounds good. Andrew > On 13 Jul 2023, at 16:45, José Armando García Sancio > wrote: > > Hi Andrew, > > On Thu, Jul 13, 2023 at 8:35 AM Andrew Schofield > wrote: >> I have a question about José’s comment (2). I can see that it’s possible for >> multiple >> partitions to

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Hi Andrew, On Thu, Jul 13, 2023 at 8:35 AM Andrew Schofield wrote: > I have a question about José’s comment (2). I can see that it’s possible for > multiple > partitions to change leadership to the same broker/node and it’s wasteful to > repeat > all of the connection information for each

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Hi, Thanks for the KIP, Mayank. I have a question about José’s comment (2). I can see that it’s possible for multiple partitions to change leadership to the same broker/node and it’s wasteful to repeat all of the connection information for each topic-partition. But, I think it’s important to

Re: [VOTE] KIP-941 Range queries to accept null lower and upper bounds

Hi all, Thanks to everyone who participated in the voting and the discussion. I'll close it since it has been open for over 72 hours, and we have sufficient votes. KIP-941 has been accepted with the following +1 votes (and no +0 or -1 votes): - Sophie Blee-Goldman (binding) - Matthias J. Sax

Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Hi Mayank, thanks for the KIP. I look forward to this improvement for new clients. Some comments below. On Thu, Jul 13, 2023 at 7:15 AM Mayank Shekhar Narula wrote: > Following KIP is up for discussion. Thanks for your feedback Regarding the FETCH response changes: 1. Tagged field 2 already

Re: [DISCUSS] KIP-660: Pluggable ReplicaPlacer

Mickael, have you had some time to review this by any chance? On Tue, Jun 20, 2023 at 5:23 PM Viktor Somogyi-Vass < viktor.somo...@cloudera.com> wrote: > Hey all, > > I'd like to revive this discussion. I've created >

[DISCUSS] KIP-951: Leader discovery optimisations for the client

Hi everyone Following KIP is up for discussion. Thanks for your feedback. https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client -- Regards, Mayank Shekhar Narula

[Vote] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

Hi Everyone! I would like to start the vote on KIP-949 details is here https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy Thanks Omnia

[jira] [Created] (KAFKA-15187) Add headers to partition method.

Jacob Tomy created KAFKA-15187: -- Summary: Add headers to partition method. Key: KAFKA-15187 URL: https://issues.apache.org/jira/browse/KAFKA-15187 Project: Kafka Issue Type: New Feature

Re: [VOTE] KIP-852: Optimize calculation of size for log in remote tier

+1 (non-binding) Thanks for the KIP! Jorge. On Thu, 13 Jul 2023 at 12:26, Luke Chen wrote: > +1 (binding) from me. > > Thanks for the KIP! > > Luke > > On Sun, Jul 2, 2023 at 11:49 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > +1 (non-binding). Thanks for the KIP! >

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

Thanks Divij. I was confusing with the metric tags used by clients that are based on topic and partition. Ideally partition label could be at a DEBUG recording level, but that's outside the scope of this KIP. Looks good to me, thanks again! Jorge. On Wed, 12 Jul 2023 at 15:55, Divij Vaidya

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

Hi Chris, The implementation should be very small so backporting this to 3.1 and 3.2 would be perfect for this case if you or any other committer are okay with approving the backporting. Do we need to state this in KIP as well or not? Also, I’ll open a vote for the KIP today and prepare the pr

[jira] [Created] (KAFKA-15186) AppInfo metrics don't contain the client-id

Mickael Maison created KAFKA-15186: -- Summary: AppInfo metrics don't contain the client-id Key: KAFKA-15186 URL: https://issues.apache.org/jira/browse/KAFKA-15186 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-15185) Consumers using the latest strategy may lose data after the topic adds partitions

RivenSun created KAFKA-15185: Summary: Consumers using the latest strategy may lose data after the topic adds partitions Key: KAFKA-15185 URL: https://issues.apache.org/jira/browse/KAFKA-15185 Project:

Re: [VOTE] KIP-852: Optimize calculation of size for log in remote tier

+1 (binding) from me. Thanks for the KIP! Luke On Sun, Jul 2, 2023 at 11:49 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > +1 (non-binding). Thanks for the KIP! > > — > Kamal > > On Mon, 7 Nov 2022 at 2:20 AM, John Roesler wrote: > > > Hi Divij, > > > > Thanks for the KIP!

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

Hi Divij, One minor comment: remoteLogSize takes 2 parameters, but in the code snippet, you only provide 1 parameter. Otherwise, LGTM Thank you. Luke On Wed, Jul 12, 2023 at 8:56 PM Divij Vaidya wrote: > Jorge, > About API name: Good point. I have changed it to remoteLogSize instead of >