Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1298

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.3 #105

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1297

See

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

Jun, a few thoughts on 10 > using the pipelining approach to write to ZK for better throughput and using conditional writes for correctness; For writes to ZK from the KRaft controller, I think we can reuse some or all of the code in KafkaZkClient which does the MultiOp correctness CheckOp along

[jira] [Created] (KAFKA-14305) KRaft Metadata Transactions

David Arthur created KAFKA-14305: Summary: KRaft Metadata Transactions Key: KAFKA-14305 URL: https://issues.apache.org/jira/browse/KAFKA-14305 Project: Kafka Issue Type: New Feature

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

Hi, Colin, 10. "That all goes away in the new mode, and we just have some code which analyzes __cluster_metadata and reflects it in 1) updates to ZK and 2) messages sent out to brokers." Hmm, I am not sure it's that simple. Some of the complexity of the ZK-based controller are (1) using the

Re: [DISCUSS] solutions for broker OOM caused by many producer IDs

Hi Luke & Justine, Thanks for looking into this issue, we have been experiencing this because of rouge clients as well. > 3. Having a limit to the number of active producer IDs (sort of like an LRU >cache) >-> The idea here is that if we hit a misconfigured client, we will expire >the older

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

Thanks for the discussion so far, everyone! Mickael 1) In this case, I think the KRaft controller could avoid sending UMR or LISR to this broker and not include it in LiveBrokers that are sent to the rest of the cluster. This would effectively fence it off from the rest of the cluster. The

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi, David, Thanks for the reply. 60. Hmm, could you explain why KStreams needs the full set of partition assignments? I am also not sure how this will be implemented based on the protocol. Since HeartBeatResponse sends the assigned partitions in phases (those that don't need to wait for

[jira] [Created] (KAFKA-14304) ZooKeeper to KRaft Migration

David Arthur created KAFKA-14304: Summary: ZooKeeper to KRaft Migration Key: KAFKA-14304 URL: https://issues.apache.org/jira/browse/KAFKA-14304 Project: Kafka Issue Type: New Feature

Re: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

Hey Chris, Thanks for your quick reply! Design 1: I think this is a completely reasonable way to implement a V1, without any more complicated pagination. I had not considered that a REST extension would be able to add pagination, but upon reflection that makes a lot of sense. I like the

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

On Thu, Oct 13, 2022, at 11:44, Jun Rao wrote: > Hi, Colin, > > Thanks for the reply. > > 10. This is a bit on the implementation side. If you look at the existing > ZK-based controller, most of the logic is around maintaining an in-memory > state of all the resources (broker, topic, partition,

Re: [VOTE] KIP-876: Time based cluster metadata snapshots

Thanks, José! +1 (binding) Colin On Fri, Oct 14, 2022, at 07:54, David Arthur wrote: > Thanks for the KIP, José! > > +1 (binding) > > -David > > On Fri, Oct 14, 2022 at 2:48 AM David Jacot wrote: >> >> +1 (binding) >> >> Thanks for the KIP! >> >> Le ven. 14 oct. 2022 à 05:47, deng ziming a >>

Re: [ANNOUNCE] New committer: Deng Ziming

Fantastic! Congratulations, Ziming! -David On Tue, Oct 11, 2022 at 12:51 PM José Armando García Sancio wrote: > > Congratulations Ziming. Well deserved and I much appreciate your > contributions to the project. > > -- > -José -- David Arthur

Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

Hi Ashwin, Thanks for your thoughts. Regarding your questions: 1. The response would show the offsets that are visible to the source connector, so it would combine the contents of the two topics, giving priority to offsets present in the connector-specific topic. I'm imagining a follow-up

Re: [VOTE] KIP-876: Time based cluster metadata snapshots

Thanks for the KIP, José! +1 (binding) -David On Fri, Oct 14, 2022 at 2:48 AM David Jacot wrote: > > +1 (binding) > > Thanks for the KIP! > > Le ven. 14 oct. 2022 à 05:47, deng ziming a > écrit : > > > Thanks for this KIP, > > > > +1 for this(binding). > > > > -- > > Best, > > Ziming > > > >

Re: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

Hi Greg, Thanks for your thoughts. RE your design questions: 1. The responses in the REST API may grow fairly large for sink connectors that consume from a large number of Kafka topic partitions, and source connectors that store a wide range of source partitions. If there is a large amount of

PR review request

Hi team, Please review the PRs: * https://github.com/apache/kafka/pull/12735  Replace EasyMock and PowerMock with Mockito in connect/runtime/ErrorHandlingTaskTest * https://github.com/apache/kafka/pull/12739 Replace EasyMock and PowerMock with Mockito |

[jira] [Created] (KAFKA-14303) Producer.send without record key and batch.size=0 goes into infinite loop

Igor Soarez created KAFKA-14303: --- Summary: Producer.send without record key and batch.size=0 goes into infinite loop Key: KAFKA-14303 URL: https://issues.apache.org/jira/browse/KAFKA-14303 Project:

[jira] [Created] (KAFKA-14302) Infinite probing rebalance if a changelog topic got emptied

Damien Gasparina created KAFKA-14302: Summary: Infinite probing rebalance if a changelog topic got emptied Key: KAFKA-14302 URL: https://issues.apache.org/jira/browse/KAFKA-14302 Project: Kafka

Re: [VOTE] KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi David, I made a final pass and LGTM now. +1 from me. Luke On Wed, Oct 5, 2022 at 12:32 AM Guozhang Wang wrote: > Hello David, > > I've made my final pass on the doc and I think it looks good now. +1. > > > Guozhang > > On Wed, Sep 14, 2022 at 1:37 PM Guozhang Wang wrote: > > > Thanks

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Thanks for the update. Yes, I think using similar way as KIP-868 to fix this issue makes sense. Let's consider it in the future. Luke On Fri, Oct 14, 2022 at 5:16 PM David Jacot wrote: > Hi Luke, > > Thanks for your questions. > > > 1. We will store the "targetAssignment" into log now. But as

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Luke, Thanks for your questions. > 1. We will store the "targetAssignment" into log now. But as we know, there's max batch size limit (default 1MB), which means, we cannot support 1M partitions in one group (actually, it should be less than 60k partitions since we'll store {topicID+partition

Re: [DISCUSS] (continued) KIP-848: The Next Generation of the Consumer Rebalance Protocol

Hi Jun, Thanks for your comments. Please find my answers below. 60. Yes, we need both. PartitionAssignor.onAssignment is here to inform the customer assignor about the assignment decision taken with the full set of assigned partitions regardless of whether they are already revoked or not and the

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.3 #104

See

Jenkins build became unstable: Kafka » Kafka Branch Builder » trunk #1296

See

Re: [DISCUSS] KIP-876: Time based cluster metadata snapshots

Hi José, Thanks for the KIP. Adding support to generate snapshot based on time makes sense to me. The only thing I'd like to point out is the compatibility section. Since this new config is default to 1 hour, which means if users explicitly set the config

Re: [VOTE] KIP-876: Time based cluster metadata snapshots

+1 (binding) Thanks for the KIP! Le ven. 14 oct. 2022 à 05:47, deng ziming a écrit : > Thanks for this KIP, > > +1 for this(binding). > > -- > Best, > Ziming > > > On Oct 14, 2022, at 8:11 AM, José Armando García Sancio > wrote: > > > > Hello all, > > > > I would like to start voting for