A question about topic limit in produceBytesIn

Another question about using kafka recently: Currently, Kafka's write throttle only supports the user dimension and clientId dimension of the request. In fact, such a situation is often encountered in actual use: a topic entry traffic suddenly increases, and the resource bottleneck is about to

A question about the reassign task in kafka

One question were found in the process of using kafka recently: Why does Kafka start fetching from the leader's logStartOffset when Kafka is doing the topic reassign task (such as cluster expansion brokers)? If the amount of data stored locally by the partition leader is large, the disk IO will

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1757

See

[jira] [Resolved] (KAFKA-14883) Broker state should be "observer" in KRaft quorum

[ https://issues.apache.org/jira/browse/KAFKA-14883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-14883. --- Fix Version/s: 3.5.0 Resolution: Fixed > Broker state should be "observer" in KRaft quorum >

[jira] [Resolved] (KAFKA-9550) RemoteLogManager - copying eligible log segments to remote storage implementation

[ https://issues.apache.org/jira/browse/KAFKA-9550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-9550. -- Resolution: Fixed > RemoteLogManager - copying eligible log segments to remote storage > implementatio

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1756

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #107

See

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1755

See Changes: -- [...truncated 367175 lines...] [2023-04-13T00:35:19.675Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 94 > KafkaZkClientTest > testControl

[jira] [Resolved] (KAFKA-14561) Improve transactions experience for older clients by ensuring ongoing transaction

[ https://issues.apache.org/jira/browse/KAFKA-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-14561. - Fix Version/s: 3.5.0 Resolution: Fixed merged the PR to trunk. > Improve transactions experience

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1754

See Changes: -- [...truncated 556315 lines...] [2023-04-12T21:31:11.321Z] Gradle Test Run :connect:runtime:integrationTest > Gradle Test Executor 132 > org.apache.kafka.c

[jira] [Created] (KAFKA-14901) Flaky test ExactlyOnceSourceIntegrationTest.testConnectorReconfiguration

Greg Harris created KAFKA-14901: --- Summary: Flaky test ExactlyOnceSourceIntegrationTest.testConnectorReconfiguration Key: KAFKA-14901 URL: https://issues.apache.org/jira/browse/KAFKA-14901 Project: Kafka

[jira] [Resolved] (KAFKA-14898) [ MirrorMaker ] sync.topic.configs.enabled not working as expected

[ https://issues.apache.org/jira/browse/KAFKA-14898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Harris resolved KAFKA-14898. - Resolution: Fixed > [ MirrorMaker ] sync.topic.configs.enabled not working as expected > ---

[jira] [Created] (KAFKA-14900) Flaky test AuthorizerTest failing with NPE

Greg Harris created KAFKA-14900: --- Summary: Flaky test AuthorizerTest failing with NPE Key: KAFKA-14900 URL: https://issues.apache.org/jira/browse/KAFKA-14900 Project: Kafka Issue Type: Test

[jira] [Created] (KAFKA-14899) Revisit Action Queue

Justine Olshan created KAFKA-14899: -- Summary: Revisit Action Queue Key: KAFKA-14899 URL: https://issues.apache.org/jira/browse/KAFKA-14899 Project: Kafka Issue Type: Sub-task Rep

[jira] [Resolved] (KAFKA-14894) MetadataLoader must call finishSnapshot after loading a snapshot

[ https://issues.apache.org/jira/browse/KAFKA-14894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-14894. -- Fix Version/s: 3.5.0 Reviewer: David Arthur Resolution: Fixed > MetadataLoader

[jira] [Created] (KAFKA-14898) [ MirrorMaker ] sync.topic.configs.enabled not working as expected

Srinivas Boga created KAFKA-14898: - Summary: [ MirrorMaker ] sync.topic.configs.enabled not working as expected Key: KAFKA-14898 URL: https://issues.apache.org/jira/browse/KAFKA-14898 Project: Kafka

[jira] [Created] (KAFKA-14897) 3.4.0 release notes does not mention fixing KAFKA-14696

Gray created KAFKA-14897: Summary: 3.4.0 release notes does not mention fixing KAFKA-14696 Key: KAFKA-14897 URL: https://issues.apache.org/jira/browse/KAFKA-14897 Project: Kafka Issue Type: Task

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

Hi Chris, Thanks, that makes sense - I hadn't considered the case where the worker itself becomes a zombie. Thanks, Yash On Wed, Apr 12, 2023 at 10:40 PM Chris Egerton wrote: > Hi Yash, > > Great, we can use the transactional ID for task zero for now 👍 > > As far as why: we'd need to fence out

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1753

See

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

Hi Yash, Great, we can use the transactional ID for task zero for now 👍 As far as why: we'd need to fence out that producer in the event that tasks for the connector are brought up while the alter offsets request is still taking place, since we'd want to make sure that the offsets aren't altered

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

Hi Chris and Greg, The current implementation does already use the transactional ID for task 0 so no complaints from me. Although I'm not sure I follow the concerns w.r.t zombie fencing? In which cases would we need to fence out the transactional producer instantiated for altering offsets? Thanks

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

Hi Sagar, I'm sorry, I'm still not convinced that this design solves the problem(s) it sets out to solve in the best way possible. I tried to highlight this in my last email: > In general, it seems like we're trying to solve two completely different problems with this single KIP: adding framework

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

Hello Colin, Thank you for the response! 1. I have attached the compatibility matrix in the KIP under the section Compatibility, Deprecation, and Migration Plan. 2. I believe the answer to how many bridge releases (for Kafka) will be needed to upgrade from 2.0 to 4.0 based on the compatibility ma

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

Hi Greg, I hadn't considered the implications W/R/T zombie fencing. I agree that using the transactional ID for task 0 is better in that case. Yash (who is implementing this part, cc'd), does this seem reasonable to you? Cheers, Chris On Tue, Apr 11, 2023 at 3:23 PM Greg Harris wrote: > Chri

[jira] [Resolved] (KAFKA-14890) Kafka initiates shutdown due to connectivity problem with Zookeeper and FatalExitError from ChangeNotificationProcessorThread

[ https://issues.apache.org/jira/browse/KAFKA-14890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ron Dagostino resolved KAFKA-14890. --- Resolution: Duplicate Duplicate of https://issues.apache.org/jira/browse/KAFKA-14887 > Kafk

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1752

See

Zookeeper upgrade 3.4.14 to 3.5.7

Hi Kafka , zookeeper experts Is it possible to upgrade the 3.4.14 version of zookeeper cluster in a rolling fashion (one by one node) to 3.5.7 zookeeper version. Would the cluster work with a possible combination of 3.4.14 and 3.5.7 . Please advise .

Re: [DISCUSS] Apache Kafka 3.5.0 release

Hi Mickael, I'd like to ask for some more days for KIP-405 tiered storage PRs to include in v3.5. Currently, we have 1 PR under reviewing ( https://github.com/apache/kafka/pull/13535), and 1 PR soon will be opened for review. After these 2 PRs merged, we can have an "Early Access" for tiered stora

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1751

See

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

Hi Divij, Thanks for the follow-up. A few comments/questions. 100. The stated motivation to increase the number of partitions above 50 is scale. Are we sure that 50 partitions is not enough to cover all valid use cases? An upper bound in the range 1 to 10 MB/s of ingress per partition gives 50 to

[jira] [Resolved] (KAFKA-14889) RemoteLogManager - allow consumer fetch records from remote storage implementation

[ https://issues.apache.org/jira/browse/KAFKA-14889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-14889. --- Resolution: Duplicate > RemoteLogManager - allow consumer fetch records from remote storage > imple

[jira] [Resolved] (KAFKA-14891) Fix rack-aware range assignor to improve rack-awareness with co-partitioning

[ https://issues.apache.org/jira/browse/KAFKA-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-14891. Fix Version/s: 3.5.0 Reviewer: David Jacot Resolution: Fixed > Fix rack-aw