Re: kafka-producer-perf-test.sh transaction support?

Hi Anindya, Currently, setting transaction.id in producer props won't enable transactions. Users need to set --transaction-duration-ms to enable transactions in kafka-producer-perf-test. We recognize this can be confusing, so there's a JIRA ticket for it:

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

+1. Thanks, Josep! Colin On Mon, Jul 29, 2024, at 10:32, Chris Egerton wrote: > Thanks for running the release, Josep! > > > On Mon, Jul 29, 2024, 13:31 'Josep Prat' via kafka-clients > wrote: >> The Apache Kafka community is pleased to announce the release for Apache >> Kafka 3.8.0 >> >>

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

Thanks for running the release, Josep! On Mon, Jul 29, 2024, 13:31 'Josep Prat' via kafka-clients < kafka-clie...@googlegroups.com> wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 3.8.0 > > This is a minor release and it includes fixes and improvements

[ANNOUNCE] Apache Kafka 3.8.0

The Apache Kafka community is pleased to announce the release for Apache Kafka 3.8.0 This is a minor release and it includes fixes and improvements from 456 JIRAs. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.8.0/RELEASE_NOTES.html

Re: Kafka Read only - disable producing

Could you enable ACLs on your cluster and not grant any principals permission to write to the cluster? On Fri, Jul 26, 2024, 11:13 Rotem S wrote: > Hi, I would like to force disable producing to the kafka cluster and still > allow consuming once the storage disk space reaches a certain

Re: [VOTE] 3.8.0 RC3

Thank you all who reviewed this release candidate and the previous ones. This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Chris Egerton * Greg Harris * Mickael Maison Committers: * Igor Soarez Community: * Murali Basani * Jakub Scholz * Krishna

Re: [VOTE] 3.8.0 RC3

Hi Josep, Thanks for running the release! I am +1 (non binding) Downloaded from sources https://github.com/apache/kafka/archive/refs/tags/3.8.0-rc3.tar.gz and built with java 17. I performed the following validations: 1. I deleted a partition.metadata for a topic manually, and tried starting

Kafka Read only - disable producing

Hi, I would like to force disable producing to the kafka cluster and still allow consuming once the storage disk space reaches a certain threshold, we tried reducing the max message bytes and producer byte rate but it didnt work out well, The best solution we found was to make the

Re: Fw: Producer ordering

Not 100% sure if librdkafka provides the exact some semantics as the Kafka Java producer (but I believe it does for this case; but might be worth to check the docs?). If `send()` gives an error back, ordering would not be guaranteed any longer if the retry happens at the application level.

Re:Re: Re: request to be contributor

Hi Josep, Thank you for your suggestion. I have added a comment to the Jira issue you mentioned Best, Yang At 2024-07-24 19:44:36, "Josep Prat" wrote: >Hi Yang, > >I think you need to add a comment here: >https://issues.apache.org/jira/browse/INFRA-25451 > >Best, > >On Wed, Jul 24,

Re: Re: request to be contributor

Hi Yang, I think you need to add a comment here: https://issues.apache.org/jira/browse/INFRA-25451 Best, On Wed, Jul 24, 2024 at 1:36 PM yangxuze wrote: > Hi Josep, > > Thanks for your prompt reply. I now can assign jira to myself, but I have > been unable to successfully register

Re:Re: request to be contributor

Hi Josep, Thanks for your prompt reply. I now can assign jira to myself, but I have been unable to successfully register Confluence/Wiki account. The sign up page shows "This installation of Confluence is not set up to permit public signup. Please contact the site administrators for more

Re: [VOTE] 3.8.0 RC3

+1 (non-binding). As with the previous RCs, I used the staged Scala 2.13 binaries and the staged Maven artifacts. All seems to work fine. Thanks! Jakub On Tue, Jul 23, 2024 at 12:07 PM Josep Prat wrote: > Hello Kafka users, developers and client-developers, > > This is the fourth candidate for

Fw: Producer ordering

Hi, My apology if this is a question which is already been answered before. Use Case: I am reviewing an application (producer) which is using librdKafka as a 3rd party library to interact with Kafka brokers (3= 1L+2F). This 3rd party lib has been configured for retries. Its a single topic with

Re: request to be contributor

Hi, Can we run multiple kafka streams with the same stream-id. On Wed, Jul 24, 2024 at 12:15 PM Josep Prat wrote: > Hi Yang, > > I added you to Jira. I couldn't find your username for the Wiki page > (Confluence) so I didn't set that up. But you are now all set in regards to > Jira. Let me

Re: request to be contributor

Hi Yang, I added you to Jira. I couldn't find your username for the Wiki page (Confluence) so I didn't set that up. But you are now all set in regards to Jira. Let me know if you have any questions. Best, On Wed, Jul 24, 2024 at 5:11 AM yangxuze wrote: > Dear Apache Kafka Team, > > > > > My

Second attempt of migration of ZK to Kraft after rollback fails

Hi, Currently I'm testing this workflow: 1. Migrate ZK to Kraft (v3.7.0); 2. Stop migration process after "Enter Migration Mode on the brokers" or "Migrating brokers to KRaft" completes (this leaves the opportunity to perform rollback); 3. Perform rollback; 4. Migrate ZK to Kraft

request to be contributor

Dear Apache Kafka Team, My name is Xuze Yang, JIRA ID is Xuze Yang, I kindly request to be added to the contributors list for Apache Kafka. Being part of this list would allow me to be assigned to JIRA tickets and work on them. Thank you for considering my request. Best regards, Yang.

Re: [VOTE] 3.8.0 RC3

Hi Josep, +1 (binding) I performed the following validations: 1. I formatted a new log directory and verified that the stray log message is no longer present (KAFKA-17148) 2. I configured a connect worker with a custom plugin path and plugin.discovery=service_load and verified that plugin

Fwd: [VOTE] 3.8.0 RC3

Forwarding my response to the other mailing list threads; apologies for missing the reply-all the first time! -- Forwarded message - From: Chris Egerton Date: Tue, Jul 23, 2024 at 3:45 PM Subject: Re: [VOTE] 3.8.0 RC3 To: Hi Josep, Thanks for running this release! I'm +1

Re: [VOTE] 3.8.0 RC3

Hi Josep, Thanks for running this release! I'm +1 (binding). To verify, I: - Built from source using Java 11 with both: - - the 3.8.0-rc3 tag on GitHub - - the source artifact from https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka-3.8.0-src.tgz - Checked signatures and checksums - Ran the

Re: [VOTE] 3.8.0 RC3

Here is the link to the system tests: https://confluent-open-source-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.8/2024-07-22--001.ffbb03b2-61f4-4ebb-ae1f-af5c753682fb--1721733000--confluentinc--3.8--9a2b34b68c/report.html The Quota tests are known to fail in this CI system. Regarding

[VOTE] 3.8.0 RC3

Hello Kafka users, developers and client-developers, This is the fourth candidate for release of Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker *

Re: [VOTE] 3.8.0 RC2

+1 (non-binding). I used the staged Scala 2.13 binaries and the staged Maven artifacts. All seems to work fine. Thanks! Jakub On Thu, Jul 18, 2024 at 11:19 AM Josep Prat wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate for release of Apache Kafka

Kafka Consumer Fetch position Out Of Range error

Hi, I'm using Kafka 2.5.1 broker and Kafka Connect Confluent image 7.1.1. We are using a sink connector to read from Kafka. We occasionally see Fetch Position OutOfRange error like this [2024-07-19 00:54:59,456] INFO [Consumer >

[VOTE] 3.8.0 RC2

Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker *

Re: KRaft Migration Prod Ready?

Thanks Luke, Just a follow up question, is it possible to migrate using dual-mode for the broker/controller? I.e. with `process.roles=broker,controller` set? Specifically in Kubernetes? Or is the my understanding correct that the correct order of the migration in kube would be: 1. Start

Re: [VOTE] 3.8.0 RC1

Hi all, I'm cancelling this VOTE thread, and I'll generate a new RC once the fix is merged. Best, On Wed, Jul 17, 2024 at 8:55 AM Josep Prat wrote: > Thanks Greg for taking a look and providing a fix! > > Best, > > -- > Josep Prat > Open Source Engineering Director, Aiven >

Proxy Support For Kafka Clients Java

Hello,  I would like to know if there any ways to use kafka clients with proxy. Note: Broker URLs cannot be fetched before running consumer and connections are made to multiple kafka clusters.

Re: [VOTE] 3.8.0 RC1

Thanks Greg for taking a look and providing a fix! Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht

Re: [VOTE] 3.8.0 RC1

Hi Josep, I found this blocker regression: https://issues.apache.org/jira/browse/KAFKA-17150 Summary: Connector configurations that specified converters with aliases (e.g. JsonConverter instead of org.apache.kafka.connect.json.JsonConverter) previously worked, and in this RC they throw validation

Re: [VOTE] 3.8.0 RC1

+1 (non-binding). I used the staged Scala 2.13 binaries and the staged Maven artifacts. All seems to work fine. Thanks! Jakub On Mon, Jul 15, 2024 at 5:53 PM Josep Prat wrote: > Hello Kafka users, developers and client-developers, > > This is the second release candidate for Apache Kafka

Kafka 3.5.2 is able to read __consumer_offsets from partitions with metadata version 3 when inter.broker.protocol.version is set to 2.0

Hi all, I am working on upgrading the Kafka cluster from the version 2.0.1 to 3.5.2. The main concern in that update is passing version 2.1 which has the following note in the upgrade document: ``` Note that 2.1.x contains a change to the internal schema used to store consumer offsets. Once the

[VOTE] 3.8.0 RC1

Hello Kafka users, developers and client-developers, This is the second release candidate for Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker * KIP-1036:

Re: [VOTE] 3.8.0 RC0

Hi all, I'm cancelling the VOTE thread for 3.8.0-RC0. I submitted a PR with the backport https://github.com/apache/kafka/pull/16593 and I'll generate a new RC as soon as it's merged. Best, On Sat, Jul 13, 2024 at 7:09 PM Josep Prat wrote: > Thanks for reviewing the RC Jakub, > > If you can

Buffer Utilization Metric

Hello, I have set up a grafana dashboard that has the Log Cleaner metrics added when we upgraded to Apache Kafka 3.5. We didn’t have these metrics before when we were using Kafka 2.7. There is one metric that always show 0%, and this is the Log Cleaner Max Buffer Utilization Percent

Re: [VOTE] 3.8.0 RC0

Thanks for reviewing the RC Jakub, If you can open a PR with this fix pointing to the 3.8 branch I could cut another RC. Best! -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117

Re: [VOTE] 3.8.0 RC0

Hi Josep, Thanks for the RC. I gave it a quick try and ran into issues with an application using the Kafka Admin API that looks like this issue: https://issues.apache.org/jira/browse/KAFKA-16905 ... given that this breaks what was working fine with Kafka 3.7, can the fix be backported to 3.8.0

Re: Requesting a Confluence account to create a KIP for a PR

It's not in our control. The INFRA team needs to create accounts manually at this point, and we don't know at what cadence they find time to do this. Sorry :( If an account is create, I believe you should get an email. We know that this current workaround is very tedious, but there is

Requesting a Confluence account to create a KIP for a PR

Hello, I created a PR about my JIRA-16707 (finally in WIP as I will have to do a KIP). I asked for an account for Confluence in JIRA https://issues.apache.org/jira/browse/INFRA-25451 , I didn't see an answer yet (or I missed it). Can anyone help me? Best regards

[VOTE] 3.8.0 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker * KIP-1036:

Re: Error creating a topic with 10k partitions but not altering existing topic to 10k partitions? Why

Kafka 3.6.2 – thanks, Paul From: Ömer Şiar Baysal Date: Thursday, 11 July 2024 at 9:57 PM To: users@kafka.apache.org Subject: Re: Error creating a topic with 10k partitions but not altering existing topic to 10k partitions? Why [You don't often get email from osiarbay...@gmail.com. Learn why

Re: [EXTERNAL] Changing consumer group in live

@Mohamed Depending on topic retention and offset, and how much data in topic, your app could start reprocessing lot of data and eventually struggling in terms of performance. [Logo, company name Description automatically generated] Jose Manuel Vega Monroy Software Engineer Team Leader Direct:

StreamThread shutdown calls completeShutdown only in CREATED state

Hi Kafka Dev/Users, While running tests in `StreamThreadTest.java` in kafka/streams, I noticed the test left many lingering threads. Though the class runs `shutdown` after each test, the shutdown only executes `completeShutdown` if the StreamThread is in CREATED state. See

Re: Error creating a topic with 10k partitions but not altering existing topic to 10k partitions? Why

Hi, I have never seen PolicyViolationException before with vanilla Kafka. What is the flavor? This may caused by the controller mutation rate checks introduced by KIP-599, it may be triggered for non-existing topics but not for the existing resources. Hope this help you track it down. OSB On

Error creating a topic with 10k partitions but not altering existing topic to 10k partitions? Why

Hi – just curious if anyone can suggest why the following occurs: 1 – try to create a topic with 10,000 partitions with Kafka CLI (kafka-topics.sh) Fails with ERROR org.apache.kafka.common.errors.PolicyViolationException: Unable to perform excessively large batch operation. 2- create a topic

Re: Changing consumer group in live

In the end you create a new consumer group. That's totally fine. The old consumer group will still "exist", ie, the committed offsets for the group are still there, until they will be eventually purged by the brokers (I think the default is 7 days to keep the metadata of consumer groups,

Re: Kafka 20k topics metadata update taking long time

Following up to check if someone can help. And is this the right approach or should i open a Jira ticket for same. Thanks, Amit On Fri, Jul 5, 2024 at 9:42 AM Amit Chopra wrote: > But 20K topics is also not a realistic assumption to have. > This is our existing scale with AWS kinesis 20k

Re: Installing kafka

Hi Christian, Are you sure are in the same directory you downloaded the kafka_2.13-3.7.1.tgz file to? You might need to change the command to point to the right directory maybe. Best -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 |

Changing consumer group in live

Hello, Let us consider one application in a consumer group consuming events from Kafka in live. Just like any code change, I changed the consumer group name of the application to a new group. I understand my application might start reading from the beginning offset or the latest based on the

Installing kafka

Hi there, when wanting to install Kafka using the command tar -xzf kafka_2.13-3.7.1.tgz my Mac says no, produces the following error message: tar: Error opening archive: Failed to open 'kafka_2.13-3.7.1.tgz' Any idea how I can still install Kafka? Thanks for helping. Christian

Re: Kafka 20k topics metadata update taking long time

Just to make sure I am following correctly, we do not see issues with 1 topic 20k partitions. With 25 brokers, the load is fairly distributed and we can achieve aggregate throughput of ~2.5 GBps. With 20k topics with 1 partition each, we see much reduced throughput. Thanks, Amit On Fri, Jul 5,

Re: Kafka 20k topics metadata update taking long time

But 20K topics is also not a realistic assumption to have. This is our existing scale with AWS kinesis 20k shards. And we are moving over to kafka and thus testing with equivalent scale. We are looking at an ingestion rate of 2.5 GBps. I don't see an alarming difference in the latency results

Re: MM2, how to configure offset-syncs topic name

Hello, If it's possible for your conf : could you try something like: *Setting 2 partitions on your topic A *run both MM2 at the same time with the same group of consumer but different clientID. In this way, Kafka Broker shoulds affect one partition by MM2 clientID, and so, messages will

Re: KRaft Migration Prod Ready?

Hi Upesh, Yes, it's supported. The doc will be updated soon (after v3.8.0 released). Thanks. Luke On Wed, Jul 3, 2024 at 6:02 AM Upesh Desai wrote: > Hello, > > > > We have been a bit confused as to if KRaft migration from ZooKeeper is > production ready? It seems that the Kafka docs still

Re: Kafka 20k topics metadata update taking long time

OK so repeating with Java Kafka producer there is no problem – it’s specific to the Kafka CLI Producer! Paul From: Brebner, Paul Date: Friday, 5 July 2024 at 1:21 PM To: users@kafka.apache.org Subject: Re: Kafka 20k topics metadata update taking long time EXTERNAL EMAIL - USE CAUTION when

Re: Kafka 20k topics metadata update taking long time

Repeating my tests today with a bit more caution I can get up to around 47,000 partitions for a single topic before the producer fails with a bootstrap broker disconnected warning (in practice the producer cannot send), here’s a graph of the producer time (to send 1k messages) using producer

Kafka Connect Worker Provisioning/Optimization Considerations

I have 100+ sink connectors running with 100+ topics each with roughly 3 partitions per topic. How would you configure resources (mem and cpu) to optimally handle this level of load. What would be your considerations? Also, When considering this load, should i be thinking about it as an aggregated

Re: Kafka Connect Limits

Thank you for having a look at this. I agree that the only way to really gauge load is to look at lag. But the connector tasks should not crash and die because of load. I will raise this with SF. On Wed, Jul 3, 2024 at 7:14 PM Greg Harris wrote: > Hey Burton, > > Thanks for your question and

MM2, how to configure offset-syncs topic name

Hello I have a source kafka cluster and 2 destination kafka clusters to which i want to mirror messages _alternatively_, with this I mean that if topic A was mirrored up to offset=3 to destination cluster 1 I want mirrormaker to mirror offset=4 onwards to destination cluster 2 I dont care for

Re: Kafka 20k topics metadata update taking long time

Hi – interesting, I had maybe similar problems today when “testing” the limits of a Kafka cluster for max partitions – I could create a topic with lots of partitions (ok so more than sensible, taking into account RF=3 over 1M partitions) – but trying to send a message failed with a meta-data

Re: Kafka 20k topics metadata update taking long time

This is interesting. But 20K topics is also not a realistic assumption to have. I don't see an alarming difference in the latency results from the two scenarios. Also even when cached, the metadata needs to be refreshed anyway. On Thu, Jul 4, 2024 at 4:37 AM Amit Chopra wrote: > Hey, > > I

Re: Kafka Connect Limits

Hey Burton, Thanks for your question and bug report. The exception you included does not indicate that your connectors are overloaded. The primary way of diagnosing an overloaded connector is the consumer lag metric, and if you're seeing acceptable lag, that should indicate that your connectors

Kafka 20k topics metadata update taking long time

Hey, I wanted to raise a performance issue that I encountered while testing on a large scale with Kafka. The issue pertains to low throughput with 20k topics (each topic with 1 partition) compared to good throughput when using 1 topic with 20k partitions. Scenario: I have a setup of kafka with

Kafka Connect Limits

Hi, I have 100+ sink connectors running with 100+ topics each with roughly 3 partitions per topic. There are running on K8s on 10 pods with 6 cpus and 32Gig mem. The connector in question is Snowflake's sink connector v2.2.0. This worked in the mini batch mode SNOWPIPE, but once i switched over

KRaft Migration Prod Ready?

Hello, We have been a bit confused as to if KRaft migration from ZooKeeper is production ready? It seems that the Kafka docs still indicate it is NOT ready: https://kafka.apache.org/documentation/#kraft_zk_migration But other sites such as Confluent and Strimzi do say it is supported: -

Re: [ANNOUNCE] Apache Kafka 3.7.1

A correction to a funny mistake in my previous message: That's *61 contributors* to this release, not 1086! -- Igor Soarez On Mon, Jul 1, 2024, at 11:25 AM, Igor Soarez wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 3.7.1 > > This is a bug fix

[ANNOUNCE] Apache Kafka 3.7.1

The Apache Kafka community is pleased to announce the release for Apache Kafka 3.7.1 This is a bug fix release and it includes fixes and improvements. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.7.1/RELEASE_NOTES.html You can

Re: Kafka Streams branching return type affects bindings when using multiple output bindings: intentional behaviour?

Hi Matthias, Thanks for the reply! The binder expects an array and their doc about multiple output bindings uses the new branching methods without noting the possible "out of order" bindings (they use the branch map values to create an array like in my code snippet). For now, I've manually

Assistance Needed: Retrieving Current User in CreateTopicPolicy for Topic Creation Restriction

Dear Kafka Community, I am reaching out for guidance on how to retrieve the current user who is attempting to create a topic within the CreateTopicPolicy implementation in Apache Kafka. Our goal is to restrict topic creation based on a configured maximum limit for each user. While working on

Trying to reproduce Auth-Issue, code changes would be helpful

Hi! We're having a Kafka 3.4.1 cluster in use which we access through a Scala Library "zio-Kafka". This lib itself uses the Java Client in Version 3.6.1 We have to authenticate with the broker through Kerberos/SAML. It appears that the broker regularly queries LDAP to see if the token is

Re: MirrorMaker2 Offset Replication Issue: MirrorCheckpointConnector Doesn't Replicate Final Offsets In Migrations

Hi Mehrtens, That is expected with the current implementation of offset translation. It makes a tradeoff that translation further from the end of the topic is less accurate, and it won't "converge" to a perfect translation. The only "perfect" translation that is offered is when the consumer group

Re: MirrorMaker2 Offset Replication Issue: MirrorCheckpointConnector Doesn't Replicate Final Offsets In Migrations

HI - Apologies for the delay, it took me a while to find time to learn how to build from source and incorporate that into my test environment. I've built off of the 3.7.1-rc2 tag which includes the KAFKA-15905 fix. Unfortunately, this did not fix the issue. With 3.7.1-rc2, offset replication

Re: [VOTE] 3.7.1 RC2

+1 (non-binding). I used the staged binaries (based on Scala 2.13) and Maven artifacts to run my tests. All seems to work fine. Thanks & Regards Jakub On Wed, Jun 19, 2024 at 10:55 AM Igor Soarez wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate for

Kafka outage due to partial node failure on Kubernetes

Hi all, I have an issue that spans Kafka and K8s... Do you think a Kafka bug is appropriate? Is there an alternative configuration to prevent this from happening again? Would it be any different with KRaft? Here's what happened: * A big disruption occurs on a node running the kafka-2 broker.

Re: Kafka Streams branching return type affects bindings when using multiple output bindings: intentional behaviour?

Jenny, thanks for reaching out. Yes, we plan to remove the old variant of `branch()` (which was deprecated with KIP-418) in the 4.0 release (https://issues.apache.org/jira/browse/KAFKA-12824) which is planned for end of this year. KIP-418 changes the pattern from relying on order to relying

Kafka Streams branching return type affects bindings when using multiple output bindings: intentional behaviour?

Hello, KIP-418 (>=2.8) improved the branching return type from KStream<>[] to HashMap. This changes the output binding when using multiple branches in the Kafka Streams binder of Spring Cloud Stream - the bindings are possibly wrong because the map storing the KStream branches is unordered.

Looking for reviewers on PR-16361 -JIRA-16707

Hello, I'm looking for reviewer(s) for PR: https://github.com/apache/kafka/pull/16361 Jira Opened is https://issues.apache.org/jira/browse/KAFKA-16707 If I missed a step to be in a correct process, may I ask you few guidance? Best regards Franck

[VOTE] 3.7.1 RC2

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 3.7.1. This is a bugfix release with several fixes. Release notes for the 3.7.1 release: https://home.apache.org/~soarez/kafka-3.7.1-rc2/RELEASE_NOTES.html *** Please download, test

Re: [EXTERNAL] Re: Kafka upgrade recommendation

Hi, Thanks for the answer. However I still do not have a clear picture on how we should go from version 2.2.1 which we are on at the moment. Options 1. Straight jump from 2.2.1 to 3.6.3 2. Jump from 2.2.1 to 2.6.2 and then the second jump from 2.6.2. to 3.6.3 What’s you recommendation

Re: [EXTERNAL] Re: Kafka upgrade recommendation

Hi Sejal, Thanks for reaching out. What would be your recommended strategy to go from 2.2.1 into 3.6.2 version? Can we securely do it in one jump or we should use some step in between (e.g. to jump form 2.2.1 to 2.6.2 and then from 2.6.2 to 3.6.2)? Thanks, [A blue background with white text

Re: [EXTERNAL] Re: Kafka upgrade recommendation

First off, I'd suggest reading https://kafka.apache.org/documentation/#upgrade pretty carefully. It has your exact question more or less defined and outlined. As I mentioned, they are putting a lot of effort into how the world should migrate

Re: [EXTERNAL] Re: Kafka upgrade recommendation

Hey Wojciech, It has been a long time since we did the jump but we had waited until 3.3.x release when we did the jump and there were a lot of manual steps at that time. We then had another manual step when going to either 3.4 or 3.5 don't remember which but that one was easier cause we did

What's going on here?

Hey folks, I'm quite confused here. I have a random partition that has 4 replications instead of 3. I don't have a clue how to fix it or what the ramifications of this are either. There are no repartitioning happening or reassignments or anything at all. This looks like the metadata is just

Re: Kafka upgrade recommendation

I would suggest going from 2.6.2 to 3.6.2 and then stopping and waiting until 3.7.1 comes out. There is a pretty serious flaw in 3.7.0 that hit us and we ended up having to downgrade to 3.6.2 to recover. The bug is fixed but not going to be released until 3.7.1 comes out. The bug prevented

Re: Making connector fault tolerant

yes, we create connectors on the fly and in some cases after a new deployment msk cluster's public access is disabled thereby making the bootstrap brokers present there unavailable to currently running connectors. I am curious to know if it's possible to momentarily catch exceptions in such cases

Kafka upgrade recommendation

Hi there, We have few clusters in AWS world and being forced to bump cluster version. So, we are considering options to be in safer way possible, mitigating risks, and without outage as possible (our business is 24/7). Upgrade would be from 2.2.1 / 2.6.2 to 3.7.0 (MSK recommended version).

Re: Making connector fault tolerant

Hi Akash, if your connector doesn't have the appropriate permissions for a topic then it can't run - so I'm not sure what value there is in trying to handle or tolerate that kind of exception. If someone is changing ACLs in such a way that it breaks a connector, then you probably want that

Re: KafkaConsumer taking time to determine offset

Hi Akash, Thanks for your question. When you say: > kafkaconsumer misses some of the records written into the kafka source topic. Do you mean that there are records in the source topic that are never returned by Consumer#poll(), SourceTask#poll(), or written to the target topic by KafkaConnect?

Making connector fault tolerant

Hello, we have a requirement to make kafka connector more fault tolerant for our use, where in we don't want them to fail for some kinds of errors, i.e error where bootstrap broker are missing or if we don't have sufficient permission to read data from topic (we are reading from aws msk). we

KafkaConsumer taking time to determine offset

Hello, we have a usecase where we use kafkaConsumer in a SourceTask of a source connector to poll messages from an aws msk. if we try to produce data into the source topic immediately after the connector gets into the running state we sometimes notice that kafkaconsumer misses some of the records

Re: Kafka rebalance

Hi Sebastian, Thanks for your question! A consumer joining a group should always notify the group coordinator and the leader consumer, and the partition.assignment.strategy [1] will determine the rebalance protocol, and which partitions are affected by the rebalance. For example, I expect that

Re: Kafka rebalance

While I haven't reached that point because with repartitioning and aggregation and various other things happening with in the stream having additional topics a simple 5 input topics with 5 partitions can end up with hundreds of total topic partitions overall. But I'm 99% confident that it will

Kafka rebalance

Hello, If I have a consumer group with more members than the number of partition of a topic, adding a consumer to the group will still trigger a rebalancing of partitions to the group? Imagine the partitions are already perfectly balanced, ie each consumer has 1 partition. Then reblancing won't

Re: [VOTE] 3.7.1 RC1

Now attaching the container build reports, which I seem to have forgotten to include before. -- Igor kafka/test:test (alpine 3.19.1) === Total: 0 (HIGH: 0, CRITICAL: 0)

[VOTE] 3.7.1 RC1

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.7.1. This is a bugfix release with several fixes. Release notes for the 3.7.1 release: https://home.apache.org/~soarez/kafka-3.7.1-rc1/RELEASE_NOTES.html *** Please download, test and

Increase in produce times after kafka 3.7 upgrade

Hello, recently we upgraded kafka from 2.8.0 to 3.7.0 and noticed a 5x value increase from the kafka.request.produce.time.avg datadog metric. There was also a considerable increase in the kafka.request.metadata.time.avg value. It is worth noting the 99percentile metrics are comparable to before

Re: I want to subscribe

Follow instruction from the webpage: https://kafka.apache.org/contact On 6/4/24 9:01 AM, outof2...@sina.com wrote: I want to subscribe

Question about moving kafka brokers between AZs in AWS

Hello, Here is the scenario. We have a kafka/zookeeper ensemble, where kafka and zookeeper are installed on the same node (instance). There are three zk nodes and two kafka nodes. In other words, out of the three zk nodes, kafka is present in two. Now, these two kafka brokers are in different AZ

I want to subscribe

I want to subscribe

  1   2   3   4   5   6   7   8   9   10   >