Re: [DISCUSS] Mirroring the Repartition Topic: MirrorMaker2 and Streams interaction

Hey everyone, Bumping this topic to see if anyone has experience migrating Streams jobs between clusters with Mirror Maker, and how you handled repartition topics. Thanks! Greg On Mon, Aug 12, 2024 at 8:55 AM Greg Harris wrote: > Hi all, > > We were recently working to set up a MirrorMaker2 fl

Re: Facing issue while producing

Hello Akash, I am getting below error logs sometimes when a kafka producer is going to produce some messages to the topic. *Logs:-* org.apache.kafka.common.errors.NotLeaderOrFollowerException: For requests intended only for the leader, this error indicates that the broker is not the current leade

Re: SmallRye bootstrap.servers value no longer accepted in kafka-clients v3.8.0

Looking at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(List, ClientDnsLookup) shows that error message is referring to a singular url from the list. So the issue is earlier in the chain - the list is not being decoded from the CSV. > On 12 Sep 2024, at 03:03, Josep Prat wr

Re: [ANNOUNCE] New committer: Jeff Kim

congrats! On Tue, Sep 10, 2024 at 5:02 PM Kirk True wrote: > Congratulations Jeff! > > > On Sep 9, 2024, at 6:25 PM, Matthias J. Sax wrote: > > > > Congrats! > > > > On 9/9/24 12:34 PM, José Armando García Sancio wrote: > >> Congratulations Jeff! > >> On Mon, Sep 9, 2024 at 11:45 AM Justine Ols

Re: SmallRye bootstrap.servers value no longer accepted in kafka-clients v3.8.0

Hi Bart, I'm not an expert in this area, but looking at the change [1] in question and the usage of this regex, I'd say it was never the intention to accept comma separated hosts with ports (and optional protocol). The only 2 usages of HOST_PORT_PATTERN can be found in lines 538 [2] and 548 [3] in

Re: Issue - Kafka server shutting down automatically

Hi Brijendra, I suspect that the path you are using for log.dir is being updated/modified/deleted etc. Can you ensure that this path -> D:\kafka\kafka_logs is not being cleaned/modified in anyway (manually/automatically/by some application)? Are you going to use windows server or linux in producti

Re: Will there be a 3.7.2?

Hi Douglas, Thanks for your interest in Apache Kafka! At this time, we don't have a timeline for 3.7.2 yet, and no committer has volunteered. Usually that happens once there are a sufficient number of backported patches accumulated or a security patch is necessary. At this time, the next planned

Re: Handling SASL Authentication Issues in running Kafka Consumers and Reconfiguration

Hi, Why do you even bother catching the exception if you could change the connection details for the clients? You would need to create a new listener, each client then required to be reconfigured to connect that, introducing a code/configuration change anyways. Good luck, Ömer Şiar Baysal On T

Re: AW: Unexpected Tombstone records after kafka streams update 3.7.1

fixed yet. There is a PR from March '24. This bug seems urgent to me. Are there any plans about merging the PR? -Kevin -Ursprüngliche Nachricht- Von: Matthias J. Sax Gesendet: Freitag, 30. August 2024 04:19 An: users@kafka.apache.org Betreff: Re: Unexpected Tombstone records a

Re: community supported releases

yes. thank you. this answers my question. From: Josep Prat Date: Friday, August 30, 2024 at 8:38 AM To: users@kafka.apache.org Cc: Tim Carroll Subject: Re: community supported releases You don't often get email from josep.p...@aiven.io. Learn why this is important<https:

Re: community supported releases

Hi Tim, I replied to your question on the previous email you sent. Didn't you receive it? *Hi Tim,* *Thanks for your interest in Apache Kafka.* *The page you mention is still contains valid information regarding the End of Life cycle: Apache Kafka maintains the latest 3 versions. Currently, th

Kafka consumer always re-join

] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-kafka-group-15, groupId=kafka-group] (Re-)joining group 2024-08-30 08:35:10.070 INFO 5000 --- [tainer#29-0-C-1] o.a.k.c.c.internals.AbstractCoordinator : [Consumer clientId=consumer-kafka-group-3, groupId=kafka-group] Join group

Re: Facing issue while producing

Hello Akash, I will try by changing compression type and test if the issue persists, but for now the issue is resolved I guess. I just changed replication_factor of the topic to 3. It was like a 3-node kafka cluster and replication_factor was configured 2 for the topic. On Wed, Aug 28, 2024 at 2:

Re: Unexpected Tombstone records after kafka streams update 3.7.1

Sounds like https://issues.apache.org/jira/browse/KAFKA-16394 -Matthias On 8/29/24 02:35, Vogel, Kevin, DP EPS, BN, extern, external wrote: Hello there, I searched the Apache Jira for a bug report on this topic but couldn't find one. Maybe anyone else has noticed something similar or knows mo

Re: community supported releases

Hi Tim, Thanks for your interest in Apache Kafka. The page you mention is still contains valid information regarding the End of Life cycle: Apache Kafka maintains the latest 3 versions. Currently, this means 3.6.2, 3.7.1 and 3.8.0. In a few weeks we hope to have released 3.9.0, which would then i

Re: Kafka routing

Hello Kafka brokers are designed to be dumb i.e. they simply act as a pipe to transmit messages from producer to consumer. They don’t have capabilities to selectively filter or perform any analytics/transformation of the messages in flights. In fact, a brokers doesn’t ever read the key/value pair

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

Thanks, Luke. Both the static and dynamic options for configuring unclean leader election now work in 3.9, thanks to your help. Dima, if you have only a transient need to do an unclean leader election, using the kafka-leader-election.sh script is still the best way. It avoids the risk for forge

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

Thank you! This helps alot Sent from Mobile On Wed, Aug 28, 2024 at 19:48 Luke Chen wrote: > Hi Dima, > > Before v3.9.0, brokers in KRaft mode didn't honor > `unclean.leader.election.enable` config. > So, the static config and dynamic config didn't work. > The only thing works before v3.9.0 i

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

Hi Dima, Before v3.9.0, brokers in KRaft mode didn't honor `unclean.leader.election.enable` config. So, the static config and dynamic config didn't work. The only thing works before v3.9.0 is using he admin API or cli to run `kafka-leader-election.sh` with `unclean` option. Thanks. Luke On Thu,

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

Thanks Luke, I meant in pre 3.9.x, like 3.6.x or 3.7.x branch for example. ttyl Dima On Wed, Aug 28, 2024 at 7:36 PM Luke Chen wrote: > Hi Dima, > > > I assume that unclean leader election exists as a static config in kafka > kraft, but is still unavailable dynamically? Or is it unavailable

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

Hi Dima, > I assume that unclean leader election exists as a static config in kafka kraft, but is still unavailable dynamically? Or is it unavailable in general? Let me make it clear. In v3.9.0, the `unclean.leader.election.enable` in KRaft will work, either static config or dynamic config. It's

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

Hi Luke, I assume that unclean leader election exists as a static config in kafka kraft, but is still unavailable dynamically? Or is it unavailable in general? Thanks! ttyl Dima On Wed, Aug 28, 2024 at 3:45 AM Luke Chen wrote: > Hi all, > > In v3.9.0, we're planning to support the `unclean.l

Re: [ANNOUNCE] New committer: Lianet Magrans

Well deserved! Best, Bruno On 8/28/24 6:37 PM, Bill Bejeck wrote: Congrats Lianet! On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax wrote: Congrats! Very well deserved! On 8/28/24 9:18 AM, Justine Olshan wrote: Congratulations Lianet!! Justine On Wed, Aug 28, 2024 at 8:58 AM David Arthur

Re: [ANNOUNCE] New committer: Lianet Magrans

Congrats Lianet! On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax wrote: > Congrats! Very well deserved! > > On 8/28/24 9:18 AM, Justine Olshan wrote: > > Congratulations Lianet!! > > Justine > > > > On Wed, Aug 28, 2024 at 8:58 AM David Arthur wrote: > > > >> Congrats, Lianet! > >> > >> On Wed

Re: [ANNOUNCE] New committer: Lianet Magrans

Congrats! Very well deserved! On 8/28/24 9:18 AM, Justine Olshan wrote: Congratulations Lianet!! Justine On Wed, Aug 28, 2024 at 8:58 AM David Arthur wrote: Congrats, Lianet! On Wed, Aug 28, 2024 at 11:48 AM Mickael Maison wrote: Congratulations Lianet! On Wed, Aug 28, 2024 at 5:40 PM J

Re: Facing issue while producing

Also what is the version of Java you are running? On Wed, Aug 28, 2024 at 11:44 AM Akash Jain wrote: > Hi Vikram, please share some more details: > >1. producer version >2. broker version >3. broker side config you mentioned is "snappy", can you check what is >it on topic level a

Re: Facing issue while producing

Hi Vikram, please share some more details: 1. producer version 2. broker version 3. broker side config you mentioned is "snappy", can you check what is it on topic level as well 4. do you face this issue when you use any other algorithm instead of snappy, say lz4? Try with topic

Re: Facing issue while producing

Hello Akash, It's the same on the broker side as well. *Broker side config:-* compression.type=snappy On Mon, Aug 26, 2024 at 9:30 PM Akash Jain wrote: > And what is it on broker? > > On Sunday, August 25, 2024, Vikram Singh .invalid> > wrote: > > > Hello Akash, > > > > > > Yes, i am using co

Re: Facing issue while producing

And what is it on broker? On Sunday, August 25, 2024, Vikram Singh wrote: > Hello Akash, > > > Yes, i am using compression type in producer side configuration as > mentioned below, > > And the compression type I am using is *snappy*. > > > *Compression on producer :-* > > producerProperties.put(

Re: Facing issue while producing

Hello Akash, Yes, i am using compression type in producer side configuration as mentioned below, And the compression type I am using is *snappy*. *Compression on producer :-* producerProperties.put("compression.type",CloudKafkaProducerConfig. COMPRESSION_TYPE); On Fri, Aug 23, 2024 at 5:56 P

Re: MirrorMaker2 Question

Hi Harry, Thanks for your question! There were negative lag/data loss bugs fixed some time back [1, 2] so check what version of the MirrorMaker connectors you're using. I would recommend using the latest version of MM2 (3.8.0) if possible. After upgrading you will need to clear the checkpoints t

Re: Facing large election time in Kafka

Hey Atul Leadership information is propagated to the rest of the brokers by the controller. In Zk mode, the controller may take a long time to start up because it needs to fetch all state from Zookeeper. Rolling restart makes it worse because you may end up having controller re-election multiple

Re: Facing issue while producing

Hi Vikram. Can you share you code snippet? Are you using compression on producer/broker side? On Friday, August 23, 2024, Vikram Singh wrote: > Hello, > > I am facing issues while producing messages on kafka topics. I am facing > this issue randomly. Please help me by referring to the logs below

Re: Facing large election time in Kafka

Hi Akash, thanks for the quick reply. By the term leader i mean partition leader. I agree with you as upgrading to Kraft based cluster will help in reducing the partitions leader election time, but as we already have Zookeeper based Kafka cluster upgrading to Kraft based cluster will be bit chal

Re: Facing large election time in Kafka

For Kafka with Zookeeper the recovery time is proportional to the number of partitions in the cluster. So theoretically speaking the behaviour is consistent - it will take time. Kraft based Kafka clusters (since Kafka v3.3) are much much better with clusters with a large number of partitions such a

Re: Facing large election time in Kafka

HI Atul you use the word 'leader'. You mean the 'controller'? Or you referring to the leader for each of the partitions? On Fri, Aug 23, 2024 at 7:44 AM Atul Sharma wrote: > Hi, > We are currently facing a prolonged leader election time, approx 2 mins, in > a Kafka cluster (version 2.8.2) that i

Re: Handling load distribution evenly, with ordering

In your producer application, you could write a logic for key1 & key2 to be produced to separate topics with appropriate number of partitions for that key according to your throughput requirements. For the consumer application you can configure it to consumer from multiple topics or can have instan

Re: producer fast fail

Is there a way for our producer to fail quickly if not available? No. Client will just try to re-connect internally. They don't rise any error if they disconnect. On 8/22/24 5:45 AM, Keith Brown wrote: Sometimes out is unavailable. Our producer gives this error [kafka-producer-ne

Re: java kafka multithreaded

I don't think there is any example, but seems to be more or less "straightforward" to do? However, I am wondering about the usage pattern? Does not sound like a good fit for Kafka. Producers and consumer should be long lived. It's an anti pattern to use short lived clients (Kafka is optimized

Re: sending and receiving message

What do you mean by "timeout a consumer"? Given that a consumer calls `poll()` and `poll()` might also return with no data, the consumer code is in full control anyway... I want to make sure a certain message gets pushed if not then exit. This sounds like writing. How it it related to a co

Re: MirrorMaker 2 : Offset issues since 3.5.x+

Hi Tom, Thanks for your question! In Kafka 3.5 there was an overhaul of the offset translation logic to avoid a data loss bug. This [1] is the ticket that I've been using to represent that overhaul. In the implementation for that fix, we traded a lot of availability (read: consumer lag) for corre

RE: Re: Kafka seems to ignore socket.receive.buffer.bytes and socket.send.buffer.bytes bytes

Hi Ömer Many thanks for your response. The bandwidth between the two sites is many orders of magnitude greater than the 1.5 MB throughput I’m able to achieve, so I’m confident that the issue isn’t a case of low bandwidth. I have used the -1 setting on the broker (as well as setting explicitly

Re: Kafka seems to ignore socket.receive.buffer.bytes and socket.send.buffer.bytes bytes

Hi Austin, I think it is worth to mention also the network bandwidth without even Kafka involved. You can create a test bench with basic Linux toolkit like ncat. Tuning socket buffers hardly make sense if the bandwidth is already low between the producer and remote site. When OS is configured t

Re: Handling load distribution evenly, with ordering

Thanks Akash Jain for your detailed explanation. I have answered your queries. Yes as you said the key 1 and key 2 doing more produces. So the skew is happening. In which case you will have to get creative with your topic > design Can you please guide on how to do dynamic segregation of such ke

Re: Handling load distribution evenly, with ordering

Hi Karthick, The choice has to be yours depending on what you want to achieve. I understand you want to achieve even distribution of messages across your partitions. This depends on the following factors: - The frequency of keys - Hashing logic itself What you can control is the hashing lo

Re: sending and receiving message

Thankyou. Is there a way to timeout a consumer after 60secs? I want to make sure a certain message gets pushed if not then exit. On Tue, Aug 20, 2024 at 8:29 AM Tiansu Yu wrote: > > You need to run on a separate process for producer and consumer (on the same > machine is fine). > > Read it here

Re: Question about config.action.reload

Hi Ran, The config.action.reload, subscribe, unsubscribe, and ConfigChangeCallback features are partially-implemented. As far as I can tell if you're using only AK connect-runtime and config providers, none of it is functional. If you have a config provider that implements TTLs, the runtime will

Re: Handling load distribution evenly, with ordering

Hi Akash Jain Thanks for the reply seeking help for the same to choose hashing logics. Please refer/suggest any. On Sat, Aug 17, 2024 at 10:21 AM Akash Jain wrote: > Hi Karthick. You could implement your own custom partitioner. > > On Saturday, August 17, 2024, Karthick wrote: > > > Hi Team, >

Re: sending and receiving message

You need to run on a separate process for producer and consumer (on the same machine is fine). Read it here for simple CLI consumer, e.g. https://www.conduktor.io/kafka/kafka-consumer-cli-tutorial/ > On 20. Aug 2024, at 14:15, Keith Brown wrote: > > I am new to kafka. I started the server a

Re: Handling load distribution evenly, with ordering

Hi Karthick. You could implement your own custom partitioner. On Saturday, August 17, 2024, Karthick wrote: > Hi Team, > > I'm using Kafka partitioning to maintain field-based ordering across > partitions, but I'm experiencing data skewness among the partitions. I have > 96 partitions, and I'm s

Re: Network partition leaves topic-partition leader as sole ISR despite min.isr=2 and producer acks=all settings

Hi Kamal, Thank you for the information. It's great to hear that this is being actively developed. I'm curious if you (or anyone else on the list) is aware of improvements that could reduce the chance of hitting this failure mode? One improvement I found KIP-497

Re: Kafka 3.5.2 is able to read __consumer_offsets from partitions with metadata version 3 when inter.broker.protocol.version is set to 2.0

Hi Dethmix, > The interesting thing is that this doesn't happen if I rollback inter.broker.protocol.version to 2.0 using Kafka 3.5.2 binaries. Cluster is able to start and read all the data from the __consumer_offsets partitions with metadata version 3. The question is that expected behavior? I t

Re: Kafka 3.5.2 is able to read __consumer_offsets from partitions with metadata version 3 when inter.broker.protocol.version is set to 2.0

Hi all, I do understand that everybody is excited about the Kafka 3.8 release, but this question is still actual :) Any updates. Thanks, On Tue, Jul 16, 2024 at 2:54 PM dethmix dethmix wrote: > Hi all, > > I am working on upgrading the Kafka cluster from the version 2.0.1 to > 3.5.2. The main

Re: Network partition leaves topic-partition leader as sole ISR despite min.isr=2 and producer acks=all settings

Hi Sabit, Thanks for reporting the issue! This is the last standing replica problem and is being fixed in KIP-966. You can go through the below blog to understand it in detail: https://jack-vanlightly.com/blog/2023/8/17/kafka-kip-966-fixing-the-last-replica-standing-issue#:~:text=Rule%20number%20

Re: JBOD support with Kraft in 3.8.0

Hi Peter, Thanks for pointing this out. The statement in the 3.8 upgrade notes is the correct one. JBOD in KRaft is no longer an early access feature. It seems we forgot to remove the early access notice from the list of missing features in KRaft after the 3.8 release. I created the following P

Re: Old Ticket (OutOfMemoryError in SaslClientAuthenticator during server restart)

Hi Andreas, Thank you for the reminder, it is unfortunate that this slipped through the cracks. This is certainly something that we should fix for everyone, so you don't have to maintain patches yourself. I'll investigate some more and get back to you. Thanks! Greg On Fri, Aug 2, 2024 at 12:33 A

Re: kafka-producer-perf-test.sh transaction support?

Hi Anindya, Currently, setting transaction.id in producer props won't enable transactions. Users need to set --transaction-duration-ms to enable transactions in kafka-producer-perf-test. We recognize this can be confusing, so there's a JIRA ticket for it: https://issues.apache.org/jira/browse/

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

+1. Thanks, Josep! Colin On Mon, Jul 29, 2024, at 10:32, Chris Egerton wrote: > Thanks for running the release, Josep! > > > On Mon, Jul 29, 2024, 13:31 'Josep Prat' via kafka-clients > wrote: >> The Apache Kafka community is pleased to announce the release for Apache >> Kafka 3.8.0 >> >> T

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

Thanks for running the release, Josep! On Mon, Jul 29, 2024, 13:31 'Josep Prat' via kafka-clients < kafka-clie...@googlegroups.com> wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 3.8.0 > > This is a minor release and it includes fixes and improvements fr

Re: Kafka Read only - disable producing

Could you enable ACLs on your cluster and not grant any principals permission to write to the cluster? On Fri, Jul 26, 2024, 11:13 Rotem S wrote: > Hi, I would like to force disable producing to the kafka cluster and still > allow consuming once the storage disk space reaches a certain threshold

Re: [VOTE] 3.8.0 RC3

:"client-id","override_quota":true} > > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@ > > {"quota_type":"user","override_quota":false} > > - > > > tests/kafkatest/tests/connect/connect_distributed_test.py::Connect

Re: [VOTE] 3.8.0 RC3

Hi Josep, Thanks for running the release! I am +1 (non binding) Downloaded from sources https://github.com/apache/kafka/archive/refs/tags/3.8.0-rc3.tar.gz and built with java 17. I performed the following validations: 1. I deleted a partition.metadata for a topic manually, and tried starting th

Re: Fw: Producer ordering

Not 100% sure if librdkafka provides the exact some semantics as the Kafka Java producer (but I believe it does for this case; but might be worth to check the docs?). If `send()` gives an error back, ordering would not be guaranteed any longer if the retry happens at the application level. Ord

Re:Re: Re: request to be contributor

Hi Josep, Thank you for your suggestion. I have added a comment to the Jira issue you mentioned Best, Yang At 2024-07-24 19:44:36, "Josep Prat" wrote: >Hi Yang, > >I think you need to add a comment here: >https://issues.apache.org/jira/browse/INFRA-25451 > >Best, > >On Wed, Jul 24,

Re: Re: request to be contributor

Hi Yang, I think you need to add a comment here: https://issues.apache.org/jira/browse/INFRA-25451 Best, On Wed, Jul 24, 2024 at 1:36 PM yangxuze wrote: > Hi Josep, > > Thanks for your prompt reply. I now can assign jira to myself, but I have > been unable to successfully register Confluence/W

Re: [VOTE] 3.8.0 RC3

+1 (non-binding). As with the previous RCs, I used the staged Scala 2.13 binaries and the staged Maven artifacts. All seems to work fine. Thanks! Jakub On Tue, Jul 23, 2024 at 12:07 PM Josep Prat wrote: > Hello Kafka users, developers and client-developers, > > This is the fourth candidate for

Re: request to be contributor

Hi, Can we run multiple kafka streams with the same stream-id. On Wed, Jul 24, 2024 at 12:15 PM Josep Prat wrote: > Hi Yang, > > I added you to Jira. I couldn't find your username for the Wiki page > (Confluence) so I didn't set that up. But you are now all set in regards to > Jira. Let me know

Re: request to be contributor

Hi Yang, I added you to Jira. I couldn't find your username for the Wiki page (Confluence) so I didn't set that up. But you are now all set in regards to Jira. Let me know if you have any questions. Best, On Wed, Jul 24, 2024 at 5:11 AM yangxuze wrote: > Dear Apache Kafka Team, > > > > > My na

Re: [VOTE] 3.8.0 RC3

at 12:51 PM Chris Egerton wrote: > Forwarding my response to the other mailing list threads; apologies for > missing the reply-all the first time! > > -- Forwarded message - > From: Chris Egerton > Date: Tue, Jul 23, 2024 at 3:45 PM > Subject: Re: [VOTE] 3.8.0

Re: [VOTE] 3.8.0 RC3

Hi Josep, Thanks for running this release! I'm +1 (binding). To verify, I: - Built from source using Java 11 with both: - - the 3.8.0-rc3 tag on GitHub - - the source artifact from https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka-3.8.0-src.tgz - Checked signatures and checksums - Ran the qu

Re: [VOTE] 3.8.0 RC3

Here is the link to the system tests: https://confluent-open-source-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.8/2024-07-22--001.ffbb03b2-61f4-4ebb-ae1f-af5c753682fb--1721733000--confluentinc--3.8--9a2b34b68c/report.html The Quota tests are known to fail in this CI system. Regarding th

Re: [VOTE] 3.8.0 RC2

+1 (non-binding). I used the staged Scala 2.13 binaries and the staged Maven artifacts. All seems to work fine. Thanks! Jakub On Thu, Jul 18, 2024 at 11:19 AM Josep Prat wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate for release of Apache Kafka 3.8

Re: KRaft Migration Prod Ready?

Software Developer ude...@itrsgroup.com ITRS From: Luke Chen Date: Friday, July 5, 2024 at 3:41 AM To: ude...@itrsgroup.com.invalid Cc: users Subject: Re: KRaft Migration Prod Ready? Hi Upesh, Yes, it's supported. The doc will be updated soon (after v3.8.0 released). Thanks. Luke On Wed

Re: [VOTE] 3.8.0 RC1

Hi all, I'm cancelling this VOTE thread, and I'll generate a new RC once the fix is merged. Best, On Wed, Jul 17, 2024 at 8:55 AM Josep Prat wrote: > Thanks Greg for taking a look and providing a fix! > > Best, > > -- > Josep Prat > Open Source Engineering Director, Aiven > jos

Re: [VOTE] 3.8.0 RC1

Thanks Greg for taking a look and providing a fix! Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht C

Re: [VOTE] 3.8.0 RC1

Hi Josep, I found this blocker regression: https://issues.apache.org/jira/browse/KAFKA-17150 Summary: Connector configurations that specified converters with aliases (e.g. JsonConverter instead of org.apache.kafka.connect.json.JsonConverter) previously worked, and in this RC they throw validation

Re: [VOTE] 3.8.0 RC1

+1 (non-binding). I used the staged Scala 2.13 binaries and the staged Maven artifacts. All seems to work fine. Thanks! Jakub On Mon, Jul 15, 2024 at 5:53 PM Josep Prat wrote: > Hello Kafka users, developers and client-developers, > > This is the second release candidate for Apache Kafka 3.8.0.

Re: [VOTE] 3.8.0 RC0

Hi all, I'm cancelling the VOTE thread for 3.8.0-RC0. I submitted a PR with the backport https://github.com/apache/kafka/pull/16593 and I'll generate a new RC as soon as it's merged. Best, On Sat, Jul 13, 2024 at 7:09 PM Josep Prat wrote: > Thanks for reviewing the RC Jakub, > > If you can ope

Re: [VOTE] 3.8.0 RC0

Thanks for reviewing the RC Jakub, If you can open a PR with this fix pointing to the 3.8 branch I could cut another RC. Best! -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117

Re: [VOTE] 3.8.0 RC0

Hi Josep, Thanks for the RC. I gave it a quick try and ran into issues with an application using the Kafka Admin API that looks like this issue: https://issues.apache.org/jira/browse/KAFKA-16905 ... given that this breaks what was working fine with Kafka 3.7, can the fix be backported to 3.8.0 as

Re: Requesting a Confluence account to create a KIP for a PR

It's not in our control. The INFRA team needs to create accounts manually at this point, and we don't know at what cadence they find time to do this. Sorry :( If an account is create, I believe you should get an email. We know that this current workaround is very tedious, but there is unfortu

Re: Error creating a topic with 10k partitions but not altering existing topic to 10k partitions? Why

Kafka 3.6.2 – thanks, Paul From: Ömer Şiar Baysal Date: Thursday, 11 July 2024 at 9:57 PM To: users@kafka.apache.org Subject: Re: Error creating a topic with 10k partitions but not altering existing topic to 10k partitions? Why [You don't often get email from osiarbay...@gmail.com. Lear

Re: [EXTERNAL] Changing consumer group in live

@Mohamed Depending on topic retention and offset, and how much data in topic, your app could start reprocessing lot of data and eventually struggling in terms of performance. [Logo, company name Description automatically generated] Jose Manuel Vega Monroy Software Engineer Team Leader Direct:

Re: Error creating a topic with 10k partitions but not altering existing topic to 10k partitions? Why

Hi, I have never seen PolicyViolationException before with vanilla Kafka. What is the flavor? This may caused by the controller mutation rate checks introduced by KIP-599, it may be triggered for non-existing topics but not for the existing resources. Hope this help you track it down. OSB On

Re: Changing consumer group in live

In the end you create a new consumer group. That's totally fine. The old consumer group will still "exist", ie, the committed offsets for the group are still there, until they will be eventually purged by the brokers (I think the default is 7 days to keep the metadata of consumer groups, after

Re: Kafka 20k topics metadata update taking long time

Following up to check if someone can help. And is this the right approach or should i open a Jira ticket for same. Thanks, Amit On Fri, Jul 5, 2024 at 9:42 AM Amit Chopra wrote: > But 20K topics is also not a realistic assumption to have. > This is our existing scale with AWS kinesis 20k shards

Re: Installing kafka

Hi Christian, Are you sure are in the same directory you downloaded the kafka_2.13-3.7.1.tgz file to? You might need to change the command to point to the right directory maybe. Best -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 |

Re: Kafka 20k topics metadata update taking long time

, 2024 at 12:08 AM Brebner, Paul wrote: > OK so repeating with Java Kafka producer there is no problem – it’s > specific to the Kafka CLI Producer! Paul > > From: Brebner, Paul > Date: Friday, 5 July 2024 at 1:21 PM > To: users@kafka.apache.org > Subject: Re: Kafka 20k t

Re: Kafka 20k topics metadata update taking long time

But 20K topics is also not a realistic assumption to have. This is our existing scale with AWS kinesis 20k shards. And we are moving over to kafka and thus testing with equivalent scale. We are looking at an ingestion rate of 2.5 GBps. I don't see an alarming difference in the latency results from

Re: MM2, how to configure offset-syncs topic name

Hello, If it's possible for your conf : could you try something like: *Setting 2 partitions on your topic A *run both MM2 at the same time with the same group of consumer but different clientID. In this way, Kafka Broker shoulds affect one partition by MM2 clientID, and so, messages will be

Re: KRaft Migration Prod Ready?

Hi Upesh, Yes, it's supported. The doc will be updated soon (after v3.8.0 released). Thanks. Luke On Wed, Jul 3, 2024 at 6:02 AM Upesh Desai wrote: > Hello, > > > > We have been a bit confused as to if KRaft migration from ZooKeeper is > production ready? It seems that the Kafka docs still ind

Re: Kafka 20k topics metadata update taking long time

OK so repeating with Java Kafka producer there is no problem – it’s specific to the Kafka CLI Producer! Paul From: Brebner, Paul Date: Friday, 5 July 2024 at 1:21 PM To: users@kafka.apache.org Subject: Re: Kafka 20k topics metadata update taking long time EXTERNAL EMAIL - USE CAUTION when

Re: Kafka 20k topics metadata update taking long time

dedicated Kraft controllers. Curious if there is a time out setting somewhere for the client meta-data request? Paul [cid:image001.png@01DACEDE.1DAE3A70] From: Brebner, Paul Date: Thursday, 4 July 2024 at 3:44 PM To: users@kafka.apache.org Subject: Re: Kafka 20k topics metadata update taking

Re: Kafka Connect Limits

o prevent SinkTaskContext#offset() from being > called, but I'm not sure why it isn't effective. > You should contact the Snowflake Connector maintainers and know that they > are exposed to this exception. > > I'll re-open this issue on the framework

Re: Kafka 20k topics metadata update taking long time

@kafka.apache.org Subject: Re: Kafka 20k topics metadata update taking long time [You don't often get email from sabarish@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] EXTERNAL EMAIL - USE CAUTION when clicking links or attachments This is interesting

Re: Kafka 20k topics metadata update taking long time

This is interesting. But 20K topics is also not a realistic assumption to have. I don't see an alarming difference in the latency results from the two scenarios. Also even when cached, the metadata needs to be refreshed anyway. On Thu, Jul 4, 2024 at 4:37 AM Amit Chopra wrote: > Hey, > > I wante

Re: Kafka Connect Limits

ut I'm not sure why it isn't effective. You should contact the Snowflake Connector maintainers and know that they are exposed to this exception. I'll re-open this issue on the framework side to see if we can find a solution to fix this for other connectors. Thanks, Greg [1] https

Re: [ANNOUNCE] Apache Kafka 3.7.1

A correction to a funny mistake in my previous message: That's *61 contributors* to this release, not 1086! -- Igor Soarez On Mon, Jul 1, 2024, at 11:25 AM, Igor Soarez wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 3.7.1 > > This is a bug fix release

Re: Kafka Streams branching return type affects bindings when using multiple output bindings: intentional behaviour?

iOS<https://aka.ms/o0ukef> From: Matthias J. Sax Sent: Thursday, June 20, 2024 9:25:10 PM To: users@kafka.apache.org Subject: Re: Kafka Streams branching return type affects bindings when using multiple output bindings: intentional behaviour? Jenny, thanks for reach

Re: MirrorMaker2 Offset Replication Issue: MirrorCheckpointConnector Doesn't Replicate Final Offsets In Migrations

Hi Mehrtens, That is expected with the current implementation of offset translation. It makes a tradeoff that translation further from the end of the topic is less accurate, and it won't "converge" to a perfect translation. The only "perfect" translation that is offered is when the consumer group

  1   2   3   4   5   6   7   8   9   10   >