[VOTE] 2.5.1 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 2.5.1. Apache Kafka 2.5.1 is a bugfix release and fixes 72 issues since the 2.5.0 release. Please see the release notes for more information. Release notes for the 2.5.1 release: https:/

Re: ktable - ktable join

I am not sure if I understand the question correctly, but a 1:n join in Kafka Streams does not miss any data. As explained in a previous answer, you can consider the join eventual consistent and if you stop sending new data to the input topic, the "final" join result will be exact. Also, records

Re: kafka-streams merge + aggregate vs merge + to topic + from topic + aggregate

If you use `merge()` merge preserve the (relative) order for each input, but the result will contain records for both inputs interleaved. For example: topicA-p0: A B C topicB-p0: X Y Z In the output KStream, A will be before B, and B will be before C. Similar for X, Y, Z. How A,B,C and X,

Re: ktable join & data loss / indeterminism risk prevention

I am not sure if I fully understand what your question is? Are you talking about stream-table or table-table join? For (1), why do you `merge()`? The merge operator is defined on KStreams, not KTable and a merge is also not a join? -Matthias On 7/15/20 3:27 AM, Dumitru-Nicolae Marasoui wrote:

Re: Kafka compatibility with ZK

Hi Peter, Thanks, that's very helpful! We'll go on with upgrading to 2.4.1 and not touching our ZK 3.4.6 for now. I can report back if that actually causes issues just in case anybody is interested. On Thu, Jul 23, 2020 at 2:00 PM Peter Bukowinski wrote: > Agreed. We use a cloudera distribution

Re: Kafka compatibility with ZK

Agreed. We use a cloudera distribution of zookeeper, that is versioned at 3.4.5 (plus a bunch of backported patches) with kafka 2.4 and haven’t had any issues. > On Jul 23, 2020, at 1:19 PM, Andrey Klochkov wrote: > > We are running a separate ZK cluster and its version is not really tied to >

Re: Kafka compatibility with ZK

We are running a separate ZK cluster and its version is not really tied to the version of Kafka we're using. I have seen the Confluent compatibility matrix and based on that our *current* version of Kafka is not compatible with our version of ZK, and we haven't seen any problems with that. My susp

Re: Kafka compatibility with ZK

Zookeeper is not part of the kafka project and must be installed separately. Confluent maintain a version compatibility table you can use as a reference: https://docs.confluent.io/current/installation/versions-interoperability.html#zk

Re: Kafka compatibility with ZK

Hi, AFAIK, ZK is packed with Kafka. So if you upgrade to 2.4.1 you’ll get what is in 2.4.1. It’s a little different however, if you’re hosting ZK in a different host running independently of Kafka. What’s your situation ? On Thu, 23 Jul 2020 at 21:02, Andrey Klochkov wrote: > Hello, > We're

Kafka compatibility with ZK

Hello, We're upgrading our Kafka from 1.1.0 to 2.4.1 and I'm wondering if ZK needs to be upgraded too (we're currently on 3.4.6). The upgrade guide says that "kafka has switched to the XXX version of ZK" but never says if switching to a newer ZK is mandatory or not. What are the guidelines on keepi

Upgrade connectors logging from log4j to log4j2

Hi Team I have some source connector, which is using the logging provided by kafka-connect framework. Now I need to change the log level dynamically at runtime i.e without restarting the worker process. I found out that changing log level is not possible with log4j, so I decided to upgrade to lo

custom KafkaAvroDeserializerConfig

Hello kafka community, We have certain configs in the KafkaAvroDeserializer config that we would like to have them logged scrambled due to a password present on registry url. We could implement our own KafkaAvroDeserializerConfig but we are not sure where to configure it to use our own, Do you have

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi All, I was checking it more and found this (we use librdkafka to put data into kafka topics): https://docs.confluent.io/5.0.0/clients/librdkafka/classRdKafka_1_1Producer.html#ab90a30c5e5fb006a3b4004dc4c9a7923 As docs say they use microseconds: virtual ErrorCode produce

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi Alexandre, According to kafka broker logs it happens even faster each 5-30 sec. Regards, Vitalii. On Thu, Jul 23, 2020 at 11:15 AM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Hi Vitalii, > > The timestamps provided by your producers are in microseconds, whereas > Kafka expects

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi Vitalii, The timestamps provided by your producers are in microseconds, whereas Kafka expects milliseconds epochs. This could be the reason for over-rolling. When you had the default roll time value of a week, did you experience segment rolls every 15 minutes or so? Thanks, Alexandre Le jeu.

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi Vitali, When I ran into it it was latest time being very large. Until we could get the messages set right we set segment.ms to maxint so it only rolled based on size. Cheers William On Thu, 23 Jul 2020 at 4:46 pm, Vitalii Stoianov < vitalii.stoianov...@gmail.com> wrote: > Hi William, > > > ./