Case of joining multiple streams/tables

Hi, I have checked the documentation and what I see that we can join two streams or tables at a given time. I have a case where I have multiple streams which I need to join based on common key. As of now I am first joining two and the result of that with next and so on. Is there a way or any cas

Re: [VOTE] 2.4.0 RC2

Hi all, We merged all the blockers and released a new release candidate RC3. Hopefully this will be a final RC for 2.4 release. Thanks, On Thu, Dec 5, 2019 at 6:56 AM Israel Ekpo wrote: > Thanks @Randall for the update. I will run the tests on RC3 when it becomes > available. > > On Wed, Dec

[VOTE] 2.4.0 RC3

Hello Kafka users, developers and client-developers, This is the fourth candidate for release of Apache Kafka 2.4.0. This release includes many new features, including: - Allow consumers to fetch from closest replica - Support for incremental cooperative rebalancing to the consumer rebalance prot

Re: [VOTE] 2.4.0 RC2

Thanks @Randall for the update. I will run the tests on RC3 when it becomes available. On Wed, Dec 4, 2019 at 12:42 PM Randall Hauch wrote: > Ah, I merged https://issues.apache.org/jira/browse/KAFKA-9258 last night > and forgot to send an email to this thread. > > Thanks! > > Randall > > On Wed,

Re: Broker Interceptors

Hi Thomas, Thanks for your interest in KIP-388. As Ignacio and Radai have mentioned, this is our (LinkedIn's) implementation of KIP-388. The implementation and deployment of this broker-side observer has been worki

Re: [VOTE] 2.4.0 RC2

Ah, I merged https://issues.apache.org/jira/browse/KAFKA-9258 last night and forgot to send an email to this thread. Thanks! Randall On Wed, Dec 4, 2019 at 10:42 AM Eric Lalonde wrote: > > I have also shared a pull request for the changes I have made to the test > > script > > > > https://gith

Re: [VOTE] 2.4.0 RC2

> I have also shared a pull request for the changes I have made to the test > script > > https://github.com/elalonde/kafka/pull/2 Thanks, this has been merged.

RE: Broker Interceptors

Thanks for the responses. I did worry about the challenge of exposing a vast number of internal classes with general interceptor framework. A less general solution more along the lines of the producer/consumer interceptors on the client would satisfy the majority of use cases. If we are smart,

Re: Reducing streams startup bandwidth usage

Oh, good! On Tue, Dec 3, 2019, at 23:29, Alessandro Tagliapietra wrote: > Testing on staging shows that a restart on exception is much faster and the > stream starts right away which I think means we're reading way less data > than before! > > What I was referring to is that, in Streams, the keys

Re: Auto Scaling in Kafka

Akash, take a look at LinkedIn's Cruise Control project. It can automatically rebalance partitions across brokers, etc. Ryanne On Wed, Dec 4, 2019, 12:10 AM Goel, Akash Deep wrote: > Hi , > > Is it possible to auto scale Kafka? If it is not directly supported, then > is there automated way of a

Kafka consumer metrics

Hi everyone, we want to get Kafka consumer group metrics (throttling and byte rate, for example). We have done this already, using: 1. JMX Mbean of the Kafka consumer Java application 2. CLI utility: *bin/kafka-consumer-groups.sh --describe --group group_name --bootstrap-server localhost:port*.

Re: Ordering of messages in the same kafka streams sub-topology with multiple sinks for the same topic

Hi John, Thanks for you explanation. I guess I'll have to adapt my code to assume there's no ordering guarantee between different "branches". On 2019/12/03 15:59:59, "John Roesler" wrote: > Hi Vasily,> > > Probably in this case, with the constraints you're providing, the first > branch wou

Re: [External] Auto Scaling in Kafka

If you run in the cloud I think there are some solutions. If you think you might need to add brokers in the future then make sure you create enough partitions ahead of time. It is easier to add brokers and have them take over existing partitions than it is to re-partition data. I don't think