What metrics to use to diagnose performance degradation?

Hi, I have a Kafka cluster and two topics it, X and Y. I have two unrelated applications, one reads from topic X and one from topic Y. Those applications don’t share any resources except for Kafka and K8S clusters. In both application the processing is very basic and no changes were made to it.

Number of topics to which provider sends

Hi, Background: Java, Kafka 2.1.0 I have application that sends to two different topics. In theory I can use the same provider. Are there any advantages to having provider per topic? I looked for best practices for this matter but didn’t find any... Thanks, Victoria

Keys and partitions

Hi, I have userId as a key. Many users have moderate amounts of data but some users have more and some users have huge amount of data. I have been thinking about the following aspects of partitioning: 1. If two or more large users will fall into same partition I might end up with large

Re: Disk space - sharp increase in usage

, "Peter Bukowinski" wrote: CAUTION: This message was sent from outside the company. Do not click links or open attachments unless you recognize the sender and know the content is safe. > On Jun 2, 2020, at 12:56 AM, Victoria Zuberman wrote:

Disk space - sharp increase in usage

Hi, Background: Kafka cluster 7 brokers, with 4T disk each version 2.3 (recently upgraded from 0.1.0 via 1.0.1) Problem: Used disk space went from 40% to 80%. Looking for root cause. Suspects: 1. Incoming traffic Ruled out, according to metrics no significant change in “bytes in” for

Re: Partitioning issue when a broker is going down

r and know the content is safe. > On May 17, 2020, at 11:45 AM, Victoria Zuberman wrote: > > Regards acks=all: > - > Interesting point. Will check acks and min.insync.replicas values. > If I understand the root cause that you are

Re: Partitioning issue when a broker is going down

ts on the number of topic partitions? If some partitions go offline, the cluster still knows how many there are supposed to be, so I’m curious what is reporting 10 when there should be 15. -- Peter > On May 17, 2020, at 10:36 AM, Victoria Zuberman wrote: > > Hi,

Partitioning issue when a broker is going down

Hi, Kafka cluster with 3 brokers, version 1.0.1. Topic with 15 partitions, replication factor 2. All replicas in sync. Bringing down one of the brokers (ungracefully), then adding a broker in version 1.0.1 During this process, are we expected either of the following to happen: 1. Some of

Re: Using Kafka AdminUtils

I’ve used the AdminClient for this kind of thing before. It’s the official java client for administrative actions like creating topics. You can create topics with any partition count, replication, or any other config. I hope this helps, John On Sat, Feb 15, 2020, at 22:41, Victoria Zube

Using Kafka AdminUtils

Hi, I have an application based on Kafka Streams. It reads from Kafka topic (I call this topic “input topic”). That topic has many partitions and their number varies based on the env in which application is running. I don’t want to create different input topics manually. Configuration of