Cross-cluster mirror making

Hi, I read on The Definitive Guide to Kafka that when using cross-cluster kafka mirroring, when possible, consume from a remote datacenter rather than produce to a remote datacenter. Why is that the case? -Husna

Information regarding Kafka 2.10-0.10.1.1 version

Hi, I am currently trying to research answers for the following questions. Can you please let me know where/how could i find these in the configuration. 1) Average record size (KB) for data written to each kafka topic 2) Average number of events written to each kafka topic per day Thanks, Pavan

Question Related to Zookeeper and kafka java error message.

Afternoon all, I was wondering if anyone has seen the following messages appearing after deployment of a quorum. I am running the latest versions of kafka and zookeeper available. kafka system 1 error OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from

max.block.ms and TimeoutException on startup question

Hi I've a question I'm hoping the Kafka community can help me puzzle out. In our setup we've configured *max.block.ms * to be 100 ms. For normal KafkaProducer.send(record, exceptionHandler) calls this works fine. But when our app is starting up it'll typically take ~500

Filebeat 6.1 not publishing logs to kafka 0.11.0.1 and kafka 0.10.2.0

Hi All, I am using filebeat 6.1 and kafka 0.11.0.1 to push logs. From filebeat logs I could see that communication is getting established but then logs are not getting pushed. Also when I updated filebeat's output.kafka.version to 0.11.0.1 I get error message : 2018/02/08

Filebeat 6.1 is not pushing logs to kafka

Hi All, I am using filebeat 6.1 and kafka 0.11.0.1 to push logs. From filebeat logs I could see that communication is getting established but then logs are not getting pushed. Also when I updated filebeat's output.kafka.version to 0.11.0.1 I get error message : 2018/02/08

Re: [VOTE] 1.0.1 RC0

Just a heads up that we had a fix for KAFKA-6529 land to fix a file descriptor leak. So this RC is dead and I'll be generating RC1 soon. Thanks, Ewen On Wed, Feb 7, 2018 at 11:06 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Ewen, > > +1 > > Building from source and running the

Re: [Streams] partition expansion of input streams

Hello Matthias, No is not too late, I understand the complexity of change of modify the topic size. The approach to oversize the partition is implemented from day one, but having multiple topics one each domain subject, I want to consider the case of someone miss sizing a specific domain. I

Re: [Streams] partition expansion of input streams

Changing source topic partitions for stateful application is very difficult. The best thing is to overpartition source topics to avoid this issue in the first place. (Too late for you I guess.) If you really need to change the partitions count, the cleanest solution is, to create a second input

[Streams] partition expansion of input streams

Hello, I start to think in the problem of expanding our source partitions, it looks like a stateless stream will handle the expansion properly after Rebalancing. In the case of a state-full then the problem seems different, first problem is the changelog itself, the expected definition will miss

Re: Partition rebalancing when both leader and follower is down?

Can you clear up what you mean by "both leader and replica nodes are down for a given partition" If all the nodes are down for a partition you can't move the partition as you don't have any data to copy. (As this is a fundamental part of re-balancing and leader election. I don't think it will

Need to know kafka client api version

Currently the facing usecase requires to know the kafka client API(library) version for our kafka cluster. We cant directly go and fetch kafka library version that client is using from client. Moreover the kafka cluster is on .11.* version. Moreover to accomplish the usecase, we have enabled kafka